dylanyunlong/project_6_89d52222
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

feat(cccl): integrate missing CCCL directories — python/, ci/, .agent/, docs/, test/

Browse Source 
Sparse-checkout from NVIDIA/cccl main branch to complete cccl_upstream:

Added:
- python/cuda_cccl/ (226 files) — Python bindings for device-level algorithms
  Critical for muh toolchain: cuda.compute.reduce_into, scan, radix_sort, etc.
  Includes 204 .py files with full test coverage for all 27 algorithms
- ci/ (163 files) — Build/test infrastructure
  build_cub.sh, test_cub.sh, build_and_test_targets.sh, matrix.yaml
  Directly maps to our [INFRA-CI] and [INFRA-BUILD] items
- .agent/skills/ (7 files) — NVIDIA's own agent skills for CCCL
  cccl-style/SKILL.md, cccl-test/SKILL.md, sass-diff/SKILL.md
- docs/ (491 files) — Official CCCL documentation
  CI references, CMake guides, Python compute docs, libcudacxx PTX docs
- test/ (12 files) — Top-level integration tests (cuda_smoke, stdpar)
- Root configs: .clang-format, .clang-tidy, CONTRIBUTING.md, pyproject.toml
- CLAUDE.md symlink → AGENTS.md (NVIDIA's standard)

cccl_upstream now mirrors full NVIDIA/cccl structure:
  Before: 42M (cub + thrust + libcudacxx + cudax + c + examples + benchmarks)
  After:  53M (+python +ci +docs +.agent +test +configs)

This completes the CCCL base needed for:
- [muh-bench] items: ci/util/build_and_test_targets.sh for targeted builds
- [CCCL-verify] items: python/cuda_cccl/tests/ as reference implementations
- [CCCL-test] items: ci/test_cub.sh, ci/test_thrust.sh
- Agent workflow: .agent/skills/ for consistent style and test patterns
This commit is contained in:
muh-bot
2026-08-07 02:34:33 +00:00
parent 3f97dca7ad
commit 2a7ca101d7
908 changed files with 121615 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
cccl_upstream/docs/.gitignore vendored Normal file
View File

@@ -0,0 +1,8 @@
_build
_repo
env
api
*png
cubimg
*/auto_api.rst
*/api/

41
cccl_upstream/docs/404.html Normal file
View File

@@ -0,0 +1,41 @@
<!DOCTYPE html>
<html lang="en" data-content_root="./" >
<head>
<title>404 &#8212; CUDA Core Compute Libraries</title>
</head>
<ul>
<div id="linkGen"></div>
</ul>
<script type="module">
import Fuse from 'https://cdn.jsdelivr.net/npm/fuse.js@7.0.0/dist/fuse.mjs'
// This page assumes that the user is already in a versioned subtree
// It is no longer the root 404 handler and is instead delagated to.
var localUri = window.location.pathname;
var hostUrl = window.location.protocol + "//" + window.location.host
// Check if user query is missing a branch prefix, e.g. cccl/thrust/vector.h
var rootPathMatch = localUri.match(/^\/cccl(?:\/unstable|\/branch\/\d+\.\d+\.\d+)/)
// Remove leading prefix, this is fed as a URI search argument to the appropriate 404 handler.
var searchPathMatch = localUri.replace(/^\/cccl(?:\/unstable|\/branch\/\d+\.\d+\.\d+)?/, "")
// Project default root leads to unstable
var rootPath = "/cccl/unstable"
// If a root path was found, use it instead
if (rootPathMatch !== null) {
rootPath = rootPathMatch[0]
}
// User was trying to reach a path, forward to appropriate 404 helper
if (searchPathMatch) {
// Redirect to the helper page, including the query
window.location.href =
hostUrl + rootPath + "/404_helper.html?path=" + searchPathMatch
}
// Redirect to index otherwise
else {
window.location.href = hostUrl + rootPath
}
</script>
</html>

63
cccl_upstream/docs/404_helper.inc.html Normal file
View File

@@ -0,0 +1,63 @@
<ul>
<div id="linkGen"></div>
</ul>
<script type="module">
import Fuse from 'https://cdn.jsdelivr.net/npm/fuse.js@7.0.0/dist/fuse.mjs'
// This page assumes that the user is already in a versioned subtree
// It is no longer the root 404 handler and is instead delagated to.
var localUri = window.location.pathname;
var hostUrl = window.location.protocol + "//" + window.location.host
let params = new URLSearchParams(document.location.search);
var rootPath = localUri.replace(/\/404_helper\.html/, "")
var rootUrl = hostUrl + rootPath
var pagelistUrl = rootUrl + "/pagelist.txt"
// // Branch is versioned if it leads with `unstable` or `vX.Y.Z`
// var isBranchVersioned = localUri.match(/^\/cccl\/(unstable|v\d+\.\d+\.\d+)/);
// // If the branch is not versioned, we redirect to a versioned branch keeping the URI.
// if (!isBranchVersioned) {
// window.location.href = hostUrl + '/' + localUri.replace(/^cccl\//g, "cccl/unstable");
// }
var searchString = params.get("path");
if (searchString === "/" || searchString === null) {
window.location.href = rootUrl + "/index.html"
}
if (searchString !== null) {
fetch(pagelistUrl)
.then((response) => response.text())
.then((text) => {
const pagelist = text.split(",")
const options = {
includeScore: true,
threshold: 0.8,
ignoreLocation: false
}
// cccl/device_vector.html => 'cccl 'device 'vector
console.log("query:" + searchString)
const fuse = new Fuse(pagelist, options)
// Include the ending token in the search to prioritize short matches
const result = fuse.search(searchString + "(end)", {limit: 10})
const listItem = '<li><a href=\"{2}{0}\">{0}</a> - Score: {1}</li>'.replace(/\{2\}/g, rootUrl)
document.getElementById('linkGen').innerHTML = ""
result.forEach(element => {
const targetUri = element.item.replace("(end)", "")
const targetScore = element.score.toFixed(2)
document.getElementById('linkGen').innerHTML += listItem.replace(/\{0\}/g, targetUri).replace(/\{1\}/g, targetScore)
// The match is near perfect, just link directly to it.
if (targetScore <= 0.02 ) {
window.location.href = rootUrl + targetUri
}
});
})
}
</script>

9
cccl_upstream/docs/404_helper.rst Normal file
View File

@@ -0,0 +1,9 @@
:orphan:
404
===
The page you're seeking could not be found. Here is a list of similar pages:
.. raw:: html
:file: 404_helper.inc.html

1
cccl_upstream/docs/VERSION.md Normal file
View File

@@ -0,0 +1 @@
3.6

0
cccl_upstream/docs/_ext/__init__.py Normal file
View File

1453
cccl_upstream/docs/_ext/auto_api_generator.py Normal file
View File

File diff suppressed because it is too large Load Diff

32
cccl_upstream/docs/_static/deduplicate_toc.js vendored Normal file
View File

@@ -0,0 +1,32 @@
// Clean up the "On this page" sidebar for C++ API pages.
//
// Two problems caused by Breathe's per-overload anchor generation:
// 1. toc-h4 entries: Breathe adds a bare redundant "transform()" child anchor under each
// overload's section heading. Always remove them.
// 2. Duplicate toc-h3 entries: when all overloads share the same display name
// (e.g. "ExclusiveSum()"), keep only the first occurrence.
document.addEventListener('DOMContentLoaded', function() {
var tocNav = document.getElementById('pst-page-toc-nav');
if (!tocNav)
return;
tocNav.querySelectorAll('li.toc-h4').forEach(function(li) {
var label = li.textContent.trim();
if (label.endsWith('()')) {
li.remove();
}
});
var seen = new Set();
tocNav.querySelectorAll('li.toc-h3').forEach(function(li) {
var label = li.textContent.trim();
if (!label.endsWith(')')) {
return;
}
if (seen.has(label)) {
li.remove();
} else {
seen.add(label);
}
});
});

BIN
cccl_upstream/docs/_static/nvidia-logo.png vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 126 KiB

41
cccl_upstream/docs/_static/search_custom.css vendored Normal file
View File

@@ -0,0 +1,41 @@
.cccl-search-breadcrumbs {
display: flex;
flex-wrap: wrap;
font-size: 0.72em;
line-height: normal;
list-style: none;
padding: 0;
}
.cccl-search-breadcrumbs .breadcrumb-item {
align-items: center;
display: flex;
font-weight: 700;
margin: 0;
padding: 0;
white-space: nowrap;
}
.cccl-search-breadcrumbs .breadcrumb-item a {
color: var(--pst-color-text-muted);
margin: 0.1875rem;
overflow-x: hidden;
text-decoration: none;
text-overflow: ellipsis;
}
.cccl-search-breadcrumbs .breadcrumb-item a:hover {
color: var(--pst-color-link-hover);
text-decoration: underline;
text-decoration-skip-ink: none;
text-decoration-thickness: max(3px, 0.1875rem, 0.12em);
text-underline-offset: 0.1578em;
}
.cccl-search-breadcrumbs .breadcrumb-item + .breadcrumb-item::before {
color: var(--pst-color-text-muted);
content: var(--pst-breadcrumb-divider);
font: var(--fa-font-solid);
font-size: 0.8rem;
padding: 0 0.5rem;
}

265
cccl_upstream/docs/_static/search_postprocess.js vendored Normal file
View File

@@ -0,0 +1,265 @@
"use strict";
(function () {
const maxBreadcrumbResults = 10;
const decodeEntities = (value) =>
String(value || "")
.replace(/&lt;/g, "<")
.replace(/&gt;/g, ">")
.replace(/&amp;/g, "&")
.replace(/&quot;/g, '"')
.replace(/&#39;|&apos;/g, "'")
.replace(/&nbsp;/g, " ")
.replace(/&hellip;/g, "...");
const getDocLinkSuffix = () =>
(typeof DOCUMENTATION_OPTIONS !== "undefined" &&
DOCUMENTATION_OPTIONS.LINK_SUFFIX) ||
".html";
const pageInfoCache = new Map();
const getDocTitle = (docName) => {
if (
typeof Search === "undefined" ||
!Search._index ||
!Array.isArray(Search._index.docnames) ||
!Array.isArray(Search._index.titles)
) {
return null;
}
const index = Search._index.docnames.indexOf(docName);
return index >= 0 ? Search._index.titles[index] : null;
};
const getResultDocNameFromHref = (href) => {
if (!href) {
return null;
}
const withoutAnchor = String(href).split("#", 1)[0];
const linkSuffix = getDocLinkSuffix();
const suffixIndex = withoutAnchor.lastIndexOf(linkSuffix);
if (suffixIndex < 0) {
return null;
}
let docName = withoutAnchor.slice(0, suffixIndex);
if (docName.startsWith("./")) {
docName = docName.slice(2);
}
const contentRoot =
document?.documentElement?.dataset?.content_root || "";
if (contentRoot && docName.startsWith(contentRoot)) {
docName = docName.slice(contentRoot.length);
}
return docName.replace(/^\/+/, "");
};
const getPageInfo = async (docName) => {
if (pageInfoCache.has(docName)) {
return pageInfoCache.get(docName);
}
const infoPromise = (async () => {
const pageUrl = `${docName}${getDocLinkSuffix()}`;
const response = await fetch(pageUrl);
const html = await response.text();
const parsed = new DOMParser().parseFromString(html, "text/html");
const pageHeading = parsed.querySelector("h1");
const breadcrumbLinks = Array.from(
parsed.querySelectorAll(".breadcrumb-item a.nav-link"),
);
const breadcrumbs = breadcrumbLinks
.map((breadcrumbLink) => {
const rawHref = breadcrumbLink.getAttribute("href");
if (!rawHref) {
return null;
}
return {
href: new URL(rawHref, response.url).href,
title: breadcrumbLink.textContent?.trim() || null,
};
})
.filter((breadcrumb) => breadcrumb && breadcrumb.title);
return {
pageTitle:
pageHeading?.textContent?.replace(/#\s*$/, "").trim() ||
getDocTitle(docName) ||
null,
breadcrumbs,
};
})().catch(() => null);
pageInfoCache.set(docName, infoPromise);
return infoPromise;
};
const addBreadcrumbTrail = async (listItem) => {
if (
!listItem ||
listItem.dataset.ccclBreadcrumbsAttached === "true" ||
listItem.dataset.ccclBreadcrumbsPending === "true"
) {
return;
}
listItem.dataset.ccclBreadcrumbsPending = "true";
const primaryLink = listItem.querySelector("a");
if (!primaryLink) {
delete listItem.dataset.ccclBreadcrumbsPending;
return;
}
const primaryTitle = primaryLink.textContent?.trim() || "";
const href = primaryLink.getAttribute("href");
const docName = getResultDocNameFromHref(href);
if (!docName) {
delete listItem.dataset.ccclBreadcrumbsPending;
return;
}
const pageInfo = await getPageInfo(docName);
if (!pageInfo) {
delete listItem.dataset.ccclBreadcrumbsPending;
return;
}
const pageTitle = pageInfo.pageTitle || getDocTitle(docName);
const breadcrumbs = [...(pageInfo.breadcrumbs || [])];
if (pageTitle && primaryTitle && pageTitle !== primaryTitle) {
breadcrumbs.push({
href: `${docName}${getDocLinkSuffix()}`,
title: pageTitle,
});
}
if (breadcrumbs.length === 0) {
delete listItem.dataset.ccclBreadcrumbsPending;
return;
}
const breadcrumbContainer = document.createElement("div");
breadcrumbContainer.className = "cccl-search-breadcrumbs";
breadcrumbs.forEach((breadcrumb) => {
const breadcrumbItem = document.createElement("span");
breadcrumbItem.className = "breadcrumb-item";
const breadcrumbLink = document.createElement("a");
breadcrumbLink.href = breadcrumb.href;
breadcrumbLink.textContent = breadcrumb.title;
breadcrumbItem.appendChild(breadcrumbLink);
breadcrumbContainer.appendChild(breadcrumbItem);
});
listItem.insertBefore(breadcrumbContainer, primaryLink.nextSibling);
listItem.dataset.ccclBreadcrumbsAttached = "true";
delete listItem.dataset.ccclBreadcrumbsPending;
};
const installResultDecorator = () => {
if (
typeof Search === "undefined" ||
Search.__ccclResultDecoratorInstalled ||
typeof MutationObserver === "undefined"
) {
return;
}
const originalPerformSearch = Search.performSearch;
Search.performSearch = (...args) => {
const result = originalPerformSearch(...args);
const output = Search.output;
if (!output) {
return result;
}
const decorateTopResults = () => {
Array.from(output.querySelectorAll("li"))
.slice(0, maxBreadcrumbResults)
.forEach(addBreadcrumbTrail);
};
if (Search.__ccclResultsObserver) {
Search.__ccclResultsObserver.disconnect();
}
decorateTopResults();
const observer = new MutationObserver((mutations) => {
decorateTopResults();
});
observer.observe(output, { childList: true, subtree: true });
Search.__ccclResultsObserver = observer;
Search.__ccclResultDecoratorInstalled = true;
return result;
};
};
const installPostprocess = () => {
if (typeof Search === "undefined" || Search.__ccclDedupInstalled) {
return;
}
const originalPerformSearch = Search._performSearch;
Search._performSearch = (...args) => {
const results = originalPerformSearch(...args);
// Sphinx keeps results in low->high score order and displays via pop().
// Walk from the end so we see the best-ranked result first, but prefer
// canonical page links without anchors when collapsing duplicates.
const chosen = new Map();
for (let i = results.length - 1; i >= 0; --i) {
const result = results[i];
const title = String(result[1] || "").toLowerCase();
const filename = String(result[5] || "");
const key = `${filename}\0${title}`;
const anchor = String(result[2] || "");
const existing = chosen.get(key);
if (!existing) {
chosen.set(key, result);
continue;
}
const existingAnchor = String(existing[2] || "");
const prefersCurrent = existingAnchor && !anchor;
if (prefersCurrent) {
chosen.set(key, result);
}
}
const deduped = [];
const emitted = new Set();
for (let i = 0; i < results.length; ++i) {
const result = results[i];
const title = String(result[1] || "").toLowerCase();
const filename = String(result[5] || "");
const key = `${filename}\0${title}`;
if (emitted.has(key)) {
continue;
}
const winner = chosen.get(key);
if (winner) {
winner[1] = decodeEntities(winner[1]);
winner[3] = decodeEntities(winner[3]);
deduped.push(winner);
emitted.add(key);
}
}
return deduped;
};
Search.__ccclDedupInstalled = true;
};
installPostprocess();
installResultDecorator();
})();

224
cccl_upstream/docs/_static/search_scorer.js vendored Normal file
View File

@@ -0,0 +1,224 @@
"use strict";
const _normalizeSearchSymbol = (value) =>
(value || "")
.toLowerCase()
.replace(/&lt;|&gt;|&amp;|&quot;|&#39;|&apos;|&nbsp;|&hellip;/g, " ")
.replace(/[^a-z0-9:]+/g, "");
const _splitSymbolWords = (value) =>
(value || "")
.replace(/&lt;|&gt;|&amp;|&quot;|&#39;|&apos;|&nbsp;|&hellip;/g, " ")
.replace(/::/g, " ")
.replace(/([a-z0-9])([A-Z])/g, "$1 $2")
.replace(/([A-Z]+)([A-Z][a-z])/g, "$1 $2")
.toLowerCase()
.match(/[a-z0-9]+/g) || [];
const _getSearchQuery = () => {
try {
return new URLSearchParams(window.location.search).get("q") || "";
} catch {
return "";
}
};
var Scorer = {
// Keep strong object-name bias at the base layer.
objNameMatch: 80,
objPartialMatch: 35,
objPrio: {
0: 25, // highest-priority API objects
1: 10,
2: -10,
},
objPrioDefault: 0,
title: 15,
partialTitle: 7,
term: 5,
partialTerm: 2,
score: (result) => {
const [docName, title, anchor, descr, baseScore, filename] = result;
let score = baseScore;
const trimmedTitle = (title || "").trim();
const trimmedAnchor = (anchor || "").trim();
const trimmedDescription = (descr || "").trim();
const trimmedFilename = (filename || "").trim();
const trimmedDocName = (docName || "").trim();
const lowerDescription = trimmedDescription.toLowerCase();
const lowerFilename = trimmedFilename.toLowerCase();
const lowerDocName = trimmedDocName.toLowerCase();
const query = _getSearchQuery().trim();
const lowerQuery = query.toLowerCase();
const normalizedQuery = _normalizeSearchSymbol(query);
const titleParts = trimmedTitle.split("::").filter(Boolean);
const symbolDepth = titleParts.length;
const leaf = titleParts.length
? titleParts[titleParts.length - 1]
: trimmedTitle;
const parentTitle =
titleParts.length > 1 ? titleParts.slice(0, -1).join("::") : "";
const normalizedLeaf = _normalizeSearchSymbol(leaf);
const normalizedParent = _normalizeSearchSymbol(parentTitle);
const normalizedLeafOnlyQuery = _normalizeSearchSymbol(
query.includes("::") ? query.split("::").pop() : query,
);
const leafWords = _splitSymbolWords(leaf);
const queryWords = _splitSymbolWords(query);
const simpleQuery =
lowerQuery &&
!/[.:/_]/.test(lowerQuery) &&
/^[a-z0-9]+$/.test(lowerQuery);
const looksLikeNamespaceQualified =
/^[a-zA-Z_]\w*(::[a-zA-Z_]\w*)+/.test(trimmedTitle);
const isExactFunctionishTitle =
/::[A-Za-z_]\w*$/.test(trimmedTitle); // e.g. thrust::transform
const isClassishTitle =
/::[A-Z]\w*$/.test(trimmedTitle); // e.g. cub::DeviceRadixSort
const isParameterLike =
/(template parameter|function parameter)/i.test(trimmedDescription);
const isMemberLike =
/(C\+\+ (member|type|property))/i.test(trimmedDescription);
const isCallableLike =
/(C\+\+ function\b|C\+\+ class\b|C\+\+ struct\b)/i.test(trimmedDescription);
const isTopLevelSymbol = looksLikeNamespaceQualified && symbolDepth <= 2;
const isNestedSymbol = symbolDepth >= 3;
const isConstructorLike =
isNestedSymbol &&
_normalizeSearchSymbol(titleParts[symbolDepth - 2]) === normalizedLeaf;
const hasQueryInFilename =
lowerQuery && lowerFilename.includes(lowerQuery);
const hasQueryInDocName = lowerQuery && lowerDocName.includes(lowerQuery);
const isEnumeratorLike =
/(C\+\+ enumerator\b)/i.test(trimmedDescription) || /^[A-Z0-9_]+$/.test(leaf);
const isInternalHelperLike =
/(policy|dispatch|state|status|callback|preference|layout|runningprefixop|emptycallback|op)/i.test(
trimmedTitle,
) ||
/(TileState|Policy|Dispatch|Callback|Preference|Layout|RunningPrefixOp|Status|EmptyCallback)/.test(
trimmedTitle,
);
const isPythonModuleLike = /(Python module\b)/i.test(trimmedDescription);
const leafStartsWithQueryWord =
queryWords.length === 1 && leafWords[0] === queryWords[0];
const leafEndsWithQueryWord =
queryWords.length === 1 &&
leafWords.length > 0 &&
leafWords[leafWords.length - 1] === queryWords[0];
const leafQueryRemainderWords = queryWords.length === 1
? leafWords.filter((word) => word !== queryWords[0])
: [];
const hasCompactQueryWordRemainder =
leafStartsWithQueryWord &&
leafQueryRemainderWords.length > 0 &&
leafQueryRemainderWords.length <= 2;
const hasHelperSuffix =
/(Strategy|Policy|State|Status|Callback|Preference|Layout|Type|Op|Match|Functor|Tag|Traits|Descriptor|Counts)$/.test(
leaf,
);
// Strong bias toward actual API symbols.
if (isExactFunctionishTitle) score += 35;
if (isClassishTitle) score += 20;
// Small boost for anchored entries; these are often object targets.
if (trimmedAnchor) score += 5;
// Penalize taxonomy/concept pages that match lots of body text.
if (
lowerDescription.includes("thrust::") ||
lowerDescription.includes("cub::") ||
lowerDescription.includes("cuda::")
) {
score += 8;
}
// Query-aware ranking: prefer canonical symbol pages over nested members.
if (normalizedQuery) {
if (normalizedLeaf === normalizedLeafOnlyQuery) {
score += isTopLevelSymbol ? 180 : 35;
} else if (
normalizedLeafOnlyQuery &&
normalizedLeaf.includes(normalizedLeafOnlyQuery)
) {
score += 15;
}
}
if (isNestedSymbol) score -= 25;
if (isParameterLike) score -= 80;
if (isMemberLike) score -= 35;
if (isConstructorLike) score -= 30;
if (isCallableLike && isTopLevelSymbol) score += 20;
// Prefer libcudacxx/cuda symbols over thrust equivalents on ties.
if (
normalizedLeaf === normalizedLeafOnlyQuery &&
/^cuda::/.test(trimmedTitle)
) {
score += 12;
}
// For plain keyword queries, prefer pages that match in title/path metadata.
if (simpleQuery) {
if (hasQueryInFilename || hasQueryInDocName) score += 45;
if (isInternalHelperLike) score -= 100;
if (isPythonModuleLike) score -= 30;
if (hasHelperSuffix) score -= 80;
if (isTopLevelSymbol && isCallableLike && !hasHelperSuffix) score += 40;
if (
leafStartsWithQueryWord &&
isTopLevelSymbol &&
!isEnumeratorLike &&
!isInternalHelperLike
) {
score += 75;
}
if (
leafEndsWithQueryWord &&
isTopLevelSymbol &&
!isEnumeratorLike &&
!isInternalHelperLike
) {
score += 45;
}
// For broad prefix-style queries like "block", prefer compact public API
// names over longer compound variants or helper-like extensions.
if (
hasCompactQueryWordRemainder &&
isTopLevelSymbol &&
!isEnumeratorLike &&
!isInternalHelperLike &&
!hasHelperSuffix
) {
score += 70 - 15 * (leafQueryRemainderWords.length - 1);
}
// If a nested member matches the query but its parent symbol also does,
// prefer the parent page/class over the member overload.
if (
isNestedSymbol &&
normalizedLeaf === normalizedLeafOnlyQuery &&
normalizedParent.includes(normalizedLeafOnlyQuery)
) {
score -= 70;
}
if (
isTopLevelSymbol &&
normalizedLeaf.includes(normalizedLeafOnlyQuery) &&
normalizedLeaf !== normalizedLeafOnlyQuery
) {
score += 70;
}
}
return score;
},
};

12
cccl_upstream/docs/_templates/search.html vendored Normal file
View File

@@ -0,0 +1,12 @@
{% extends "!nvidia_sphinx_theme/search.html" %}
{% block extrahead %}
{{ super() }}
<link rel="stylesheet" href="{{ pathto('_static/search_custom.css', 1) }}" />
{% endblock %}
{% block scripts %}
<script src="{{ pathto('_static/search_scorer.js', 1) }}"></script>
{{ super() }}
<script src="{{ pathto('_static/search_postprocess.js', 1) }}"></script>
{% endblock %}

310
cccl_upstream/docs/cccl/3.0_migration_guide.rst Normal file
View File

@@ -0,0 +1,310 @@
.. _cccl-3.0-migration-guide:
CCCL 2.x ‐ CCCL 3.0 migration guide
===================================
The CCCL team plans breaking changes carefully and only conducts them at major releases.
The CCCL 2.8 release came with many deprecations to prepare for the breaking changes conducted in CCCL 3.0.
This page summarizes the changes and helps migrating from CCCL 2.x to CCCL 3.0.
See also the `list of all deprecated APIs in CCCL 2.8 <https://github.com/NVIDIA/cccl/issues/3700>`_
and the `list of breaking changes in CCCL 3.0 <https://github.com/NVIDIA/cccl/issues/101>`_.
CUDA Toolkit changes
--------------------
CCCL is moving to its own include directory within the CUDA Toolkit. This may cause build failures and some initial confusion.
This section will have some suggestions and mitigations to help maintain builds across both CUDA12 and future releases.
The CTK-provided includes are changing in the following ways:
+-------------------------------+------------------------------------+
| **Before CUDA 13.0** | **After CUDA 13.0** |
+-------------------------------+------------------------------------+
| `${CTK_ROOT}/include/cuda/` | `${CTK_ROOT}/include/cccl/cuda/` |
+-------------------------------+------------------------------------+
| `${CTK_ROOT}/include/cub/` | `${CTK_ROOT}/include/cccl/cub/` |
+-------------------------------+------------------------------------+
| `${CTK_ROOT}/include/thrust/` | `${CTK_ROOT}/include/cccl/thrust/` |
+-------------------------------+------------------------------------+
Due to these changes, and the fact that NVCC by default includes its own directories, you may encounter errors when including
CCCL headers in source files that are compiled *only* by the host compiler.
For example, when compiling with GCC or MSVC alone, you may see ``<cuda/...>``, ``<cub/...>``, or ``<thrust/...>`` headers missing.
To mitigate this there are several solutions available depending on your build system:
- **DO NOT** prefix missing includes with ``<cccl/>`` -- This will break.
- CMake: link ``CCCL::CCCL`` to your target.
- Example: ``target_link_library(${MY_TARGET} PRIVATE CCCL::CCCL)``
- Non-CMake: Directly include the CUDA Toolkit's CCCL directory. (Make/Other)
- Example: Add CCCL as an include flag ``-I${CTK_ROOT}/include/cccl``
- Use a non-bundled CCCL. CCCL is available and maintained independently of the CTK.
- `See here for compatibility. <https://github.com/NVIDIA/cccl?tab=readme-ov-file#cuda-toolkit-ctk-compatibility>`_
Removed macros
--------------
* ``CUB_IS_INT128_ENABLED``: No replacement
* ``CUB_MAX(a, b)``: Use the ``cuda::std::max(a, b)`` function instead
* ``CUB_MIN(a, b)``: Use the ``cuda::std::min(a, b)`` function instead
* ``CUB_QUOTIENT_CEILING(a, b)``: Use ``cuda::ceil_div(a, b)`` instead
* ``CUB_QUOTIENT_FLOOR(a, b)``: Use plain integer division ``a / b`` instead
* ``CUB_ROUND_DOWN_NEAREST(a, b)``: Use ``cuda::round_down(a, b)`` instead
* ``CUB_ROUND_UP_NEAREST(a, b)``: Use ``cuda::round_up(a, b)`` instead
* ``CUB_RUNTIME_ENABLED``: No replacement
* ``CUB_USE_COOPERATIVE_GROUPS``: No replacement
* ``CUDA_CUB_RET_IF_FAIL``: No replacement
* ``[THRUST|CUB]_CLANG_VERSION``: No replacement
* ``[THRUST|CUB]_DEVICE_COMPILER*``: No replacement
* ``[THRUST|CUB]_GCC_VERSION``: No replacement
* ``[THRUST|CUB]_HOST_COMPILER*``: No replacement
* ``[THRUST|CUB]_INCLUDE_DEVICE_CODE``: No replacement
* ``[THRUST|CUB]_INCLUDE_HOST_CODE``: No replacement
* ``[THRUST|CUB]_IS_DEVICE_CODE``: No replacement
* ``[THRUST|CUB]_IS_HOST_CODE``: No replacement
* ``[THRUST|CUB]_MSVC_VERSION_FULL``: No replacement
* ``[THRUST|CUB]_MSVC_VERSION``: No replacement
* ``THRUST_CDP_DISPATCH``: No replacement (Support for CUDA Dynamic Parallelism V1 (CDPv1) has been removed, see below)
* ``THRUST_DECLTYPE_RETURNS_WITH_SFINAE_CONDITION``: No replacement
* ``THRUST_DECLTYPE_RETURNS``: No replacement
* ``THRUST_DEVICE_CODE``: No replacement
* ``THRUST_HOST_BACKEND``: Use ``THRUST_HOST_SYSTEM`` instead
* ``THRUST_INLINE_CONSTANT``: Use ``inline constexpr`` instead
* ``THRUST_INLINE_INTEGRAL_MEMBER_CONSTANT``: Use ``static constexpr`` instead
* ``THRUST_LEGACY_GCC``: No replacement
* ``THRUST_MODERN_GCC_REQUIRED_NO_ERROR``: No replacement
* ``THRUST_MODERN_GCC``: No replacement
* ``THRUST_MVCAP``: No replacement
* ``THRUST_NODISCARD``: Use ``[[nodiscard]]`` instead
* ``THRUST_RETOF1``: No replacement
* ``THRUST_RETOF2``: No replacement
* ``THRUST_RETOF``: No replacement
* ``THRUST_TUNING_ARCH``: No direct replacement. Use compiler-specific ``__CUDA_ARCH__`` (nvcc) or ``__NVCOMPILER_CUDA_ARCH__`` (nvc++) instead
Removed functions and classes
-----------------------------
* ``_ReadWriteBarrier`` and ``__thrust_compiler_fence``: Use ``cuda::atomic`` instead
* ``cub::*Kernel``: Any CUB kernel entrypoint is considered an implementation detail. No public exposure is provided.
* ``cub::Agent*``: CUB agents were considered implementation details and have all been moved to internal namespaces. No public exposure is provided.
* ``cub::AliasTemporaries``: No replacement
* ``cub::ArrayWrapper``: Use ``cuda::std::array`` instead
* ``cub::BAR``: No replacement
* ``cub::BaseTraits::CATEGORY``: Use the facilities from ``<cuda/std/type_traits>`` instead
* ``cub::BaseTraits::NULL_TYPE``: No replacement
* ``cub::BaseTraits::PRIMITIVE``: Use the facilities from ``<cuda/std/type_traits>`` instead
* ``cub::BFI``: Use ``cuda::bitfield_insert`` instead
* ``cub::BinaryOpHasIdxParam::HAS_PARAM``: Use ``cub::BinaryOpHasIdxParam::value`` instead
* ``cub::ConstantInputIterator``: Use ``thrust::constant_iterator`` instead
* ``cub::CountingInputIterator``: Use ``thrust::counting_iterator`` instead
* ``cub::CTA_SYNC_AND``: Use ``__syncthreads_and()`` instead
* ``cub::CTA_SYNC_OR``: Use ``__syncthreads_or()`` instead
* ``cub::CTA_SYNC``: Use ``__syncthreads()`` instead
* ``cub::Device*Policy``: Those policy hubs are considered implementation details. No public exposure is provided.
* ``cub::DeviceSpmv``: Use `cuSPARSE <https://docs.nvidia.com/cuda/cusparse>`_ instead
* ``cub::Difference``: Use ``cuda::std::minus`` instead
* ``cub::DivideAndRoundUp``: Use ``cuda::round_up`` instead
* ``cub::Division``: Use ``cuda::std::divides`` instead
* ``cub::Equality``: Use ``cuda::std::equal_to`` instead
* ``cub::FFMA_RZ``: No replacement
* ``cub::FMUL_RZ``: No replacement
* ``cub::FpLimits<T>``: Use ``cuda::std::numeric_limits<T>`` instead
* ``cub::GridBarrier``: Use the APIs from cooperative groups instead
* ``cub::GridBarrierLifetime``: Use the APIs from cooperative groups instead
* ``cub::IADD3``: No replacement
* ``cub::Inequality``: Use ``cuda::std::not_equal_to`` instead
* ``cub::Int2Type``: Use ``cuda::std::integral_constant`` instead
* ``cub::IterateThreadLoad``: No replacement
* ``cub::IterateThreadStore``: No replacement
* ``cub::KernelConfig``: No replacement
* ``cub::LaneId()``: Use ``cuda::ptx::get_sreg_laneid()`` instead
* ``cub::LaneMaskGe()``: Use ``cuda::ptx::get_sreg_lanemask_ge()`` instead
* ``cub::LaneMaskGt()``: Use ``cuda::ptx::get_sreg_lanemask_gt()`` instead
* ``cub::LaneMaskLe()``: Use ``cuda::ptx::get_sreg_lanemask_le()`` instead
* ``cub::LaneMaskLt()``: Use ``cuda::ptx::get_sreg_lanemask_lt()`` instead
* ``cub::MakePolicyWrapper``: No replacement
* ``cub::Max``: Use ``cuda::maximum`` instead
* ``cub::max``: Use ``cuda::std::max`` instead
* ``cub::MemBoundScaling``: No replacement
* ``cub::Min``: Use ``cuda::minimum`` instead
* ``cub::min``: Use ``cuda::std::min`` instead
* ``cub::Mutex``: Use ``std::mutex`` instead
* ``cub::PolicyWrapper``: No replacement
* ``cub::PRMT``: Use ``cuda::ptx::prmt()`` instead
* ``cub::RegBoundScaling``: No replacement
* ``cub::SHFL_IDX_SYNC``: Use ``__shfl_sync()`` instead
* ``cub::SHL_ADD``: No replacement
* ``cub::SHR_ADD``: No replacement
* ``cub::Sum``: Use ``cuda::std::plus`` instead
* ``cub::Swap(a, b)``: Use ``cuda::std::swap(a, b)`` instead
* ``cub::ThreadTrap()``: Use ``cuda::std::terminate()`` instead
* ``cub::TransformInputIterator``: Use ``thrust::transform_iterator`` instead
* ``cub::TripleChevronFactory``: No replacement for now, we are working on a new kernel launch facility
* ``cub::ValueCache``: No replacement
* ``cub::WARP_ALL``: Use ``__all_sync()`` instead
* ``cub::WARP_ANY``: Use ``__any_sync()`` instead
* ``cub::WARP_BALLOT``: Use ``__ballot_sync()`` instead
* ``cub::WARP_SYNC``: Use ``__syncwarp()`` instead
* ``cub::WarpId()``: Use ``cuda::ptx::get_sreg_warpid()`` instead
* ``thrust::*::[first_argument_type|second_argument_type|result_type]``: The nested aliases have been removed for all function object types: ``thrust::[plus|minus|multiplies|divides|modulus|negate|square|equal_to|not_equal_to|greater|less|greater_equal|less_equal|logical_and|logical_or|logical_not|bit_and|bit_or|bit_xor|identity|maximum|minimum|project1st|project2nd]``. No replacement.
* ``thrust::[unary|binary]_function``: No replacement. If you inherit from one of these types, just remove those base classes.
* ``thrust::[unary|binary]_traits``: No replacement.
* ``thrust::async::*``: No replacement for now. We are working on a C++26 senders implementation. For make a thrust algorithm skip syncing, use ``thrust::cuda::par_nosync`` as execution policy.
* ``thrust::bidirectional_universal_iterator_tag``: No replacement
* ``thrust::conjunction_value<Ts...>``: Use ``cuda::std::bool_constant<(Ts && ...)>`` instead
* ``thrust::conjunction_value_v<Ts...>``: Use a fold expression: ``Ts && ...`` instead
* ``thrust::cuda_cub::core::*``: Those are considered implementation details. No public exposure is provided.
* ``thrust::cuda_cub::counting_iterator_t``: Use ``thrust::counting_iterator`` instead
* ``thrust::cuda_cub::identity``: Use ``cuda::std::identity`` instead
* ``thrust::cuda_cub::launcher::triple_chevron``: No replacement for now, we are working on a new kernel launch facility
* ``thrust::cuda_cub::terminate``: Use ``cuda::std::terminate()`` instead
* ``thrust::cuda_cub::transform_input_iterator_t``: Use ``thrust::transform_iterator`` instead
* ``thrust::cuda_cub::transform_pair_of_input_iterators_t``: Use ``thrust::transform_iterator of a thrust::zip_iterator`` instead
* ``thrust::disjunction_value<Ts...>``: Use ``cuda::std::bool_constant<(Ts || ...)>`` instead
* ``thrust::disjunction_value_v<Ts...>``: Use a fold expression: ``Ts || ...`` instead
* ``thrust::forward_universal_iterator_tag``: No replacement
* ``thrust::identity<T>``: Use ``cuda::std::identity`` instead. If ``thrust::identity`` was used to perform a cast to ``T``, please define your own function object.
* ``thrust::input_universal_iterator_tag``: No replacement
* ``thrust::negation_value<T>``: Use ``cuda::std::bool_constant<!T>`` instead
* ``thrust::negation_value_v<T>``: Use a plain negation ``!T``
* ``thrust::not[1|2]``: Use ``cuda::std::not_fn`` instead
* ``thrust::null_type``: No replacement
* ``thrust::numeric_limits<T>``: Use ``cuda::std::numeric_limits<T>`` instead
* ``thrust::optional<T>``: Use ``cuda::std::optional<T>`` instead.
* ``thrust::output_universal_iterator_tag``: No replacement
* ``thrust::random_access_universal_iterator_tag``: No replacement
* ``thrust::remove_cvref[_t]``: Use ``cuda::std::remove_cvref[_t]`` instead
* ``thrust::void_t``: Use ``cuda::std::void_t`` instead
Deprecations with planned removal
---------------------------------
* ``CUB_LOG_SMEM_BANKS``: No replacement
* ``CUB_LOG_WARP_THREADS``: No replacement
* ``CUB_MAX_DEVICES``: No replacement
* ``CUB_PREFER_CONFLICT_OVER_PADDING``: No replacement
* ``CUB_PTX_LOG_SMEM_BANKS``: No replacement
* ``CUB_PTX_LOG_WARP_THREADS``: No replacement
* ``CUB_PTX_PREFER_CONFLICT_OVER_PADDING``: No replacement
* ``CUB_PTX_SMEM_BANKS``: No replacement
* ``CUB_PTX_SUBSCRIPTION_FACTOR``: No replacement
* ``CUB_PTX_WARP_THREADS``: No replacement
* ``CUB_SMEM_BANKS``: No replacement
* ``CUB_SUBSCRIPTION_FACTOR``: No replacement
* ``CUB_WARP_THREADS``: No replacement
* ``THRUST_FALSE``: No replacement
* ``THRUST_PREVENT_MACRO_SUBSTITUTION``: No replacement
* ``THRUST_STATIC_ASSERT(expr)``: Use ``static_assert(expr)`` instead
* ``THRUST_TRUE``: No replacement
* ``THRUST_UNKNOWN``: No replacement
* ``THRUST_UNUSED_VAR``: No replacement
* ``cub::BFE``: Use ``cuda::bitfield_extract`` instead
* ``cub::MergePathSearch``: No replacement
* ``cub::Traits<T>::Max()``: Use ``cuda::std::numeric_limits<T>::max()`` instead
* ``cub::Traits<T>::Min()``: Use ``cuda::std::numeric_limits<T>::min()`` instead
* ``thrust::iterator_difference[_t]<T>``: Use ``cuda::std::iterator_traits<T>::difference_type`` or ``cuda::std::iter_difference_t<T>`` instead
* ``thrust::iterator_pointer[_t]<T>``: Use ``cuda::std::iterator_traits<T>::pointer`` instead
* ``thrust::iterator_reference[_t]<T>``: Use ``cuda::std::iterator_traits<T>::reference`` or ``cuda::std::iter_reference_t<T>`` instead
* ``thrust::iterator_traits<T>``: Use ``cuda::std::iterator_traits<T>`` instead
* ``thrust::iterator_value[_t]<T>``: Use ``cuda::std::iterator_traits<T>::value_type`` or ``cuda::std::iter_value_t<T>`` instead
API breaks
----------
* ``cub::Block*``: All trailing ``int LEGACY_PTX_ARCH`` template parameters have been removed
* ``cub::CachingAllocator``: The constructor taking a trailing ``bool debug`` parameter has been removed
* ``cub::Device*``: All overloads with a trailing ``bool debug_synchronous`` parameter have been removed
* ``cub::Dispatch*``: All Boolean template parameters have been replaced by enumerations to increase readability
* ``cub::Dispatch*``: All policy hub template parameters have been moved to the back of the template parameters list
* ``cub::DispatchScan[ByKey]``: The offset type must be an unsigned type of at least 4-byte size
* ``cuda::ceil_div``: Now returns the common type of its arguments
* ``thrust::pair``: Is now an alias to ``cuda::std::pair`` and no longer a distinct type
* ``thrust::tabulate_output_iterator``: The ``value_type`` has been fixed to be ``void``
* ``thrust::transform_iterator``: Upon copying, will now always copy its contained function. If the contained function is neither copy constructible nor copy assignable, the iterator fails to compile when attempting to be copied.
* ``thrust::tuple``: Is now an alias to ``cuda::std::tuple`` and no longer a distinct type
* ``thrust::universal_host_pinned_memory_resource``: The alias has changed to a different memory resource, potentially changing pointer types derived from an allocator/container using this memory resource.
* The following Thrust function object types have been made aliases to the equally-named types in ``cuda::std::``: ``thrust::[plus|minus|multiplies|divides|modulus|negate|equal_to|not_equal_to|greater|less|greater_equal|less_equal|logical_and|logical_or|logical_not|bit_and|bit_or|bit_xor|identity|maximum|minimum]``. No replacement.
* ``CUB_DEFINE_DETECT_NESTED_TYPE``: The generated detector trait no longer provides a ``::VALUE`` member. Use ``::value`` instead.
Iterator traits
^^^^^^^^^^^^^^^
``cuda::std::iterator_traits`` will now correctly recognize user-provided specializations of ``std::iterator_traits``.
All of Thrust's iterator traits have been redefined in terms of ``cuda::std::iterator_traits``,
and users should prefer to use iterator traits from libcu++.
``thrust::iterator_traits`` can no longer be specialized.
Users should prefer to specialize ``cuda::std::iterator_traits`` instead of ``std::iterator_traits`` when necessary,
to make their iterators work equally in device code.
CUB Traits
^^^^^^^^^^
The functionality and internal use of ``cub::Traits`` has been minimized, because libcu++ provides better and standard alternatives.
Only the use in CUB's radix sort implementation for bit-twiddling remains.
Floating-point limits should be obtained using ``cuda::std::numeric_limits<T>`` instead of ``cub::FpLimits<T>``.
Classification of types should be done with the facilities from ``<cuda/std/type_traits>`` and ``<cuda/type_traits>``,
notably with ``cuda::std::is_signed[_v]``, ``cuda::std::is_integral[_v]``, etc.
There is an important difference for extended floating point types though:
Since ``cuda::std::is_floating_point[_v]`` will only recognize C++ standard floating point types,
``cuda::is_floating_point[_v]`` must be used to correctly classify extended floating point types like ``__half`` or ``__nv_bfloat16``.
``cub::BaseTraits`` and ``cub::Traits`` can no longer be specialized for custom types, and ``cub::FpLimits`` has been removed.
We acknowledge the need to provide user-defined floating point types though,
e.g., registering a custom half type with CUB to be used in radix sort.
Therefore, users can still specialize ``cub::NumericTraits`` for their custom floating point types,
inheriting from ``cub::BaseTraits`` and providing the necessary information for the type.
Additionally, the traits from libcu++ have to be specialized as well:
For example, a custom floating point type ``my_half`` could be registered with CUB and libcu++ like this:
.. code:: cpp
template <>
inline constexpr bool ::cuda::is_floating_point_v<my_half> = true;
template <>
class ::cuda::std::numeric_limits<my_half> {
public:
static constexpr bool is_specialized = true;
static __host__ __device__ my_half max() { return /* TODO */; }
static __host__ __device__ my_half min() { return /* TODO */; }
static __host__ __device__ my_half lowest() { return /* TODO */; }
};
template <>
struct CUB_NS_QUALIFIER::NumericTraits<my_half> : BaseTraits<FLOATING_POINT, true, uint16_t, my_half> {};
Behavioral changes
------------------
* ``cub::DeviceReduce::[Arg][Max|Min]``: Will now use ``cuda::std::numeric_limits<T>::[max|min]()`` instead of ``cub::Traits`` to determine the initial value
* ``cuda::std::mdspan``: The implementation was entirely rewritten and you may experience subtle behavioral changes
* ``thrust::transform_iterator``: The logic to determine the reference type has been reworked, especially wrt. to functions that return references to their own arguments (e.g., ``thrust::identity``).
* ``thrust::transform_iterator::difference_type``: The logic to select the difference type has been reworked. It's now either ``int`` or ``ptrdiff``.
ABI breaks
----------
* All of libcu++'s old ABI namespaces have been removed
Platform support
----------------
* At least C++17 is required
* At least clang 14 is required
* At least GCC 7 is required
* On Windows, at least Visual Studio 2019 is required (MSC_VER >= 1920)
* Intel ICC (``icpx``) is no longer supported
* At least CUDA Toolkit 12.0 is required
* Support for CUDA Dynamic Parallelism V1 (CDPv1) has been removed
* At least a GPU with compute capability 50 (Maxwell) is required

94
cccl_upstream/docs/cccl/config_macros.rst Normal file
View File

@@ -0,0 +1,94 @@
.. _cccl-config:
CCCL configuration macros
=========================
The CUDA Core Compute Libraries provide a set of macros to enable or disable specific features. These macros must be defined before any CCCL source file is included. The recommended way is to define them as the predefined compiler macros, for example:
.. code-block:: bash
nvcc -DCCCL_DISABLE_SOME_FEATURE src.cu
.. important::
These macros should be defined consistently in the whole project. Defining them only for some translation units may lead to unexpected compile time and runtime behaviour.
Assertion Control Macros
------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_ENABLE_ASSERTIONS | Enables assertions in both host and device code. Implied by compiling in debug mode. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_ENABLE_DEVICE_ASSERTIONS | Enables assertions in device code, independent of debug mode. Implied by ``CCCL_ENABLE_ASSERTIONS``. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_ENABLE_HOST_ASSERTIONS | Enables assertions in host code, independent of debug mode. Implied by ``CCCL_ENABLE_ASSERTIONS``. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
C++ Feature Control Macros
--------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_EXCEPTIONS | Disables throwing exceptions. Each ``throw`` is replaced by a call to ``cuda::std::terminate()``. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_RTTI | Disables use of runtime type information. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_MSVC_TRADITIONAL_PREPROCESSOR_WARNING | Disables diagnostics emitted when using MSVC's traditional preprocessor. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
CUDA Feature Control Macros
---------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_ARCH_DEPENDENT_NAMESPACE | Disables architecture dependent name mangling of kernels. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_CDP | Disables use of CUDA Dynamic Parallelism. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_CTK_COMPATIBILITY_CHECK | Disables the check whether NVCC's version matches the CUDA Toolkit version. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_DEVICE_RUNTIME | Disables use of CUDA device runtime APIs (``<cuda_device_runtime.h>``), thus makes some APIs that are ``__host__ __device__`` to be ``__host__`` only. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_LAUNCH_BOUNDS | Disables use of ``__launch_bounds__`` attribute. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_PDL | Disables use of Programmatic Dependent Launch. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Deprecation Diagnostics Suppression Macros
------------------------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_DEPRECATED_API | Disables deprecated API diagnostics. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_DEPRECATED_COMPILER | Disables deprecated compiler diagnostics. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_DEPRECATED_CPP_DIALECT | Disables deprecated C++ dialect diagnostics. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_IGNORE_DEPRECATED_CUDA_BELOW_12 | Disables deprecated CUDA compiler diagnostics. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Third Party Libraries Interoperability
--------------------------------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_DLPACK | Disables inclusion of DLPack header and APIs. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
Type Support
------------
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_BF16_SUPPORT | Disables use and library support for the ``__nv_bfloat16`` type. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_FLOAT128_SUPPORT | Disables use and library support for the ``__float128`` type. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_FP16_SUPPORT | Disables use and library support for the ``__half`` type. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_INT128_SUPPORT | Disables use and library support for the ``__int128`` type. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_LONG_DOUBLE_SUPPORT | Disables use and library support for the ``long double`` type. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_NVFP4_SUPPORT | Disables use and library support for the ``__nv_fp4_eNmM`` types. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_NVFP6_SUPPORT | Disables use and library support for the ``__nv_fp6_eNmM`` types. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_DISABLE_NVFP8_SUPPORT | Disables use and library support for the ``__nv_fp8_eNmM`` types. Also disables support for smaller NV floating point types. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_ENABLE_EXPERIMENTAL_HOST_ATOMICS_128B | Enables experimental support for 128b atomics in host code. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| CCCL_GCC_HAS_EXTENDED_NUMERIC_LITERALS | Must be enabled in addition to passing ``-fext-numeric-literals`` with GCC to enable ``__float128`` support. |
+---------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+
.. note::
``<cuda_fpN.h>`` headers contain dependencies on other ``<cuda_fpM.h>`` headers, thus for example defining ``CCCL_DISABLE_BF16_SUPPORT`` will disable support for NVIDIA 8-bit, 6-bit and 4-bit floating point types, too.

14
cccl_upstream/docs/cccl/contributing.rst Normal file
View File

@@ -0,0 +1,14 @@
.. _cccl-contributing:
Contributing to the CUDA Core Compute Libraries
===============================================
.. toctree::
:maxdepth: 1
contributing/code_of_conduct
We welcome contributions - just send us a pull request!
You can find detailed instructions on `GitHub <https://github.com/NVIDIA/cccl/blob/main/CONTRIBUTING.md>`_.
libcu++ uses the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.

105
cccl_upstream/docs/cccl/contributing/code_of_conduct.rst Normal file
View File

@@ -0,0 +1,105 @@
Code of Conduct
===============
Overview
--------
This document defines the Code of Conduct followed and enforced for
NVIDIA C++ Core Compute Libraries.
Intended Audience
~~~~~~~~~~~~~~~~~
- Community
- Developers
- Project Leads
Our Pledge
----------
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our
project and our community a harassment-free experience for everyone,
regardless of age, body size, disability, ethnicity, sex
characteristics, gender identity and expression, level of experience,
education, socio-economic status, nationality, personal appearance,
race, religion, or sexual identity and orientation.
Our Standards
-------------
Examples of behavior that contributes to creating a positive environment
include:
- Using welcoming and inclusive language.
- Being respectful of differing viewpoints and experiences.
- Gracefully accepting constructive criticism.
- Focusing on what is best for the community.
- Showing empathy towards other community members.
Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery and unwelcome sexual
attention or advances.
- Trolling, insulting/derogatory comments, and personal or political
attacks.
- Public or private harassment.
- Publishing others' private information, such as a physical or
electronic address, without explicit permission.
- Other conduct which could reasonably be considered inappropriate.
Our Responsibilities
--------------------
Project maintainers are responsible for clarifying the standards of
acceptable behavior and are expected to take appropriate and fair
corrective action in response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit,
or reject comments, commits, code, wiki edits, issues, and other
contributions that are not aligned to this Code of Conduct, or to ban
temporarily or permanently any contributor for other behaviors that they
deem inappropriate, threatening, offensive, or harmful.
Scope
-----
This Code of Conduct applies both within project spaces and in public
spaces when an individual is representing the project or its community.
Examples of representing a project or community include using an
official project email address, posting via an official social media
account, or acting as an appointed representative at an online or
offline event. Representation of a project may be further defined and
clarified by project maintainers.
Enforcement
-----------
Instances of abusive, harassing, or otherwise unacceptable behavior may
be reported by contacting cpp-conduct@nvidia.com. All complaints will be
reviewed and investigated and will result in a response that is deemed
necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an
incident. Further details of specific enforcement policies may be posted
separately.
Project maintainers who do not follow or enforce the Code of Conduct in
good faith may face temporary or permanent repercussions as determined
by other members of the project's leadership.
Attribution
-----------
This Code of Conduct was taken from the `NVIDIA
RAPIDS <https://docs.rapids.ai/resources/conduct/>`_ project, which was
adapted from the `Contributor Covenant version
1.4 <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>`_.
Please see this `FAQ <https://www.contributor-covenant.org/faq>`_ for
answers to common questions about this Code of Conduct.
Contact
-------
Please email cpp-conduct@nvidia.com for any Code of Conduct related
matters.

106
cccl_upstream/docs/cccl/determinism.rst Normal file
View File

@@ -0,0 +1,106 @@
.. _cccl-determinism:
Determinism
===========
Determinism describes whether an algorithm produces the *same result* every time it is run with the
same input. For many parallel algorithms this is not automatic. For reductions and scans, for example,
the order in which partial results are combined depends on how work is scheduled across thousands of
threads, and that schedule can change between launches or between GPUs. When the combining operator is
not perfectly associative — most notably floating-point addition, where ``(a + b) + c`` need not equal
``a + (b + c)`` — a different combining order yields a (slightly) different result, so the output is no
longer identical from one run to the next.
What counts as the "same result" is defined *per algorithm*. For reductions and scans it means a
*bitwise-identical* output. For other algorithms it can be weaker: a deterministic top-k, for example,
guarantees the same *set* of selected items, while the order of those items within the output is a
separate guarantee that an algorithm may expose on its own.
CCCL lets users state the determinism guarantee they need as an explicit *requirement* on an
algorithm, rather than relying on implementation-defined behavior. The library then either
selects an implementation that satisfies the requirement or rejects the call at compile time if the
requirement cannot be met for the given types and operator.
Determinism guarantees
----------------------
By *reproducible* we mean: given the same inputs, an algorithm returns the same output, in the sense
defined for that algorithm (see above). What the guarantees below differ in is the *scope* of that
reproducibility — across repeated runs, across hardware, or not at all. CCCL models three levels,
defined in ``cuda::execution::determinism``:
``not_guaranteed``
No reproducibility guarantee. The result is a valid answer, but it may differ from one invocation to
the next — even on the same GPU with the same input. This is usually the fastest option.
``run_to_run``
The result is reproducible across repeated runs *on the same GPU*, with the same input, build,
tuning, and launch configuration. It may still differ on a *different* GPU architecture.
``gpu_to_gpu``
The strongest guarantee: the result is reproducible across repeated runs *and across different GPU
architectures* — the same inputs yield the same bits whether the algorithm runs on, say, an Ampere or
a Hopper GPU. This is the most constrained option, is not available for every type/operator
combination, and is typically the slowest.
The guarantees are ordered from weakest to strongest:
``not_guaranteed`` ⊆ ``run_to_run`` ⊆ ``gpu_to_gpu``. A ``gpu_to_gpu`` result is also reproducible
run-to-run, and a ``run_to_run`` result is a valid (but stronger-than-required) answer wherever
``not_guaranteed`` would be accepted.
For types and operators that are exactly associative (see
:ref:`cuda::is_associative_v <libcudacxx-extended-api-functional-operator-properties>`; for example, integral
addition with well-known operators), every invocation is already reproducible across runs and GPUs, so the
stronger guarantees come for free and the library simply selects the fastest valid implementation.
.. warning::
``gpu_to_gpu``/``run_to_run`` reproducibility is guaranteed for a *fixed* CCCL and CUDA Toolkit version, not
across versions. If a policy selector is specified to change the used tuning, then reproducibility is only
guaranteed for identical tunings. The bitwise result may also change between CCCL or CUDA Toolkit releases as
algorithms, reduction structures, or tuning evolve.
Requesting a determinism guarantee
-----------------------------------
Determinism is expressed as a *requirement* and passed to an algorithm through its execution
environment using ``cuda::execution::require``:
.. code-block:: c++
#include <cuda/execution>
// Request run-to-run reproducibility for this call.
auto env = cuda::execution::require(cuda::execution::determinism::run_to_run);
The requirement may be combined with other environment properties — such as a stream or a memory
resource — into a single environment:
.. code-block:: c++
auto determinism = cuda::execution::require(cuda::execution::determinism::run_to_run);
auto env = cuda::std::execution::env{cuda::stream_ref{stream}, memory_resource, determinism};
Passing a determinism property *without* wrapping it in ``require`` is a compile-time error
(*"Determinism should be used inside requires to have an effect."*). ``require`` turns the property
into a *requirement*, which is what the algorithm honors — this prevents a stray determinism property
from being silently ignored.
If an algorithm cannot satisfy the requested guarantee for the given value type and operator, the call
fails to compile with a diagnostic explaining the constraint. If the guarantee can be satisfied by a
weaker-but-sufficient implementation (for example, an exactly-associative operator under
``gpu_to_gpu``), the library transparently selects it.
Where it is used
----------------
Determinism requirements are consumed today by several ``cub`` device algorithms. See the
:ref:`CUB determinism guide <cub-determinism>` for the per-algorithm support matrix, the exact
type/operator constraints, and some examples.
Further reading
---------------
- `Controlling Floating-Point Determinism in NVIDIA CCCL
<https://developer.nvidia.com/blog/controlling-floating-point-determinism-in-nvidia-cccl/>`_ — a
deeper walkthrough of the three guarantees and the implementation strategies behind them.

136
cccl_upstream/docs/cccl/development/build_and_bisect_tools.rst Normal file
View File

@@ -0,0 +1,136 @@
.. _build-and-bisect-tools:
Build and Bisect Utilities
==========================
``build_and_test_targets.sh``
-----------------------------
:file:`ci/util/build_and_test_targets.sh` configures, builds, and tests selected
CMake targets.
Options
~~~~~~~
- ``--preset <name>`` - choose a CMake preset.
- ``--cmake-options <str>`` - extra arguments for the preset configuration.
- ``--configure-override <cmd>`` - run a custom configuration command instead of
a preset. When used, ``--preset`` and ``--cmake-options`` are ignored.
- ``--build-targets <targets>`` - space separated Ninja targets. If omitted,
nothing builds.
- ``--ctest-targets <regex>`` - space separated CTest ``-R`` patterns. If
omitted, nothing runs.
- ``--lit-precompile-tests <paths>`` - space separated libcudacxx lit test paths
to precompile (no run). Paths are relative to ``libcudacxx/test/libcudacxx/``.
- ``--lit-tests <paths>`` - space separated libcudacxx lit test paths to execute.
Paths are relative to ``libcudacxx/test/libcudacxx/``.
- ``--custom-test-cmd <cmd>`` - arbitrary command executed after build/tests.
Combine with ``.devcontainer/launch.sh -d`` to reproduce CI commands inside a
container and choose a CUDA toolkit and host compiler:
``.devcontainer/launch.sh -d [--cuda <XX.Y>] [--host <compiler>] [--gpus all] -- <script>``
Examples
~~~~~~~~
Build a single CUB test locally::
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator"
Build the same test for SM90 using a CMake option::
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--cmake-options "-DCMAKE_CUDA_ARCHITECTURES=90" \
--build-targets "cub.cpp20.test.iterator"
Build the test for SM90 with a configure override::
ci/util/build_and_test_targets.sh \
--configure-override "ci/build_cub.sh -configure -arch 90" \
--build-targets "cub.cpp20.test.iterator"
Build **and run** a single CUB test locally::
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator"
Build and run a single CUB test in a devcontainer with specific CTK and host::
.devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator"
Precompile the libcudacxx lit suite::
ci/util/build_and_test_targets.sh \
--preset libcudacxx \
--build-targets libcudacxx.test.lit.precompile
Precompile a single libcudacxx lit test (no execution)::
ci/util/build_and_test_targets.sh \
--preset libcudacxx \
--lit-precompile-tests \
"std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
Execute one or more libcudacxx lit tests::
ci/util/build_and_test_targets.sh \
--preset libcudacxx \
--lit-tests \
"std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
``git_bisect.sh``
-----------------
:file:`ci/util/git_bisect.sh` wraps ``git bisect`` around the build/test helper.
It accepts all ``build_and_test_targets.sh`` options plus:
- ``--good-ref <rev>`` - Optional; known good commit, tag, or branch. ``-Nd`` means
"N days ago." Defaults to the latest release version tag.
- ``--bad-ref <rev>`` - Optional; known bad commit. ``-Nd`` means "N days ago."
Defaults to ``origin/main``.
Examples
~~~~~~~~
Local CUB bisection from latest release to origin/main::
ci/util/git_bisect.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator"
Devcontainer CUB bisection from last week::
.devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \
ci/util/git_bisect.sh \
--preset cub-cpp20 \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator" \
--good-ref -7d
Compute-sanitizer example for regression introduced between 3-4 weeks ago::
.devcontainer/launch.sh -d --cuda 12.9 --host gcc13 --gpus all \
--env CCCL_TEST_MODE=compute-sanitizer-initcheck \
--env C2H_SEED_COUNT_OVERRIDE=1 \
-- ci/util/git_bisect.sh \
--preset "cub-cpp20" \
--build-targets "cub.cpp20.test.iterator" \
--ctest-targets "cub.cpp20.test.iterator" \
--good-ref -28d \
--bad-ref -21d
Workflow/Bisect
---------------
A ``Workflow/Bisect`` GitHub Actions job runs ``git_bisect.sh`` on a remote
runner. Launch it from **Actions → Git Bisect → Run workflow**. Provide
any desired runner label, refs, preset, targets, or launch arguments. The job
log streams bisect progress, and the run's **Summary** page renders the final
Markdown report with culprit commit, PR, reproduction steps, and more.

95
cccl_upstream/docs/cccl/development/debugger_setup.rst Normal file
View File

@@ -0,0 +1,95 @@
.. _cccl-development-module-debugger-setup:
=================
General Debugging
=================
Debugger Pretty Printers
========================
libcudacxx ships custom pretty printers for its types under
``libcudacxx/share/libcudacxx``. They render CCCL types in a readable form and, for
device-accessible data, copy the contents back to the host so the elements can be
inspected. Two independent implementations are provided:
- ``libcudacxx/share/libcudacxx/gdb`` - printers for GDB.
- ``libcudacxx/share/libcudacxx/lldb`` - printers for LLDB.
Each directory has an ``__init__.py`` entry point that registers every printer. The
repository root contains a ``.gdbinit`` and a ``.lldbinit`` that load the matching entry
point for you, so the simplest way to enable the printers is to let the debugger pick up
these files.
.. important::
``lldb`` and ``gdb`` only inspect the following locations when looking for init
dotfiles (in the given order):
#. Home directory config files (usually ``~/.lldbinit`` or ``~/.config/gdb/gdbinit`` on
Linux, but check the respective manuals for specifics).
#. The current working directory.
They do **not** walk up the directory stack like most tools. So if you have a
``.lldbinit`` in the parent directory, ``lldb`` will **not** load it. For this reason,
you **must** run the debugger from the root CCCL directory in order for automatic
loading of the pretty printers to work.
In addition to not loading parent directory dot-files, ``gdb`` or ``lldb`` will load not
dotfiles unless you explicitly allow them. The following sections explain how to enable
this for each debugger.
.. note::
The following is **not** needed when working inside a devcontainer. devcontainers
already have the following set up.
If they don't, and automatic loading of the pretty printers does not work, then this is
a bug and should be fixed.
It is only needed for bare metal builds.
GDB
---
By default GDB does not source a ``.gdbinit`` from the current directory, and it guards
auto-loaded scripts with the ``auto-load safe-path`` setting. Add the repository root to
your ``~/.gdbinit`` (or ``~/.config/gdb/gdbinit`` if you have ``XDG_CONFIG_HOME`` set) so
the project's ``.gdbinit`` is trusted and loaded::
add-auto-load-safe-path /absolute/path/to/cccl
set auto-load local-gdbinit on
Launch GDB from the repository root and the printers should register automatically.
Verify that the printers are active with ``info pretty-printer``.
To load the printers without depending on the working directory - for example
from a global ``~/.gdbinit`` - ``source`` the entry point by absolute path
instead::
source /absolute/path/to/cccl/libcudacxx/share/libcudacxx/gdb/__init__.py
``source`` runs the script directly and is not subject to the ``auto-load safe-path``
restriction.
LLDB
----
LLDB only reads ``.lldbinit`` from your home directory unless you opt in to loading one
from the current working directory. Enable that once in your ``~/.lldbinit`` (``lldb``
seemingly does not respect ``XDG_CONFIG_HOME``)::
settings set target.load-cwd-lldbinit true
This is a trust decision, since the local file runs arbitrary Python. Launch LLDB from the
repository root and the project's ``.lldbinit`` imports the formatters automatically.
To load the formatters without depending on the working directory, add the absolute path
to your ``~/.lldbinit`` instead::
command script import "/absolute/path/to/cccl/libcudacxx/share/libcudacxx/lldb/__init__.py"
The entry point's ``__lldb_init_module`` hook defines and enables an LLDB type category
for the formatters. Print any CCCL value with the usual commands (``v``, ``frame
variable``, or ``dwim-print``).

23
cccl_upstream/docs/cccl/development/index.rst Normal file
View File

@@ -0,0 +1,23 @@
.. _cccl-development-module:
CCCL Development Guide
======================
.. toctree::
:hidden:
:maxdepth: 1
macro
testing
build_and_bisect_tools
visibility
debugger_setup
This living document serves to describe the internal details and the development process of CCCL libraries.
Documentation:
- :doc:`CCCL Internal Macros <macro>`
- :doc:`CCCL Testing Utilities <testing>`
- :doc:`CCCL Bisect And Targeted Build/Test Helpers <build_and_bisect_tools>`
- :doc:`General Debugging <debugger_setup>`

553
cccl_upstream/docs/cccl/development/macro.rst Normal file
View File

@@ -0,0 +1,553 @@
.. _cccl-development-module-macros:
CCCL Internal Macros
====================
The document describes the main *internal* macros used by CCCL. They are not intended to be used by end users, but for development of CCCL features only. We reserve the right to change them at any time without warning.
----
Compiler Macros
---------------
**Host compiler macros**:
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(CLANG)`` | Clang |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(GCC)`` | GCC |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(NVHPC)`` | Nvidia HPC compiler |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(MSVC)`` | Microsoft Visual Studio |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(MSVC2019)`` | Microsoft Visual Studio 2019 |
+------------------------------+---------------------------------------------+
| ``_CCCL_COMPILER(MSVC2022)`` | Microsoft Visual Studio 2022 |
+------------------------------+---------------------------------------------+
The ``_CCCL_COMPILER`` function-like macro can also be used to check the version of a compiler.
.. code:: cpp
_CCCL_COMPILER(MSVC, <, 19, 24)
_CCCL_COMPILER(GCC, >=, 9)
*Note*: When used without specifying a minor version number, the macro will only test against
the compiler's major version number. For example, when the compiler is ``gcc-9.1``, the macro
``_CCCL_COMPILER(GCC, >, 9)`` will be ``false`` even though ``9.1`` is greater than ``9``.
**CUDA compiler macros**:
+--------------------------------+-------------------------+
| ``_CCCL_CUDA_COMPILER(NVCC)`` | Nvidia compiler |
+--------------------------------+-------------------------+
| ``_CCCL_CUDA_COMPILER(NVHPC)`` | Nvidia HPC compiler |
+--------------------------------+-------------------------+
| ``_CCCL_CUDA_COMPILER(NVRTC)`` | Nvidia Runtime Compiler |
+--------------------------------+-------------------------+
| ``_CCCL_CUDA_COMPILER(CLANG)`` | Clang |
+--------------------------------+-------------------------+
The ``_CCCL_CUDA_COMPILER`` function-like macro can also be used to check the version of a CUDA compiler.
.. code:: cpp
_CCCL_CUDA_COMPILER(NVCC, <, 12, 3)
_CCCL_CUDA_COMPILER(CLANG, >=, 14)
*Note*: ``_CCCL_CUDA_COMPILER(...)`` check may result in a ``true`` value even during the compilation of a C++ source
file. Use ``_CCCL_CUDA_COMPILATION()`` to check for the compilation of a CUDA source file.
**CUDA identification/version macros**:
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_CUDA_COMPILATION()`` | CUDA code is being compiled |
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_HOST_COMPILATION()`` | Compiling host code, ``true`` when executing the CUDA host pass or compiling a C++ source file |
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_DEVICE_COMPILATION()`` | Compiling device code, ``true`` when executing the CUDA device pass |
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_CUDACC_BELOW(12, 7)`` | CUDA version below 12.7 when compiling a CUDA source file |
+----------------------------------+------------------------------------------------------------------------------------------------+
| ``_CCCL_CUDACC_AT_LEAST(12, 7)`` | CUDA version at least 12.7 when compiling a CUDA source file |
+----------------------------------+------------------------------------------------------------------------------------------------+
*Note*: When compiling CUDA code with ``nvc++`` both ``_CCCL_HOST_COMPILATION()`` and ``_CCCL_DEVICE_COMPILATION()`` result in a ``true`` value.
**PTX macros**:
+----------------------+-------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_PTX_ARCH()`` | Alias of ``__CUDA_ARCH__`` with value equal to 0 if a CUDA compiler is not available |
+----------------------+-------------------------------------------------------------------------------------------------------------------+
| ``__cccl_ptx_isa`` | PTX ISA version available with the current CUDA compiler, e.g. PTX ISA 8.4 (``840``) is available from CUDA 12.4 |
+----------------------+-------------------------------------------------------------------------------------------------------------------+
*Note*: When compiling CUDA code with ``nvc++`` the ``_CCCL_PTX_ARCH()`` macro expands to ``0``.
----
Architecture Macros
-------------------
The following macros are used to check the target architecture. They comply with the compiler supported by the CUDA toolkit. Compilers outside the CUDA toolkit may define such macros in a different way.
+------------------------------+---------------------------------------------------+
| ``_CCCL_HOST_ARCH(ARM64)`` | ARM 64-bit, including MSVC emulation |
+------------------------------+---------------------------------------------------+
| ``_CCCL_HOST_ARCH(X86_64)`` | X86 64-bit. False on ARM 64-bit MSVC emulation |
+------------------------------+---------------------------------------------------+
----
OS Macros
---------
+-----------------------+---------------------------------+
| ``_CCCL_OS(WINDOWS)`` | Windows, including NVRTC LLP64 |
+-----------------------+---------------------------------+
| ``_CCCL_OS(LINUX)`` | Linux, including NVRTC LP64 |
+-----------------------+---------------------------------+
| ``_CCCL_OS(ANDROID)`` | Android |
+-----------------------+---------------------------------+
| ``_CCCL_OS(QNX)`` | QNX |
+-----------------------+---------------------------------+
----
Execution Space
---------------
**Functions**
+-----------------------+-----------------------+
| ``_CCCL_HOST`` | Host function |
+-----------------------+-----------------------+
| ``_CCCL_DEVICE`` | Device function |
+-----------------------+-----------------------+
| ``_CCCL_HOST_DEVICE`` | Host/Device function |
+-----------------------+-----------------------+
In addition, ``_CCCL_EXEC_CHECK_DISABLE`` disables the execution space check for the NVHPC compiler
**Target Macros**
+---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
| ``NV_IF_TARGET(TARGET, (CODE))`` | Enable ``CODE`` only if ``TARGET`` is satisfied. |
+---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
| ``NV_IF_ELSE_TARGET(TARGET, (IF_CODE), (ELSE_CODE))`` | Enable ``CODE_IF`` if ``TARGET`` is satisfied, ``CODE_ELSE`` otherwise. |
+---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
| ``NV_DISPATCH_TARGET(TARGET1, (TARGET1_CODE), ..., TARGET_N, (TARGET_N_CODE))`` | Enable a single code block if any of ``TARGET_i`` is satisfied. |
+---------------------------------------------------------------------------------+--------------------------------------------------------------------------+
Possible ``TARGET`` values:
+---------------------------+-------------------------------------------------------------------+
| ``NV_ANY_TARGET`` | Any target |
+---------------------------+-------------------------------------------------------------------+
| ``NV_IS_HOST`` | Host-code target |
+---------------------------+-------------------------------------------------------------------+
| ``NV_IS_DEVICE`` | Device-code target |
+---------------------------+-------------------------------------------------------------------+
| ``NV_PROVIDES_SM_<VER>`` | SM architecture is at least ``VER``, e.g. ``NV_PROVIDES_SM_80`` |
+---------------------------+-------------------------------------------------------------------+
| ``NV_IS_EXACTLY_SM_<NN>`` | SM architecture is exactly ``VER``, e.g. ``NV_IS_EXACTLY_SM_80`` |
+---------------------------+-------------------------------------------------------------------+
Usage example:
.. code-block:: c++
NV_IF_TARGET(NV_IS_DEVICE, (auto x = threadIdx.x; return x;));
NV_IF_ELSE_TARGET(NV_IS_HOST, (return 0;), (auto x = threadIdx.x; return x;));
NV_DISPATCH_TARGET(NV_PROVIDES_SM_90, (return "Hopper+";),
NV_IS_EXACTLY_SM_75, (return "Turing";),
NV_IS_HOST, (return "Host";))
*Pitfalls*:
* All target macros generate the code in a local scope, i.e. ``{ code }``.
* ``NV_DISPATCH_TARGET`` is *NOT* a switch statement. It enables the code associated with the first condition satisfied.
* The target macros take ``code`` as an argument, so it is *not* possible to use any conditional compilation, .e.g ``#if _CCCL_STD_VER >= 20`` within a target macro
----
CUDA attributes
---------------
+------------------------------+----------------------------------------------------------+
| ``_CCCL_GRID_CONSTANT`` | Grid constant kernel parameter |
+------------------------------+----------------------------------------------------------+
| ``_CCCL_GLOBAL_CONSTANT`` | Host/device global scope constant (``inline constexpr``) |
+------------------------------+----------------------------------------------------------+
----
CUDA Toolkit macros
-------------------
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_CTK()`` | CUDA toolkit is available if ``_CCCL_CUDA_COMPILER()`` evaluates to a ``true`` value or if ``cuda_runtime_api.h`` was found |
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_CTK_BELOW`` | CUDA toolkit version below 12.7 |
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_CTK_AT_LEAST(12, 7)`` | CUDA toolkit version at least 12.7 |
+-------------------------------+-----------------------------------------------------------------------------------------------------------------------------+
Non-standard Types Support
--------------------------
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_INT128()`` | ``__int128`` and ``__uint128_t`` for 128-bit integer are supported and enabled |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_NVFP8()`` | ``__nv_fp8_e5m2/__nv_fp8_e4m3/__nv_fp8_e8m0`` data types are supported and enabled. Prefer over ``__CUDA_FP8_TYPES_EXIST__`` |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_NVFP16()`` | ``__half/__half2`` data types are supported and enabled. Prefer over ``__CUDA_FP16_TYPES_EXIST__`` |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_NVBF16()`` | ``__nv_bfloat16/__nv_bfloat162`` data types are supported and enabled. Prefer over ``__CUDA_BF16_TYPES_EXIST__`` |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_HAS_FLOAT128()`` | ``__float128`` for 128-bit floating-point are supported and enabled |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------+
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_INT128_SUPPORT`` | Disable ``__int128/__uint128_t`` support |
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_NVFP8_SUPPORT`` | Disable ``__nv_fp8_e5m2/__nv_fp8_e4m3/__nv_fp8_e8m0`` support |
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_NVFP16_SUPPORT`` | Disable ``__half/__half2`` support |
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_NVBF16_SUPPORT`` | Disable ``__nv_bfloat16/__nv_bfloat162`` support |
+-----------------------------------+-------------------------------------------------------------------------+
| ``CCCL_DISABLE_FLOAT128_SUPPORT`` | Disable ``__float128`` support |
+-----------------------------------+-------------------------------------------------------------------------+
+-----------------------------------+-------------------------------------------------------------------------+
| ``_LIBCUDACXX_HAS_NVFP16()`` | ``__half/__half2`` host/device are supported (CUDA 12.2+) |
+-----------------------------------+-------------------------------------------------------------------------+
| ``_LIBCUDACXX_HAS_NVBF16()`` | ``__nv_bfloat16/__nv_bfloat162`` host/device are supported (CUDA 12.2+) |
+-----------------------------------+-------------------------------------------------------------------------+
----
C++ Language Macros
-------------------
The following macros are required only if the target C++ version does not support the corresponding attribute
+-----------------------------+----------------------------------------------------------+
| ``_CCCL_STD_VER`` | C++ standard version, e.g. ``#if _CCCL_STD_VER >= 2017`` |
+-----------------------------+----------------------------------------------------------+
| ``_CCCL_CONSTEXPR_CXX20`` | Enable ``constexpr`` for C++20 or newer |
+-----------------------------+----------------------------------------------------------+
| ``_CCCL_CONSTEXPR_CXX23`` | Enable ``constexpr`` for C++23 or newer |
+-----------------------------+----------------------------------------------------------+
| ``_CCCL_HAS_EXCEPTIONS()`` | Features can use exceptions, e.g ``bad_optional_access`` |
+-----------------------------+----------------------------------------------------------+
**Concept-like Macros**:
+------------------------+--------------------------------------------------------------------------------------------+
| ``_CCCL_TEMPLATE(X)`` | ``template`` clause |
+------------------------+--------------------------------------------------------------------------------------------+
| ``_CCCL_REQUIRES(X)`` | ``requires`` clause |
+------------------------+--------------------------------------------------------------------------------------------+
| ``_CCCL_AND`` | Traits conjunction only used with ``_CCCL_REQUIRES`` |
+------------------------+--------------------------------------------------------------------------------------------+
Usage example:
.. code-block:: c++
_CCCL_TEMPLATE(typename T)
_CCCL_REQUIRES(is_integral_v<T> _CCCL_AND(sizeof(T) > 1))
.. code-block:: c++
_CCCL_TEMPLATE(typename T)
_CCCL_REQUIRES(is_arithmetic_v<T> _CCCL_AND (!is_integral_v<T>))
**Portable feature testing**:
+--------------------------+--------------------------------------------------+
| ``_CCCL_HAS_BUILTIN(X)`` | Portable ``__has_builtin(X)`` |
+--------------------------+--------------------------------------------------+
| ``_CCCL_HAS_FEATURE(X)`` | Portable ``__has_feature(X)`` |
+--------------------------+--------------------------------------------------+
**Portable attributes**:
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_ASSUME(EXPR)`` | Portable ``[[assume]]`` attribute (before C++23) |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_NO_UNIQUE_ADDRESS`` | Portable ``[[no_unique_address]]`` attribute |
+----------------------------------+------------------------------------------------------------------------------+
| ``CCCL_DEPRECATED`` | Portable ``[[deprecated]]`` attribute (before C++14) |
+----------------------------------+------------------------------------------------------------------------------+
| ``CCCL_DEPRECATED_BECAUSE(MSG)`` | Portable ``[[deprecated]]`` attribute with custom message (before C++14) |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_FORCEINLINE`` | Portable "always inline" attribute |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_PURE`` | Portable "pure" function attribute |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_CONST`` | Portable "constant" function attribute |
+----------------------------------+------------------------------------------------------------------------------+
| ``_CCCL_LIFETIMEBOUND`` | Portable "lifetime bound" function attribute |
+----------------------------------+------------------------------------------------------------------------------+
**Portable Builtin Macros**:
+---------------------------------------+--------------------------------------------+
| ``_CCCL_UNREACHABLE()`` | Portable ``__builtin_unreachable()`` |
+---------------------------------------+--------------------------------------------+
| ``_CCCL_BUILTIN_EXPECT(X)`` | Portable ``__builtin_expected(X)`` |
+---------------------------------------+--------------------------------------------+
| ``_CCCL_BUILTIN_PREFETCH(X[, Y, Z])`` | Portable ``__builtin_prefetch(X, Y, Z)`` |
+---------------------------------------+--------------------------------------------+
**Portable Keyword Macros**
+-----------------------------+--------------------------------------------+
| ``_CCCL_RESTRICT`` | Portable ``restrict`` keyword |
+-----------------------------+--------------------------------------------+
| ``_CCCL_ALIGNAS(X)`` | Portable ``alignas(X)`` keyword (variable) |
+-----------------------------+--------------------------------------------+
| ``_CCCL_ALIGNAS_TYPE(X)`` | Portable ``alignas(X)`` keyword (type) |
+-----------------------------+--------------------------------------------+
| ``_CCCL_PRAGMA(X)`` | Portable ``_Pragma(X)`` keyword |
+-----------------------------+--------------------------------------------+
**Portable Pragma Macros**
+--------------------------------+-------------------------------------------+
| ``_CCCL_PRAGMA_UNROLL(N)`` | Portable ``#pragma unroll N`` pragma |
+--------------------------------+-------------------------------------------+
| ``_CCCL_PRAGMA_UNROLL_FULL()`` | Portable ``#pragma unroll`` pragma |
+--------------------------------+-------------------------------------------+
| ``_CCCL_PRAGMA_NOUNROLL()`` | Portable ``#pragma nounroll`` pragma |
+--------------------------------+-------------------------------------------+
**Conditional Constant Evaluation Macros**
In C++23, the ``if consteval`` statement (`link <https://en.cppreference.com/w/cpp/language/if.html>`_) was introduced. CCCL mimics the behaviour with a set of macros that expand to an implementation supported by the compiler. If the compiler doesn't support any kind of conditional constant evaluation, the macros expand to predefined fallback values.
+------------------------------------+-----------------------------------------------------------------------------------+
| ``_CCCL_IF_CONSTEVAL`` | Equivalent to ``if consteval`` statement (fallbacks to ``if constexpr (false)``) |
+------------------------------------+-----------------------------------------------------------------------------------+
| ``_CCCL_IF_CONSTEVAL_DEFAULT`` | Equivalent to ``if consteval`` statement (fallbacks to ``if constexpr (true)``) |
+------------------------------------+-----------------------------------------------------------------------------------+
| ``_CCCL_IF_NOT_CONSTEVAL`` | Equivalent to ``if !consteval`` statement (fallbacks to ``if constexpr (true)``) |
+------------------------------------+-----------------------------------------------------------------------------------+
| ``_CCCL_IF_NOT_CONSTEVAL_DEFAULT`` | Equivalent to ``if !consteval`` statement (fallbacks to ``if constexpr (false)``) |
+------------------------------------+-----------------------------------------------------------------------------------+
**Exception Macros**
CUDA doesn't support exceptions in device code, however, sometimes we need to write host/device functions that use exceptions on host and ``__trap()`` on device. CCCL provides a set of macros that should be used in place of the standard C++ keywords to make the code compile in both, host and device code.
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_TRY`` | Replacement for the ``try`` keyword. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_CATCH (X)`` | Replacement for the ``catch (/*X*/)`` statement. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_CATCH_ALL`` | Replacement for the ``catch (...)`` statement. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_THROW(X, ...)`` | Replacement for the ``throw X(...)`` expression. ``X`` must be fully qualified type, without the leading ``::``. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
| ``_CCCL_RETHROW`` | Replacement for the plain ``throw`` expression. |
+-----------------------------+------------------------------------------------------------------------------------------------------------------+
*Note*: The ``_CCCL_CATCH`` clause must always introduce a named variable, like: ``_CCCL_CATCH(const exception_type& var)``.
.. note::
``_CCCL_THROW`` requires to include the ``<stdexcept>`` header, regardless exceptions are enabled or not.
Example:
.. code-block:: c++
__host__ __device__ void* alloc(cuda::std::size_t nbytes)
{
if (void* ptr = cuda::std::malloc(nbytes))
{
return ptr;
}
_CCCL_THROW(std::bad_alloc); // on device calls cuda::std::terminate()
}
__host__ __device__ void do_something(int* buff)
{
_CCCL_THROW(std::runtime_error, "Something went wrong"); // on device calls cuda::std::terminate()
}
__host__ __device__ void fn(cuda::std::size_t n)
{
int* buff{};
_CCCL_TRY
{
buff = reinterpret_cast<int*>(alloc(n * sizeof(int)));
do_something(buff);
}
_CCCL_CATCH ([[maybe_unused]] const std::bad_alloc& e) // must be always named
{
std::fprintf(stderr, "Failed to allocate memory\n"); // We can directly call host-only functions
cuda::std::terminate();
}
_CCCL_CATCH_ALL // or _CCCL_CATCH_FALLTHOUGH
{
cuda::std::free(buff);
_CCCL_RETHROW;
}
}
__global__ void kernel()
{
fn(10);
}
int main()
{
fn(10);
return 0;
}
----
Visibility Macros
-----------------
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_VISIBILITY_HIDDEN`` | Hidden visibility attribute (e.g. ``__attribute__((visibility("hidden")))``) |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_HIDE_FROM_ABI`` | Hidden visibility (i.e. ``inline``, not exported, not instantiated) |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_API`` | Host/device function with hidden visibility. Most CCCL functions are hidden with this attribute |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_HOST_API`` | Host function with hidden visibility. Most CCCL functions are hidden with this attribute |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_DEVICE_API`` | Device function with hidden visibility. Most CCCL functions are hidden with this attribute |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
| ``_CCCL_KERNEL_ATTRIBUTES`` | Global function with hidden visibility. Most CCCL functions are hidden with this attribute |
+-------------------------------+-----------------------------------------------------------------------------------------------------+
----
Other Common Macros
-------------------
+-----------------------------+--------------------------------------------+
| ``_CCCL_TO_STRING(X)`` | ``X`` to literal string |
+-----------------------------+--------------------------------------------+
| ``_CCCL_DOXYGEN_INVOKED`` | Defined during Doxygen parsing |
+-----------------------------+--------------------------------------------+
----
Debugging Macros
----------------
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``_CCCL_ASSERT(COND, MSG)`` | Portable, conditional CCCL `assert()` macro. Requires (``CCCL_ENABLE_ASSERTIONS`` or a debug build) |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``_CCCL_VERIFY(COND, MSG)`` | Portable, always-on `assert()` reserved for critical checks that are always required |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``_CCCL_ENABLE_ASSERTIONS`` | Enable assertions |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``CCCL_ENABLE_HOST_ASSERTIONS`` | Enable host-side assertions |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``CCCL_ENABLE_DEVICE_ASSERTIONS`` | Enable device-side assertions |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
| ``_CCCL_ENABLE_DEBUG_MODE`` | Enable debug mode (and assertions) |
+-----------------------------------+-------------------------------------------------------------------------------------------------------------+
----
Warning Suppression Macros
--------------------------
+-----------------------------+--------------------------------------------+
| ``_CCCL_DIAG_PUSH`` | Portable ``#pragma push`` |
+-----------------------------+--------------------------------------------+
| ``_CCCL_DIAG_POP`` | Portable ``#pragma pop`` |
+-----------------------------+--------------------------------------------+
**Compiler-specific Suppression Macros**:
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_DIAG_SUPPRESS_CLANG(X)`` | Suppress clang warning, e.g. ``"-Wattributes"`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_DIAG_SUPPRESS_GCC(X)`` | Suppress gcc warning, e.g. ``"-Wattributes"`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_DIAG_SUPPRESS_NVHPC(X)`` | Suppress nvhpc warning, e.g. ``expr_has_no_effect`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_DIAG_SUPPRESS_MSVC(X)`` | Suppress msvc warning, e.g. ``4127`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_BEGIN_NV_DIAG_SUPPRESS(X)`` | Start to suppress nvcc warning, e.g. ``177`` |
+-------------------------------------+-------------------------------------------------------------+
| ``_CCCL_END_NV_DIAG_SUPPRESS()`` | End to suppress nvcc warning |
+-------------------------------------+-------------------------------------------------------------+
Usage example:
.. code-block:: c++
_CCCL_DIAG_PUSH
_CCCL_DIAG_SUPPRESS_GCC("-Wattributes")
// code ..
_CCCL_DIAG_POP
----
Freestanding support
--------------------------
We - partially - support building CCCL headers in freestanding mode, for example JIT compilation with NVRTC.
+-----------------------------+----------------------------------------------------+
| ``_CCCL_HOSTED()`` | "Normal" compilation mode with host STL support |
+-----------------------------+----------------------------------------------------+
| ``_CCCL_FREESTANDING()`` | Freestanding compilation mode, no host STL support |
+-----------------------------+----------------------------------------------------+
| ``_CCCL_HOSTJIT()`` | Freestanding compilation mode, with host compiler |
+-----------------------------+----------------------------------------------------+
Usage example:
.. code-block:: c++
#if _CCCL_HOSTED()
# include <iostream> // Host STL header not available in freestanding
#endif // _CCCL_HOSTED()
// code ..
Similarly we also provide macros to detect which host standard library is available
+-----------------------------------+----------------------------------------------------+
| ``_CCCL_HAS_HOST_STD_LIB()`` | Whether a known host standard library is available |
+-----------------------------------+----------------------------------------------------+
| ``_CCCL_HOST_STD_LIB(LIBSTDCXX)`` | libstdc++ is available as host standard library |
+-----------------------------------+----------------------------------------------------+
| ``_CCCL_HOST_STD_LIB(LIBCXX)`` | libc++ is available as host standard library |
+-----------------------------------+----------------------------------------------------+
| ``_CCCL_HOST_STD_LIB(STL)`` | MSVC STL is available as host standard library |
+-----------------------------------+----------------------------------------------------+
Usage example:
.. code-block:: c++
#if _CCCL_HAS_HOST_STD_LIB()
_CCCL_BEGIN_NAMESPACE_STD
# if _CCCL_HOST_STD_LIB(STL)
template <class _Tp, size_t _Size>
class array;
# else // ^^^ _CCCL_HOST_STD_LIB(STL) ^^^ / vvv !_CCCL_HOST_STD_LIB(STL) vvv
template <class _Tp, size_t _Size>
struct array;
# endif // !_CCCL_HOST_STD_LIB(STL)
_CCCL_END_NAMESPACE_STD
#endif // _CCCL_HAS_HOST_STD_LIB()

138
cccl_upstream/docs/cccl/development/testing.rst Normal file
View File

@@ -0,0 +1,138 @@
.. _cccl-development-module-testing:
======================
CCCL Testing Utilities
======================
This document describes utilities provided for implementing the *internal* CCCL tests.
They are not intended to be used by end users, but for development of CCCL features only.
We reserve the right to change them at any time without warning.
----
-------------------------------------------------------------------
Test Variants: Generating Multiple Executables from a Single Source
-------------------------------------------------------------------
Some of CCCL's tests are very slow to build and are capable of exhausting RAM
during compilation/linking. To avoid such issues, large tests are split into
multiple executables to take advantage of parallel computation and reduce memory
usage.
CCCL facilitates this by providing a CMake-based solution for automatically generating multiple
test executables from a single source file. This is done by using one or more ``%PARAM%`` comments
in the test's source code, each of which defines a parameter that will be split across multiple
executables.
The CMake functions that implement this feature are in ``cmake/CCCLTestParams.cmake``.
An example of their usage is provided below.
Using ``%PARAM%``
-----------------
The ``%PARAM%`` hint provides an automated method of generating multiple test
executables from a single source file. To use it, add one or more special
comments to the test source file::
// %PARAM% [definition] [label] [values]
CMake will parse the source file and extract these comments, using them to
generate multiple test executables for the full cartesian product of values.
- ``definition`` will be used as a preprocessor definition name. By convention,
these begin with ``TEST_``.
- ``label`` is a short, human-readable label that will be used in the test
executable's name to identify the test variant.
- ``values`` is a colon-separated list of values used during test generation. Only
numeric values have been tested.
Example
*******
A source file containing the following hints::
// %PARAM% TEST_FOO foo 0:1:2
// %PARAM% TEST_LAUNCH lid 0:1
will generate six variants with unique preprocessor definitions:
+-----------------------------+-------------------------------------------+
| Executable Name | Preprocessor Definitions |
+=============================+===========================================+
| ``<name_base>.foo_0.lid_0`` | ``-DTEST_FOO=0 -DTEST_LAUNCH=0 VAR_ID=0`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_0.lid_1`` | ``-DTEST_FOO=0 -DTEST_LAUNCH=1 VAR_ID=1`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_1.lid_0`` | ``-DTEST_FOO=1 -DTEST_LAUNCH=0 VAR_ID=2`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_1.lid_1`` | ``-DTEST_FOO=1 -DTEST_LAUNCH=1 VAR_ID=3`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_2.lid_0`` | ``-DTEST_FOO=2 -DTEST_LAUNCH=0 VAR_ID=4`` |
+-----------------------------+-------------------------------------------+
| ``<name_base>.foo_2.lid_1`` | ``-DTEST_FOO=2 -DTEST_LAUNCH=1 VAR_ID=5`` |
+-----------------------------+-------------------------------------------+
Changing ``%PARAM%`` Hints
**************************
Since CMake does not automatically reconfigure the build when source files are
modified, CMake will need to be rerun manually whenever the ``%PARAM%`` comments
change.
Using the CMake Variant Functions
---------------------------------
``cmake/CCCLTestParams.cmake`` provides the functions that implement this functionality.
See that file for detailed documentation. An example of their usage is:
.. code-block:: cmake
set(test_src <path_to_source_file>)
set(test_name <test_name_derived_from_test_src>)
# Parse %PARAM% comments from the source file and generate lists of labels/definitions:
cccl_parse_variant_params("${test_src}" num_variants variant_labels variant_defs)
if (num_variants EQUAL 0)
# Add test with no variants named `test_name` here. Example:
add_executable("${test_name}" "${test_src}")
add_test(NAME "${test_name}" COMMAND "${test_name}")
else() # Has variants:
# Optional: log the detected variant info to CMake's VERBOSE output stream:
cccl_log_variant_params("${test_name}" ${num_variants} variant_labels variant_defs)
# Subtract 1 to support the inclusive endpoint of foreach(...RANGE...):
math(EXPR var_range_end "${num_variants} - 1")
foreach(var_idx RANGE ${var_range_end})
# Get the variant label and definitions for the current index:
cccl_get_variant_data(variant_labels variant_defs ${var_idx} var_label var_defs)
set(var_name "${test_name}.${var_label}")
# Add the test with the current variant label and definitions.
# Example:
add_executable("${var_name}" "${test_src}")
target_compile_definitions("${var_name}" PRIVATE ${var_defs})
add_test(NAME "${var_name}" COMMAND "${var_name}")
endforeach()
endif()
Debugging
---------
Running CMake with ``--log-level=VERBOSE`` will print out extra information about
all detected test variants.
Additional Info
---------------
Ideally, only parameters that directly influence template instantiations
should be split out in this way. If changing a parameter doesn't change a
template type, the same template instantiations will be compiled into multiple
executables. This defeats the purpose of splitting up the test since the
compiler will generate redundant code across the new split executables.
The best candidate parameters for splitting are input value types, rather than
integral parameters like ``BLOCK_THREADS``, etc. Splitting by value type allows more
infrastructure (data generation, validation) to be reused. Splitting other
parameters can cause build times to increase since type-related infrastructure
has to be rebuilt for each test variant.

63
cccl_upstream/docs/cccl/development/visibility.rst Normal file
View File

@@ -0,0 +1,63 @@
. _cccl-development-visibility:
Symbol Visibility
==================
.. toctree::
:glob:
:maxdepth: 1
visibility/host_stub_visibility
visibility/device_kernel_visibility
visibility/different_architectures
Using kernels in shared libraries is a known source of issues. This relates to the visibility of the kernel functions
and their host stubs, but also ODR violations that arise from compiling with different CUDA architectures.
To alleviate those issues we have derived the following rules to ensure that users can safely rely on CCCL features in
shared libraries.
1. Every kernel should be annotated as ``hidden`` through ``_CCCL_KERNEL_ATTRIBUTES``
2. Every function or type that eventually calls a kernel in a subsequent function call or member function must be put in
a namespace that disambiguates the CUDA architectures the library was compiled with.
3. It is important that an API accepting kernel pointers (e.g. ``triple_chevron``) always resides in the same
library as the code taking this pointers.
In the following we will give a more detailed overview over the different problems and why we settled on above rules.
Problem 1: Selecting the right kernel stub
-------------------------------------------
Consider a project that links two shared libraries ``lib_a`` and ``lib_b`` that involve a kernel call of some global
``kernel`` template. The compiler will generate a stub function that handles actually launching the kernel via the CUDA
runtime. Prior to CTK 13.0 that stub function has weak linkage, so if both libraries try to launch ``kernel`` only one
host stub will be selected and the other kernel launch might silently fail. See the compiler teams
`blog post <https://developer.nvidia.com/blog/cuda-c-compiler-updates-impacting-elf-visibility-and-linkage/>`_ about the
recent changes to kernel visibility.
A more detailed description can be found :ref:`here <cccl-development-visibility-host-stub-visibility>`.
Problem 2: Calling kernels from inside a shared library
--------------------------------------------------------
This is quite similar to Problem 1 above. Again a project links two shared libraries ``lib_a`` and ``lib_b``. However,
this time we call a library function ``foo`` that takes a function pointer to a kernel as an argument and invokes it.
If ``foo`` has weak external linkage we might end up calling ``lib_b::foo`` from inside ``lib_b`` instead of
``lib_a::foo``, or vice versa. The CUDA runtime from ``lib_a`` will not be able to call the kernel function pointer we
passed from ``lib_b``.
A more detailed description can be found :ref:`here <cccl-development-visibility-device-kernel-visibility>`.
Problem 3: Libraries compiled for different architectures
----------------------------------------------------------
This is orthogonal to the visibility of the functions themself but relates to ODR
(`_one definition rule_ <https://en.cppreference.com/w/cpp/language/definition.html>`_) violations in case libraries are
compiled for different architectures. As new architectures come out, we adopt new features to provide the best possible
performance for all existing architectures.
However, consider a kernel that relies on hardware dependent tuning or can leverage runtime features that are only
available on certain hardware. If we build 2 libraries for different architectures then the kernel implementation
will be different between the two libraries, but the kernel itself is mangled as the same symbol.
A more detailed description can be found :ref:`here <cccl-development-visibility-different-architectures>`.

133
cccl_upstream/docs/cccl/development/visibility/device_kernel_visibility.rst Normal file
View File

@@ -0,0 +1,133 @@
.. _cccl-development-visibility-device-kernel-visibility:
Device Kernel Visibility Issue
-------------------------------
Consider the following simple translation unit (TU):
.. code-block:: cpp
template <class T>
__global__ void kernel(T *val) {
::printf("kernel: set val = 42\n");
*val = 42;
}
int main() {
int *ptr{};
kernel<<<1, 1>>>(ptr);
}
The cuda compiler frontend will turn this into:
.. code-block:: cpp
template< class T>
static void __wrapper__device_stub_kernel(T *&ptr) {
::cudaLaunchKernel(0, 0, 0, 0, 0, 0);
}
// stub host function
template< class T>
void kernel(T *ptr) {
__wrapper__device_stub_kernel<T>(ptr);
}
int main() {
int *ptr{};
(__cudaPushCallConfiguration(1, 1)) ? (void)0 : kernel(ptr);
}
static void __device_stub__Z6kernelIiEvPT_(int *__par0) {
__cudaLaunchPrologue(1);
__cudaSetupArgSimple(__par0, 0UL);
__cudaLaunch(((char *)((void ( *)(int *))kernel )));
}
template<> void __wrapper__device_stub_kernel(int *&__cuda_0) {
__device_stub__Z6kernelIiEvPT_( (int *&)__cuda_0);
}
The CUDA runtime is going to use the address of ``template<> void kernel(T *ptr)`` (in the following ``h_kernel``)
as a key in the host stub function (``h_kernel``) - device function (``d_kernel``) mapping. This works fine if
there is only a single source of truth for the stub function ``h_kernel``.
However, imagine that there are two shared libraries: ``lib_a`` and ``lib_b`` both instantiating different ``kernel``
instances, e.g ``d_kernel<int>`` and ``d_kernel<size_t>``.
.. code-block:: cmake
project(DeviceKernelVisibility CUDA CXX)
add_executable(device_kernel_visibility main.cu)
add_library(lib_a SHARED tu_a.cu)
add_library(lib_b SHARED tu_b.cu)
target_link_libraries(device_kernel_visibility PRIVATE lib_a lib_b)
Each library will have it's own fatbinary: ``d_kernel<int>_a`` and ``d_kernel<size_t>_b`` as well as host stub functions
``h_kernel<int>_a`` and ``h_kernel<size_t>_b``.
=== ============= ============
lib host device
=== ============= ============
a 0xh_kernel_a 0xd_kernel_a
b 0xh_kernel_b 0xd_kernel_b
=== ============= ============
In contrast to
:ref:`Problem 1 <cccl-development-visibility-host-stub-visibility>` the host stubs will get a different mangled name
and so the right stub function will always be selected.
Now imagine that both libraries are going to defer launching of their kernels to a function ``foo`` common to both
``lib_a`` and ``lib_b``, which has weak external linkage. This might happen in ``CUB``, because it launches
kernels through the ``thrust::triple_chevron`` helper.
Similar to :ref:`Problem 1 <cccl-development-visibility-host-stub-visibility>` the linker will pick one of the two
weak symbols and subsequently ``lib_a`` will try to pass its own kernel ``d_kernel<int>_a`` to ``lib_b::foo``.
However, the CUDA runtime in ``lib_b`` will not find any kernel registered at the address of ``d_kernel<int>_a`` and
will fail to launch the kernel.
A simple example program that exemplifies this can be found
`on github <https://github.com/NVIDIA/cccl/tree/main/docs/cub/developer/visibility/examples/device_kernel_visibility>`_
.. code-block:: bash
./device_kernel_visibility/device_kernel_visibility
a: kernel stub address: 0x7fdec19e13eb <== launching kernel_a from a
a: kernel is in mapping: no error
b: launched kernel
a: kernel: set val = 42
a: synchronized stream
a: copied from device to host
a: out: 42
a: kernel was launched: out == 42 <== everything is fine
a: defers launch to b
b: kernel stub address: 0x7fdec19e13eb <== launch kernel_a from b
b: kernel NOT found in mapping: invalid device function <== kernel_a is not found in b mapping
b: FAILED to launch kernel <== unable to launch the kernel from b
b: synchronized stream
b: copied from device to host
b: out: 0
b: kernel was NOT actually launched: out != 42
b: kernel stub address: 0x7fdec19333eb <== launch kernel_b from b
b: kernel is in mapping: no error
b: launched kernel
b: kernel: set val = 42
b: synchronized stream
b: copied from device to host
b: out: 42
b: kernel was launched: out == 42 <== everything is fine
b: defers launch to a
a: kernel stub address: 0x7fdec19333eb <== launching kernel_b from a
a: kernel NOT found in mapping: invalid device function <== same issue as above
a: FAILED to launch kernel
b: kernel: set val = 42
a: synchronized stream
a: copied from device to host
a: out: 42
a: kernel was launched: out == 42 <== kernel launch somehow succeeded

186
cccl_upstream/docs/cccl/development/visibility/different_architectures.rst Normal file
View File

@@ -0,0 +1,186 @@
.. _cccl-development-visibility-different-architectures:
Linking TUs compiled with different architectures
--------------------------------------------------
Consider the following simple library:
.. code-block:: cpp
template <int... Archs>
__host__ __device__ constexpr int sum_archs() noexcept {
return (Archs + ... + 0);
}
// kernel with architecture dependent symbol name and functionality
template <class T, auto Archs = sum_archs<__CUDA_ARCH_LIST__>()>
__global__ void kernel(T *val) {
*val = sum_archs<__CUDA_ARCH_LIST__>();
}
__attribute__((visibility("hidden"))) inline int use_kernel() {
int *d_val{};
cudaMalloc(&d_val, sizeof(d_val));
kernel<<<1, 1>>>(d_val);
int ret;
if (cudaMemcpy(&ret, d_val, sizeof(size_t), cudaMemcpyDeviceToHost) !=
cudaSuccess) {
std::printf("c: FAILED to copy from device to host\n");
}
return ret;
}
template <class T = int>
struct some_class_with_kernel {
T val_;
some_class_with_kernel();
__forceinline__ some_class_with_kernel(T) { val_ = use_kernel(); }
};
We have a kernel that does some architecture dependent work. This could be relying on some hardware feature that is
dependent on the current architecture.
.. code-block:: cpp
#include "kernel.cuh"
int main() {
some_class_with_kernel with_inline{1};
std::printf("a: value of class with inlined constructor: %d\n",
with_inline.val_);
some_class_with_kernel from_library{};
std::printf("a: value of class with constructor from library: %d\n",
from_library.val_);
}
Importantly, one of the constructors for that class is put into a shared library, whereas the other one happens to be
inlined. If a user now links two different libraries, the outcome of the initialization of ``some_class_with_kernel``
will depend on whether the inlined constructor is called and which of the libraries is loaded first by the linker.
Even worse, the state of a class depends on whether the constructor has been inlined or not and the order in which
the linker loads the libraries.
.. code-block:: cmake
project(CUBVisDifferentArchitectures CUDA CXX)
add_library(cubvis_different_architectures_lib_a SHARED tu_a.cu)
set_target_properties(cubvis_different_architectures_lib_a PROPERTIES CUDA_ARCHITECTURES "86;90a")
add_library(cubvis_different_architectures_lib_b SHARED tu_b.cu)
set_target_properties(cubvis_different_architectures_lib_b PROPERTIES CUDA_ARCHITECTURES "75;86;90a")
add_executable(cubvis_different_architectures main.cu)
set_target_properties(cubvis_different_architectures PROPERTIES CUDA_ARCHITECTURES "75;86")
target_link_libraries(cubvis_different_architectures PRIVATE
cubvis_different_architectures_lib_a
cubvis_different_architectures_lib_b)
add_executable(cubvis_different_architectures_switched main.cu)
set_target_properties(cubvis_different_architectures_switched PROPERTIES CUDA_ARCHITECTURES "75;86")
target_link_libraries(cubvis_different_architectures_switched PRIVATE
cubvis_different_architectures_lib_b
cubvis_different_architectures_lib_a)
Execution the two libraries will result in the following:
.. code-block::
./different_architectures/different_architectures
a: value of class with inlined constructor: 1610 <<<--- from main
a: value of class with constructor from library: 1760 <<<--- from lib_a
./different_architectures/different_architectures_switched
a: value of class with inlined constructor: 1610 <<<--- from main
a: value of class with constructor from library: 2510 <<<--- from lib_b
One solution would be to bake the architectures into the symbol name of the class, either via a defaulted template
argument or an inline namespace. That way the usage of the non-inlined kernel would result in a linker error, because
we did not provide a matching implementation.
.. code-block::
tmpxft_00048dff_00000000-6_main.compute_86.cudafe1.cpp:(.text.startup+0xc0):
undefined reference to `some_class_with_kernel<int, 5120ul>::some_class_with_kernel()'
However, if all the functionality is within a non-inlined function we would still get different results, because all
kernel definitions would be internal to the respective library.
.. code-block:: cpp
// In tu_a.cu and tu_b.cu
void non_inlined_function() {
some_class_with_kernel with_inline{1};
std::printf("a: value of class with inlined constructor: %d\n",
with_inline.val_);
some_class_with_kernel from_library{};
std::printf("a: value of class with constructor from library: %d\n",
from_library.val_);
}
// In main.cu
#include "kernel.cuh"
void non_inlined_function();
int main() {
some_class_with_kernel with_inline{1};
std::printf("a: value of class with inlined constructor: %d\n",
with_inline.val_);
non_inlined_function();
}
Executing this binary will give us again:
.. code-block::
./different_architectures/different_architectures
a: value of class with inlined constructor: 1610 <<<--- from main
a: value of class with inlined constructor: 1760 <<<--- from lib_a
a: value of class with constructor from library: 1760 <<<--- from lib_a
./different_architectures/different_architectures_switched
a: value of class with inlined constructor: 1610 <<<--- from main
a: value of class with inlined constructor: 2510 <<<--- from lib_a
a: value of class with constructor from library: 2510 <<<--- from lib_b
So there is not functional way we can solve this problem generically, because the moment a user actually uses any type
of function that executes a kernel and puts that function into a shared library there is no guarantee which function
is selected. The same happens if the user builds a type
.. code-block:: cpp
class user_defined_with_kernel {
some_class_with_kernel val;
user_defined_with_kernel();
__forceinline__ user_defined_with_kernel(T input) : val(input)
{}
};
void function_that_uses_kernel_inside();
If ``user_defined_with_kernel`` is ever baked into a library we would be back with the same exact problem,
just one layer up. The user would need to know that ``some_class_with_kernel`` uses a kernel and then annotate *their*
classes and functions appropriately. This is neither realistic nor feasible.
Lets circle back to the previous statement: ``This is bad.`` Is it really though?
Lets look at the prime example ``thrust::device_vector``, which uses a kernel for initialization. What happens if we
accidentally run the kernel from another shared library compiled with different architectures? Worst case we are
eating some performance regressions because the kernel will not utilize advanced features of a new architecture,
but in the end the result of calling that kernel will not change the outcome.
This is because the kernel call is consistent *within* each library. As long as the user facing API does not rely on
specific internals of a kernel to be called -which it should not-, then any of the two libraries will do.
Finally, the architectures that are passed around in ``__CUDA_ARCH_LIST__`` do *not* discriminate architecture families.
There is currently no programmatic way to discriminate a library that has been compiled for ``SM90a`` from one that was
compiled for ``SM90``. This is because the architecture specific macros are only available on device not on host.

10
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/CMakeLists.txt Normal file
View File

@@ -0,0 +1,10 @@
project(DeveloperGuideDeviceKernelVisibility CUDA CXX)
add_executable(device_kernel_visibility main.cu)
add_library(device_kernel_visibility_liba SHARED tu_a.cu)
add_library(device_kernel_visibility_lib_b SHARED tu_b.cu)
target_link_libraries(
device_kernel_visibility
PRIVATE device_kernel_visibility_lib_a device_kernel_visibility_lib_b
)

10
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/kernel.cuh Normal file
View File

@@ -0,0 +1,10 @@
#pragma once
#include <cstdio>
template <class T>
__global__ void kernel(char ln, T* val)
{
printf("%c: kernel: set val = 42\n", ln);
*val = 42;
}

8
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/main.cu Normal file
View File

@@ -0,0 +1,8 @@
void a();
void b();
int main()
{
a();
b();
}

132
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_a.cu Normal file
View File

@@ -0,0 +1,132 @@
#include "kernel.cuh"
void b_launch(void (*k)(char, size_t*), char c, size_t* d_out);
void a_launch(void (*k)(char, int*), char c, int* d_out)
{
void* ptr = reinterpret_cast<void*>(k);
printf("a: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
cudaMemset(d_out, 0, sizeof(int));
k<<<1, 1>>>(c, d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("a: FAILED to launch kernel\n");
}
else
{
printf("a: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("a: FAILED to synchronize stream\n");
}
else
{
printf("a: synchronized stream\n");
}
int h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("a: FAILED to copy from device to host\n");
}
else
{
printf("a: copied from device to host\n");
}
printf("a: out: %d\n", h_out);
if (h_out != 42)
{
printf("a: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("a: kernel was launched: out == 42\n");
}
printf("\n");
}
void a()
{
cudaGetLastError();
size_t* d_out{};
cudaMalloc(&d_out, sizeof(size_t));
cudaMemset(d_out, 0, sizeof(size_t));
void* ptr = reinterpret_cast<void*>(kernel<size_t>);
printf("a: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
kernel<<<1, 1>>>('a', d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("a: FAILED to synchronize stream\n");
}
else
{
printf("a: synchronized stream\n");
}
size_t h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("a: FAILED to copy from device to host\n");
}
else
{
printf("a: copied from device to host\n");
}
printf("a: out: %d\n", static_cast<int>(h_out));
if (h_out != 42)
{
printf("a: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("a: kernel was launched: out == 42\n");
}
cudaMemset(d_out, 0, sizeof(size_t));
printf("\n");
printf("a: defers launch to b\n");
b_launch(kernel<size_t>, 'b', d_out);
}

132
cccl_upstream/docs/cccl/development/visibility/examples/device_kernel_visibility/tu_b.cu Normal file
View File

@@ -0,0 +1,132 @@
#include "kernel.cuh"
void a_launch(void (*k)(char, int*), char c, int* d_out);
void b_launch(void (*k)(char, size_t*), char c, size_t* d_out)
{
void* ptr = reinterpret_cast<void*>(k);
printf("b: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
k<<<1, 1>>>(c, d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("b: FAILED to synchronize stream\n");
}
else
{
printf("b: synchronized stream\n");
}
size_t h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("b: FAILED to copy from device to host\n");
}
else
{
printf("b: copied from device to host\n");
}
printf("b: out: %d\n", static_cast<int>(h_out));
if (h_out != 42)
{
printf("b: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("b: kernel was launched: out == 42\n");
}
printf("\n");
}
void b()
{
cudaGetLastError();
int* d_out{};
cudaMalloc(&d_out, sizeof(int));
cudaMemset(d_out, 0, sizeof(int));
void* ptr = reinterpret_cast<void*>(kernel<int>);
printf("b: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
kernel<<<1, 1>>>('b', d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("b: FAILED to synchronize stream\n");
}
else
{
printf("b: synchronized stream\n");
}
int h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("b: FAILED to copy from device to host\n");
}
else
{
printf("b: copied from device to host\n");
}
printf("b: out: %d\n", h_out);
if (h_out != 42)
{
printf("b: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("b: kernel was launched: out == 42\n");
}
cudaMemset(d_out, 0, sizeof(int));
printf("\n");
printf("b: defers launch to a\n");
a_launch(kernel<int>, 'b', d_out);
}

35
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/CMakeLists.txt Normal file
View File

@@ -0,0 +1,35 @@
project(DeveloperGuideDifferentArchitectures CUDA CXX)
add_library(different_architectures_lib_a SHARED tu_a.cu)
set_target_properties(
different_architectures_lib_a
PROPERTIES CUDA_ARCHITECTURES "86;90a"
)
add_library(different_architectures_lib_b SHARED tu_b.cu)
set_target_properties(
different_architectures_lib_b
PROPERTIES CUDA_ARCHITECTURES "75;86;90a"
)
add_executable(different_architectures main.cu)
set_target_properties(
different_architectures
PROPERTIES CUDA_ARCHITECTURES "75;86"
)
target_link_libraries(
different_architectures
PRIVATE different_architectures_lib_a different_architectures_lib_b
)
add_executable(different_architectures_switched main.cu kernel.cu)
set_target_properties(
different_architectures_switched
PROPERTIES CUDA_ARCHITECTURES "75;86"
)
target_link_libraries(
different_architectures_switched
PRIVATE different_architectures_lib_b different_architectures_lib_a
)

42
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/kernel.cuh Normal file
View File

@@ -0,0 +1,42 @@
#pragma once
#include <cstdint>
#include <cstdio>
template <int... Archs>
__attribute__((visibility("hidden"))) __host__ __device__ constexpr int sum_archs()
{
return (Archs + ... + 0);
}
template <class T, auto Archs = sum_archs<__CUDA_ARCH_LIST__>()>
__attribute__((visibility("hidden"))) __global__ void kernel(char ln, T* val)
{
printf("%c: kernel: set val = %i\n", ln, sum_archs<__CUDA_ARCH_LIST__>());
*val = sum_archs<__CUDA_ARCH_LIST__>();
}
__attribute__((visibility("hidden"))) __forceinline__ int use_kernel()
{
int* d_val{};
cudaMalloc(&d_val, sizeof(size_t));
kernel<<<1, 1>>>(d_val);
int ret;
if (cudaMemcpy(&ret, d_val, sizeof(size_t), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("c: FAILED to copy from device to host\n");
}
return ret;
}
template <class T = int>
struct some_class_with_kernel
{
T val_;
some_class_with_kernel();
__forceinline__ some_class_with_kernel(T)
{
val_ = use_kernel();
}
};

11
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/main.cu Normal file
View File

@@ -0,0 +1,11 @@
#include "kernel.cuh"
void non_inlined_function();
int main()
{
some_class_with_kernel with_inline{1};
printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
non_inlined_function();
}

16
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_a.cu Normal file
View File

@@ -0,0 +1,16 @@
#include "kernel.cuh"
template <class T>
some_class_with_kernel<T>::some_class_with_kernel()
{
val_ = use_kernel();
}
void non_inlined_function()
{
some_class_with_kernel with_inline{1};
printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
some_class_with_kernel from_library{};
printf("a: value of class with constructor from library: %d\n", from_library.val_);
}

16
cccl_upstream/docs/cccl/development/visibility/examples/different_architectures/tu_b.cu Normal file
View File

@@ -0,0 +1,16 @@
#include "kernel.cuh"
template <class T>
some_class_with_kernel<T>::some_class_with_kernel()
{
val_ = use_kernel();
}
void non_inlined_function()
{
some_class_with_kernel with_inline{1};
printf("a: value of class with inlined constructor: %d\n", with_inline.val_);
some_class_with_kernel from_library{};
printf("a: value of class with constructor from library: %d\n", from_library.val_);
}

10
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/CMakeLists.txt Normal file
View File

@@ -0,0 +1,10 @@
project(DeveloperGuideHostStubVisibility CUDA CXX)
add_executable(host_stub_visibility main.cu)
add_library(host_stub_visibility_lib_a SHARED tu_a.cu)
add_library(host_stub_visibility_lib_b SHARED tu_b.cu)
target_link_libraries(
host_stub_visibility
PRIVATE host_stub_visibility_lib_a host_stub_visibility_lib_b
)

10
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/kernel.cuh Normal file
View File

@@ -0,0 +1,10 @@
#pragma once
#include <cstdio>
template <class T>
__global__ void kernel(char ln, T* val)
{
printf("%c: kernel: set val = 42\n", ln);
*val = 42;
}

8
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/main.cu Normal file
View File

@@ -0,0 +1,8 @@
void a();
void b();
int main()
{
a();
b();
}

64
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_a.cu Normal file
View File

@@ -0,0 +1,64 @@
#include "kernel.cuh"
void a()
{
int* d_out{};
cudaMalloc(&d_out, sizeof(int));
cudaMemset(d_out, 0, sizeof(int));
void* ptr = reinterpret_cast<void*>(kernel<int>);
printf("a: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("a: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("a: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
kernel<<<1, 1>>>('a', d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("a: FAILED to synchronize stream\n");
}
else
{
printf("a: synchronized stream\n");
}
int h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("a: FAILED to copy from device to host\n");
}
else
{
printf("a: copied from device to host\n");
}
printf("a: out: %d\n", h_out);
if (h_out != 42)
{
printf("a: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("a: kernel was launched: out == 42\n");
}
printf("\n");
}

65
cccl_upstream/docs/cccl/development/visibility/examples/host_stub_visibility/tu_b.cu Normal file
View File

@@ -0,0 +1,65 @@
#include "kernel.cuh"
void b()
{
int* d_out{};
cudaMalloc(&d_out, sizeof(int));
cudaMemset(d_out, 0, sizeof(int));
void* ptr = reinterpret_cast<void*>(kernel<int>);
printf("b: kernel stub address: %p\n", ptr);
cudaFunction_t func{};
if (cudaError_t error = cudaGetFuncBySymbol(&func, ptr))
{
printf("b: kernel NOT found in mapping: %s\n", cudaGetErrorString(error));
}
else
{
printf("b: kernel is in mapping: %s\n", cudaGetErrorString(error));
}
kernel<<<1, 1>>>('b', d_out);
if (cudaPeekAtLastError() != cudaSuccess)
{
printf("b: FAILED to launch kernel\n");
}
else
{
printf("b: launched kernel\n");
}
if (cudaStreamSynchronize(0) != cudaSuccess)
{
printf("b: FAILED to synchronize stream\n");
}
else
{
printf("b: synchronized stream\n");
}
int h_out{};
if (cudaMemcpy(&h_out, d_out, sizeof(int), cudaMemcpyDeviceToHost) != cudaSuccess)
{
printf("b: FAILED to copy from device to host\n");
}
else
{
printf("b: copied from device to host\n");
}
printf("b: out: %d\n", h_out);
if (h_out != 42)
{
printf("b: kernel was NOT actually launched: out != 42\n");
}
else
{
printf("b: kernel was launched: out == 42\n");
}
printf("\n");
}

115
cccl_upstream/docs/cccl/development/visibility/host_stub_visibility.rst Normal file
View File

@@ -0,0 +1,115 @@
.. _cccl-development-visibility-host-stub-visibility:
Host Stub Visibility Issue
---------------------------
Consider the following simple translation unit (TU):
.. code-block:: cpp
#include <cstdio>
#include <cuda/memory>
template <class T>
__global__ void kernel(T *val) {
printf("kernel: set val = 42\n");
*val = 42;
}
__device__ int val;
int main() {
kernel<<<1, 1>>>(cuda::get_device_address(val));
}
The CUDA compiler frontend will turn this into:
.. code-block:: cpp
template< class T>
static void __wrapper__device_stub_kernel(T *&ptr) {
::cudaLaunchKernel(0, 0, 0, 0, 0, 0);
}
// stub host function
template< class T>
void kernel(T *ptr) {
__wrapper__device_stub_kernel<T>(ptr);
}
int main() {
int *ptr{};
(__cudaPushCallConfiguration(1, 1)) ? (void)0 : kernel(ptr);
}
static void __device_stub__Z6kernelIiEvPT_(int *__par0) {
__cudaLaunchPrologue(1);
__cudaSetupArgSimple(__par0, 0UL);
__cudaLaunch(((char *)((void ( *)(int *))kernel )));
}
template<> void __wrapper__device_stub_kernel(int *&__cuda_0) {
__device_stub__Z6kernelIiEvPT_( (int *&)__cuda_0);
}
The CUDA runtime is going to use the address of ``template<> void kernel(T *ptr)`` (in the following ``h_kernel``)
as a key in the host stub function (``h_kernel``) - device function (``d_kernel``) mapping. This works fine if
there is only a single source of truth for the stub function ``h_kernel``.
However, imagine that there are two shared libraries: ``lib_a`` and ``lib_b`` both using the same ``kernel`` instance.
.. code-block:: cmake
project(HostStubVisibility CUDA CXX)
add_executable(host_stub_visibility main.cu)
add_library(lib_a SHARED tu_a.cu)
add_library(lib_b SHARED tu_b.cu)
target_link_libraries(host_stub_visibility PRIVATE lib_a lib_b)
Each library will have its own fatbinary: ``d_kernel_a`` and ``d_kernel_b``, but the compiler
generated host stub function ``h_kernel`` has weak external linkage, so after dynamic linkage, we'll end up having
only one of them.
=== ===================== ============
lib host device
=== ===================== ============
a 0xh_kernel_a 0xd_kernel_a
b 0xh_kernel_a <- issue 0xd_kernel_b
=== ===================== ============
Since there's a clash of stub function addresses, only one entry stored. When ``lib_b`` queries for the
kernel using its address of ``h_kernel``, it's visible, although it might point to ``lib_a``'s fatbinary.
The opposite case might happen as well, depending on loading order, linker etc and is undefined behavior.
Launching ``d_kernel`` from ``lib_b`` is not possible and leads to random errors. For instance, there seems to be
some per CUDART global state. When the ``__cudaPushCallConfiguration`` is called in ``lib_b``, it affects the state of
``cudart_b``, but the launch happens through ``h_kernel``, which is in ``lib_a``.
This sometimes leads to ``__global__ function call is not configured``. However, there might also be no error at all,
and the kernel launch is silently skipped.
A simple example program that exemplifies this can be found
`on github <https://github.com/NVIDIA/cccl/tree/main/docs/cub/developer/visibility/examples/host_stub_visibility>`_
.. code-block:: bash
:./host_stub_visibility/host_stub_visibility
a: kernel stub address: 0x7f43318a415d <== same address as in B
a: kernel is in mapping: no error <== kernel is found in the mapping
b: launched kernel
a: kernel: set val = 42
a: synchronized stream
a: copied from device to host
a: out: 42
a: kernel was launched: out == 42
b: kernel stub address: 0x7f43318a415d <== same address as in A
b: kernel is in mapping: no error <== kernel is found in the mapping
b: launched kernel
b: synchronized stream
b: copied from device to host
b: out: 0
b: kernel was NOT actually launched: out != 42 <== silent failure

12
cccl_upstream/docs/cccl/license.rst Normal file
View File

@@ -0,0 +1,12 @@
.. _cccl-license:
License
========
The CUDA Core Compute Libraries use a mix of several licenses, mostly for historical reasons.
libcu++ uses the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.
CUB mostly uses the `3-Clause BSD License <https://github.com/NVIDIA/cccl/blob/main/cub/LICENSE.TXT>`_.
Thrust mostly the `Apache License v2.0 <https://github.com/NVIDIA/cccl/blob/main/thrust/LICENSE>`_.
Newly developed code will be under the `Apache License v2.0 with LLVM Exceptions <https://llvm.org/LICENSE.txt>`_.

35
cccl_upstream/docs/cccl/tma.rst Normal file
View File

@@ -0,0 +1,35 @@
.. _cccl-tma:
Tensor Memory Accelerator (TMA)
===============================
The Tensor Memory Accelerator (TMA) is a hardware feature available on Hopper (SM90) and newer GPUs
that enables efficient asynchronous memory copies of tensor data between global and (cluster) shared memory.
The use of TMA is required to reach SOL memory throughput for some workloads,
notable those where the necessary load/store vectorization, unrolling, or pipelining are limited by the register file or other factors.
CCCL offer several tools to help users leverage TMA in their applications.
In general, we recommend users to reach for high-level algorithms if they fit their problem.
Several algorithms, like ``cub::DeviceTransform``, ``cub::DeviceMerge``, ``cub::DeviceScan`` already use TMA internally today,
with many Thrust algorithms building on those.
And more algorithms will be added over time.
Relying on high level algorithms leaves the complexity of implementing and tuning TMA to CCCL team,
while providing users with safer interfaces, high productivity and SOL performance from the start.
If direct use of TMA is required to author new kernels, CCCL offers the following tools to help users get started,
from high-level to low-level:
- ``cub::BlockLoadToShared`` coming soon :)
- :ref:`cuda::memcpy_async <libcudacxx-extended-api-asynchronous-operations-memcpy-async>`
- :ref:`cuda::device::memcpy_async_tx <libcudacxx-extended-api-asynchronous-operations-memcpy-async-tx>`
- :ref:`cuda::ptx::cp_async_bulk* variants <libcudacxx-ptx-instructions>`
``cub::BlockLoadToShared`` and ``cuda::memcpy_async`` have fallback implementations for pre-Hopper GPUs,
using ``cp.async``/``LDGSTS`` on Ampere (SM80+) and ordinary loads/stores on older architectures.
Furthermore, they gracefully handle unaligned data and copying regions of arbitrary size.
The various ``cuda::ptx::cp_async_bulk*`` versions and ``cuda::device::memcpy_async_tx``
are thin wrappers of the corresponding PTX instructions
and provide no fallback path on older GPUs and also require the copied data to be aligned and sized appropriately.
Some further TMA-related utilities are provided by the :ref:`libcu++ extended API <libcudacxx-extended-api-tma>`.

284
cccl_upstream/docs/conf.py Normal file
View File

@@ -0,0 +1,284 @@
# CCCL Documentation Configuration File
# Generated to replace repo-docs with direct Sphinx usage
import os
import sys
from datetime import datetime
# Add extension directory to path
sys.path.insert(0, os.path.abspath("_ext"))
# Add Python CCCL package to path for autodoc
python_package_path = os.path.abspath("../python/cuda_cccl")
if os.path.exists(python_package_path):
sys.path.insert(0, python_package_path)
# Note: numpy is installed as a real dependency (see requirements.txt)
# This avoids issues with type annotations using union syntax (ndarray | type)
# -- Project information -----------------------------------------------------
project = "CUDA Core Compute Libraries"
copyright = f"{datetime.now().year}, NVIDIA Corporation"
author = "NVIDIA Corporation"
# Version information
_env_version = os.environ.get("SPHINX_CCCL_VER")
if _env_version:
release = _env_version
else:
try:
with open("VERSION.md", "r", encoding="utf-8") as f:
release = f.read().strip()
except Exception:
release = "unstable"
version = release
# -- General configuration ---------------------------------------------------
extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
"sphinx.ext.intersphinx",
"sphinx.ext.napoleon",
"sphinx.ext.extlinks",
"sphinx.ext.mathjax",
"sphinx.ext.graphviz",
"sphinx.ext.doctest",
"myst_parser", # MyST parser for markdown support
"breathe", # For Doxygen integration - has built-in embed:rst support
# "exhale", # Disabled - causing build timeouts, API docs handled by breathe
"sphinx_design", # For dropdown, card, and other directives
"sphinx_copybutton",
"nbsphinx",
# "rst_processor", # Disabled - breathe handles embed:rst natively
"auto_api_generator", # Automatically generate API reference pages from Doxygen XML
]
# Breathe configuration for Doxygen integration
breathe_projects = {
"cub": "_build/doxygen/cub/xml",
"thrust": "_build/doxygen/thrust/xml",
"libcudacxx": "_build/doxygen/libcudacxx/xml",
"cudax": "_build/doxygen/cudax/xml",
}
breathe_default_project = "cub"
breathe_default_members = ("members", "undoc-members")
breathe_show_enumvalue_initializer = True
breathe_domain_by_extension = {"cuh": "cpp", "h": "cpp", "hpp": "cpp"}
# Configure cpp domain to handle cub namespace
cpp_index_common_prefix = ["cub::"]
# Preprocessor definitions for Breathe to handle CCCL macros
cpp_id_attributes = [
"__device__",
"__host__",
"__global__",
"__forceinline__",
"_CCCL_HOST_DEVICE",
"_CCCL_DEVICE",
"_CCCL_HOST",
"_CCCL_FORCEINLINE",
"_CCCL_API",
"_CCCL_HOST_API",
"_CCCL_DEVICE_API",
"_CCCL_NODEBUG_API",
"_CCCL_NODEBUG_HOST_API",
"_CCCL_NODEBUG_DEVICE_API",
"_CCCL_TRIVIAL_API",
"_CCCL_TRIVIAL_HOST_API",
"_CCCL_TRIVIAL_DEVICE_API",
]
cpp_paren_attributes = ["__declspec", "__align__"]
# Add support for .rst and .md files
source_suffix = {
".rst": "restructuredtext",
".md": "markdown",
}
templates_path = ["_templates"]
# Exclude patterns
exclude_patterns = [
"_build",
"_repo",
"tools",
"VERSION.md",
"Thumbs.db",
".DS_Store",
"env/**", # Virtual environment
"**/.pytest_cache",
"**/__pycache__",
"*.pyc",
"*.pyo",
]
# -- Options for HTML output -------------------------------------------------
html_theme = "nvidia_sphinx_theme"
html_logo = "_static/nvidia-logo.png"
html_baseurl = (
os.environ.get("CCCL_DOCS_BASE_URL", "https://nvidia.github.io/cccl/").rstrip("/")
+ "/"
)
html_theme_options = {
"icon_links": [
{
"name": "GitHub",
"url": "https://github.com/NVIDIA/cccl",
"icon": "fa-brands fa-github",
"type": "fontawesome",
}
],
"navigation_depth": 4,
"show_toc_level": 2,
"navbar_start": ["navbar-logo"],
"navbar_end": ["theme-switcher", "navbar-icon-links"],
"footer_start": ["copyright"],
"footer_end": ["sphinx-version"],
"sidebar_includehidden": True,
"collapse_navigation": False,
"switcher": {
"json_url": f"{html_baseurl}nv-versions.json",
"version_match": release,
},
}
html_static_path = ["_static"] if os.path.exists("_static") else []
# Images directory
if os.path.exists("img"):
html_static_path.append("img")
html_js_files = ["deduplicate_toc.js"]
html_title = "CUDA Core Compute Libraries"
# -- Options for extensions --------------------------------------------------
# Intersphinx mapping
intersphinx_mapping = {
"python": ("https://docs.python.org/3/", None),
"numpy": ("https://numpy.org/doc/stable/", None),
}
# MyST parser configuration
myst_enable_extensions = [
"colon_fence",
"deflist",
"html_image",
]
# Napoleon settings
napoleon_google_docstring = True
napoleon_numpy_docstring = True
napoleon_include_init_with_doc = False
napoleon_include_private_with_doc = False
napoleon_include_special_with_doc = True
napoleon_use_admonition_for_examples = False
napoleon_use_admonition_for_notes = False
napoleon_use_admonition_for_references = False
napoleon_use_ivar = False
napoleon_use_param = True
napoleon_use_rtype = True
napoleon_preprocess_types = False
napoleon_type_aliases = None
# Autodoc settings
autodoc_default_options = {
"members": True,
"member-order": "bysource",
"special-members": "__init__",
"undoc-members": True,
"exclude-members": "__weakref__",
}
# Enable type hints to be shown in the documentation
autodoc_type_hints = "description"
autodoc_type_aliases = {
"Operator": "Operator",
}
# Set Python domain primary for intersphinx
primary_domain = "py"
# Mock imports for Python documentation - these modules may not be installed
autodoc_mock_imports = [
"numba",
"numba.core",
"numba.core.cgutils",
"numba.core.extending",
"numba.core.typing",
"numba.core.typing.ctypes_utils",
"numba.core.typing.templates",
"numba.cuda",
"numba.cuda.cudadecl",
"numba.cuda.dispatcher",
"numba.extending",
"numba.types",
"cuda.bindings",
"cuda.bindings.driver",
"cuda.bindings.runtime",
"cuda.core",
"cuda.core.experimental",
"cuda.core.experimental._utils",
"cuda.core.experimental._utils.cuda_utils",
"cuda.pathfinder",
"llvmlite",
"llvmlite.ir",
# numpy is installed as a real dependency (see requirements.txt)
"numpydoc_test_module", # Mock to avoid import errors
"cupy",
"cuda.compute._bindings",
"cuda.compute._bindings_impl",
]
# External links configuration
extlinks = {
"github": ("https://github.com/NVIDIA/cccl/blob/main/%s", "%s"),
}
# Exhale not used - API documentation is handled directly through breathe directives
# Napoleon configuration (handles NumPy-style docstrings)
# Note: numpydoc settings removed as Napoleon is used instead
# Config copybutton
# Suppress specific warning categories that arise from breathe (Doxygen-to-Sphinx
# bridge) limitations. These cannot be fixed in our source headers or RST files.
#
# See also _BREATHE_SKIP_SYMBOLS in _ext/auto_api_generator.py for symbols that
# are excluded from page generation entirely due to unparsable declarations.
suppress_warnings = [
# Breathe walks each Doxygen XML file independently. When a symbol appears
# in both a namespace XML and a class/group XML (which is normal for Doxygen),
# breathe emits the C++ declaration twice, triggering a duplicate-declaration
# warning. There is no way to control this from our side without patching
# breathe's XML traversal.
"cpp.duplicate_declaration",
# When breathe expands doxygenfunction/doxygenvariable directives, it writes
# the resolved C++ signature into RST. Signatures containing default argument
# values (e.g. ``= {}``) or complex SFINAE expressions produce RST that the
# docutils parser cannot handle (mismatched inline-literal markers, unexpected
# braces, etc.). The source C++ is valid; the issue is that what breathe
# emits as RST is not valid RST.
"docutils",
]
copybutton_prompt_text = ">>> |$ |# "
autosummary_imported_members = False
autosummary_generate = True
autoclass_content = "class"
def setup(app):
if os.path.exists("_static/custom.css"):
app.add_css_file("custom.css")

101
cccl_upstream/docs/contributors/how_tos/adding_build_target.rst Normal file
View File

@@ -0,0 +1,101 @@
.. _infra-cmake-adding-build-target:
Adding a build target
=====================
In most cases, new C++ test and example sources are detected automatically.
Their executables should appear after re-configuring / re-building.
When in doubt, check the CMakeLists.txt in the source directory to discover the conventions.
This document is intended for the rare case where a developer needs to build project infrastructure from scratch.
``cccl_add_executable`` wraps ``add_executable`` with CCCL's standard target
configuration: dialect handling, output directories, metatarget registration,
and clang-tidy integration. Use it for every test, example, benchmark, and tool
in the tree. The function lives in ``cmake/CCCLAddExecutable.cmake``.
Signature
---------
::
cccl_add_executable(target_name
SOURCES <source1> [source2 ...]
[ADD_CTEST]
[NO_METATARGETS]
[NO_CLANG_TIDY]
[METATARGET_PATH <path>]
[DIALECT <standard>]
)
The first positional argument is the target name. The remaining arguments are
keyword options:
.. list-table::
:header-rows: 1
:widths: 25 15 60
* - Argument
- Kind
- Effect
* - ``SOURCES``
- Required
- Source files for the executable. The function hard-fails with a fatal
error if absent.
* - ``ADD_CTEST``
- Flag
- Registers a CTest with the same name as the target, running the
executable with no arguments.
* - ``NO_METATARGETS``
- Flag
- Skips metatarget registration. The target builds only by its own name.
* - ``NO_CLANG_TIDY``
- Flag
- Skips clang-tidy integration for these sources.
* - ``METATARGET_PATH``
- One value
- Dotted path placing the target in the metatarget hierarchy. Defaults to
``target_name``.
* - ``DIALECT``
- One value
- C++ standard override for this target (for example ``17`` or ``20``).
Add a test target
-----------------
**Step 1. Call the function with SOURCES and ADD_CTEST.** Name the target with
its dotted hierarchy path so the metatarget system groups it correctly::
cccl_add_executable(cub.test.device_reduce
SOURCES test_device_reduce.cu
ADD_CTEST
)
This creates the executable ``cub.test.device_reduce``, registers a CTest of the
same name, and adds it to the ``cub`` and ``cub.test`` metatargets.
**Step 2. Link the target's dependencies.** ``cccl_add_executable`` configures
the target but does not link libraries. Add them after the call::
target_link_libraries(cub.test.device_reduce
PRIVATE
cub.compiler_interface
cccl.c2h
)
Place the target in the hierarchy
---------------------------------
By default the metatarget path equals the target name. A target named
``foo.bar.baz`` builds via metatargets ``foo`` and ``foo.bar``. Running
``ninja foo`` builds every descendant; ``ninja foo.bar`` builds that subtree.
Use ``METATARGET_PATH`` to decouple the target name from its hierarchy
position. Rare; advanced use only.
Exclude a target from the hierarchy
-----------------------------------
Pass ``NO_METATARGETS`` for targets that should not appear in the test
hierarchy: benchmarks, standalone tools, and anything outside the
build-everything-and-test workflow. Rare; advanced use only.

14
cccl_upstream/docs/contributors/how_tos/index.rst Normal file
View File

@@ -0,0 +1,14 @@
How Tos
=======
How-to guides for contributors.
.. toctree::
:maxdepth: 1
install_build_test
targeted_builds
launching
reproducing_ci
override_matrix
adding_build_target

162
cccl_upstream/docs/contributors/how_tos/install_build_test.rst Normal file
View File

@@ -0,0 +1,162 @@
.. _infra-install-build-test:
Install, Build, Test
====================
CCCL provides several developer interfaces for working with the codebase.
Purpose-built scripts exist that drive routine work, and using the wrong tool can waste hours of developer time.
For example, the ``ci/test_*.sh`` scripts can take hours to run a full validation suite, while the
``ci/util/build_and_test_targets.sh`` tool configures, builds, and runs a small subset in seconds.
Prebuilt development containers simplify working in specific toolchains and environments.
Core members can launch GitHub Actions benchmarking / bisection workflows that run on cloud infrastructure.
Pick a path by goal
-------------------
.. list-table::
:header-rows: 1
:widths: 34 38 14 14
* - Goal
- Tools
- Type
- Availability
* - Install CCCL headers to a prefix
- ``ci/install_cccl.sh``
- Script
- Public
* - Build and run a specific test
- ``ci/util/build_and_test_targets.sh``
- Script
- Public
* - Build or test an entire project
- ``ci/build_<project>.sh`` / ``ci/test_<project>.sh``
- Script
- Public
* - Bisect a regression
- ``ci/util/git_bisect.sh``, git-bisect.yml
- Script, GHA
- Public / members
* - Request a benchmark comparison
- ``ci/bench/bench.sh``, bench.yml, ``ci/bench.yaml``
- Script, GHA, PR tool
- Public / members
* - Custom build
- ``cmake --preset``
- Script
- Public
Install only
~~~~~~~~~~~~
``ci/install_cccl.sh <prefix>`` copies CCCL's headers and CMake config files into a prefix
directory. CCCL is header-only, so the install has no build step and finishes in seconds.
Use it when a downstream project needs ``find_package(CCCL)`` against a fixed checkout, or when you
want CCCL on a system include path without cloning into the consumer's tree.
Build and run a specific test
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``ci/util/build_and_test_targets.sh`` configures one preset, builds the Ninja targets you name,
and runs the CTest or lit tests you name. It is the fast-iteration path for a single test or a
handful of targets.
::
./ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.test.iterator" \
--ctest-targets "cub.test.iterator"
Use it when you are fixing one test and want a tight edit-build-run loop. Building tests does not
require a GPU; running them does.
Need a specific CTK or host compiler? Launch the matching container with ``.devcontainer/launch.sh``
first, then run the script inside it. Valid toolchain combinations are in the
``devcontainers:`` section of ``ci/matrix.yaml``; launching is covered at
:ref:`infra-devcontainer-launching`.
Build or test an entire project
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``ci/build_<project>.sh`` and ``ci/test_<project>.sh`` build or test a full project across a host
compiler, C++ standard, and architecture set. These are the scripts CI runs, so they reproduce a
CI job exactly.
::
./ci/build_cub.sh -cxx g++ -std 17 -arch "75;80;90"
./ci/test_cub.sh -cxx g++ -std 17 -arch "75;80;90"
Use them to reproduce a CI failure or to validate a project end to end before pushing. A full
project build takes hours; a targeted ``build_and_test_targets.sh`` run takes minutes. Test scripts
require a GPU.
Need a specific toolchain? Run these inside a devcontainer launched with ``.devcontainer/launch.sh``
(:ref:`infra-devcontainer-launching`). CI failure logs print the exact container and arguments to
reproduce the job.
Bisect a regression
~~~~~~~~~~~~~~~~~~~
``ci/util/git_bisect.sh`` walks the commit history between a good and a bad ref, building and
testing each candidate, until it pins the commit that introduced a regression. It takes the same
``--preset``, ``--build-targets``, and ``--ctest-targets`` arguments as
``build_and_test_targets.sh``.
::
./ci/util/git_bisect.sh \
--preset cub-cpp20 \
--build-targets "cub.test.iterator" \
--ctest-targets "cub.test.iterator" \
--good-ref v1.13.0 \
--bad-ref origin/main
Use it when a test passes on an old ref and fails on ``main`` and you need the offending commit.
``--repeat N`` re-runs multiple times to help catch intermittent failures.
Need a specific toolchain? Run the script inside a devcontainer launched with
``.devcontainer/launch.sh`` (:ref:`infra-devcontainer-launching`).
Members can run the same bisect remotely on CI machines through the `Git Bisect workflow
<https://github.com/NVIDIA/cccl/actions/workflows/git-bisect.yml>`_: choose a runner, set the good
and bad refs and target arguments, and dispatch. Source is ``.github/workflows/git-bisect.yml``.
Request a benchmark comparison
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compare benchmark results between two refs from the browser, a local script, or a pull request.
Members can dispatch the `Benchmark Compare workflow
<https://github.com/NVIDIA/cccl/actions/workflows/bench.yml>`_ from the browser: choose "Run
workflow", set the base and test refs and the CUB and Python filters, and it runs on the CI GPU
pool. Source is ``.github/workflows/bench.yml``.
Locally, ``ci/bench/bench.sh <base> <test>`` runs the same comparison against checked-out refs::
./ci/bench/bench.sh origin/main HEAD --cub-filter "^cub\.bench\.copy\.memcpy\.base$"
It wraps ``ci/bench/compare_git_refs.sh`` and ``ci/bench/compare_paths.sh``; call those directly
when you already have two checkouts.
To benchmark inside a PR, edit ``ci/bench.yaml`` to set GPUs and filters and push; PR CI detects the
diff from ``ci/bench.template.yaml`` and dispatches the jobs. Reset ``ci/bench.yaml`` to match the
template before merging. Argument behavior and artifact layout live in ``ci/bench/README.md``.
Full CMake control
~~~~~~~~~~~~~~~~~~
``cmake --preset <name>`` configures a build directory directly.
This is discouraged, but available for custom workflows / tool integrations.
::
cmake --preset all-dev
cmake --build --preset all-dev
ctest --preset all-dev
List the presets with ``cmake --list-presets``. The ``all-dev`` preset enables every library,
test, and example against your native GPU; per-library presets like ``cub-cpp20`` scope the build
to one library and standard. The :ref:`infra-cmake-preset-reference` catalogs the full preset set.

89
cccl_upstream/docs/contributors/how_tos/launching.rst Normal file
View File

@@ -0,0 +1,89 @@
.. _infra-devcontainer-launching:
Launching a container
=====================
``.devcontainer/launch.sh`` starts a CCCL development container with a chosen CUDA
toolkit and host compiler, mounts the repo, and either opens VSCode or drops you into
a shell. It is Linux-only (including WSL2). With no toolchain flags, it uses the default devcontainer
in ``.devcontainer/devcontainer.json``, which uses the latest CTK + gcc.
Launch in VSCode
----------------
**Open the default container.** Run the script with no flags.
.. code-block:: bash
.devcontainer/launch.sh
The script copies the selected ``devcontainer.json`` into a temporary directory and
opens VSCode against it.
**Select a toolchain.** Pass ``--cuda`` and ``--host`` to open a specific variant.
.. code-block:: bash
.devcontainer/launch.sh --cuda <cuda-version> --host <host-compiler>
The temporary-directory copy lets you run multiple variants of the same environment
side by side, each in its own VSCode window.
Launch directly in Docker
-------------------------
**Drop into a shell.** Add ``--docker`` to skip VSCode and run a bash shell inside the
container.
.. code-block:: bash
.devcontainer/launch.sh --docker --cuda <cuda-version> --host <host-compiler>
The container mounts the repo at ``/home/coder/cccl`` and removes itself on exit. Any
trailing arguments after the flags run as a command instead of an interactive shell.
Specify the toolchain
---------------------
``--cuda`` selects the CUDA toolkit version. ``--host`` selects the host compiler.
The two flags resolve to ``.devcontainer/cuda<cuda>-<host>/devcontainer.json``; an
unknown combination exits with an error.
Valid values come from the devcontainers located under ``.devcontainer/``.
Pass through GPUs
-----------------
**Add host GPUs.** Pass ``--gpus all`` to expose every host GPU to the container.
.. code-block:: bash
.devcontainer/launch.sh --docker --cuda <cuda-version> --host <host-compiler> --gpus all
``--gpus`` takes any Docker GPU request string. It overrides the ``hostRequirements.gpu``
default read from the devcontainer config. Without it, the container starts without GPU
access, which is sufficient for building tests.
Launch from a git worktree
--------------------------
``launch.sh`` handles linked worktrees automatically. A worktree's ``.git`` is a file
pointing at the main repository's git directory, which the container cannot reach through
the worktree mount alone. The script bind-mounts the main repo's git common directory at
its host path so git operations resolve inside the container.
The ``cccl-build`` and ``cccl-wheelhouse`` Docker volumes are shared across all worktrees
and the main checkout, so build artifacts collide between them. Do not run multiple
worktree containers concurrently unless you are careful to avoid conflicts.
First-time git auth in a worktree needs the main checkout's ``.config/gh``. If the main
checkout has never run its container, ``launch.sh`` warns that startup will block on an
interactive ``gh auth login``. Launch the main checkout's container once, complete the
login, then re-launch the worktree.
Forward SSH keys
----------------
If ``SSH_AUTH_SOCK`` is set in your environment, ``launch.sh`` forwards the agent socket
into the container automatically. No flag is required. Git operations over SSH inside the
container use your host agent's keys.

109
cccl_upstream/docs/contributors/how_tos/override_matrix.rst Normal file
View File

@@ -0,0 +1,109 @@
.. _infra-ci-override-matrix:
Override matrix
===============
The override matrix scopes a pull request's CI to a chosen subset of jobs.
When the ``workflows.override`` key in ``ci/matrix.yaml`` is non-empty, it
replaces the entire ``pull_request`` matrix. The PR runs only the override
jobs, and branch protection blocks the merge until the override is empty
again.
Use the override matrix to:
- Test a new compiler's nightly / weekly jobs from the PR before merging.
- Validate a compiler-specific or GPU-specific fix against one combo.
- Test CI infrastructure changes that need only a few jobs to validate.
- Debug a nightly failure by running only the combos that failed.
The override matrix is a temporary scoping tool, not a permanent matrix
edit. Every override entry must be removed before the PR lands.
Add an override entry
---------------------
The override lives at the top of ``ci/matrix.yaml`` under
``workflows.override``. The default value is empty. Override entries use the
same syntax as ``pull_request`` entries.
**Step 1. Pick the combo to test.** Identify the exact job, project,
compiler, CTK version, and GPU you need. For a compiler-specific fix, this is
one ``cxx`` value. For a nightly failure, copy the failing entry from the
``nightly`` workflow.
**Step 2. Add the entry under** ``override``. Edit ``ci/matrix.yaml`` and
add one mapping under the ``override:`` key:
.. code-block:: yaml
workflows:
override:
- {jobs: ['test'], project: 'thrust', std: 'max', ctk: '<ctk>', cxx: '<compiler>', gpu: '<gpu>'}
pull_request:
- <...>
Choose ``ctk``, ``cxx``, and ``gpu`` values from the existing
``pull_request`` entries in ``ci/matrix.yaml``. Each field scopes the run:
.. list-table::
:header-rows: 1
:widths: 20 80
* - Field
- Meaning
* - ``jobs``
- Job types to run. A ``test`` entry auto-generates any build jobs it
depends on.
* - ``project``
- Which project to build or test (``thrust``, ``cub``, ``libcudacxx``, ``cudax``, ...).
* - ``std``
- C++ standard. ``max`` selects the highest standard the combo supports.
* - ``ctk``
- CUDA Toolkit version. A ``<major>.X`` suffix selects the newest image for that major version.
* - ``cxx``
- Host compiler. A single value runs one compiler; an array expands to several jobs.
* - ``gpu``
- GPU runner model. Required for ``test`` jobs.
Field defaults and the full tag list live in the ``tags`` section of
``ci/matrix.yaml``.
**Step 3. Trim turnaround with targeted builds.** A full project build is
slow. To build and run a single test target instead, use ``project:
'target'`` and pass ``args`` to ``ci/util/build_and_test_targets.sh``:
.. code-block:: yaml
workflows:
override:
- {jobs: ['run_gpu'], project: 'target', ctk: '<ctk>', cxx: '<compiler>', gpu: '<gpu>',
args: '--preset <preset> --build-targets "<target>" --ctest-targets "<target>"'}
The ``run_cpu`` and ``run_gpu`` jobs map directly to
``build_and_test_targets.sh``. See that script for the available ``args``,
covered in :doc:`/cccl/development/build_and_bisect_tools`.
**Step 4. Reduce overhead further with skip tags.** Combine the override
with :ref:`[skip-*] tags <infra-ci-skip-tags>` in the last commit message to drop
devcontainer, docs, and third-party canary jobs:
.. code-block:: bash
git commit -m "Debug <compiler> <project> failure [skip-vdc][skip-docs][skip-tpt]"
Run scoped CI and merge
-----------------------
The override is temporary: scoped jobs first, full matrix before merge.
1. **Add the override entry and push.** Only the override jobs run; the
``pull_request`` matrix is skipped while ``workflows.override`` is non-empty.
2. **Iterate until the override jobs pass.** Each push reruns only those
jobs, keeping turnaround short.
3. **Empty the override and push again.** With ``workflows.override`` reset
to empty, the full ``pull_request`` matrix runs — the suite that gates merge.
4. **Merge once the full matrix is green.**

121
cccl_upstream/docs/contributors/how_tos/reproducing_ci.rst Normal file
View File

@@ -0,0 +1,121 @@
.. _infra-ci-reproducing-locally:
Reproducing CI locally
======================
A failing CI job prints everything needed to reproduce it. Open the failed
job from the PR's checks list and expand the log.
On failure: reproduction block
------------------------------
When a job exits non-zero, the runner emits an
**Instructions to Reproduce CI Failure Locally** block. It contains two steps:
**Step 1** — clone the repository at the exact SHA under test:
.. code-block:: bash
git clone --branch <branch> --single-branch https://github.com/NVIDIA/cccl.git \
&& cd cccl && git checkout <sha>
**Step 2** — launch the same container and re-run the same command:
.. code-block:: bash
.devcontainer/launch.sh -d -c <cuda> -H <host> -- <command>
Copy both lines verbatim. The ``-c`` and ``-H`` values are the CTK version and
host compiler for this job. ``<command>`` is the ``ci/*.sh`` invocation the
runner used, with all flags.
.. note::
GPU test jobs omit the GPU flag from the printed command. Add ``--gpus all``
when reproducing a test job on a machine with a GPU.
Job Inputs block
----------------
The **Job Inputs** group near the top of every log records the full job
configuration, printed before the container launches:
.. list-table::
:header-rows: 1
:widths: 30 70
* - Field
- Meaning
* - ``Job command``
- The ``ci/*.sh`` script and flags the runner will execute
* - ``JOB_ID``
- Unique job identifier; pass it to ``create_mock_job_env.sh`` to reproduce the job's
environment locally
* - ``JOB_CUDA``
- CTK version (e.g. ``13.3``)
* - ``JOB_HOST``
- Host compiler identifier (e.g. ``gcc15``)
* - ``JOB_IMAGE``
- Full RAPIDS devcontainer image tag pulled for this job
* - ``JOB_RUNNER``
- Runner label; labels containing ``-gpu-`` mean the job used a GPU
* - ``JOB_ENVIRONMENT``
- Extra environment variables injected into the container
These are the same values that feed the ``launch.sh`` invocation in the
failure block.
Override matrix entry
---------------------
The **Override matrix entry** block, also printed before the container
launches, contains a YAML snippet:
.. code-block:: yaml
- {jobs: [...], project: '...', ctk: '...', cxx: '...', ...}
Paste this into the ``workflows.override`` list in ``ci/matrix.yaml`` and push
to re-run only that specific job in CI without waiting for the full matrix.
See :ref:`infra-ci-override-matrix` for the override workflow.
Mock the job environment
------------------------
.. note::
This step is only needed when debugging the CCCL workflow / artifact scripts (rare).
More project contributors can safely skip this.
To run the ``ci/util/workflow/`` and ``ci/util/artifacts/`` scripts outside CI —
debugging artifact upload, download, or producer resolution locally — recreate the
job's environment with ``create_mock_job_env.sh``:
.. code-block:: bash
ci/util/create_mock_job_env.sh <run_id> <job_id>
Run it inside a devcontainer. It takes the ``<run_id>`` and ``<job_id>`` from the job
log (``GITHUB_RUN_ID`` and ``JOB_ID``), sets the ``GITHUB_*`` variables CI exports,
sources the workflow and artifact helpers, clears stale local artifact directories, and
drops into a shell that mimics the in-container CI environment. Every job log prints the
exact command under ``Mock with:``.
Tighten the loop
----------------
The full ``ci/build_*.sh`` and ``ci/test_*.sh`` scripts build and run an
entire project's test suite. Once the container reproduces the failure, use
``ci/util/build_and_test_targets.sh`` to build and run only the failing target:
.. code-block:: bash
.devcontainer/launch.sh --docker --cuda 13.0 --host gcc14 --gpus all -- \
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.test.iterator" \
--ctest-targets "cub.test.iterator"
This rebuilds one target instead of the full project. For preset, target, and
lit-test flags, and for ``ci/util/git_bisect.sh`` to find the introducing commit,
see :doc:`/cccl/development/build_and_bisect_tools`.

75
cccl_upstream/docs/contributors/how_tos/targeted_builds.rst Normal file
View File

@@ -0,0 +1,75 @@
.. _infra-ci-targeted-builds:
Build and run targeted tests
============================
The ``ci/build_*.sh`` and ``ci/test_*.sh`` scripts build and run all headers,
tests, examples, etc for a single project. It is the right tool for reproducing
a CI job, but it is slow when you are iterating on a single test.
``ci/util/build_and_test_targets.sh`` builds and runs a named subset of CMake
targets against one preset. Use it to compile one test, run one CTest pattern,
or execute one libcudacxx lit test without rebuilding the rest of the project.
The full flag reference is in :doc:`/cccl/development/build_and_bisect_tools`.
Build a single CUB test
-----------------------
CCCL tests usually have a single name for their CMake target, ninja target, and CTest target.
It uniquely encodes the project, path, and test case, eg: ``cub.test.iterator``.
#. **Configure and build the target.** Pass the preset and the metatarget to
``--build-targets``::
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.test.iterator"
With no ``--ctest-targets``, the script configures and compiles, then stops.
Compiling a test does not require a GPU.
#. **Run the target.** Add ``--ctest-targets`` with a CTest ``-R`` regex. The
metatarget name works directly as the pattern::
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.test.iterator" \
--ctest-targets "cub.test.iterator"
If ``--build-targets`` is omitted, the script assumes the targets are already built and
skips to testing. Running tests may require a GPU.
Run a libcudacxx lit test
-------------------------
Some libcudacxx tests run under lit, not CTest. Pass lit test paths relative to
``libcudacxx/test/libcudacxx/``.
#. **Execute one lit test.** Use ``--lit-tests`` with the test path::
ci/util/build_and_test_targets.sh \
--preset libcudacxx \
--lit-tests \
"std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
#. **Precompile without running.** Use ``--lit-precompile-tests`` to compile the
test with a no-op executor. This catches compile errors without a GPU::
ci/util/build_and_test_targets.sh \
--preset libcudacxx \
--lit-precompile-tests \
"std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"
Run inside a devcontainer
-------------------------
To build against a specific CUDA toolkit and host compiler, wrap the invocation
with ``.devcontainer/launch.sh -d``. Valid CTK and host compiler values are
listed in the ``.devcontainer`` directory. Pass ``--gpus all`` when the run
needs a device::
.devcontainer/launch.sh -d --cuda <CTK> --host <compiler> --gpus all -- \
ci/util/build_and_test_targets.sh \
--preset cub-cpp20 \
--build-targets "cub.test.iterator" \
--ctest-targets "cub.test.iterator"

11
cccl_upstream/docs/contributors/index.rst Normal file
View File

@@ -0,0 +1,11 @@
.. _contributors-index:
Contributor Docs
================
This section covers the build, test, and debug workflows a CCCL contributor uses day to day.
.. toctree::
:maxdepth: 1
how_tos/index

70
cccl_upstream/docs/cpp.rst Normal file
View File

@@ -0,0 +1,70 @@
.. _cccl-cpp-libraries:
CUDA C++ Core Libraries
=======================
.. toctree::
:hidden:
:maxdepth: 3
libcudacxx/index
cub/index
thrust/index
cudax/index
cccl/tma
cccl/determinism
cccl/config_macros
cccl/3.0_migration_guide
cccl/development/index
cccl/contributing
cccl/license
Welcome to the CUDA Core Compute Libraries (CCCL) libraries for C++.
The concept for the CCCL C++ librarires grew organically out of the Thrust,
CUB, and libcudacxx projects that were developed independently over the years
with a similar goal: to provide high-quality, high-performance, and
easy-to-use C++ abstractions for CUDA developers. Naturally, there was a lot
of overlap among the three projects, and it became clear the community would
be better served by unifying them into a single repository.
- :doc:`libcu++ <libcudacxx/index>`
is the CUDA C++ Standard Library. It provides an implementation of the C++
Standard Library that works in both host and device code. Additionally, it
provides abstractions for CUDA-specific hardware features like
synchronization primitives, cache control, atomics, and more.
- :doc:`CUB <cub/index>`
is a lower-level, CUDA-specific library designed for speed-of-light parallel
algorithms across all GPU architectures. In addition to device-wide
algorithms, it provides *cooperative algorithms* like block-wide reduction
and warp-wide scan, providing CUDA kernel developers with building blocks to
create speed-of-light, custom kernels.
- :doc:`Thrust <thrust/index>`
is the C++ parallel algorithms library which inspired the introduction of
parallel algorithms to the C++ Standard Library. Thrust's high-level
interface greatly enhances programmer productivity while enabling performance
portability between GPUs and multicore CPUs via configurable backends that
allow using multiple parallel programming frameworks (such as CUDA, TBB, and
OpenMP).
- :doc:`Cuda Experimental <cudax/index>`
is a library of experimental features that are still in the design process.
The main goal of the CCCL C++ libraries is to fill a similar role that the
Standard C++ Library fills for Standard C++: provide general-purpose,
speed-of-light tools to CUDA C++ developers, allowing them to focus on
solving the problems that matter. Unifying these projects is the first step
towards realizing that goal.
CUDA C++ Tile support
======================
`CUDA Tile <https://developer.nvidia.com/cuda/tile#section-more-resources>`_ introduces a new way to program GPUs at a higher level than SIMT.
Due to the additional constraints of tile compilation :doc:`CUB <cub/index>`, :doc:`Thrust <thrust/index>`
and :doc:`Cuda Experimental <cudax/index>` are currently not supported in a tile program.
There is partial support for :doc:`libcu++ <libcudacxx/index>` for tile programs.
The current support matrix can be found in the :doc:`libcu++ documentation <libcudacxx/tile>`

183
cccl_upstream/docs/cub/Doxyfile Normal file
View File

@@ -0,0 +1,183 @@
# Doxyfile for CUB
PROJECT_NAME = CUB
OUTPUT_DIRECTORY = ../_build/doxygen/cub
CREATE_SUBDIRS = NO
GENERATE_HTML = NO
GENERATE_LATEX = NO
GENERATE_XML = YES
XML_OUTPUT = xml
XML_PROGRAMLISTING = YES
INPUT = ../../cub/cub \
../../cub/cub/thread \
../../cub/cub/warp \
../../cub/cub/block \
../../cub/cub/device \
../../cub/cub/device/dispatch/tuning \
../../cub/cub/grid \
../../cub/cub/iterator
RECURSIVE = YES
EXCLUDE_PATTERNS = */detail/* */dispatch/dispatch_* */dispatch/kernels/* */kernels/* */test/* */examples/*
EXCLUDE_SYMBOLS = *detail* CUB_DETAIL*
FILE_PATTERNS = *.cuh *.h
EXTENSION_MAPPING = cuh=C++ cu=C++
# Documentation extraction settings
EXTRACT_ALL = YES
EXTRACT_PRIVATE = NO
EXTRACT_STATIC = YES
EXTRACT_LOCAL_CLASSES = YES
HIDE_UNDOC_MEMBERS = NO
HIDE_UNDOC_CLASSES = NO
SHOW_INCLUDE_FILES = YES
INLINE_INHERITED_MEMB = YES
FULL_PATH_NAMES = YES
STRIP_FROM_PATH = ../../cub
STRIP_FROM_INC_PATH = ../../cub
SHORT_NAMES = NO
# Parsing settings
JAVADOC_AUTOBRIEF = YES
QT_AUTOBRIEF = NO
MULTILINE_CPP_IS_BRIEF = NO
INHERIT_DOCS = YES
SEPARATE_MEMBER_PAGES = NO
TAB_SIZE = 4
BUILTIN_STL_SUPPORT = YES
# Preprocessing
ENABLE_PREPROCESSING = YES
MACRO_EXPANSION = YES
EXPAND_ONLY_PREDEF = NO
SEARCH_INCLUDES = YES
SKIP_FUNCTION_MACROS = YES
# IMPORTANT: Aliases for custom commands
# The rst alias enables embedding reStructuredText in doxygen comments
ALIASES = "rst=\verbatim embed:rst:leading-asterisk"
ALIASES += "endrst=\endverbatim"
ALIASES += "rststar=\verbatim embed:rst:leading-asterisk"
ALIASES += "inlinerst=\verbatim embed:rst:inline"
# Key aliases that are used within @rst blocks (from repo.toml)
ALIASES += "smemwarpreuse=A subsequent ``__syncwarp()`` warp-wide barrier should be invoked after calling this method if the collective's temporary storage (e.g., ``temp_storage``) is to be reused or repurposed."
ALIASES += "smemreuse=A subsequent ``__syncthreads()`` threadblock barrier should be invoked after calling this method if the collective's temporary storage (e.g., ``temp_storage``) is to be reused or repurposed."
ALIASES += "smemreuse{1}=After any operation, a subsequent ``__syncthreads()`` barrier is required if the collective's \1 is to be reused or repurposed"
ALIASES += "smemstorage{1}=The operations exposed by \1 require a temporary memory allocation of this nested type for thread communication. This opaque storage can be allocated directly using the ``__shared__`` keyword. Alternatively, it can be aliased to externally allocated memory (shared or global) or ``union``'d with other storage allocation types to facilitate memory reuse."
ALIASES += "granularity=Efficiency is increased with increased granularity ``ITEMS_PER_THREAD``. Performance is also typically increased until the additional register pressure or shared memory allocation size causes SM occupancy to fall too low. Consider variants of ``cub::BlockLoad`` for efficiently gathering a :ref:`blocked arrangement <flexible-data-arrangement>` of elements across threads."
ALIASES += "blocksize=The number of threads in the block is a multiple of the architecture's warp size"
ALIASES += "ptxversion=The PTX compute capability for which to to specialize this collective, formatted as per the ``__CUDA_ARCH__`` macro (e.g., 750 for sm_75). Useful for determining the collective's storage requirements for a given device from the host. (Default: the value of ``__CUDA_ARCH__`` during the current compiler pass)"
ALIASES += "blockcollective{1}=Every thread in the block uses the \1 class by first specializing the \1 type, then instantiating an instance with parameters for communication, and finally invoking one or more collective member functions."
ALIASES += "warpcollective{1}=Every thread in the warp uses the \1 class by first specializing the \1 type, then instantiating an instance with parameters for communication, and finally invoking or more collective member functions."
ALIASES += "devicestorage=Temporary storage for this operation. If ``d_temp_storage`` is ``nullptr``, the required size is written to ``temp_storage_bytes`` without dereferencing iterators or launching kernels. Otherwise, ``d_temp_storage`` must point to a device-accessible allocation of at least ``temp_storage_bytes`` bytes. No special alignment is required. See :ref:`device-temp-storage` for usage guidance."
ALIASES += "devicestorageP=This operation requires a relatively small allocation of temporary device storage that is ``O(P)``, where ``P`` is the number of streaming multiprocessors on the device (and is typically a small constant relative to the input size ``N``)."
ALIASES += "devicestorageNP=This operation requires an allocation of temporary device storage that is ``O(N+P)``, where ``N`` is the length of the input and ``P`` is the number of streaming multiprocessors on the device."
ALIASES += "devicestorageNCP=This operation requires a relatively small allocation of temporary device storage that is ``O(N/C + P)``, where ``N`` is the length of the input, ``C`` is the number of concurrent threads that can be actively scheduled on each streaming multiprocessor (typically several thousand), and ``P`` is the number of streaming multiprocessors on the device."
ALIASES += "cdp_class{1}= - Dynamic parallelism. \1 methods can be called within kernel code on devices in which CUDA dynamic parallelism is supported."
ALIASES += "determinism{1}= - Determinism. The default reproducibility guarantee is ``\1``. A different guarantee can be requested through the execution environment with ``cuda::execution::require``. See :ref:`Determinism in CUB <cub-determinism>` for the supported guarantees."
ALIASES += "iterator=(may be a simple pointer type)"
ALIASES += "offset_size1=(Consider using 32-bit values as offsets/lengths/etc. For example, ``int`` will typically yield better performance than ``size_t`` in 64-bit memory mode.)"
ALIASES += "offset_size2=Careful consideration should be given to the size of integer types used for offsets and lengths. Many (if not most) scenarios will only require 32-bit offsets (e.g., ``int``). 64-bit offset types (e.g., ``size_t`` on 64-bit memory mode) can consume a significant amount of thread storage resources, adversely affecting processor occupancy and performance."
ALIASES += "rowmajor=For multi-dimensional blocks, threads are linearly ranked in row-major order."
ALIASES += "blocked=Assumes a :ref:`blocked arrangement <flexible-data-arrangement>` of (*block-threads* * *items-per-thread*) items across the thread block, where *thread*\ :sub:`i` owns the *i*\ :sup:`th` range of *items-per-thread* contiguous items. For multi-dimensional thread blocks, a row-major thread ordering is assumed."
ALIASES += "striped=Assumes a :ref:`striped arrangement <flexible-data-arrangement>` of (*block-threads* * *items-per-thread*) items across the thread block, where *thread*\ :sub:`i` owns items (*i*), (*i* + *block-threads*), ..., (*i* + (*block-threads* * (*items-per-thread* - 1))). For multi-dimensional thread blocks, a row-major thread ordering is assumed."
ALIASES += "warpstriped=Assumes a *warp-striped arrangement* of elements across threads, where warp\ :sub:`i` owns the *i*\ :sup:`th` range of (*warp-threads* * *items-per-thread*) contiguous items, and each thread owns items (*i*), (*i* + *warp-threads*), ..., (*i* + (*warp-threads* * (*items-per-thread* - 1)))."
ALIASES += "linear_performance{1}=The work-complexity of \1 as a function of input size is linear, resulting in performance throughput that plateaus with problem sizes large enough to saturate the GPU."
ALIASES += "plots_below=Performance plots for other scenarios can be found in the detailed method descriptions below."
ALIASES += "identityzero=This operation assumes the value of obtained by the ``T``'s default constructor (or by zero-initialization if no user-defined default constructor exists) is suitable as the identity value \"zero\" for addition."
ALIASES += "lookback=`decoupled look-back <https://research.nvidia.com/publication/single-pass-parallel-prefix-scan-decoupled-look-back>`_"
# Predefined macros (based on repo.toml doxygen_predefined)
PREDEFINED = __device__= \
__host__= \
__global__= \
__forceinline__= \
"__declspec(x)=" \
"__align__(x)=" \
__cccl_lib_mdspan \
"CUB_NAMESPACE_BEGIN=namespace cub {" \
"CUB_NAMESPACE_END=}" \
"CUB_NS_PREFIX=" \
"CUB_NS_POSTFIX=" \
"CUB_NS_QUALIFIER=cub::" \
"CUB_DETAIL_MAGIC_NS_BEGIN=" \
"CUB_DETAIL_MAGIC_NS_END=" \
"_CCCL_AND=&&" \
"_CCCL_CONCEPT=constexpr bool " \
"_CCCL_CONSTEXPR_FRIEND=friend " \
"_CCCL_CONSTEXPR_CXX20=constexpr" \
"_CCCL_CONSTEXPR_CXX23=constexpr" \
"_CCCL_CTK_AT_LEAST(x, y)=1" \
"_CCCL_CTK_BELOW(x, y)=0" \
"_CCCL_CUDACC_AT_LEAST(x, y)=1" \
"_CCCL_CUDACC_BELOW(x, y)=0" \
_CCCL_DEVICE= \
_CCCL_DIAG_PUSH= \
_CCCL_DIAG_POP= \
"_CCCL_DIAG_SUPPRESS_CLANG(x)=" \
"_CCCL_DIAG_SUPPRESS_GCC(x)=" \
"_CCCL_DIAG_SUPPRESS_MSVC(x)=" \
"_CCCL_DIAG_SUPPRESS_NVHPC(x)=" \
_CCCL_DOXYGEN_INVOKED \
_CCCL_EXEC_CHECK_DISABLE= \
_CCCL_FORCEINLINE= \
"_CCCL_GLOBAL_CONSTANT=inline constexpr" \
"_CCCL_HAS_CTK()=1" \
_CCCL_HIDE_FROM_ABI= \
_CCCL_HOST= \
_CCCL_HOST_DEVICE= \
"_CCCL_REQUIRES(x)= ::cuda::std::enable_if_t<x, int> = 0>" \
_CCCL_STD_VER=2020 \
_CCCL_SUPPRESS_DEPRECATED_PUSH= \
_CCCL_SUPPRESS_DEPRECATED_POP= \
"_CCCL_TEMPLATE(x)=template<x, " \
"_CCCL_TRAIT(x, y)=x<y>::value" \
"_CCCL_TRAILING_REQUIRES(x)=-> x requires " \
_CCCL_TYPE_VISIBILITY_DEFAULT= \
_CCCL_TYPE_VISIBILITY_HIDDEN= \
_CCCL_API=inline \
_CCCL_HOST_DEVICE_API=inline \
_CCCL_DEVICE_API=inline \
_CCCL_HOST_API=inline \
_CCCL_NODEBUG_API=inline \
_CCCL_NODEBUG_DEVICE_API=inline \
_CCCL_NODEBUG_HOST_API=inline \
_CCCL_TRIVIAL_API=inline \
_CCCL_TRIVIAL_DEVICE_API=inline \
_CCCL_TRIVIAL_HOST_API=inline \
_CCCL_VISIBILITY_DEFAULT= \
_CCCL_VISIBILITY_HIDDEN= \
_CCCL_LIFETIMEBOUND= \
_CCCL_TRY=try \
_CCCL_CATCH=catch \
"_CCCL_CATCH_ALL=catch (...)" \
"_CCCL_CATCH_FALLTHROUGH=" \
_CCCL_PUBLIC_API=inline \
_CCCL_PUBLIC_DEVICE_API=inline \
_CCCL_PUBLIC_HOST_API=inline \
"_CUDAX_CONSTEXPR_FRIEND=friend" \
"_LIBCUDACXX_HAS_SPACESHIP_OPERATOR()=1" \
"CCCL_DEPRECATED=" \
"CCCL_DEPRECATED_BECAUSE(x)=" \
"CCCL_IGNORE_DEPRECATED_CPP_DIALECT" \
"CUB_DISABLE_NAMESPACE_MAGIC" \
"CUB_IGNORE_NAMESPACE_MAGIC_ERROR" \
"CUB_RUNTIME_FUNCTION=" \
"THRUST_FWD(x)=x" \
"THRUST_NAMESPACE_BEGIN=namespace thrust {" \
"THRUST_NAMESPACE_END=}" \
"THRUST_PREVENT_MACRO_SUBSTITUTION" \
"_CCCL_HOSTED()=1" \
_CCCL_DOXYGEN_INVOKED
# Quiet mode
QUIET = YES
WARNINGS = YES
WARN_AS_ERROR = FAIL_ON_WARNINGS
WARN_IF_UNDOCUMENTED = NO
WARN_IF_DOC_ERROR = YES
WARN_IF_INCOMPLETE_DOC = NO
WARN_NO_PARAMDOC = NO

376
cccl_upstream/docs/cub/benchmarking.rst Normal file
View File

@@ -0,0 +1,376 @@
Benchmarks
*************************************
.. TODO(bgruber): this guide applies to Thrust as well. We should rename it to "CCCL Benchmarks" and move it out of CUB
CUB comes with a set of `NVBench <https://github.com/NVIDIA/nvbench>`_-based benchmarks for its algorithms,
which can be used to measure the performance of CUB on your system on a variety of workloads.
The integration with NVBench allows to archive and compare benchmark results,
which is useful for continuous performance testing, detecting regressions, tuning, and optimization.
This guide gives an introduction into CUB's benchmarking infrastructure.
Building benchmarks
--------------------------------------------------------------------------------
CUB benchmarks are build as part of the CCCL CMake infrastructure.
Starting from scratch:
.. code-block:: bash
git clone https://github.com/NVIDIA/cccl.git
cd cccl
mkdir build
cd build
cmake .. --preset=benchmark
You clone the repository, create a build directory and configure the build with CMake.
The preset `benchmark` takes care of everything.
.. TODO(bgruber): do we have a public NVIDIA maintained table I can link here instead?
We use Ninja as CMake generator in this guide, but you can use any other generator you prefer.
You can then proceed to build the benchmarks.
You can list the available cmake build targets with, if you intend to only build selected benchmarks:
.. code-block:: bash
ninja -t targets | grep '\.bench\.'
cub.bench.adjacent_difference.subtract_left.base: phony
cub.bench.copy.memcpy.base: phony
...
cub.bench.transform.babelstream3.base: phony
cub.bench.transform_reduce.sum.base: phony
We also provide a target to build all benchmarks:
.. code-block:: bash
ninja cub.all.benches
.. _cub-benchmarking-running:
Running a benchmark
--------------------------------------------------------------------------------
After we built a benchmark, we can run it as follows:
.. code-block:: bash
./bin/cub.bench.adjacent_difference.subtract_left.base\
-d 0\
--stopping-criterion entropy\
--json base.json\
--md base.md
In this command, `-d 0` indicates that we want to run on GPU 0 on our system.
Setting `--stopping-criterion entropy` is advisable since it reduces runtime
and increase confidence in the resulting data.
It's not set as default yet, because NVBench is still evaluating it.
By default, NVBench will print the benchmark results to the terminal as Markdown.
`--json base.json` will save the detailed results in a JSON file as well for later use.
`--md base.md` will save the Markdown output to a file as well,
so you can easily view the results later without having to parse the JSON.
More information on what command line options are available can be found in the
`NVBench documentation <https://github.com/NVIDIA/nvbench/blob/main/docs/cli_help.md>`__.
The expected terminal output is something along the following lines (also saved to `base.md`),
shortened for brevity:
.. code-block:: bash
# Log
Run: [1/8] base [Device=0 T{ct}=I32 OffsetT{ct}=I32 Elements{io}=2^16]
Pass: Cold: 0.004571ms GPU, 0.009322ms CPU, 0.00s total GPU, 0.01s total wall, 334x
Run: [2/8] base [Device=0 T{ct}=I32 OffsetT{ct}=I32 Elements{io}=2^20]
Pass: Cold: 0.015161ms GPU, 0.023367ms CPU, 0.01s total GPU, 0.02s total wall, 430x
...
# Benchmark Results
| T{ct} | OffsetT{ct} | Elements{io} | Samples | CPU Time | Noise | GPU Time | Noise | Elem/s | GlobalMem BW | BWUtil |
|-------|-------------|------------------|---------|------------|---------|------------|--------|---------|--------------|--------|
| I32 | I32 | 2^16 = 65536 | 334x | 9.322 us | 104.44% | 4.571 us | 10.87% | 14.337G | 114.696 GB/s | 14.93% |
| I32 | I32 | 2^20 = 1048576 | 430x | 23.367 us | 327.68% | 15.161 us | 3.47% | 69.161G | 553.285 GB/s | 72.03% |
...
If you are only interested in a subset of workloads, you can restrict benchmarking as follows:
.. code-block:: bash
./bin/cub.bench.adjacent_difference.subtract_left.base ...\
-a 'T{ct}=I32'\
-a 'OffsetT{ct}=I32'\
-a 'Elements{io}[pow2]=[24,28]'\
The `-a` option allows you to restrict the values for each axis available for the benchmark.
See the `NVBench documentation <https://github.com/NVIDIA/nvbench/blob/main/docs/cli_help_axis.md>`__.
for more information on how to specify the axis values.
If the specified axis does not exist, the benchmark will terminate with an error.
If you want to plot the benchmark results, you can use the following script:
.. code-block:: bash
PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_plot_bwutil.py base.json
The `-a` option is supported to restrict the values for some axes as well,
which is useful if you want to plot only a subset of workloads.
Use the `-b` option to select a specific benchmark by name
in case your JSON file contains results for multiple benchmarks.
Multiple benchmarks are selected by repeating the `-b` option.
.. code-block:: bash
PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_plot_bwutil.py \
-b base -a Elements{io}[pow2]=28 base.json
.. _cub-benchmarking-comparing:
Comparing benchmark results
--------------------------------------------------------------------------------
Let's say you have a modification that you'd like to benchmark.
To compare the performance you have to build and run the benchmark as described above for the unmodified code,
saving the results to a JSON file, e.g. `base.json`.
Then, you apply your code changes (e.g., switch to a different branch, git stash pop, apply a patch file, etc.),
rebuild and rerun the benchmark, saving the results to a different JSON file, e.g. `new.json`.
You can now compare the two result JSON files using, assuming you are still in your build directory:
.. code-block:: bash
PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_compare.py base.json new.json
The `PYTHONPATH` environment variable may not be necessary in all cases.
The script will print a Markdown report showing the runtime differences between each variant of the two benchmark run.
This could look like this, again shortened for brevity:
.. code-block:: bash
| T{ct} | OffsetT{ct} | Elements{io} | Ref Time | Ref Noise | Cmp Time | Cmp Noise | Diff | %Diff | Status |
|---------|---------------|----------------|------------|-------------|------------|-------------|------------|---------|----------|
| I32 | I32 | 2^16 | 4.571 us | 10.87% | 4.096 us | 0.00% | -0.475 us | -10.39% | FAIL |
| I32 | I32 | 2^20 | 15.161 us | 3.47% | 15.143 us | 3.55% | -0.018 us | -0.12% | PASS |
...
In addition to showing the absolute and relative runtime difference,
NVBench reports the noise of the measurements,
which corresponds to the relative standard deviation.
It then reports with statistical significance in the `Status` column
how the runtime changed from the base to the new version.
You can reduce the output to runs with larger differences using the `--threshold-diff` option,
passing the minimum percentage for a run to be shown, e.g., `0.05` for 5%.
.. code-block:: bash
PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_compare.py \
--threshold-diff 0.05 base.json new.json
You can also plot the comparison by adding the `--plot` argument.
It's reasonable to combine this with the `-a` option again
to restrict the values for some axes.
.. code-block:: bash
PYTHONPATH=./_deps/nvbench-src/python/scripts ./_deps/nvbench-src/python/scripts/nvbench_compare.py \
-a Elements{io}[pow2]=28 --plot base.json new.json
Running all benchmarks directly from the command line
--------------------------------------------------------------------------------
To get a full snapshot of CUB's performance, you can run all benchmarks and save the results.
For example, inside a build directory you can run:
.. code-block:: bash
ninja cub.all.benches
benchmarks=$(ls bin | grep cub.bench); n=$(echo $benchmarks | wc -w); i=1; \
for b in $benchmarks; do \
echo "=== Running $b ($i/$n) ==="; \
./bin/$b -d 0 --stopping-criterion entropy --json $b.json --md $b.md; \
((i++)); \
done
This will generate one JSON and one Markdown file for each benchmark.
You can archive those files for later comparison or analysis.
Running all benchmarks via tuning scripts (alternative)
--------------------------------------------------------------------------------
The benchmark suite can also be run using the :ref:`tuning infrastructure <cub-tuning-infra>`.
The tuning infrastructure handles building benchmarks itself, because it records the build times.
Therefore, it's critical that you run it in a clean build directory without any build artifacts.
Running cmake is enough. Alternatively, you can also clean your build directory.
Furthermore, the tuning scripts require some additional python dependencies, which you have to install:
.. code-block:: bash
ninja clean
pip install --user fpzip pandas scipy
To select the appropriate CUDA GPU, first identify the GPU ID by running `nvidia-smi`, then set the
desired GPU using `export CUDA_VISIBLE_DEVICES=x <https://docs.nvidia.com/cuda/cuda-c-programming-guide/#cuda-environment-variables>`_,
where `x` is the ID of the GPU you want to use (e.g., `1`).
This ensures your application uses only the specified GPU.
We can then run the full benchmark suite from the build directory with:
.. code-block:: bash
export CUDA_VISIBLE_DEVICES=0 # or any other GPU ID
PYTHONPATH=../benchmarks/scripts ../benchmarks/scripts/run.py
You can expect the output to look like this:
.. code-block:: bash
&&&& RUNNING bench
ctk: 12.2.140
cub: 812ba98d1
&&&& PERF cub_bench_adjacent_difference_subtract_left_base_T_ct__I32___OffsetT_ct__I32___Elements_io__pow2__16 4.095999884157209e-06 -sec
&&&& PERF cub_bench_adjacent_difference_subtract_left_base_T_ct__I32___OffsetT_ct__I32___Elements_io__pow2__20 1.2288000107218977e-05 -sec
&&&& PERF cub_bench_adjacent_difference_subtract_left_base_T_ct__I32___OffsetT_ct__I32___Elements_io__pow2__24 0.00016998399223666638 -sec
&&&& PERF cub_bench_adjacent_difference_subtract_left_base_T_ct__I32___OffsetT_ct__I32___Elements_io__pow2__28 0.002673664130270481 -sec
...
The tuning infrastructure will build and execute all benchmarks and their variants one after each other,
reporting the time in seconds it took to execute the benchmarked region.
It's also possible to benchmark a subset of algorithms and workloads, by running in a build directory:
.. code-block:: bash
export CUDA_VISIBLE_DEVICES=0 # or any other GPU ID
PYTHONPATH=../benchmarks/scripts ../benchmarks/scripts/run.py -R '.*scan.exclusive.sum.*' -a 'Elements{io}[pow2]=[24,28]' -a 'T{ct}=I32'
&&&& RUNNING bench
ctk: 12.6.77
cccl: v2.7.0-rc0-265-g32aa6aa5a
&&&& PERF cub_bench_scan_exclusive_sum_base_T_ct__I32___OffsetT_ct__U32___Elements_io__pow2__28 0.003194367978721857 -sec
&&&& PERF cub_bench_scan_exclusive_sum_base_T_ct__I32___OffsetT_ct__U64___Elements_io__pow2__28 0.00319383991882205 -sec
&&&& PASSED bench
The `-R` option allows you to specify a regular expression for selecting benchmarks.
The `-a` restricts the values for an axis across all benchmarks
See the `NVBench documentation <https://github.com/NVIDIA/nvbench/blob/main/docs/cli_help_axis.md>`__.
for more information on how to specify the axis values.
Contrary to running a benchmark directly,
the tuning infrastructure will just ignore an axis value if a benchmark does not support,
run the benchmark regardless, and continue.
The tuning infrastructure stores results in an SQLite database called :code:`cccl_meta_bench.db` in the build directory.
This database persists across tuning runs.
If you interrupt the benchmark script and then launch it again, only missing benchmark variants will be run.
Comparing results of multiple tuning databases
--------------------------------------------------------------------------------
Benchmark results captured in different tuning databases can be compared as well:
.. code-block:: bash
<cccl_git_root>/benchmarks/scripts/compare.py -o cccl_meta_bench1.db cccl_meta_bench2.db
This will print a Markdown report showing the runtime differences and noise for each variant.
Furthermore, you can plot the results, which requires additional python packages:
.. code-block:: bash
pip install fpzip pandas matplotlib seaborn tabulate PyQt5 colorama
You can plot one or more tuning databases as a bar chart or a box plot (add `--box`):
.. code-block:: bash
<cccl_git_root>/benchmarks/scripts/sol.py cccl_meta_bench.db ...
This is useful to display the current performance of CUB as captured in a single tuning database,
or visually compare the performance of CUB across different tuning databases
(from different points in time, on different GPUs, etc.).
Dumping benchmark results from a tuning database
--------------------------------------------------------------------------------
The resulting database contains all samples, which can be extracted into JSON files:
.. code-block:: bash
<cccl_git_root>/benchmarks/scripts/analyze.py -o ./cccl_meta_bench.db
This will create a JSON file for each benchmark variant next to the database.
For example:
.. code-block:: bash
cat cub_bench_scan_exclusive_sum_base_T_ct__I32___OffsetT_ct__U32___Elements_io__pow2__28.json
[
{
"variant": "base ()",
"elapsed": 2.6299014091,
"center": 0.003194368,
"bw": 0.8754671386,
"samples": [
0.003152896,
0.0031549439,
...
],
"Elements{io}[pow2]": "28",
"base_samples": [
0.003152896,
0.0031549439,
...
],
"speedup": 1
}
]
Profiling benchmarks with Nsight Compute
--------------------------------------------------------------------------------
If you want to see profiling metrics on source code level,
you have to recompile your benchmarks with the `-lineinfo` option.
With cmake, you can just add `-DCMAKE_CUDA_FLAGS=-lineinfo` when invoking cmake in the `build` directory:
.. code-block:: bash
cmake .. --preset=benchmark -DCMAKE_CUDA_FLAGS=-lineinfo
To profile the kernels, use the `ncu` command.
A typical invocation, if you work on a remote cluster, could look like this:
.. code-block:: bash
ncu --set full --import-source yes -o base.ncu-rep -f ./bin/thrust.bench.transform.basic.base -d 0 --profile
The option `--set full` instructs `ncu` to collect all metrics.
This requires rerunning some kernels and takes more time.
`--import-source yes` imports the source code into the report file,
so you can see metrics not only in SASS but also in your source code,
even if you copy the resulting report away from the source code.
`-o base.ncu-rep` specifies the output file and `-f` overwrites the output file if it already exists.
`--profile` tells NVBench to run only one iteration, which speeds up profiling.
For inspecting the profiling report, we recommend using the GUI of Nsight Compute.
If you run on a remote machine, you may want to copy the report `base.ncu-rep` back to your local workstation,
before viewing the report using `ncu-ui`:
.. code-block:: bash
scp <remote hostname>:<cccl repo directory>/build/base.ncu-rep .
ncu-ui base.ncu-rep
The version of `ncu-ui` needs to be at least as high as the version of `ncu` used to create the report.
Authoring benchmarks
--------------------------------------------------------------------------------
CUB's benchmarks serve a dual purpose.
They are used to measure and compare the performance of CUB and to tune CUB's algorithms.
More information on how to create new benchmarks is provided in the :ref:`CUB tuning infrastructure guide <cub-tuning-infra>`.

26
cccl_upstream/docs/cub/block_wide.rst Normal file
View File

@@ -0,0 +1,26 @@
.. _block-module:
Block-Wide "Collective" Primitives
==================================================
.. toctree::
:glob:
:hidden:
:maxdepth: 2
api/block
CUB block-level algorithms are specialized for execution by threads in the same CUDA thread block:
* :cpp:class:`cub::BlockAdjacentDifference` computes the difference between adjacent items partitioned across a CUDA thread block
* :cpp:class:`cub::BlockDiscontinuity` flags discontinuities within an ordered set of items partitioned across a CUDA thread block
* :cpp:struct:`cub::BlockExchange` rearranges data partitioned across a CUDA thread block
* :cpp:class:`cub::BlockHistogram` constructs block-wide histograms from data samples partitioned across a CUDA thread block
* :cpp:class:`cub::BlockLoad` loads a linear segment of items from memory into a CUDA thread block
* :cpp:class:`cub::BlockMergeSort` sorts items partitioned across a CUDA thread block
* :cpp:class:`cub::BlockRadixSort` sorts items partitioned across a CUDA thread block using radix sorting method
* :cpp:struct:`cub::BlockReduce` computes reduction of items partitioned across a CUDA thread block
* :cpp:class:`cub::BlockRunLengthDecode` decodes a run-length encoded sequence partitioned across a CUDA thread block
* :cpp:struct:`cub::BlockScan` computes a prefix scan of items partitioned across a CUDA thread block
* :cpp:struct:`cub::BlockShuffle` shifts items partitioned across a CUDA thread block
* :cpp:class:`cub::BlockStore` stores items partitioned across a CUDA thread block to a linear segment of memory

76
cccl_upstream/docs/cub/determinism.rst Normal file
View File

@@ -0,0 +1,76 @@
.. _cub-determinism:
Determinism
===============
Several ``cub`` device algorithms let you request a reproducibility guarantee for a call. The concepts
behind the three guarantees — ``not_guaranteed``, ``run_to_run``, and ``gpu_to_gpu`` — and the meaning
of *reproducibility* are described in the :ref:`CCCL determinism overview <cccl-determinism>`. This
page documents how to request a guarantee for a CUB algorithm and which algorithms support which
guarantees.
Requesting a guarantee
----------------------
A determinism guarantee is passed to a device algorithm through its execution environment using
``cuda::execution::require``. The example below requests run-to-run reproducibility for
``cub::DeviceReduce::Sum``:
.. literalinclude:: ../../cub/test/catch2_test_device_reduce_env_api.cu
:language: c++
:dedent:
:start-after: example-begin sum-env-determinism
:end-before: example-end sum-env-determinism
The general rules for requesting a guarantee are described in the
:ref:`CCCL determinism overview <cccl-determinism>`.
Each CUB algorithm has its own default guarantee, applied when none is requested, and its own type and
operator constraints for each guarantee, summarized below. Requesting a guarantee that an algorithm
does not support is rejected at compile time.
Support matrix
--------------
.. list-table::
:header-rows: 1
:widths: 30 18 18 18 16
* - Algorithm
- ``not_guaranteed``
- ``run_to_run``
- ``gpu_to_gpu``
- Default
* - ``cub::DeviceReduce`` (``Reduce``, ``Sum``, ``Min``, ``Max``, ``TransformReduce``, ...)
- Yes
- Yes
- Yes (partial)
- ``run_to_run``
* - ``cub::DeviceScan`` (``ExclusiveSum``, ``ExclusiveScan``, ``InclusiveSum``, ``InclusiveScan``, ...)
- Yes
- Yes (partial)
- Yes (partial)
- ``not_guaranteed``
* - ``cub::DeviceSegmentedReduce``
- Yes
- Yes
- No
- ``run_to_run``
.. note::
The set of algorithms that accept determinism requirements, and the type/operator constraints for
each guarantee, are expanding over time. The matrix above reflects the current implementation.
Algorithm-specific determinism models
--------------------------------------
The three guarantees describe the *scope* of reproducibility and fit most algorithms, where a
reproducible result means a *bitwise-identical* output. A few algorithms still use the same three
levels but extend the model with additional, algorithm-specific controls, documented on their own
pages:
- :ref:`cub::DeviceTopK <cub-topk-requirements>` — determinism applies to *set membership* (which
*K* items are selected) rather than a bitwise-identical buffer, and it adds tie-breaking
(``cuda::execution::tie_break``) and output-ordering (``cuda::execution::output_ordering``)
controls.

79
cccl_upstream/docs/cub/developer/block_scope.rst Normal file
View File

@@ -0,0 +1,79 @@
.. _cub-developer-guide-block-scope:
Block-scope
************
Overview
=========
Block-scope algorithms are provided by structures as well:
.. code-block:: c++
template <typename T,
int BLOCK_DIM_X,
BlockReduceAlgorithm ALGORITHM = BLOCK_REDUCE_WARP_REDUCTIONS,
int BLOCK_DIM_Y = 1,
int BLOCK_DIM_Z = 1>
class BlockReduce {
public:
struct TempStorage : Uninitialized<_TempStorage> {};
// (1) new constructor
__device__ __forceinline__ BlockReduce()
: temp_storage(PrivateStorage()),
linear_tid(RowMajorTid(BLOCK_DIM_X, BLOCK_DIM_Y, BLOCK_DIM_Z)) {}
__device__ __forceinline__ BlockReduce(TempStorage &temp_storage)
: temp_storage(temp_storage.Alias()),
linear_tid(RowMajorTid(BLOCK_DIM_X, BLOCK_DIM_Y, BLOCK_DIM_Z)) {}
};
While warp-scope algorithms only provide a single constructor that requires the user to provide temporary storage,
block-scope algorithms provide two constructors:
#. The default constructor that allocates the required shared memory internally.
#. The constructor that requires the user to provide temporary storage as argument.
In the case of the default constructor,
the block-level algorithm uses the ``PrivateStorage()`` member function to allocate the required shared memory.
This ensures that shared memory required by the algorithm is only allocated when the default constructor is actually called in user code.
If the default constructor is never called,
then the algorithm will not allocate superfluous shared memory.
.. code-block:: c++
__device__ __forceinline__ _TempStorage& PrivateStorage()
{
__shared__ _TempStorage private_storage;
return private_storage;
}
The ``__shared__`` memory has static semantic, so it's safe to return a reference here.
Specialization
====================================
Block-scope facilities usually expose algorithm selection to the user.
The algorithm is represented by the enumeration part of the API.
For the reduction case,
``BlockReduceAlgorithm`` is provided.
Specializations are stored in the ``cub/block/specializations`` directory.
Temporary storage usage
====================================
For block-scope algorithms,
it's unsafe to use temporary storage without synchronization:
.. code-block:: c++
using BlockReduce = cub::BlockReduce<int, 128> ;
__shared__ BlockReduce::TempStorage temp_storage;
int aggregate_1 = BlockReduce(temp_storage).Sum(thread_data_1);
// illegal, has to add `__syncthreads` between the two
int aggregate_2 = BlockReduce(temp_storage).Sum(thread_data_2);
// illegal, has to add `__syncthreads` between the two
foo(temp_storage);

530
cccl_upstream/docs/cub/developer/device_scope.rst Normal file
View File

@@ -0,0 +1,530 @@
.. _cub-developer-guide-device-scope:
Device-scope
*************
Overview
=========
Device-scope functionality is provided by classes called ``DeviceAlgorithm``,
where ``Algorithm`` is the implemented algorithm.
These classes then contain static member functions providing corresponding API entry points.
For example, device-level reduce will look like `cub::DeviceReduce::Sum`.
Here is a generic example:
.. code-block:: c++
struct DeviceAlgorithm {
// two step API
template <typename ...>
static cudaError_t Algorithm(void *d_temp_storage, size_t &temp_storage_bytes, ..., cudaStream_t stream = 0) {
// optional: minimal argument checking or setup to call dispatch layer
return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ..., stream);
}
// environment API
template <typename ..., typename Env = cuda::std::execution::env<>>
static cudaError_t Algorithm(..., const Env& env = {}) {
// optional: minimal argument checking or setup to call dispatch layer
using default_policy_selector = detail::algorithm::policy_selector_from_types<...>;
return dispatch_with_env_and_tuning<default_policy_selector>(
env, [&](auto policy_selector, void* storage, size_t& bytes, auto stream) {
return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ..., stream, policy_selector);
});
}
};
Device-scope APIs come in two flavors, two step APIs and environment APIs,
both return ``cudaError_t`` and take algorithm specific arguments.
The two step API accepts a ``stream`` as the last parameter (``NULL`` stream by default)
and the first two parameters are always ``void *d_temp_storage, size_t &temp_storage_bytes``.
The environment API just takes an environment as the last parameter (empty environment by default).
The implementation may consist of some minimal argument checking, but should forward as soon as possible to the dispatch layer.
Device-scope algorithms are implemented in files located in `cub/device/device_***.cuh`.
The two step API is called in two phases:
1. Temporary storage size is calculated and returned in ``size_t &temp_storage_bytes``.
2. ``temp_storage_bytes`` of memory is expected to be allocated and ``d_temp_storage`` is expected to be the pointer to this memory.
The following example illustrates this pattern:
.. code-block:: c++
// First call: Determine temporary device storage requirements
std::size_t temp_storage_bytes = 0;
cub::DeviceReduce::Sum(/*d_temp_storage*/ nullptr, temp_storage_bytes, d_in, d_out, num_items);
// Allocate temporary storage
thrust::device_vector<unsigned char> temp_storage(temp_storage_bytes, thrust::no_init);
// Second call: Perform algorithm
cub::DeviceReduce::Sum(temp_storage.data().get(), temp_storage_bytes, d_in, d_out, num_items);
.. warning::
Even if the algorithm doesn't need temporary storage as scratch space,
the overload with ``void *d_temp_storage, size_t &temp_storage_bytes``
still requires one byte of memory to be allocated.
The environment overloads just require a single call, but setting up the environment may be more complex:
.. code-block:: c++
// Setup environment, everything is optional
auto env = cuda::std::execution::env{
stream_ref,
memory_resource,
cuda::execution::require(requirements...),
cuda::execution::tune(policy_selectors....)
};
// Perform algorithm
cub::DeviceReduce::Sum(d_in, d_out, num_items, env);
The device layer handles the extraction of the:
* CUDA stream
* memory resource
* requirements
* guarantees (TODO(bgruber): are those public?)
* :ref:`tuning policy selectors <cub-policy-selectors>`
from the environment argument, or provide default values in case the environment does not contain them.
They typically use helper functions like ``dispatch_with_env`` and ``dispatch_with_env_and_tuning``.
Some CUB APIs require no temporary storage and may omit the ``void *d_temp_storage, size_t &temp_storage_bytes`` parameters.
Their environment overloads will also ignore any passed memory resource.
Dispatch layer
====================================
A dispatch function exists for each device-scope algorithm (e.g., ``detail::reduce::dispatch``),
and is located in ``cub/device/dispatch``.
Most device-scope algorithms share one dispatch function.
Only device-scope algorithms have dispatch functions, which are also referred to as the dispatch layer.
The dispatch layer follows a certain architecture.
The high-level control flow is represented by the code below.
A more precise description is given later.
..
TODO(bgruber): consider removing the numbers below. control flow is now easier to follow
.. code-block:: c++
// Device-scope API
cudaError_t cub::DeviceAlgorithm::Algorithm(d_temp_storage, temp_storage_bytes, ...) {
return detail::algorithm::dispatch(d_temp_storage, temp_storage_bytes, ...);
}
namespace detail::algorithm {
cudaError_t dispatch(
void *d_temp_storage, size_t &temp_storage_bytes,
...,
cudaStream_t stream, PolicySelector policy_selector = {}) {
cuda::compute_capability cc{};
ptx_compute_cap(cc);
const /*or constexpr*/ AlgorithmPolicy active_policy = policy_selector(cc);
// host-side implementation of algorithm, calls kernels
kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
}
template <typename PolicySelector>
__launch_bounds__(int(current_policy<PolicySelector>().threads_per_block))
void kernel(...) {
static constexpr auto policy = current_policy<PolicySelector>();
using agent_policy = AgentPolicy<policy.threads_per_block, policy.items_per_thread, ...>;
using agent = AgentAlgorithm<agent_policy, InputIteratorT, OffsetT, ...>;
agent a{...};
a.Process();
}
template <int ThreadsPerBlock, ...>
struct AgentPolicy { // legacy
static constexpr threads_per_block = ThreadsPerBlock;
...
};
template <typename Policy, ...>
struct AlgorithmAgent {
void Process() { ... }
};
}
Let's look at each of the building blocks closer.
The dispatch function
------------------------------------
The dispatch function is typically a simple function template called ``dispatch`` inside an algorithm-specific namespace.
It basically receives the same parameters as the public API entry point,
but they may have already been modified, extended, or generalized (to map many public APIs to the same dispatch function).
The goal of the dispatch function is to setup the execution of the algorithm on the device
by selecting the appropriate policy for the current GPU's compute capability,
preparing temporary storage and shared memory, configuring kernel launches, launching kernels, etc.
There are two style of dispatch functions, depending on whether the policy is needed as runtime or compile-time value:
.. code-block:: c++
namespace detail::algorithm {
// Dispatch version A - runtime policy
cudaError_t dispatch(
void *d_temp_storage, size_t &temp_storage_bytes,
...,
cudaStream_t stream, PolicySelector policy_selector = {}) {
cuda::compute_capability cc{};
ptx_compute_cap(cc);
const auto active_policy = policy_selector(cc); // runtime-time policy
// host-side implementation of algorithm, calls kernels
kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
}
}
The dispatch function starts by querying the target compute capability for which compiled GPU code (PTX or SASS) is available,
by calling ``ptx_compute_cap``.
If the host code does not require the policy for this compute capability at compile-time (version A),
we can just pass the compute capability to the :ref:`policy selector <cub-policy-selectors>` to obtain the tuning policy at runtime.
The values from the policy are then used to setup resources and the kernel.
The kernel is then instantiated using only the type of the policy selector (not a concrete tuning policy),
so there is only one kernel instantiation across all target architectures compiled for.
More on that later.
If the host code needs the policy as a compile-time value (version B), we have to use ``dispatch_compute_cap``.
dispatch_compute_cap
------------------------------------
``dispatch_compute_cap`` maps a runtime ``cuda::compute_capability`` to a compile-time policy value
and calls the user-provided functor ``f`` with a nullary callable (a ``policy_getter``)
that returns the policy as a compile-time constant.
The policy getter is necessary to work around a C++17 limitation.
.. code-block:: c++
namespace detail::algorithm {
// Dispatch version B - compile-time policy
cudaError_t dispatch(
void *d_temp_storage, size_t &temp_storage_bytes,
...,
cudaStream_t stream, PolicySelector policy_selector = {}) {
cuda::compute_capability cc{};
ptx_compute_cap(cc);
return dispatch_compute_cap(policy_selector, cc, [&](auto policy_getter) {
constexpr auto active_policy = policy_getter(); // compile-time policy
static_assert(active_policy.tile_size() * sizeof(T) <= 48 * 1024, "Not enough SMEM");
// host-side implementation of algorithm, calls kernels
kernel<PolicySelector><<<grid_size, active_policy.threads_per_block>>>(...);
});
}
template <typename PolicySelector, typename F>
cudaError_t dispatch_compute_cap(PolicySelector, cuda::compute_capability cc, F&& f) { // (2)
// fold over __CUDA_ARCH_LIST__, calling f with a policy_getter
// that returns the policy for the matching arch as a compile-time value
}
}
Inside the lambda, ``policy_getter()`` returns the selected policy as a constant expression,
so compile-time branching (i.e. ``if constexpr``) or static assertions using policy values is possible.
Internally, ``dispatch_compute_cap`` uses ``__CUDA_ARCH_LIST__`` / ``NV_TARGET_SM_INTEGER_LIST`` (or all known compute capabilities as fallback)
to create one instantiation of ``f`` per distinct value of ``policy_selector(cc)`` (not per compute capability).
This results in one template instantiation of ``f`` per distinct policy value.
The kernel is then again only instantiated once using the type of the policy selector.
Kernels
------------------------------------
Kernels are templated on the ``PolicySelector`` type,
which is stateless and the same for all target architectures compiled for.
.. code-block:: c++
namespace detail::algorithm {
template <typename PolicySelector, typename InputIteratorT, typename OffsetT, /* ... */>
__launch_bounds__(int(current_policy<PolicySelector>().reduce.threads_per_block))
void DeviceReduceKernel(InputIteratorT d_in, OffsetT num_items, /* ... */) {
static constexpr auto policy = current_policy<PolicySelector>();
using agent_policy = AgentPolicy<policy.threads_per_block, policy.items_per_thread, ...>;
using agent = AgentAlgorithm<agent_policy, InputIteratorT, OffsetT, ...>;
agent a{...};
a.Process();
}
template <typename PolicySelector>
constexpr auto current_policy() {
return PolicySelector{}(cuda::compute_capability{__CUDA_ARCH__ / 10}); // simplified
}
}
``PolicySelector`` must be stateless (``is_empty_v<PolicySelector>`` is ``true``),
so it can be default-constructed in device code where needed.
The utility function ``current_policy`` can only be called in device code.
It selects the target compute capability based on compiler macros of the current device compilation pass
and retrieves a tuning policy from the policy selector.
The kernel typically uses ``current_policy`` in two places,
to get the block size to define the launch bounds,
and inside the kernel to setup various sub algorithms (like agents).
Because C++17 does not allow to pass structs as Non-Type Template Parameters (NTTPs),
we cannot easily pass the policy around to other functions,
so it will be converted to legacy agent policies in many places.
Those are just structs with static data members holding the policy value as a type,
so they can be passed to templates.
Agent policy structs have historically been part of the public API, and should be removed in the future.
.. warning::
The kernel gets compiled for each target architecture (N many) that was provided to the compiler.
During each device pass, ``current_policy`` may return a different policy.
During the host pass, version A (runtime policy) compiles a single instantiation of the dispatch logic for all target architectures.
Version B (using ``dispatch_compute_cap``) compiles the dispatch logic for each distinct tuning policy (M many).
If we passed the selected tuning policy instead of the policy selector as a kernel template parameter,
the kernel template instantiation would be different for each tuning policy value and
we would compile O(M*N) kernels for version B instead of O(N).
Many kernels are short, since the functionality is extracted into the agent layer.
All the kernel does is derive the proper policy,
unwrap it to initialize the agent and call one of its ``Consume`` / ``Process`` functions.
Agents hold kernel bodies and are intended to be reused across multiple device-scope algorithms.
However, this is not a requirement and some kernels just contain their entire implementation themselves.
The default policy selector
------------------------------------
CUB contains a default policy selector for each dispatch function.
Because many dispatch functions are also used by CCCL.C,
which compiles them without proper type information,
we have to provide them in two forms,
a typeless and a typeful version.
.. code-block:: c++
struct AlgorithmPolicy { ... }; // unrelated to AgentPolicy
namespace detail::algorithm {
struct policy_selector {
type_t accum_t;
op_kind_t operation_t;
int offset_size;
int accum_size;
constexpr auto operator()(::cuda::compute_capability cc) const -> AlgorithmPolicy {
// parameter selection across target compute capabilities and input characteristics (possibly HUGE logic)
}
};
template <typename AccumT, typename OffsetT, typename ReductionOpT>
struct policy_selector_from_types {
constexpr auto operator()(cuda::compute_capability cc) const -> AlgorithmPolicy {
constexpr auto ps = policy_selector{
classify_type<AccumT>(),
classify_op<ReductionOpT>(),
int{sizeof(OffsetT)},
int{sizeof(AccumT)}
};
return ps(cc);
}
};
}
The ``policy_selector`` is intended to be used without template-parameter-based type information
and is thus suitable to be used in CCCL.C without a JIT compiler for host code.
It contains the necessary information on the algorithm's input as data members.
This policy is only used in the host code of the dispatch function.
Because it is not stateless anymore, CCCL.C overrides the kernel launcher used by CUB,
providing the kernel from a JIT-compiled instantiation that uses a proper stateless policy selector.
The kernel, and the dispatch function when not called from CCCL.C,
will use the second version of the policy selector, ``policy_selector_from_types``,
which offers proper template parameters to pass type information.
This type is stateless and delegates to a ``constexpr`` instance of the ``policy_selector``
with compile-time values derived from the template parameters.
The policy selection logic is thus the same for CCCL.C and CUB,
the type information is just provided differently.
Each dispatch function also has an associated concept for the policy selector it expects:
.. code-block:: c++
template <typename T, typename Policy>
concept policy_selector = requires(T pol_sel, cuda::compute_capability cc) {
requires std::regular<Policy>;
{ pol_sel(cc) } -> std::same_as<Policy>;
};
namespace detail::algorithm {
template <typename T>
concept algorithm_policy_selector = policy_selector<T, AlgorithmPolicy>;
}
The concept basically checks whether the policy selector can be called with a ``cuda::compute_capability``
and returns the expected policy struct.
The default policy selectors and related concept are defined in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
Backward compatibility
------------------------------------
Legacy public dispatchers (e.g. ``DispatchReduce``) are deprecated.
They continue to work by translating the ``PolicyHub`` template parameter to the new ``policy_selector``
via a ``policy_selector_from_hub`` adapter:
.. code-block:: c++
template <typename PolicyHub>
struct policy_selector_from_hub {
constexpr auto operator()(cuda::compute_capability cc) const -> AlgorithmPolicy {
// conversion logic
}
};
This allows existing user code that passes custom policy hubs to dispatchers to continue working.
The legacy dispatchers, policy hubs and related logic are scheduled for removal in CCCL 4.0.
Policies
====================================
Policies describe the configuration of agents or just kernels with respect to performance.
They must not change functional behavior, but affect how work is mapped to the hardware
by defining certain parameters (items per thread, block size, etc.),
or choosing between algorithms.
Policies must be plain semiregular aggregates to allow using them during constant evaluation,
and with designated initializers:
.. code-block:: c++
struct AlgorithmPolicy {
int threads_per_block;
int items_per_thread;
cub::BlockLoadAlgorithm load_algorithm;
cub::CacheLoadModifier load_modifier;
friend constexpr bool operator==(const AlgorithmPolicy& lhs, const AlgorithmPolicy& rhs) { ... }
friend constexpr bool operator!=(const AlgorithmPolicy& lhs, const AlgorithmPolicy& rhs) { ... }
friend std::ostream& operator<<(std::ostream& os, const AlgorithmPolicy& p) { ... }
};
Tuning policies can have various complexities and contain nested structures.
Policies are defined in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
Tunings
====================================
Because the values to parameterize an agent may vary a lot for different compile-time parameters,
the selection of values can involve complex logic.
Often, such tunings are found by experimentation or heuristic search.
See also :ref:`cub-tuning-infra`.
Tunings are expressed as logic and values inside the ``constexpr operator()`` of a policy selector.
Because of the complexity of some policy selectors, nested functions may be used.
Many policy selectors also implement a fallback logic,
where they try to find a matching tuning based on the input characteristics (policy selector data members),
but if no match is found, they fall back to an older target compute capability.
Here is an example:
.. code-block:: c++
struct sm90_tuning_values {
int items;
int threads;
int items_per_vec_load;
};
constexpr auto get_sm90_tuning(type_t accum_t, op_kind_t op, int offset_size, int accum_size)
-> std::optional<sm90_tuning_values> {
// provide tunings for certain operations, accumulator or offset types
if (op == op_kind_t::plus) {
if (accum_t == type_t::float32 && offset_size == 4 && accum_size == 4)
return sm90_tuning_values{16, 512, 2};
if (offset_size == 4 && accum_size == 8)
return sm90_tuning_values{15, 512, 2};
}
return {}; // no tuning available, causes fallback
}
struct policy_selector {
type_t accum_t;
op_kind_t operation_t;
int offset_size;
int accum_size;
constexpr auto operator()(cuda::compute_capability cc) const -> reduce_policy {
if (cc >= cuda::compute_capability{9, 0}) {
if (auto tuning = get_sm90_tuning(accum_t, operation_t, offset_size, accum_size)) {
return *tuning; // found a tuning, use it
}
// fall through to sm_80 if no matching tuning found
}
if (cc >= cuda::compute_capability{8, 0}) {
return { /* sm_80 default policy */ };
}
return { /* default policy for everything else */ };
}
};
In general, tunings are not exhaustive and usually only apply for specific combinations
of parameter values and a single compute capability.
This is because they originate from tuning benchmarks running for specific workloads on specific target architectures.
Generic fallbacks are often just retained from earlier days of CUB to not risk regressions,
or are based on heuristics trying to provide reasonable performance based on a model of the GPU architecture or algorithm.
Tunings for CUB algorithms reside in ``cub/device/dispatch/tuning/tuning_<algorithm>.cuh``.
Temporary storage usage
====================================
It's safe to reuse storage in the stream order:
.. code-block:: c++
cub::DeviceReduce::Sum(nullptr, storage_bytes, d_in, d_out, num_items, stream_1);
// allocate temp storage
cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_1);
// fine not to synchronize stream
cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_1);
// illegal, should call cudaStreamSynchronize(stream)
cub::DeviceReduce::Sum(d_storage, storage_bytes, d_in, d_out, num_items, stream_2);
Temporary storage management
====================================
Often times temporary storage for device-scope algorithms has a complex structure.
To simplify temporary storage management and make it safer,
we introduced ``cub::detail::temporary_storage::layout``:
.. code-block:: c++
cub::detail::temporary_storage::layout<2> storage_layout;
auto slot_1 = storage_layout.get_slot(0);
auto slot_2 = storage_layout.get_slot(1);
auto allocation_1 = slot_1->create_alias<int>();
auto allocation_2 = slot_1->create_alias<double>(42);
auto allocation_3 = slot_2->create_alias<char>(12);
if (condition)
{
allocation_1.grow(num_items);
}
if (d_temp_storage == nullptr)
{
temp_storage_bytes = storage_layout.get_size();
return;
}
storage_layout.map_to_buffer(d_temp_storage, temp_storage_bytes);
// different slots, safe to use simultaneously
use(allocation_1.get(), allocation_3.get(), stream);
// `allocation_2` alias `allocation_1`, safe to use in stream order
use(allocation_2.get(), stream);

17
cccl_upstream/docs/cub/developer/nvtx.rst Normal file
View File

@@ -0,0 +1,17 @@
.. _cub-developer-guide-nvtx:
NVTX
=====
The `NVIDIA Tools Extension SDK (NVTX) <https://nvidia.github.io/NVTX/>`_ is a cross-platform API
for annotating source code to provide contextual information to developer tools.
All device-scope algorithms in CUB are annotated with NVTX ranges,
allowing their start and stop to be visualized in profilers
like `NVIDIA Nsight Systems <https://developer.nvidia.com/nsight-systems>`_.
Only the public APIs available in the ``<cub/device/device_xxx.cuh>`` headers are annotated,
excluding direct calls to the dispatch layer.
NVTX annotations can be disabled by defining ``NVTX_DISABLE`` during compilation.
When CUB device algorithms are called on a stream subject to
`graph capture <https://developer.nvidia.com/blog/cuda-graphs/>`_,
the NVTX range is reported for the duration of capture (where no execution happens),
and not when a captured graph is executed later (the actual execution).

328
cccl_upstream/docs/cub/developer/test_overview.rst Normal file
View File

@@ -0,0 +1,328 @@
CUB Tests
###########################
.. warning::
CUB is in the progress of migrating to [Catch2](https://github.com/catchorg/Catch2) framework.
CUB tests rely on `CPM <https://github.com/cpm-cmake/CPM.cmake>`_ to fetch
`Catch2 <https://github.com/catchorg/Catch2>`_ that's used as our main testing framework.
Currently,
legacy tests coexist with Catch2 ones.
This guide is focused on new tests.
.. important::
Instead of including ``<catch2/catch.hpp>`` directly, use ``catch2_test_helper.h``.
.. code-block:: c++
#include <cub/block/block_scan.cuh>
#include <c2h/vector.h>
#include <c2h/catch2_test_helper.h>
Directory and File Naming
*************************************
Our tests can be found in the ``test`` directory.
Legacy tests have the following naming scheme: ``test_SCOPE_FACILITY.cu``.
For instance, here are the reduce tests:
.. code-block:: c++
test/test_warp_reduce.cu
test/test_block_reduce.cu
test/test_device_reduce.cu
Catch2-based tests have a different naming scheme: ``catch2_test_SCOPE_FACILITY.cu``.
The prefix is essential since that's how CMake finds tests
and distinguishes new tests from legacy ones.
Test Structure
*************************************
Base case
=====================================
Let's start with a simple example.
Say there's no need to cover many types with your test.
.. code-block:: c++
// 0) Define test name and tags
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]")
{
using type = std::int32_t;
constexpr int threads_per_block = 256;
constexpr int num_items = threads_per_block;
// 1) Allocate device input
c2h::device_vector<type> d_input(num_items);
// 2) Generate 3 random input arrays using Catch2 helper
c2h::gen(C2H_SEED(3), d_input);
// 3) Allocate output array
c2h::device_vector<type> d_output(d_input.size());
// 4) Copy device input to host
c2h::host_vector<key_t> h_reference = d_input;
// 5) Compute reference output
std::ALGORITHM(
thrust::raw_pointer_cast(h_reference.data()),
thrust::raw_pointer_cast(h_reference.data()) + h_reference.size());
// 6) Compute CUB output
SCOPE_ALGORITHM<threads_per_block>(d_input.data(),
d_output.data(),
d_input.size());
// 7) Compare device and host results
REQUIRE( d_input == d_output );
}
We introduce test cases with the ``C2H_TEST`` macro in (0).
This macro always takes two string arguments - a free-form test name and
one or more tags. Then, in (1), we allocate device memory using ``c2h::device_vector``.
``c2h::device_vector`` and ``c2h::host_vector`` behave similarly to their Thrust counterparts,
but are modified to provide more stable behavior in some testing edge cases.
.. important::
Always use ``c2h::host_vector<T>``/``c2h::device_vector<T>``
instead of ``thrust::host_vector<T>``/``thrust::device_vector<T>``,
unless the test code is being used for documentation examples.
Similarly, any thrust algorithms that executed on the device must be invoked with the
`c2h::device_policy` execution policy (not shown here) to support the same edge cases.
The memory is filled with random data in (2).
Generator ``c2h::gen`` takes at least two parameters.
The first one is a random generator seed.
Instead of providing a single value, we use the ``C2H_SEED`` macro.
The macro expects a number of seeds that has to be generated.
In the example above, we require three random seeds to be generated.
This leads to the whole test being executed three times
with different seed values.
Later, in (3), we allocate device output and host reference.
In (4), we allocate and populate the host input data.
Then, we perform the reference computation on the host in (5).
.. important::
Standard library algorithms (``std::``) have to be used where possible when computing reference solutions.
Afterwards, we launch the corresponding CUB algorithm in (6).
At this point, we have a reference solution on the CPU and a CUB solution on the GPU.
The two can be compared using Catch2's ``REQUIRE`` macro, which stops execution upon failure (preferred).
Catch2 also offers the ``CHECK`` macro, which continues test execution if the check fails.
If your test has to cover floating point types,
it's sufficient to replace ``REQUIRE( a == b )`` with ``REQUIRE_APPROX_EQ(a, b)``.
.. important::
Using ``c2h::gen`` for producing input data is strongly advised.
Do not use ``assert`` in tests, which is usually only enabled in Debug mode,
and we run CUB tests in Release mode.
If a custom (non-fundamental) type has to be tested, the following helper class template should be used:
.. code-block:: c++
using type = c2h::custom_type_t<c2h::accumulateable_t,
c2h::equal_comparable_t>;
Here we enumerate all the type properties that we are interested in.
The produced type ends up having ``operator==`` (from ``equal_comparable_t``)
and ``operator+`` (from ``accumulateable_t``).
More properties are available.
If a property is missing, please add it to the existing set in ``c2h``
instead of writing a custom type from scratch.
Generators
=====================================
We often need to test CUB algorithms against different inputs or problem sizes.
If these are **runtime values**, we can use the Catch2 ``GENERATE`` macro:
.. code-block:: c++
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]")
{
int num_items = GENERATE(1, 100, 1'000'000); // 0) Init. a variable with a generator
// ...
}
This will lead to the test being executed three times, once for each argument to ``GENERATE(...)``.
Multiple generators in a test inside the same scope will form the cartesian product of all combinations.
Please consult the `Catch2 documentation <https://github.com/catchorg/Catch2/blob/devel/docs/generators.md>`_
for more details.
``C2H_SEED(3)`` uses a generator expression internally.
Type Lists
=====================================
Since CUB is a generic library,
it's often required to test CUB algorithms against many types.
To do so,
it's sufficient to define a type list and provide it to the ``C2H_TEST`` macro.
This is useful for **compile-time** parameterization of tests.
.. code-block:: c++
// 0) Define type list
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types) // 1) Provide it to the test case
{
// 2) Access current type with `c2h::get`
using type = typename c2h::get<0, TestType>;
// ...
}
This will lead to the test being compiled (instantiated) and run twice.
The first run will cause ``type`` to be ``std::uint8_t``.
The second one will cause ``type`` to be ``std::uint32_t``.
.. warning::
It's important to use types from the ``<cstdint>`` header
instead of built-in types like ``char`` and ``int``.
Multidimensional Configuration Spaces
=====================================
In most cases, the input data type is not the only compile-time parameter we want to vary.
For instance, you might need to test a block algorithm for different data types
**and** different thread block sizes.
To do so, you can add another type list as follows:
.. code-block:: c++
using block_sizes = c2h::enum_type_list<int, 128, 256>;
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types, block_sizes)
{
using type = typename c2h::get<0, TestType>;
constexpr int threads_per_block = c2h::get<1, TestType>::value;
// ...
}
The code above leads to the following combinations being compiled:
- ``type = std::uint8_t``, ``threads_per_block = 128``
- ``type = std::uint8_t``, ``threads_per_block = 256``
- ``type = std::int32_t``, ``threads_per_block = 128``
- ``type = std::int32_t``, ``threads_per_block = 256``
As an example, the following test case includes both multidimensional configuration spaces
and multiple random sequence generations.
.. code-block:: c++
using block_sizes = c2h::enum_type_list<int, 128, 256>;
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types, block_sizes)
{
using type = typename c2h::get<0, TestType>;
constexpr int threads_per_block = c2h::get<1, TestType>::value;
// ...
c2h::device_vector<type> d_input(5);
c2h::gen(C2H_SEED(2), d_input);
}
The code above leads to the following combinations being compiled:
- ``type = std::uint8_t``, ``threads_per_block = 128``, 1st random generated input sequence
- ``type = std::uint8_t``, ``threads_per_block = 256``, 1st random generated input sequence
- ``type = std::int32_t``, ``threads_per_block = 128``, 1st random generated input sequence
- ``type = std::int32_t``, ``threads_per_block = 256``, 1st random generated input sequence
- ``type = std::uint8_t``, ``threads_per_block = 128``, 2nd random generated input sequence
- ``type = std::uint8_t``, ``threads_per_block = 256``, 2nd random generated input sequence
- ``type = std::int32_t``, ``threads_per_block = 128``, 2nd random generated input sequence
- ``type = std::int32_t``, ``threads_per_block = 256``, 2nd random generated input sequence
Each new generator multiplies the number of execution times by its number of seeds. That means
that if there were further more sequence generators (``c2h::gen(C2H_SEED(X), ...)``) on the
example above the test would execute X more times and so on.
Speedup Compilation Time
=====================================
Since type lists in the ``C2H_TEST`` form a Cartesian product,
compilation time grows quickly with every new dimension.
To keep the compilation process parallelized,
it's possible to rely on our ``%PARAM%`` machinery:
.. code-block:: c++
// %PARAM% BLOCK_SIZE bs 128:256
using block_sizes = c2h::enum_type_list<int, BLOCK_SIZE>;
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types, block_sizes)
{
using type = typename c2h::get<0, TestType>;
constexpr int threads_per_block = c2h::get<1, TestType>::value;
// ...
}
The comment with ``%PARAM%`` is recognized by our CMake scripts.
It leads to multiple executables being produced from a single test source.
.. code-block:: bash
bin/cub.test.scope_algorithm.bs_128
bin/cub.test.scope_algorithm.bs_256
Multiple ``%PARAM%`` comments can be specified forming another Cartesian product.
Final Test
=====================================
Let's consider the final test that illustrates all of the tools we discussed above:
.. code-block:: c++
// %PARAM% BLOCK_SIZE bs 128:256
using block_sizes = c2h::enum_type_list<int, BLOCK_SIZE>;
using types = c2h::type_list<std::uint8_t, std::int32_t>;
C2H_TEST("SCOPE FACILITY works with CONDITION", "[FACILITY][SCOPE]",
types, block_sizes)
{
using type = typename c2h::get<0, TestType>;
constexpr int threads_per_block = c2h::get<1, TestType>::value;
constexpr int max_num_items = threads_per_block;
c2h::device_vector<type> d_input(
GENERATE_COPY(take(2, random(0, max_num_items))));
c2h::gen(C2H_SEED(3), d_input);
c2h::device_vector<type> d_output(d_input.size());
SCOPE_ALGORITHM<threads_per_block>(d_input.data(),
d_output.data(),
d_input.size());
REQUIRE( d_input == d_output );
const type expected_sum = 4;
const type sum = thrust::reduce(c2h::device_policy, d_output.cbegin(), d_output.cend());
REQUIRE( sum == expected_sum);
}
Apart from discussed tools, here we also rely on ``Catch2`` to generate random input sizes
in the range ``[0, max_num_items]`` for our input vector ``d_input``.
Overall, the test will produce two executables.
Each of these executables is going to generate ``2`` input problem sizes.
For each problem size, ``3`` random vectors are generated.
As a result, we have ``12`` different tests.
The code also demonstrates the syntax and usage of ``c2h::device_policy`` with a Thrust algorithm.

24
cccl_upstream/docs/cub/developer/thread_level.rst Normal file
View File

@@ -0,0 +1,24 @@
.. _cub-developer-guide-thread-level:
Thread-level
*************
In contrast to algorithms at the warp/block/device layer,
single threaded functionality like ``cub::ThreadReduce``
is typically implemented as a sequential function and rarely exposed to the user.
.. code-block:: c++
template <
int LENGTH,
typename T,
typename ReductionOp,
typename PrefixT,
typename AccumT = detail::accumulator_t<ReductionOp, PrefixT, T>>
__device__ __forceinline__ AccumT ThreadReduce(
T (&input)[LENGTH],
ReductionOp reduction_op,
PrefixT prefix)
{
return ...;
}

157
cccl_upstream/docs/cub/developer/warp_level.rst Normal file
View File

@@ -0,0 +1,157 @@
.. _cub-developer-guide-warp-level:
Warp-level
************************************
CUB warp-level algorithms are specialized for execution by threads in the same CUDA warp.
These algorithms may only be invoked by ``1 <= n <= 32`` *consecutive* threads in the same warp.
Overview
====================================
Warp-level functionality is provided by types (classes) to provide encapsulation and enable partial template specialization.
For example, :cpp:struct:`cub::WarpReduce` is a class template:
.. code-block:: c++
template <typename T,
int LOGICAL_WARP_THREADS = 32>
class WarpReduce {
// ...
// (1) define `_TempStorage` type
// ...
_TempStorage &temp_storage;
public:
// (2) wrap `_TempStorage` in uninitialized memory
struct TempStorage : Uninitialized<_TempStorage> {};
__device__ __forceinline__ WarpReduce(TempStorage &temp_storage)
// (3) reinterpret cast
: temp_storage(temp_storage.Alias())
{}
// (4) actual algorithms
__device__ __forceinline__ T Sum(T input);
};
In CUDA, the hardware warp size is 32 threads.
However, CUB enables warp-level algorithms on "logical" warps of ``1 <= n <= 32`` threads.
The size of the logical warp is required at compile time via the ``LOGICAL_WARP_THREADS`` non-type template parameter.
This value is defaulted to the hardware warp size of ``32``.
There is a vital difference in the behavior of warp-level algorithms that depends on the value of ``LOGICAL_WARP_THREADS``:
- If ``LOGICAL_WARP_THREADS`` is a power of two - warp is partitioned into *sub*-warps,
each reducing its data independently from other *sub*-warps.
The terminology used in CUB: ``32`` threads are called hardware warp.
Groups with less than ``32`` threads are called *logical* or *virtual* warp since it doesn't correspond directly to any hardware unit.
- If ``LOGICAL_WARP_THREADS`` is **not** a power of two - there's no partitioning.
That is, only the first logical warp executes algorithm.
.. TODO: Add diagram showing non-power of two logical warps.
Temporary storage usage
====================================
Warp-level algorithms require temporary storage for scratch space and inter-thread communication.
The temporary storage needed for a given instantiation of an algorithm is known at compile time
and is exposed through the ``TempStorage`` member type definition.
It is the caller's responsibility to create this temporary storage and provide it to the constructor of the algorithm type.
It is possible to reuse the same temporary storage for different algorithm invocations,
but it is unsafe to do so without first synchronizing to ensure the first invocation is complete.
.. TODO: Add more explanation of the `TempStorage` type and the `Uninitialized` wrapper.
.. TODO: Explain if `TempStorage` is required to be shared memory or not.
.. code-block:: c++
using WarpReduce = cub::WarpReduce<int>;
// Allocate WarpReduce shared memory for four warps
__shared__ WarpReduce::TempStorage temp_storage[4];
// Get this thread's warp id
int warp_id = threadIdx.x / 32;
int aggregate_1 = WarpReduce(temp_storage[warp_id]).Sum(thread_data_1);
// illegal, has to add `__syncwarp()` between the two
int aggregate_2 = WarpReduce(temp_storage[warp_id]).Sum(thread_data_2);
// illegal, has to add `__syncwarp()` between the two
foo(temp_storage[warp_id]);
Specialization
====================================
The goal of CUB is to provide users with algorithms that abstract the complexities of achieving speed-of-light performance across a variety of use cases and hardware.
It is a CUB developer's job to abstract this complexity from the user by providing a uniform interface that statically dispatches to the optimal code path.
This is usually accomplished via customizing the implementation based on compile time information like the logical warp size, the data type, and the target architecture.
For example, :cpp:struct:`cub::WarpReduce` dispatches to two different implementations based on if the logical warp size is a power of two (described above):
.. code-block:: c++
using InternalWarpReduce = cuda::std::conditional_t<
IS_POW_OF_TWO,
detail::WarpReduceShfl<T, LOGICAL_WARP_THREADS>, // shuffle-based implementation
detail::WarpReduceSmem<T, LOGICAL_WARP_THREADS>>; // smem-based implementation
Specializations provide different shared memory requirements,
so the actual ``_TempStorage`` type is defined as:
.. code-block:: c++
using _TempStorage = typename InternalWarpReduce::TempStorage;
and algorithm implementation look like:
.. code-block:: c++
__device__ __forceinline__ T Sum(T input, int valid_items) {
return InternalWarpReduce(temp_storage)
.Reduce(input, valid_items, ::cuda::std::plus<>{});
}
``__CUDA_ARCH__`` cannot be used because it is conflicting with the PTX dispatch refactoring and limited NVHPC support.
Due to this limitation, we can't specialize on the PTX version.
``NV_IF_TARGET`` shall be used by specializations instead:
.. code-block:: c++
template <typename T, int LOGICAL_WARP_THREADS>
struct WarpReduceShfl
{
template <typename ReductionOp>
__device__ __forceinline__ T ReduceImpl(T input, int valid_items,
ReductionOp reduction_op)
{
// ... base case (SM < 80) ...
}
template <class U = T>
__device__ __forceinline__
typename std::enable_if<std::is_same_v<int, U> ||
std::is_same_v<unsigned int, U>,
T>::type
ReduceImpl(T input,
int, // valid_items
::cuda::std::plus<>) // reduction_op
{
T output = input;
NV_IF_TARGET(NV_PROVIDES_SM_80,
(output = __reduce_add_sync(member_mask, input);),
(output = ReduceImpl<::cuda::std::plus<>>(
input, LOGICAL_WARP_THREADS, ::cuda::std::plus<>{});));
return output;
}
};
Specializations are stored in the ``cub/warp/specializations`` directory.

138
cccl_upstream/docs/cub/developer_overview.rst Normal file
View File

@@ -0,0 +1,138 @@
Developer Overview
##########################
.. toctree::
:hidden:
:maxdepth: 2
developer/thread_level
developer/warp_level
developer/block_scope
developer/device_scope
developer/nvtx
developer/test_overview
This living document serves as a guide to the design of the internal structure of CUB.
CUB provides layered algorithms that correspond to the thread/warp/block/device hierarchy of threads in CUDA.
There are distinct algorithms for each layer and higher-level layers build on top of those below.
For example, CUB has four flavors of ``reduce``,
one for each layer: ``ThreadReduce, WarpReduce, BlockReduce``, and ``DeviceReduce``.
Each is unique in how it is invoked,
how many threads participate,
and on which thread(s) the result is valid.
These layers naturally build on each other.
For example, :cpp:struct:`cub::WarpReduce` uses :cpp:func:`cub::ThreadReduce`,
:cpp:struct:`cub::BlockReduce` uses :cpp:struct:`cub::WarpReduce`, etc.
:cpp:func:`cub::ThreadReduce`
- A normal function invoked and executed sequentially by a single thread that returns a valid result on that thread
- Single thread functions are usually an implementation detail and not exposed in CUB's public API
:cpp:struct:`cub::WarpReduce` and :cpp:struct:`cub::BlockReduce`
- A "cooperative" function where threads concurrently invoke the same function to execute parallel work
- The function's return value is well-defined only on the "first" thread (lowest thread index)
:cpp:struct:`cub::DeviceReduce`
- A normal function invoked by a single thread that spawns additional threads to execute parallel work
- Result is stored in the pointer provided to the function
- Function returns a ``cudaError_t`` error code
- Function does not synchronize the host with the device
The table below provides a summary of these functions:
.. list-table::
:class: table-no-stripes
:header-rows: 1
* - layer
- coop invocation
- parallel execution
- max threads
- valid result in
* - :cpp:func:`cub::ThreadReduce`
- :math:`-`
- :math:`-`
- :math:`1`
- invoking thread
* - :cpp:struct:`cub::WarpReduce`
- :math:`+`
- :math:`+`
- :math:`32`
- main thread
* - :cpp:struct:`cub::BlockReduce`
- :math:`+`
- :math:`+`
- :math:`1024`
- main thread
* - :cpp:struct:`cub::DeviceReduce`
- :math:`-`
- :math:`+`
- :math:`\infty`
- global memory
The details of how each of these layers are implemented is described below.
Common Patterns
************************************
While CUB's algorithms are unique at each layer,
there are commonalities among all of them:
- Algorithm interfaces are provided as *types* (classes)\ [1]_
- Algorithms need temporary storage
- Algorithms dispatch to specialized implementations depending on compile-time and runtime information
- Cooperative algorithms require the number of threads at compile time (template parameter)
Invoking any CUB algorithm follows the same general pattern:
#. Select the class for the desired algorithm
#. Query the temporary storage requirements
#. Allocate the temporary storage
#. Pass the temporary storage to the algorithm
#. Invoke it via the appropriate member function
An example of :cpp:struct:`cub::BlockReduce` demonstrates these patterns in practice:
.. code-block:: c++
__global__ void kernel(int* per_block_results)
{
// (1) Select the desired class
// `cub::BlockReduce` is a class template that must be instantiated for the
// input data type and the number of threads. Internally the class is
// specialized depending on the data type, number of threads, and hardware
// architecture. Type aliases are often used for convenience:
using BlockReduce = cub::BlockReduce<int, 128>;
// (2) Query the temporary storage
// The type and amount of temporary storage depends on the selected instantiation
using TempStorage = typename BlockReduce::TempStorage;
// (3) Allocate the temporary storage
__shared__ TempStorage temp_storage;
// (4) Pass the temporary storage
// Temporary storage is passed to the constructor of the `BlockReduce` class
BlockReduce block_reduce{temp_storage};
// (5) Invoke the algorithm
// The `Sum()` member function performs the sum reduction of `thread_data` across all 128 threads
int thread_data[4] = {1, 2, 3, 4};
int block_result = block_reduce.Sum(thread_data);
per_block_results[blockIdx.x] = block_result;
}
.. [1] Algorithm interfaces are provided as classes because it provides encapsulation for things like temporary storage requirements and enables partial template specialization for customizing an algorithm for specific data types or number of threads.
For more detailed descriptions of the respective algorithms levels see the individual sections below
- :ref:`thread-level algorithms<cub-developer-guide-thread-level>`
- :ref:`warp-level algorithms<cub-developer-guide-warp-level>`
- :ref:`block-scope algorithms<cub-developer-guide-block-scope>`
- :ref:`device-scope algorithms<cub-developer-guide-device-scope>`
There is additional information for :ref:`nvtx ranges <cub-developer-guide-nvtx>`

323
cccl_upstream/docs/cub/device_topk_requirements.rst Normal file
View File

@@ -0,0 +1,323 @@
:orphan:
.. _cub-topk-requirements:
Top-K: Determinism, Tie-Breaking, and Output Ordering
======================================================
This page describes how to control the result of the CUB top-k family of algorithms
(:cpp:struct:`cub::DeviceTopK` and :cpp:struct:`cub::DeviceBatchedTopK`) through the execution
environment. For :cpp:struct:`cub::DeviceBatchedTopK`, these requirements apply independently within
each segment. The same requirement model applies to every ``MaxKeys`` / ``MinKeys`` / ``MaxPairs`` /
``MinPairs`` entry point.
Two orthogonal concerns
-----------------------
Top-k algorithms answer two separate questions:
#. **Which items are returned?** (the result *set* / membership), controlled by
``cuda::execution::determinism`` and, when deterministic, optionally refined by
``cuda::execution::tie_break``.
#. **In what order are those items written to the output?** (the result *sequence*), controlled
independently by ``cuda::execution::output_ordering``.
Think of it this way: determinism (with an optional tie-break) first selects a *set* of *K* items.
Output ordering then arranges that fixed set into the output buffer. Changing output ordering never
changes *which* items are selected. Changing tie-breaking never dictates *how* equal-key items are
sequenced in the output (unless you also request a stable ordering, as described below).
**Determinism applies to set membership.** Even with a deterministic selection, the *positions* of
the selected items in the output buffer may still vary unless you also request a specific output
ordering. Non-determinism arises only when more elements compare equal at the selection boundary
than there are remaining slots in the top-*K*. For example, with *K* = 3 and four elements tied for
the third-largest position, the algorithm must choose three of the four, and that choice is the
source of variability.
**Output ordering applies to the result sequence.** Once the result set is fixed, output ordering
specifies how those *K* items are laid out in the output buffer.
.. _cub-topk-default-behavior:
Default behavior
----------------
When you do **not** specify any of these requirements, the top-k algorithms provide their strongest
reproducibility guarantees. The committed default contract is:
* ``cuda::execution::determinism::gpu_to_gpu`` for a deterministic result set,
* ``cuda::execution::tie_break::prefer_smaller_index`` to resolve ties at the selection boundary
toward the smaller (lower) source index,
* ``cuda::execution::output_ordering::stable_sorted`` to write output sorted by key, with equal
keys ordered by source index.
In other words, by default you get the same items, in the same positions, run after run and across
GPUs of the same architecture. You opt **out** of these guarantees (by requiring weaker properties
such as ``cuda::execution::determinism::not_guaranteed`` and
``cuda::execution::output_ordering::unsorted``) to obtain faster implementations.
``determinism`` and ``tie_break`` are coupled. You specify **both** of them (inside a single
``cuda::execution::require(...)``) or **neither** (to take the default). A specified ``tie_break`` of
``prefer_smaller_index`` or ``prefer_larger_index`` pins the result set across GPUs and therefore
requires ``determinism::gpu_to_gpu``. See :ref:`cub-topk-set-membership` for the full table.
.. note::
**Current support.** This initial API surface only implements the fully opted-out configuration.
For :cpp:struct:`cub::DeviceBatchedTopK` it must be requested **explicitly** as
``cuda::execution::require(cuda::execution::determinism::not_guaranteed,
cuda::execution::tie_break::unspecified, cuda::execution::output_ordering::unsorted)``
(:cpp:struct:`cub::DeviceTopK` has no tie-break dimension yet and omits the ``tie_break`` token).
The algorithms ``static_assert`` for any other combination (including an empty, no-requirement
environment), so the deterministic default described above cannot yet be exercised in code. The
deterministic, tie-broken, and (stable-)sorted modes documented here define the committed long-term
contract and will become available (including as the no-requirement default) as those code paths
land.
Requirements reference
----------------------
Determinism (``cuda::execution::determinism``)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 25 75
* - Value
- Meaning
* - ``not_guaranteed``
- No reproducibility guarantee. Among tied elements at the selection boundary, any valid subset
may be returned. Enables the fastest implementations.
* - ``run_to_run``
- The result set is identical across repeated invocations on the same GPU with the same input.
The tie-breaking policy is implementation-defined. Pinning a specific tie-break is not
available at this level and requires ``gpu_to_gpu``.
* - ``gpu_to_gpu``
- The result set is identical across different GPUs of the same architecture. This is the only
level that may be combined with an explicit ``tie_break`` (``prefer_smaller_index`` or
``prefer_larger_index``), which then fully pins the result set for a given input.
Tie-break (``cuda::execution::tie_break``)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A specified ``tie_break`` of ``prefer_smaller_index`` or ``prefer_larger_index`` pins the result set
across GPUs, so it requires ``determinism::gpu_to_gpu``. Pairing it with ``run_to_run`` or
``not_guaranteed`` is rejected at compile time. ``determinism`` and ``tie_break`` must always be
specified together (or both omitted to take the default). Use ``tie_break::unspecified`` to leave the
boundary policy to the implementation, for example alongside ``not_guaranteed`` or ``run_to_run``.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Value
- Meaning
* - ``unspecified``
- Any deterministic tie-break is acceptable, and the implementation chooses. Valid with any
determinism level (including ``not_guaranteed`` and ``run_to_run``).
* - ``prefer_smaller_index`` *(default)*
- Among elements that compare equal at the boundary, prefer those with the **smaller** source
index. Requires ``determinism::gpu_to_gpu``.
* - ``prefer_larger_index``
- Among elements that compare equal at the boundary, prefer those with the **larger** source
index. Requires ``determinism::gpu_to_gpu``.
Output ordering (``cuda::execution::output_ordering``)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 25 75
* - Value
- Meaning
* - ``unsorted``
- No guarantee on output order. The same result set may appear in different permutations across
runs.
* - ``sorted``
- Output is sorted by key value (descending for ``Max*``, ascending for ``Min*``). Among
elements with equal keys, the relative order is **unspecified**.
* - ``stable_sorted``
- Output is sorted by key value, and among equal keys the relative order matches the **input
order** (smaller source index first). With a fully pinned result set (an explicit
``tie_break``) this fully determines the output, so the result is bit-identical even across
GPUs of the same architecture.
Composing requirements
----------------------
Requirements compose into a single ``cuda::execution::require(...)`` argument, which is placed in the
execution environment alongside other properties such as a stream:
.. code-block:: c++
auto env = cuda::std::execution::env{
cuda::execution::require(
cuda::execution::determinism::gpu_to_gpu,
cuda::execution::tie_break::prefer_smaller_index,
cuda::execution::output_ordering::sorted),
stream_ref};
.. _cub-topk-set-membership:
Which items are selected?
-------------------------
Determinism and tie-break together control **set membership**. They are always specified as a pair
(or both omitted to take the default). Rows below are the ``determinism`` requirement and columns are
the paired ``tie_break`` requirement. Cells marked *(compile error)* are rejected by a ``static_assert``.
.. list-table::
:header-rows: 1
:stub-columns: 1
:widths: 22 26 26 26
* - ``determinism``
- ``tie_break::unspecified``
- ``tie_break::prefer_smaller_index``
- ``tie_break::prefer_larger_index``
* - ``not_guaranteed``
- Non-deterministic (fast path)
- *(compile error)*
- *(compile error)*
* - ``run_to_run``
- Deterministic, implementation-defined tie-break
- *(compile error)*
- *(compile error)*
* - ``gpu_to_gpu``
- Deterministic, implementation-defined tie-break
- Deterministic, ties toward the **smaller** source index
- Deterministic, ties toward the **larger** source index
Reading the table:
* A specified ``tie_break`` of ``prefer_smaller_index`` or ``prefer_larger_index`` pins the result set
across GPUs, which is a ``gpu_to_gpu`` guarantee. Requesting it alongside ``not_guaranteed`` or
``run_to_run`` is a compile error, because you must acknowledge the ``gpu_to_gpu`` determinism you
receive.
* With ``tie_break::unspecified`` the implementation chooses the boundary policy. ``run_to_run`` and
``gpu_to_gpu`` then differ only in *scope*: identical results on the same GPU versus across GPUs of
the same architecture.
* Omitting **both** requirements selects the default (``gpu_to_gpu`` with ``prefer_smaller_index``),
which is the bottom-middle cell.
.. note::
This determinism and tie_break pairing rule is currently enforced only by
:cpp:struct:`cub::DeviceBatchedTopK`. :cpp:struct:`cub::DeviceTopK` does not yet inspect
``tie_break``, so it still accepts requirement combinations that ``cub::DeviceBatchedTopK`` rejects.
The same enforcement will be added to ``cub::DeviceTopK`` in the next major release of CCCL (4.0).
Worked example: set membership x output ordering
-------------------------------------------------
Consider ``cub::DeviceTopK::MaxKeys`` with *K* = 3 on this input:
.. code-block:: text
index : 0 1 2 3 4 5
key : 10 8 8 8 6 5
The top three keys are ``10`` and two ``8``\ s. Four elements compare equal at the boundary (the
``8``\ s at indices 1, 2, 3), but only two can be kept. That is the tie. The notation ``key@index``
identifies an element by both its key and its source position (for example ``8@2`` is the ``8`` at
index 2).
The table below shows **two runs on the same input** for each combination. Compare the two runs
within a cell to see whether the output order varies. Compare across rows to see whether the set
membership varies.
.. list-table::
:header-rows: 1
:widths: 28 24 24 24
* - ``require(...)``
- ``output_ordering::unsorted``
- ``output_ordering::sorted``
- ``output_ordering::stable_sorted``
* - ``determinism::not_guaranteed,``
``tie_break::unspecified``
- | Run 1: ``[8@2, 10@0, 8@1]``
| Run 2: ``[8@3, 10@0, 8@1]``
| Different sets *and* orders
- | Run 1: ``[10@0, 8@2, 8@1]``
| Run 2: ``[10@0, 8@1, 8@3]``
| Different sets, sorted by key
- | Run 1: ``[10@0, 8@1, 8@2]``
| Run 2: ``[10@0, 8@1, 8@3]``
| Different sets, equal keys in input order
* - ``determinism::run_to_run,``
``tie_break::unspecified``
- | Run 1: ``[8@3, 10@0, 8@1]``
| Run 2: ``[10@0, 8@1, 8@3]``
| Same set ``{10@0, 8@1, 8@3}``, order may vary
- | Run 1: ``[10@0, 8@3, 8@1]``
| Run 2: ``[10@0, 8@1, 8@3]``
| Same set, equal-key order unspecified
- | Run 1: ``[10@0, 8@1, 8@3]``
| Run 2: ``[10@0, 8@1, 8@3]``
| Same set, equal keys always in input order
* - ``determinism::gpu_to_gpu,``
``tie_break::prefer_smaller_index``
- | Run 1: ``[8@2, 10@0, 8@1]``
| Run 2: ``[10@0, 8@1, 8@2]``
| Same set ``{10@0, 8@1, 8@2}``, order may vary
- | Run 1: ``[10@0, 8@2, 8@1]``
| Run 2: ``[10@0, 8@1, 8@2]``
| Same set, equal-key order unspecified
- | Run 1: ``[10@0, 8@1, 8@2]``
| Run 2: ``[10@0, 8@1, 8@2]``
| Same set, equal keys always in input order
* - ``determinism::gpu_to_gpu,``
``tie_break::prefer_larger_index``
- | Run 1: ``[8@3, 10@0, 8@2]``
| Run 2: ``[10@0, 8@2, 8@3]``
| Same set ``{10@0, 8@2, 8@3}``, order may vary
- | Run 1: ``[10@0, 8@3, 8@2]``
| Run 2: ``[10@0, 8@2, 8@3]``
| Same set, equal-key order unspecified
- | Run 1: ``[10@0, 8@2, 8@3]``
| Run 2: ``[10@0, 8@2, 8@3]``
| Same set, equal keys always in input order
Reading the matrix:
.. list-table::
:header-rows: 1
:widths: 45 55
* - Observation
- Where to look
* - Set membership varies across runs
- ``not_guaranteed`` row: Run 1 keeps ``8@2``, Run 2 keeps ``8@3``
* - Set membership fixed, order varies
- ``run_to_run`` + ``unsorted``: both runs return ``{10@0, 8@1, 8@3}`` in different permutations
* - Set membership fixed, sorted but unstable among equal keys
- ``run_to_run`` + ``sorted``: both runs start with ``10@0``, but ``8@1`` and ``8@3`` may swap
* - Fully pinned: same set and same order
- ``gpu_to_gpu`` + ``tie_break::prefer_smaller_index`` + ``stable_sorted``: both runs yield
``[10@0, 8@1, 8@2]``
* - Tie-break changes the set, not just the order
- Compare ``prefer_smaller_index`` vs ``prefer_larger_index``: ``8@2`` vs ``8@3``
Choosing requirements
---------------------
.. list-table::
:header-rows: 1
:widths: 55 45
* - Goal
- Suggested ``require(...)``
* - Maximum performance, exact result unimportant
- ``determinism::not_guaranteed, tie_break::unspecified, output_ordering::unsorted``
* - Reproducible result set, order does not matter
- ``determinism::run_to_run, tie_break::unspecified, output_ordering::unsorted``
* - Reproducible result set with an explicit boundary policy
- ``determinism::gpu_to_gpu, tie_break::prefer_{smaller,larger}_index, output_ordering::unsorted``
* - Reproducible, key-sorted output
- the above + ``output_ordering::sorted``
* - Reproducible, key-sorted output with input-order stability among ties
- the above + ``output_ordering::stable_sorted`` (a fully pinned set plus stable-sorted output
is bit-identical, including across GPUs)

120
cccl_upstream/docs/cub/device_wide.rst Normal file
View File

@@ -0,0 +1,120 @@
.. _device-module:
Device-Wide Primitives
======================
.. toctree::
:glob:
:hidden:
:maxdepth: 2
api/device
Almost all of CUB's device-wide APIs come in two flavors:
* the traditional two-phase style that requires calling the API twice and managing temporary storage explicitly,
* and the newer single-phase style where temporary storage is obtained from a memory resource in the execution environment.
Some APIs that do not require any temporary storage may also have a traditional single-phase form in addition to the newer environment-based one.
.. _device-temp-storage:
Two-Phase API (explicit temporary storage management)
+++++++++++++++++++++++++++++++++++++++++++++++++++++
Traditional two-phase APIs can be recognized by taking ``void* d_temp_storage, size_t& temp_storage_bytes`` as their first two parameters.
They follow a two-phase usage pattern that requires three steps:
1. **Query Phase**: The algorithm is called the first time with ``d_temp_storage = nullptr`` to determine the required temporary storage size.
The required size is written to ``temp_storage_bytes`` without dereferencing iterators or launching kernels.
2. **Temporary storage allocation**: The user is responsible for allocating device-accessible memory of at least ``temp_storage_bytes`` bytes.
No special alignment is required.
3. **Execution Phase**: The algorithm is called the second time with ``d_temp_storage`` pointing to the allocated device memory, performing the actual operation.
In principle, the query phase and execution phase must call the same CUB API.
This means in detail:
* **Template arguments**: The query call must use the same template arguments as the execution call, so they share the same template instantiation.
* **Argument values**: Regarding function parameters, only the values of the ``d_temp_storage``, ``temp_storage_bytes``,
and problem-size related arguments (like number of elements, number of segments, segment sizes, etc.) may be read during the query phase.
No other parameters (like input/output iterators, initial values, etc.) are accessed during the query phase, so their values may be indeterminate.
During the query phase, the API will return before launching any kernels or touching user storage.
* **Current device**: The computed temporary storage size is valid only when the execution phase runs on the same current CUDA device as the query.
Re-run the query if the current device changes between phases.
Example pattern:
.. literalinclude:: ../../cub/examples/device/example_device_reduce.cu
:language: c++
:dedent:
:start-after: example-begin temp-storage-query
:end-before: example-end temp-storage-query
Environment API (single phase)
++++++++++++++++++++++++++++++
The environment-based API is available for all CUB device-wide algorithms.
They remove the split of query/execute phase and manually obtaining the temporary storage.
Instead, the temporary storage is automatically requested from a memory resource queried from the execution environment argument.
The environment supports further properties like passing a stream or an execution requirement in addition to a memory resource.
Key properties of the environment argument:
- It is a defaulted parameter and appears as the last argument.
- Streams like `cudaStream_t` or `cuda::stream_ref` can be passed as environments directly, or added to the environment.
- You can select the memory resource (CCCL-provided or custom) used for internal allocations.
- Supported algorithms accept determinism requirements (for example, ``cuda::execution::determinism::gpu_to_gpu``).
- Multiple properties compose into a single centralized argument by wrapping them into a ``cuda::execution::env`` object.
Example pattern:
.. literalinclude:: ../../cub/examples/device/example_device_reduce_env.cu
:language: c++
:dedent:
:start-after: example-begin env-overload-setup
:end-before: example-end env-overload-setup
.. literalinclude:: ../../cub/examples/device/example_device_reduce_env.cu
:language: c++
:dedent:
:start-after: example-begin env-overload-run
:end-before: example-end env-overload-run
Further information on CUB execution environments can be found in
:ref:`Execution Environments <cub-environment>`.
API overview
++++++++++++
In the following, the various groups of CUB device-wide algorithms are listed,
linking to their respective documentation.
CUB device-level single-problem parallel algorithms:
* :cpp:struct:`cub::DeviceAdjacentDifference` computes the difference between adjacent elements residing within device-accessible memory
* :cpp:struct:`cub::DeviceFor` provides device-wide, parallel operations for iterating over data residing within device-accessible memory
* :cpp:struct:`cub::DeviceHistogram` constructs histograms from data samples residing within device-accessible memory
* :cpp:struct:`cub::DevicePartition` partitions data residing within device-accessible memory
* :cpp:struct:`cub::DeviceMerge` merges two sorted sequences in device-accessible memory into a single one
* :cpp:struct:`cub::DeviceMergeSort` sorts items residing within device-accessible memory
* :cpp:struct:`cub::DeviceRadixSort` sorts items residing within device-accessible memory using radix sorting method
* :cpp:struct:`cub::DeviceReduce` computes reduction of items residing within device-accessible memory
* :cpp:struct:`cub::DeviceRunLengthEncode` demarcating "runs" of same-valued items within a sequence residing within device-accessible memory
* :cpp:struct:`cub::DeviceScan` computes a prefix scan across a sequence of data items residing within device-accessible memory
* :cpp:struct:`cub::DeviceSelect` compacts data residing within device-accessible memory
* :cpp:struct:`cub::DeviceTransform` transforms elements from multiple input sequences into an output sequence
* :cpp:struct:`cub::DeviceTopK` finds the largest (or smallest) K items from an unordered list residing within device-accessible memory
CUB device-level segmented-problem (batched) parallel algorithms:
* :cpp:struct:`cub::DeviceSegmentedSort` computes batched sort across non-overlapping sequences of data residing within device-accessible memory
* :cpp:struct:`cub::DeviceSegmentedRadixSort` computes batched radix sort across non-overlapping sequences of data residing within device-accessible memory
* :cpp:struct:`cub::DeviceSegmentedReduce` computes reductions across multiple sequences of data residing within device-accessible memory
* :cpp:struct:`cub::DeviceSegmentedScan` computes prefix scans across multiple sequences of data residing within device-accessible memory
* :cpp:struct:`cub::DeviceBatchedTopK` finds the largest (or smallest) K items from each of multiple unordered lists (segments) residing within device-accessible memory
* :cpp:struct:`cub::DeviceCopy` provides device-wide, parallel operations for batched copying of data residing within device-accessible memory
* :cpp:struct:`cub::DeviceMemcpy` provides device-wide, parallel operations for batched copying of data residing within device-accessible memory
* :cpp:struct:`cub::DeviceFind` provides vectorized binary search algorithms

253
cccl_upstream/docs/cub/environment.rst Normal file
View File

@@ -0,0 +1,253 @@
.. _cub-environment:
Execution Environments
======================
Most CUB device-wide algorithms accept an optional *execution environment* as their last
argument. The environment is a single object that bundles everything CUB needs to know
about *how* to run the algorithm.
Part of what it carries is familiar: the CUDA stream, which CUB algorithms have always
accepted, now simply travels inside the environment. The rest are controls that had no
place in the classic API and are enabled by environments:
:ref:`determinism requirements <cub-env-determinism>`,
:ref:`custom tuning policy selectors <cub-env-tuning>`, and
:ref:`memory resources <cub-env-memory-resource>`. All properties are optional and freely
composable with each other.
This page explains what the CUB environment APIs are, and how to use them.
.. contents::
:local:
:depth: 2
Why environments?
-----------------
The classic two-phase CUB API requires three steps: query temporary-storage size, allocate,
then execute. That is fine for situations where precise control over the temporary storage
allocation is required, or when the temporary storage size needs to be queried independently
of allocating it. For example, when the temporary storage is combined with other storage, e.g.
for an output, into a single allocation. But in many cases this is not needed and the two-step
API is just repetitive boilerplate.
The environment-based single-phase API collapses all of that into one call. The algorithm
queries the environment for the properties it needs — for example, the stream to run on,
or a memory resource to allocate temporary storage from — and then executes. Properties
an algorithm does not use are simply ignored, which is what makes one environment safe to
pass to many different algorithms:
.. code-block:: c++
cub::DeviceReduce::Sum(d_input, d_output, num_items, env);
.. note::
The environment argument is entirely
optional: since it is defaulted, an algorithm can be invoked with no temporary-storage
arguments and no environment at all, and every property falls back to its default
(see :ref:`Default behavior <cub-environment-fallback>`):
.. code-block:: c++
cub::DeviceReduce::Sum(d_input, d_output, num_items);
.. _cub-env-building:
Building an environment
-----------------------
.. list explicitly what types are valid environments
Some types are already valid environments on their own and can be passed directly to an
algorithm. The most common case is a stream: ``cuda::stream_ref`` (or a raw
``cudaStream_t``) passed as the last argument is treated as an environment containing
just that stream.
.. literalinclude:: ../../cub/test/catch2_test_device_reduce_env_api.cu
:language: c++
:dedent:
:start-after: example-begin reduce-env-stream
:end-before: example-end reduce-env-stream
When you need more than one property, combine them with ``cuda::std::execution::env``.
Properties can be listed in any order. The example below
composes a stream, a memory pool for temporary storage, and a determinism requirement
(the required declarations are provided by ``<cuda/execution>``, ``<cuda/stream>``,
``<cuda/memory_pool>``, and ``<cuda/devices>``):
.. literalinclude:: ../../cub/examples/device/example_device_reduce_env.cu
:language: c++
:dedent:
:start-after: example-begin env-overload-setup
:end-before: example-end env-overload-setup
.. literalinclude:: ../../cub/examples/device/example_device_reduce_env.cu
:language: c++
:dedent:
:start-after: example-begin env-overload-run
:end-before: example-end env-overload-run
The same ``env`` object can be passed to multiple algorithm calls without rebuilding it each
time, see :ref:`cub-environment-reuse`.
How to use environments
-----------------------
The controls below are what environments enable beyond the classic API. A few algorithms
additionally accept algorithm-specific controls — e.g. tie-breaking and output-ordering
for :ref:`cub::DeviceTopK <cub-topk-requirements>`. The set of supported properties
keeps growing.
.. _cub-env-determinism:
Determinism Requirements
~~~~~~~~~~~~~~~~~~~~~~~~
Use ``cuda::execution::require`` to request a guarantee, like a reproducible/deterministic execution:
.. literalinclude:: ../../cub/test/catch2_test_device_reduce_env_api.cu
:language: c++
:dedent:
:start-after: example-begin reduce-env-determinism
:end-before: example-end reduce-env-determinism
Multiple determinism levels are available. The meaning of each is described in the
:ref:`CCCL determinism overview <cccl-determinism>`. Which algorithms support which levels,
and each algorithm's default, are listed in the :ref:`CUB determinism support matrix
<cub-determinism>`. Requesting a level an algorithm does not support is rejected at
compile time.
.. _cub-env-tuning:
Custom Tuning Policy Selectors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pass a custom policy selector through the environment to override CUB's built-in tuning.
A policy selector answers the question "which tuning parameters should this algorithm use
on this GPU?": it is a function object that CUB calls with the GPU's compute capability,
and that returns the tuning parameters — threads per block, items per thread, and so on —
packaged as the algorithm's policy (here ``cub::MergePolicy``):
.. literalinclude:: ../../cub/test/catch2_test_device_merge_env_api.cu
:language: c++
:dedent:
:start-after: example-begin merge-keys-policy-selector
:end-before: example-end merge-keys-policy-selector
The selector is wrapped with ``cuda::execution::tune`` and passed as (part of) the
environment:
.. literalinclude:: ../../cub/test/catch2_test_device_merge_env_api.cu
:language: c++
:dedent:
:start-after: example-begin merge-keys-tuning
:end-before: example-end merge-keys-tuning
.. seealso:: :ref:`cub-policy-selectors` - full guide on defining and composing policy
selectors, including the requirements a policy selector must satisfy.
.. _cub-env-memory-resource:
Memory Resources
~~~~~~~~~~~~~~~~
The memory resource controls where an algorithm's temporary storage is allocated from.
Memory resource types are valid environments on their own, so they can be passed directly
or composed with other properties:
.. code-block:: c++
auto pool = cuda::device_default_memory_pool(cuda::devices[0]);
// Pass directly...
cub::DeviceReduce::Sum(d_input, d_output, num_items, pool);
// ...or compose with other properties
auto env = cuda::std::execution::env{stream, pool};
cub::DeviceReduce::Sum(d_input, d_output, num_items, env);
Temporary storage is allocated from the memory resource on the algorithm's stream before
execution and released on the same stream afterwards. When no memory resource is present
in the environment, CUB falls back to a stream-ordered ``cudaMallocAsync``/``cudaFree`` allocator
(see :ref:`Default behavior <cub-environment-fallback>`).
.. TODO: Add Guarantees sub-section after #9278 is merged.
.. _cub-environment-reuse:
Reusing an environment across multiple calls
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
An ``env`` object can be built once and passed to as many algorithm
calls as you like:
.. code-block:: c++
auto stream = cuda::stream{cuda::devices[0]};
auto pool = cuda::device_default_memory_pool(cuda::devices[0]);
auto env = cuda::std::execution::env{cuda::stream_ref{stream}, pool};
cub::DeviceScan::ExclusiveSum(d_a, d_out_a, n, env);
cub::DeviceReduce::Sum(d_b, d_out_b, n, env);
cub::DeviceSelect::If(d_c, d_out_c, d_num_selected, n, my_predicate, env);
stream.sync();
All three calls share the same stream and memory pool. Temporary storage is allocated and
released from the pool independently for each call.
.. note::
When a tuning policy selector is embedded in the environment, it applies to *every*
algorithm that can be tuned by this selector. For example, a policy selector returning
a ``cub::ReducePolicy`` will be used by all calls to ``cub::DeviceReduce::*`` that use
the same environment. If two algorithms using the same environment need different
tunings, build separate environments.
.. _cub-environment-fallback:
Default behavior
----------------
The environment argument is optional on every algorithm that supports it. CUB applies the
following defaults when a property is absent from the environment (or when no environment
is passed at all):
.. list-table::
:header-rows: 1
:widths: 25 75
* - Property
- Default when absent
* - Stream
- The default CUDA stream (``cudaStream_t{}``).
* - Memory resource
- A stream-ordered allocator based on ``cudaMallocAsync``. Temporary storage is
allocated on the stream the algorithm runs on.
* - Determinism
- Algorithm-specific. See :ref:`cub-determinism`.
* - Policy selector
- CUB's built-in selector, returning architecture-tuned defaults for the current
device.
Missing properties never cause an error. The algorithm simply falls back to its default for
that property.
..
TODO(gonidelis): link to the developer-facing environments page (queries, CPOs, prop,
building custom environment types) once it lands via
https://github.com/NVIDIA/cccl/pull/10013
See also
--------
- :ref:`cub-determinism` - per-algorithm determinism support matrix
- :ref:`cub-policy-selectors` - defining and composing custom policy selectors
- :ref:`device-module` - overview of the device-wide API and the two-phase alternative
- :ref:`cccl-determinism` - CCCL-level determinism concepts

487
cccl_upstream/docs/cub/index.rst Normal file
View File

@@ -0,0 +1,487 @@
.. _cub-module:
CUB
==================================================
.. toctree::
:hidden:
:maxdepth: 3
Overview <self>
thread_level
warp_wide
block_wide
device_wide
environment
determinism
benchmarking
tuning
tuning_infra
API reference <api/index>
developer_overview
What is CUB?
==================================================
CUB provides state-of-the-art, reusable software components for every layer
of the CUDA programming model:
* **Parallel primitives**
* :ref:`Thread <thread-module>` primitives
* Thread-level reduction, etc.
* Safely specialized for each underlying CUDA architecture
* :ref:`Warp-wide <warp-module>` "collective" primitives
* Cooperative warp-wide prefix scan, reduction, etc.
* Safely specialized for each underlying CUDA architecture
* :ref:`Block-wide <block-module>` "collective" primitives
* Cooperative I/O, sort, scan, reduction, histogram, etc.
* Compatible with arbitrary thread block sizes and types
* :ref:`Device-wide <device-module>` primitives
* Parallel sort, prefix scan, reduction, histogram, etc.
* Compatible with CUDA dynamic parallelism
* **Utilities**
* **Fancy iterators**
* **Thread and thread block I/O**
* **PTX intrinsics**
* **Device, kernel, and storage management**
.. _collective-primitives:
CUB's collective primitives
==================================================
Collective software primitives are essential for constructing high-performance,
maintainable CUDA kernel code. Collectives allow complex parallel code to be
re-used rather than re-implemented, and to be re-compiled rather than
hand-ported.
.. figure:: ../img/cub_overview.png
:align: center
:alt: Orientation of collective primitives within the CUDA software stack
:name: fig_cub_overview
Orientation of collective primitives within the CUDA software stack
As a SIMT programming model, CUDA engenders both **scalar** and
**collective** software interfaces. Traditional software
interfaces are *scalar* : a single thread invokes a library routine to perform some
operation (which may include spawning parallel subtasks). Alternatively, a *collective*
interface is entered simultaneously by a group of parallel threads to perform
some cooperative operation.
CUB's collective primitives are not bound to any particular width of parallelism
or data type. This flexibility makes them:
* **Adaptable** to fit the needs of the enclosing kernel computation
* **Trivially tunable** to different grain sizes (threads per block, items per thread, etc.)
Thus CUB is *CUDA Unbound*.
An example (block-wide sorting)
==================================================
The following code snippet presents a CUDA kernel in which each block of ``BLOCK_THREADS`` threads
will collectively load, sort, and store its own segment of (``BLOCK_THREADS * ITEMS_PER_THREAD``)
integer keys:
.. code-block:: c++
#include <cub/block/block_load.cuh>
#include <cub/block/block_store.cuh>
#include <cub/block/block_radix_sort.cuh>
template <int BLOCK_THREADS, int ITEMS_PER_THREAD>
__global__ void BlockSortKernel(int *d_in, int *d_out)
{
// Specialize BlockLoad, BlockStore, and BlockRadixSort collective types
using BlockLoadT = cub::BlockLoad<
int, BLOCK_THREADS, ITEMS_PER_THREAD, cub::BLOCK_LOAD_TRANSPOSE>;
using BlockStoreT = cub::BlockStore<
int, BLOCK_THREADS, ITEMS_PER_THREAD, cub::BLOCK_STORE_TRANSPOSE>;
using BlockRadixSortT = cub::BlockRadixSort<
int, BLOCK_THREADS, ITEMS_PER_THREAD>;
// Allocate type-safe, repurposable shared memory for collectives
__shared__ union {
typename BlockLoadT::TempStorage load;
typename BlockStoreT::TempStorage store;
typename BlockRadixSortT::TempStorage sort;
} temp_storage;
// Obtain this block's segment of consecutive keys (blocked across threads)
int thread_keys[ITEMS_PER_THREAD];
const int block_offset = blockIdx.x * (BLOCK_THREADS * ITEMS_PER_THREAD);
BlockLoadT(temp_storage.load).Load(d_in + block_offset, thread_keys);
__syncthreads(); // Barrier for smem reuse
// Collectively sort the keys
BlockRadixSortT(temp_storage.sort).Sort(thread_keys);
__syncthreads(); // Barrier for smem reuse
// Store the sorted segment
BlockStoreT(temp_storage.store).Store(d_out + block_offset, thread_keys);
}
.. code-block:: c++
// Elsewhere in the host program: parameterize and launch a block-sorting
// kernel in which blocks of 128 threads each sort segments of 2048 keys
int *d_in = ...;
int *d_out = ...;
int num_blocks = ...;
BlockSortKernel<128, 16><<<num_blocks, 128>>>(d_in, d_out);
In this example, threads use ``cub::BlockLoad``, ``cub::BlockRadixSort``, and ``cub::BlockStore``
to collectively load, sort and store the block's segment of input items. Because these operations
are cooperative, each primitive requires an allocation of shared memory for threads to communicate
through. The typical usage pattern for a CUB collective is:
#. Statically specialize the primitive for the specific problem setting at hand, e.g.,
the data type being sorted, the number of threads per block, the number of keys per
thread, optional algorithmic alternatives, etc. (CUB primitives are also implicitly
specialized by the targeted compilation architecture.)
#. Allocate (or alias) an instance of the specialized primitive's nested ``TempStorage``
type within a shared memory space.
#. Specify communication details (e.g., the ``TempStorage`` allocation) to
construct an instance of the primitive.
#. Invoke methods on the primitive instance.
In particular, ``cub::BlockRadixSort`` is used to collectively sort the segment of data items
that have been partitioned across the thread block. To provide coalesced accesses
to device memory, we configure the ``cub::BlockLoad`` and ``cub::BlockStore`` primitives
to access memory using a striped access pattern (where consecutive threads
simultaneously access consecutive items) and then *transpose* the keys into
a :ref:`blocked arrangement <flexible-data-arrangement>` of elements across threads.
To reuse shared memory across all three primitives, the thread block statically
allocates a union of their ``TempStorage`` types.
Why do you need CUB?
==================================================
Writing, tuning, and maintaining kernel code is perhaps the most challenging,
time-consuming aspect of CUDA programming. Kernel software is where
the complexity of parallelism is expressed. Programmers must reason about
deadlock, livelock, synchronization, race conditions, shared memory layout,
plurality of state, granularity, throughput, latency, memory bottlenecks, etc.
With the exception of CUB, however, there are few (if any) software libraries of
*reusable* kernel primitives. In the CUDA ecosystem, CUB is unique in this regard.
As a `SIMT <https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#hardware-implementation>`_
library and software abstraction layer, CUB provides:
#. **Simplicity of composition**. CUB enhances programmer productivity by
allowing complex parallel operations to be easily sequenced and nested.
For example, ``cub::BlockRadixSort`` is constructed from ``cub::BlockExchange`` and
``cub::BlockRadixRank``. The latter is composed of ``cub::BlockScan``
which incorporates ``cub::WarpScan``.
.. figure:: ../img/nested_composition.png
:align: center
#. **High performance**. CUB simplifies high-performance program and kernel
development by taking care to implement the state-of-the-art in parallel algorithms.
#. **Performance portability**.
CUB primitives are specialized to match the diversity of NVIDIA hardware, continuously
evolving to accommodate new architecture-specific features and instructions. And
because CUB's device-wide primitives are implemented using flexible block-wide and
warp-wide collectives, we are able to performance-tune them to match the processor
resources provided by each CUDA processor architecture.
#. **Simplicity of performance tuning**:
* **Resource utilization**. CUB primitives allow developers to quickly
change grain sizes (threads per block, items per thread, etc.) to best match
the processor resources of their target architecture
* **Variant tuning**. Most CUB primitives support alternative algorithmic
strategies. For example, ``cub::BlockHistogram`` is parameterized to implement either
an atomic-based approach or a sorting-based approach. (The latter provides uniform
performance regardless of input distribution.)
* **Co-optimization**. When the enclosing kernel
is similarly parameterizable, a tuning configuration can be found that optimally
accommodates their combined register and shared memory pressure.
#. **Robustness and durability**. CUB just works. CUB primitives
are designed to function properly for arbitrary data types and widths of
parallelism (not just for the built-in C++ types or for powers-of-two threads
per block).
#. **Reduced maintenance burden**. CUB provides a SIMT software abstraction layer
over the diversity of CUDA hardware. With CUB, applications can enjoy
performance-portability without intensive and costly rewriting or porting efforts.
#. **A path for language evolution**. CUB primitives are designed
to easily accommodate new features in the CUDA programming model, e.g., thread
subgroups and named barriers, dynamic shared memory allocators, etc.
How do CUB collectives work?
==================================================
Four programming idioms are central to the design of CUB:
#. :ref:`Generic programming <generic-programming>`. C++ templates provide the flexibility
and adaptive code generation needed for CUB primitives to be useful, reusable, and
fast in arbitrary kernel settings.
#. :ref:`Reflective class interfaces <reflective-class-interfaces>`.
CUB collectives statically export their their resource requirements
(e.g., shared memory size and layout) for a given specialization, which allows compile-time
tuning decisions and resource allocation.
#. :ref:`Flexible data arrangement across threads <flexible-data-arrangement>`.
CUB collectives operate on data that is logically partitioned across a group of threads.
For most collective operations, efficiency is increased with increased granularity
(i.e., items per thread).
#. :ref:`Static tuning and co-tuning <static-tuning-and-co-tuning>`. Simple constants and static
types dictate the granularities and algorithmic alternatives to be employed by CUB collectives.
When the enclosing kernel is similarly parameterized, an optimal configuration can be determined
that best accommodates the combined behavior and resource consumption of all primitives within
the kernel.
.. _generic-programming:
Generic programming
--------------------------------------------------
We use template parameters to specialize CUB primitives for the particular
problem setting at hand. Until compile time, CUB primitives are not bound
to any particular:
* Data type (int, float, double, etc.)
* Width of parallelism (threads per thread block)
* Grain size (data items per thread)
* Underlying processor (special instructions, warp size, rules for bank conflicts, etc.)
* Tuning configuration (e.g., latency vs. throughput, algorithm selection, etc.)
.. _reflective-class-interfaces:
Reflective class interfaces
--------------------------------------------------
Unlike traditional function-oriented interfaces, CUB exposes its collective
primitives as templated C++ classes. The resource requirements for a specific
parameterization are reflectively advertised as members of the class. The
resources can then be statically or dynamically allocated, aliased
to global or shared memory, etc. The following illustrates a CUDA kernel
fragment performing a collective prefix sum across the threads of a thread block:
.. code-block:: c++
#include <cub/cub.cuh>
__global__ void SomeKernelFoo(...)
{
// Specialize BlockScan for 128 threads on integer types
using BlockScan = cub::BlockScan<int, 128>;
// Allocate shared memory for BlockScan
__shared__ typename BlockScan::TempStorage scan_storage;
...
// Obtain a segment of consecutive items that are blocked across threads
int thread_data_in[4];
int thread_data_out[4];
...
// Perform an exclusive block-wide prefix sum
BlockScan(scan_storage).ExclusiveSum(thread_data_in, thread_data_out);
Furthermore, the CUB interface is designed to separate parameter
fields by concerns. CUB primitives have three distinct parameter fields:
#. *Static template parameters*. These are constants that will
dictate the storage layout and the unrolling of algorithmic steps (e.g.,
the input data type and the number of block threads), and are used to specialize the class.
#. *Constructor parameters*. These are optional parameters regarding
inter-thread communication (e.g., storage allocation, thread-identifier mapping,
named barriers, etc.), and are orthogonal to the functions exposed by the class.
#. *Formal method parameters*. These are the operational inputs/outputs
for the various functions exposed by the class.
This allows CUB types to easily accommodate new
programming model features (e.g., named barriers, memory allocators, etc.)
without incurring a combinatorial growth of interface methods.
.. _flexible-data-arrangement:
Flexible data arrangement across threads
--------------------------------------------------
CUDA kernels are often designed such that each thread block is assigned a
segment of data items for processing.
.. figure:: ../img/tile.png
:align: center
:alt: Segment of eight ordered data items
:name: fig_tile
Segment of eight ordered data items
When the tile size equals the thread block size, the
mapping of data onto threads is straightforward (one datum per thread).
However, there are often performance advantages for processing more
than one datum per thread. Increased granularity corresponds to
decreased communication overhead. For these scenarios, CUB primitives
will specify which of the following partitioning alternatives they
accommodate:
.. list-table::
:class: table-no-stripes
:widths: 70 30
* - **Blocked arrangement**. The aggregate tile of items is partitioned
evenly across threads in "blocked" fashion with *thread*\ :sub:`i`
owning the *i*\ :sup:`th` segment of consecutive elements.
Blocked arrangements are often desirable for algorithmic benefits (where
long sequences of items can be processed sequentially within each thread).
- .. figure:: ../img/blocked.png
:align: center
:alt: *Blocked* arrangement across four threads
:name: fig_blocked
*Blocked* arrangement across four threads
(emphasis on items owned by *thread*\ :sub:`0`)
* - **Striped arrangement**. The aggregate tile of items is partitioned across threads in "striped"
fashion, i.e., the ``ITEMS_PER_THREAD`` items owned by each thread have logical stride
``BLOCK_THREADS`` between them. Striped arrangements are often desirable for data movement through
global memory (where
`read/write coalescing <https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html#coalesced-access-to-global-memory>`_
is an important performance consideration).
- .. figure:: ../img/striped.png
:align: center
:alt: *Striped* arrangement across four threads
:name: fig_striped
*Striped* arrangement across four threads
(emphasis on items owned by *thread*\ :sub:`0`)
The benefits of processing multiple items per thread (a.k.a., *register blocking*,
*granularity coarsening*, etc.) include:
* Algorithmic efficiency. Sequential work over multiple items in
thread-private registers is cheaper than synchronized, cooperative
work through shared memory spaces.
* Data occupancy. The number of items that can be resident on-chip in
thread-private register storage is often greater than the number of
schedulable threads.
* Instruction-level parallelism. Multiple items per thread also
facilitates greater ILP for improved throughput and utilization.
Finally, ``cub::BlockExchange`` provides operations for converting between blocked
and striped arrangements.
.. _static-tuning-and-co-tuning:
Static tuning and co-tuning
--------------------------------------------------
This style of flexible interface simplifies performance tuning. Most CUB
primitives support alternative algorithmic strategies that can be
statically targeted by a compiler-based or JIT-based autotuner. (For
example, ``cub::BlockHistogram`` is parameterized to implement either an
atomic-based approach or a sorting-based approach.) Algorithms are also
tunable over parameters such as thread count and grain size as well.
Taken together, each of the CUB algorithms provides a fairly rich tuning
space.
Whereas conventional libraries are optimized offline and in isolation, CUB
provides interesting opportunities for whole-program optimization. For
example, each CUB primitive is typically parameterized by threads-per-block
and items-per-thread, both of which affect the underlying algorithm's
efficiency and resource requirements. When the enclosing kernel is similarly
parameterized, the coupled CUB primitives adjust accordingly. This enables
autotuners to search for a single configuration that maximizes the performance
of the entire kernel for a given set of hardware resources.
How do I get started using CUB?
==================================================
CUB is a C++ header-only library, and part of the CUDA Core Compute Libraries (CCCL).
It ships as part of the CUDA Toolkit and is thus readily available when using the ``nvcc`` compiler.
Alternatively, consider fetching CCCL directly from GitHub to benefit from the latest improvements.
How is CUB different than Thrust and Modern GPU?
==================================================
CUB and Thrust
--------------------------------------------------
CUB and :ref:`Thrust <thrust-module>` share some
similarities in that they both provide similar device-wide primitives for CUDA.
However, they target different abstraction layers for parallel computing.
Thrust abstractions are agnostic of any particular parallel framework (e.g.,
CUDA, TBB, OpenMP, sequential CPU, etc.). While Thrust has a "backend"
for CUDA devices, Thrust interfaces themselves are not CUDA-specific and
do not explicitly expose CUDA-specific details (e.g., ``cudaStream_t`` parameters).
CUB, on the other hand, is slightly lower-level than Thrust. CUB is specific
to CUDA C++ and its interfaces explicitly accommodate CUDA-specific features.
Furthermore, CUB is also a library of SIMT collective primitives for block-wide
and warp-wide kernel programming.
CUB and Thrust are complementary and can be used together. In fact, the CUB
project arose out of a maintenance need to achieve better performance-portability
within Thrust by using reusable block-wide primitives to reduce maintenance and
tuning effort.
CUB and Modern GPU
--------------------------------------------------
CUB and `Modern GPU <https://github.com/moderngpu/moderngpu>`_ also
share some similarities in that they both implement similar device-wide primitives for CUDA.
However, they serve different purposes for the CUDA programming community. MGPU
is a pedagogical tool for high-performance GPU computing, providing clear and concise
exemplary code and accompanying commentary. It serves as an excellent source of
educational, tutorial, CUDA-by-example material. The MGPU source code is intended
to be read and studied, and often favors simplicity at the expense of portability and
flexibility.
CUB, on the other hand, is a production-quality library whose sources are complicated
by support for every version of CUDA architecture, and is validated by an extensive
suite of regression tests. Although well-documented, the CUB source text is verbose
and relies heavily on C++ template metaprogramming for situational specialization.
CUB and MGPU are complementary in that MGPU serves as an excellent descriptive source
for many of the algorithmic techniques used by CUB.
Contributors
==================================================
CUB is developed as open-source as part of the CUDA Core Compute Libraries (CCCL) by NVIDIA.
Open Source License
==================================================
CUB is mostly licensed under the BSD 3-Clause "New" or "Revised" License.
New files are created under the Apache-2.0 WITH LLVM-exception License.
See also our `LICENSE <https://github.com/NVIDIA/cccl/blob/main/LICENSE>`_ file.

8
cccl_upstream/docs/cub/thread_level.rst Normal file
View File

@@ -0,0 +1,8 @@
.. _thread-module:
Thread-level Primitives
==================================================
CUB thread-level algorithms are specialized for execution by a single thread.
* :cpp:func:`cub::ThreadReduce <cub::ThreadReduce>` computes reduction of a sequence of items

146
cccl_upstream/docs/cub/tuning.rst Normal file
View File

@@ -0,0 +1,146 @@
..
TODO(bgruber): rename the label below to _cub-tuning when all tuning API exposure PRs have landed
.. _cub-policy-selectors:
Tunings
================================================================================
Device-scope algorithms in CUB have many knobs that significantly impact performance (without affecting correctness).
For instance, the number of threads per block and items per thread can be tuned to maximize performance for a given device and data type.
But also algorithmic choices such as the load or store algorithm, a load vectorization size,
the used block-level algorithm, or enabling the use of the tensor memory accelerator can be tweaked.
Most device-scope algorithms in CUB accept a set of such tuning parameters,
by passing a policy selector, wrapped into :code:`cuda::execution::tune(...)`,
as part of the environment of a CUB device-scope API.
In the following, we describe this process in more detail.
Policy selectors
--------------------------------------------------------------------------------
Policy selectors are the mechanism by which CUB's device algorithms and kernels select
tuning parameters for a given workload and GPU compute capability.
A policy selector is a stateless callable that maps a :code:`cuda::compute_capability` to a policy struct
containing the tuning values for that compute capability.
Each set of CUB algorithms using a common underlying implementation defines a common policy struct,
e.g. :code:`cub::ReducePolicy`, which must be returned by a policy selector passed to those algorithms.
CUB employs internal default policy selectors providing tunings for known compute capabilities and workloads,
which are not publicly accessible to users.
Users can override CUB's policy selector for a given algorithm
by passing a custom policy selector through the algorithm's environment parameter.
For a description of how policy selectors are used internally in CUB's dispatch layer and kernels
see the corresponding :ref:`developer documentation <cub-developer-guide-device-scope>`.
Defining a policy selector
--------------------------------------------------------------------------------
A policy selector is any type with a :code:`__host__`, :code:`__device__`, :code:`constexpr`, and :code:`const` call operator
taking a ``cuda::compute_capability`` and returning the algorithm's policy struct:
.. code:: c++
template <typename T>
struct my_reduce_tuning {
__host__ __device__ constexpr auto operator()(cuda::compute_capability cc) const -> cub::ReducePolicy {
// tuning for Hopper and later
if (cc >= cuda::compute_capability(9, 0)) {
const auto pass = cub::ReducePassPolicy{
.threads_per_block = 512,
.items_per_thread = std::max(64 / sizeof(T), 1), // 8 double, 16 float, 32 half_t, ...
.vec_size = 2,
.reduce_algorithm = cub::BLOCK_REDUCE_WARP_REDUCTIONS,
.load_modifier = cub::LOAD_DEFAULT
};
return { .multi_tile = pass, .single_tile = pass };
}
// fallback for older GPUs
const auto pass = cub::ReducePassPolicy{
.threads_per_block = 256,
.items_per_thread = 12,
.vec_size = 1,
.reduce_algorithm = cub::BLOCK_REDUCE_WARP_REDUCTIONS,
.load_modifier = cub::LOAD_DEFAULT
};
return { .multi_tile = pass, .single_tile = pass };
}
};
.. warning::
The policy selector must be stateless (:code:`std::is_empty_v<T>` must be :code:`true`)
since only its type will be passed to a kernel — any captured state would be silently
lost. Policy selectors are also freely constructed and copied where needed, so they
must also be default constructible and copyable (:code:`std::semiregular<T>` must be
:code:`true`).
It can be a class template, but then only a full specialization can be passed to CUB,
e.g., :code:`my_reduce_tuning<float>`.
The implementation can use branches and helper functions arbitrarily,
as long as they can be evaluated at compile-time.
The returned tuning policy can, and probably should, contain different values for different compute capabilities or workloads.
Each CUB algorithm with an environment parameter searches the environment for a policy selector returning a matching policy struct.
If one is found, the policy selector will be used to determine the tuning values for host-side dispatch and kernel compilation.
Each CUB algorithm documents the policy struct to which it responds.
If CUB does not find a matching policy selector in the environment, it falls back to its internal default policy selector.
Multiple policy selectors returning different policy structs can be passed as part of the same environment,
and each algorithm will pick the one with the matching policy struct.
This is useful if the same environment is reused across several algorithm calls.
The policy structs themselves are simple semiregular aggregates.
They support C++20 designated initializers (i.e., the syntax :code:`{ .threads_per_block = 512, ... }`),
comparison for (in-)equality, and serialization using :code:`operator<<`.
They may occasionally contain member functions that compute derived values from the contained tuning values.
All policy structs are public types and will evolve in a non-breaking way, at least during minor releases,
by only adding new data members at the end of the struct.
Passing a policy selector to CUB device-scope algorithms
--------------------------------------------------------------------------------
Custom policy selectors are passed to CUB algorithms via the environment argument.
They first need to be wrapped by passing them to :code:`cuda::execution::tune(...)`:
.. code:: c++
cub::DeviceReduce::Reduce(
d_in, d_out, num_items, op, init,
cuda::execution::tune(my_reduce_tuning<int>{}));
Multiple tunings for different algorithms can be combined in a single environment:
.. code:: c++
auto env = cuda::execution::tune(
my_reduce_tuning<int>{},
my_scan_tuning{}
);
cub::DeviceReduce::Reduce(d_in, d_out, num_items, op, init, env);
cub::DeviceScan::ExclusiveSum(d_in, d_out, num_items, env);
An environment can carry further properties like a stream or a memory resource.
Policy selectors can simply be added to those:
.. code:: c++
auto env = cuda::std::execution::env{
stream_ref,
resource,
cuda::execution::tune(
my_reduce_tuning<int>{}, my_scan_tuning{})
};
cub::DeviceReduce::Reduce(d_in, d_out, num_items, op, init, env);
// alternatively, if we want to extend an env `other_env`
auto env = cuda::std::execution::env{
other_env,
cuda::execution::tune(my_scan_tuning{})
};
cub::DeviceReduce::Reduce(d_in, d_out, num_items, op, init, env);
CUB's benchmarks also make heavy use of policy selectors for tuning.
For more details on authoring benchmarks and their policy selectors for automatic tuning, see :ref:`cub-tuning-infra`.

629
cccl_upstream/docs/cub/tuning_infra.rst Normal file
View File

@@ -0,0 +1,629 @@
.. _cub-tuning-infra:
Automated Tuning Infrastructure
================================================================================
This page describes tuning infrastructure to automatically tune CUB device-scope algorithms for performance.
It provides a set of tools facilitating the process of selecting optimal tuning parameters for a given device and data type.
Terminology
--------------------------------------------------------------------------------
*We omit the word "tuning" but assume it in the definitions for all terms below,
so those terms may mean something else in a more generic context.*
The following three terms are fundamental to understanding the essence of CUB tuning:
* **compile-time (ct) workload**: a workload that can be recognized only at compile time.
*e.g. the combination of key type and offset type,* :code:`int16_t` *and* :code:`int32_t`
* **runtime (rt) workload**: a workload that can be recognized only at runtime.
*e.g. the number of input elements*
* **tuning parameter (or parameter)**: a parameter that can be tuned to maximize performance for a given device and data type.
*e.g. number of threads per block, items per thread*
Algorithms are tuned for different workloads. These workloads are defined as subspaces by NVBench via the benchmarks' axis.
For instance, radix sort can be tuned for different key types, different number of keys, and different distributions of keys. The tuning process is summarized in
the following statement:
.. raw:: html
<div style="display: flex; justify-content: center; align-items: center; height: 10; text-align: center; font-size: 1.5em; color: #76B900;">
"For each Compile-time Workload, we search for the best tuning parameters"
</div>
More specifically, the tuning infrastructure optimizes algorithms for specific compile-time workloads,
aggregating results across all runtime workloads.
It searches through a space of parameters to find the combination for a given compile-time workload with the highest score.
--------
Following is supplemental terminology that will be used throughout the rest of this tuning guide:
.. e.g. :math:`threads\_per\_block=128`
* **Parameter Space**: the set of all possible values for a given Tuning Parameter.
*It is specific to the algorithm*. For example the parameter space for the number of threads per block can be :math:`\{32, 64, 96, 128, \dots, 1024\}` for radix sort, but :math:`\{32, 64, 128, 256, 512\}` for merge sort.
* **Search Space**: Cartesian product of all the Parameter Spaces of a single algorithm.
For instance, the Search Space for an algorithm with tunable items per thread and threads per block might look like :math:`\{(ipt \times tpb) | ipt \in \{1, \dots, 25\} \text{and} tpb \in \{32, 64, 96, 128, \dots, 1024\}\}`.
* **Variant** - a point in the corresponding Search Space.
* **Base** - the variant that CUB uses by default.
* **Score** - a single number representing the performance for a given compile-time workload across all runtime workloads. For instance, a weighted-sum of speedups of a given variant compared to its base for all runtime workloads is a score.
.. * **Search** - a process consisting of covering all variants for all compile-time workloads to find a variant with maximal score.
.. ^^^ @giannis: again we do not want to scare a first time user with too many terms. "search" is both evident and can also be explained with an introductory sentence in the "Search Process" chapter ^^^
.. _cub-tuning-infra-authoring-benchmarks:
Authoring Benchmarks
--------------------------------------------------------------------------------
CUB benchmarks are split into multiple files based on the algorithm they are testing
and potentially further into compile-time flavors that are tuned for individually
(e.g. sorting only keys vs. key-value pairs, or reducing using sum vs. using min).
The name of the directory represents the name of the algorithm.
The filename corresponds on the flavor.
For instance, the benchmark :code:`benchmarks/bench/radix_sort/keys.cu` tests the radix sort implementation sorting only keys.
The executable file name is going to be transformed into :code:`cub.bench.radix_sort.keys.*`,
which is the benchmark name reported by the infrastructure.
+++++++++++++++
Headers
+++++++++++++++
**Benchmarks are based on NVBench.**
You start writing a benchmark by including :code:`nvbench_helper.cuh`. This contains all
necessary includes and definitions.
.. code:: c++
#include <nvbench_helper.cuh>
The next step is to define a search space. The search space is represented by a number of C++ comments.
The format consists of the :code:`%RANGE%` keyword, the parameter macro, the parameter abbreviation, and its range of values.
The range is represented by three numbers: :code:`start:end:step`.
Start and end are included.
For instance, the following code defines a search space for two parameters, the number of threads per block and items per thread.
.. code:: c++
// %RANGE% TUNE_ITEMS_PER_THREAD ipt 7:24:1
// %RANGE% TUNE_THREADS_PER_BLOCK tpb 128:1024:32
Next, you need to define a benchmark function. The function accepts :code:`nvbench::state &state` and
a :code:`nvbench::type_list`. For more details on the benchmark signature, take a look at the
`NVBench documentation <https://github.com/NVIDIA/nvbench>`_.
.. code:: c++
template <typename T, typename OffsetT>
void algname(nvbench::state &state, nvbench::type_list<T, OffsetT>)
{...}
Before proceeding further with the benchmark authoring
it is imperative to understand policy selectors and how they provide tuning values.
+++++++++++++++
Policy Selector
+++++++++++++++
The tuning value of all CUB device algorithms can be customized
by providing a custom policy selector to the environment argument of a CUB API call.
See :ref:`cub-policy-selectors` for a full explanation.
The tuning infrastructure will use the :code:`TUNE_BASE` macro to distinguish between compiling the base version (i.e. baseline) of a benchmark
and compiling a variant for a given set of tuning parameters.
When base is used, no custom policy selector is specified, so CUB's default tunings are used.
If :code:`TUNE_BASE` is not defined, we define a custom policy selector
that specifies the values for the current variant (i.e. the current set of tuning parameters),
which are derived from the parameter macros defined in the :code:`%RANGE%` comments, which define the search space.
This custom policy selector is passed to :code:`cuda::execution::tune`,
and the returned value is included in the environment passed to the CUB API.
The following code is included in the benchmark for the policy selector to be enabled
and the parameters to have effect in execution:
.. code:: c++
#if !TUNE_BASE
template <typename T>
struct policy_selector {
_CCCL_HOST_DEVICE_API constexpr auto operator()(cuda::compute_capability /*cc*/) const -> cub::ReducePolicy {
const auto [items, threads] = cub::detail::scale_mem_bound(
TUNE_THREADS_PER_BLOCK, TUNE_ITEMS_PER_THREAD, sizeof(T));
const auto pass = cub::ReducePassPolicy{
.threads_per_block = threads,
.items_per_thread = items,
.vec_size = 1 << TUNE_ITEMS_PER_VEC_LOAD_POW2,
.reduce_algorithm = cub::BLOCK_REDUCE_WARP_REDUCTIONS,
.load_modifier = cub::LOAD_DEFAULT
};
return { .multi_tile = pass, .single_tile = pass};
}
};
#endif
The custom policy selector returns a fixed policy regardless of the compute capability,
since tuning is usually done on a specific GPU and compiling only for that GPU's compute capability.
The policy selector uses all tuning parameters from the search space to form the policy used by CUB.
+++++++++
Main Body
+++++++++
The :code:`state` passed into the benchmark function allows access to runtime workload axes,
for example the number of elements to process.
*When creating containers for the input avoid to initialize data yourself.
Instead, use the* :code:`gen` *function,
which will fill the input vector with random data on GPU with no compile-time overhead.*
.. code:: c++
const auto elements = static_cast<std::size_t>(state.get_int64("Elements{io}"));
thrust::device_vector<T> in(elements);
thrust::device_vector<T> out(1);
gen(seed_t{}, in);
In addition to the benchmark runtime, NVBench can also report information on the achieved memory bandwidth.
For this, you can optionally provide information on the memory reads and writes of the algorithm to the :code:`state`:
.. code:: c++
state.add_element_count(elements);
state.add_global_memory_reads<T>(elements, "Size");
state.add_global_memory_writes<T>(1);
Finally, the actual benchmark region then calls the public CUB API with an appropriate environment.
Temporary storage allocation is handled automatically by a caching allocator provided through the environment.
The CUDA stream is provided by the :code:`nvbench::launch` parameter of the benchmark lambda.
When tuning (:code:`TUNE_BASE` is not defined),
an instance of the custom policy selector is wrapped by :code:`cuda::execution::tune` and passed to the environment as well.
.. code:: c++
caching_allocator_t alloc;
state.exec(nvbench::exec_tag::gpu | nvbench::exec_tag::no_batch,
[&](nvbench::launch &launch) {
auto env = cub_bench_env(
alloc,
launch
#if !TUNE_BASE
, cuda::execution::tune(policy_selector<T>{})
#endif // !TUNE_BASE
);
cub::DeviceReduce::Reduce(d_in, d_out, elements, op_t{}, init, env);
});
This concludes defining the benchmark function.
Now we need to tell NVBench about it.
++++++++++++++++++
NVBench Attributes
++++++++++++++++++
.. code:: c++
NVBENCH_BENCH_TYPES(algname, NVBENCH_TYPE_AXES(all_types, offset_types))
.set_name("base")
.set_type_axes_names({"T{ct}", "OffsetT{ct}"})
.add_int64_power_of_two_axis("Elements{io}", nvbench::range(16, 28, 4));
:code:`NVBENCH_BENCH_TYPES` registers the benchmark as one with multiple compile-time workloads,
which are defined by the Cartesian product of the type lists in :code:`NVBENCH_TYPE_AXES`.
:code:`set_name(...)` sets the name of the benchmark.
Only alphabetical characters, numbers and underscores are allowed in the benchmark name.
Furthermore, compile-time axes should be suffixed with :code:`{ct}`. The runtime axes might be optionally annotated
as :code:`{io}` which stands for importance-ordered. *This will tell the tuning infrastructure that
the later values on the axis are more important. If the axis is not annotated, each value will be
treated as equally important.*
When you define a type axis annotated with :code:`{ct}`, you should consider optimizing
the build time. Many variants are going to be build, but the search is considering one compile-time
use case at a time. This means that if you have many types to tune for, you'll end up having
many template specializations that you don't need. To avoid this, for each compile time axis, the tuning framework will predefine
a `TUNE_AxisName` macro with the type that's currently being tuned. For instance, if you
have the type axes :code:`T{ct}` and :code:`OffsetT` (as shown above), you can use the following
pattern to narrow down the types you compile for:
.. code:: c++
#ifdef TUNE_T
using all_types = nvbench::type_list<TUNE_T>;
#else
using all_types = nvbench::type_list<char, short, int, long, ...>;
#endif
#ifdef TUNE_OffsetT
using offset_types = nvbench::type_list<TUNE_OffsetT>;
#else
using offset_types = nvbench::type_list<int32_t, int64_t>;
#endif
This logic is already implemented if you use any of the following predefined type lists:
.. list-table:: Predefined type lists
:header-rows: 1
* - Axis name
- C++ identifier
- Included types
* - :code:`T{ct}`
- :code:`integral_types`
- :code:`int8_t, int16_t, int32_t, int64_t`
* - :code:`T{ct}`
- :code:`fundamental_types`
- :code:`integral_types` and :code:`int128_t, float, double`
* - :code:`T{ct}`
- :code:`all_types`
- :code:`fundamental_types` and :code:`complex`
* - :code:`OffsetT{ct}`
- :code:`offset_types`
- :code:`int32_t, int64_t`
You are free to define your own axis names and use the logic above for them (see the sort pairs example).
A single benchmark file can define multiple benchmarks (multiple benchmark functions registered with :code:`NVBENCH_BENCH_TYPES`).
All benchmarks in a single file must share the same compile-time axes.
**The tuning infrastructure will run all benchmarks in a single file together for the same compile-time workload
and compute a common score across all benchmarks and runtime workloads.
Unless a benchmark axis is importance-ordered, each sample contributes equally to the score.**
This is useful to tune an algorithm for multiple runtime use cases at once,
that we don't intend to provide separate tuning policies for.
Also, a large space of runtime workloads can be segmented this way,
e.g. by splitting the benchmark entry point and supplying a few low and a few high values for a runtime axis:
.. code:: c++
NVBENCH_BENCH_TYPES(algname, NVBENCH_TYPE_AXES(all_types, offset_types))
.set_name("small")
...
.add_int64_power_of_two_axis("SegmentSize", nvbench::range(0, 3, 1)); // tests sizes 2^0, 2^1, 2^2, 2^3
NVBENCH_BENCH_TYPES(algname, NVBENCH_TYPE_AXES(all_types, offset_types))
.set_name("large")
...
.add_int64_power_of_two_axis("SegmentSize", nvbench::range(12, 18, 2)); // tests sizes 2^12, 2^14, 2^16, 2^18
Search Process
--------------------------------------------------------------------------------
During the Search Process we are covering all variants for all compile-time workloads to find a variant with a maximum (at least locally) score.
To get started with tuning, you need to configure CMake.
You can use the following command:
.. code:: bash
$ mkdir build
$ cd build
$ cmake .. --preset=cub-tune
You can then run the tuning search for a specific algorithm and compile-time workload. We use a CCCL internal script for that:
.. code:: bash
$ ../benchmarks/scripts/search.py -R '.*merge_sort.*pairs' -a 'KeyT{ct}=I128' -a 'Elements{io}[pow2]=28'
cub.bench.merge_sort.pairs.trp_0.ld_1.ipt_13.tpb_6 0.6805093269929858
cub.bench.merge_sort.pairs.trp_0.ld_1.ipt_11.tpb_10 1.0774560502969677
...
This will search the space of merge sort for key-value pairs, for the key type :code:`int128_t` on :code:`2^28` elements.
The :code:`-R` and :code:`-a` options are optional. **If not specified, all benchmarks are going to be tuned.**
The :code:`-R` option can select multiple benchmarks using a regular expression.
For the axis option :code:`-a`, you can also specify a range of values like :code:`-a 'KeyT{ct}=[I32,I64]'`.
Any axis values not supported by a selected benchmark will be ignored.
The first variant :code:`cub.bench.merge_sort.pairs.trp_0.ld_1.ipt_13.tpb_6` has a score <1 and is thus generally slower than the baseline,
whereas the second variant :code:`cub.bench.merge_sort.pairs.trp_0.ld_1.ipt_11.tpb_10` has a score of >1 and is thus an improvement over the baseline.
.. warning::
Notice there is currently a limitation in :code:`search.py`
which will only execute runs for the first axis value for each axis
(independently of whether the axis is specified on the command line or not).
Tuning for multiple axis values requires multiple runs of :code:`search.py`.
Please see `this issue <https://github.com/NVIDIA/cccl/issues/2267>`_ for more information.
**Benchmarks do not need to be built a priori.** The tuning framework will handle building the benchmarks (base and variants) and running them by itself.
It will keep track of the build time for base and variants.
Sometimes, a tuning variant may lead the compiler to hang or take exceptionally long to compile.
To keep the tuning process going, if the build time of a variant exceeds a threshold, the build is cancelled.
The same applies to benchmarks running for too long.
To get quick feedback on what benchmarks are selected and how big the search space is,
you can add the :code:`-l` option:
.. code:: bash
$ ../benchmarks/scripts/search.py -R '.*merge_sort.*pairs' -a 'KeyT{ct}=I128' -a 'Elements{io}[pow2]=28' -l
ctk: 12.6.85
cccl: v2.7.0
### Benchmarks
* `cub.bench.merge_sort.pairs`: 540 variants:
* `trp`: (0, 2, 1)
* `ld`: (0, 3, 1)
* `ipt`: (7, 25, 1)
* `tpb`: (6, 11, 1)
It will list all selected benchmarks as well as the total number of variants (the magnitude of the search space)
as a result of the Cartesian product of all its tuning parameter spaces.
The tuning infrastructure stores the results in an SQLite database called :code:`cccl_meta_bench.db` in the build directory.
This database persists across tuning runs.
If you interrupt the benchmark script and then launch it again, only missing benchmark variants will be run.
Tuning on multiple GPUs
--------------------------------------------------------------------------------
Because the search process computes scores by comparing the performance of a variant to the baseline,
it has to store the baseline result in the tuning database.
The baseline is specific to the physical GPU on which it was obtained.
Therefore, a single tuning database should not be used to run the tuning search on two different GPUs, even of the same architecture.
Similarly, you should also not interrupt the search and resume it on a different GPU.
Be careful when sharing build directories over network file systems.
Check whether a build directory already contains a :code:`cccl_meta_bench.db` from a previous run before starting a new search.
..
TODO(bgruber): I don't yet understand whether we can tune a single variant on multiple GPUs.
I think this is possible, but would it then create a database per GPU (because 1 baseline per GPU)?
Does search.py do this automatically, or do I need to pass a flag? Or does this only work with our "internal extensions"?
Because the search space can be separated based on different axis values,
a tuning search can be run on multiple GPUs in parallel, even across multiple physical machines (e.g., on a cluster).
To do this, :code:`search.py` is invoked in parallel, one invocation/process per GPU,
with different axis values specified for each invocation.
A dedicated tuning database will be created per physical GPU.
If a shared filesystem is in use, make sure that :code:`search.py` is run from different directories,
so the :code:`cccl_meta_bench.db` files are placed into distinct paths.
It is recommended to drive a multi-GPU/multi-node search process from a script,
iterating the axis values and invoking :code:`search.py` for each variant.
This integrates nicely with workload managers on clusters, which allow submitting batch jobs.
In such a scenario, it is recommended to submit a job per variant.
After tuning on multiple GPUs, the results are available in multiple tuning databases, which can be analyzed together.
Analyzing the results
--------------------------------------------------------------------------------
The result of the search is stored in one or more :code:`cccl_meta_bench.db` files. To analyze the
result you can use the :code:`analyze.py` script.
The :code:`--coverage` flag will show the amount of variants that were covered per compile-time workload:
.. code:: bash
$ ../benchmarks/scripts/analyze.py --coverage
cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I32] coverage: 167 / 522 (31.9923%)
cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I64] coverage: 152 / 522 (29.1188%)
The :code:`--top N` flag will list the best :code:`N` variants for each compile-time workload:
.. code:: bash
$ ../benchmarks/scripts/analyze.py --top=5
cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I32]:
variant score mins means maxs
97 ipt_19.tpb_512 1.141015 1.039052 1.243448 1.679558
84 ipt_18.tpb_512 1.136463 1.030434 1.245825 1.668038
68 ipt_17.tpb_512 1.132696 1.020470 1.250665 1.688889
41 ipt_15.tpb_576 1.124077 1.011560 1.245011 1.722379
52 ipt_16.tpb_512 1.121044 0.995238 1.252378 1.717514
cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I64]:
variant score mins means maxs
71 ipt_19.tpb_512 1.250941 1.155738 1.321665 1.647868
86 ipt_20.tpb_512 1.250840 1.128940 1.308591 1.612382
55 ipt_17.tpb_512 1.244399 1.152033 1.327424 1.692091
98 ipt_21.tpb_448 1.231045 1.152798 1.298332 1.621110
85 ipt_20.tpb_480 1.229382 1.135447 1.294937 1.631225
The name of the variant contains the short parameter names and values used for the variant.
For each variant, a score is reported. The base has a score of 1.0, so each score higher than 1.0 is an improvement over the base.
However, because a single variant contains multiple runtime workloads, also the minimum, mean, maximum score is reported.
If all those three values are larger than 1.0, the variant is strictly better than the base.
If only the mean or max are larger than 1.0, the variant may perform better in most runtime workloads, but regress in others.
This information can be used to change the existing tuning policies in CUB. A detailed explanation of the output is presented
in the following image:
.. image:: ../images/top_results_expl.png
By default, :code:`analyze.py` will look for a file named :code:`cccl_meta_bench.db` in the current directory.
If the tuning results are available in multiple databases, e.g., after tuning on multiple GPUs,
glob expressions matching multiple databases, or just multiple file paths, can be passed as arguments as well:
.. code:: bash
$ ../benchmarks/scripts/analyze.py --top=5 <path-to-databases>/*.db
In case the tuning database(s) store(s) results for several different benchmarks,
the analysis can again be restricted using a regular expression via the :code:`-R` option:
.. code:: bash
$ ../benchmarks/scripts/analyze.py -R=".*radix_sort.keys.*" --top=5 <path-to-databases>/*.db
Variant plots
--------------------------------------------------------------------------------
The reported score for a tuning aggregates the performance across all runtime workloads.
Furthermore, NVBench collects and aggregates multiple samples for a single compile and runtime workload.
So, even though the min, mean and max score are reported for a variant,
it may be necessary to compare the distributions of raw speedups between the baseline and a variant across all runtime workloads and samples.
This is achieved using variant plots.
For more background information on this subject, we refer the reader to `this article <https://aakinshin.net/posts/shift-and-ratio-functions/>`_.
A variant plot can be generated for one or more variants using the :code:`--variants-ratio=` option and specifying the specific variant to plot.
For example:
.. code:: bash
$ ../benchmarks/scripts/analyze.py -R=".*radix_sort.keys.*" --variants-ratio='ipt_18.tpb_288' <path-to-databases>/*.db
May display a matrix of variant plots like:
.. image:: ../images/variant_plot.png
In the image above we see twelve diagrams for the Cartesian product of the :code:`Entropy` (horizontally) and :code:`Elements{io}` (vertically) runtime axes.
The compile-time axes are fixed for one matrix of variant plots.
Across each variant plot's x-axis, the speedup over the baseline (y-axis) is represented.
The baseline is shown as a straight horizontal red line at 1.
The found tuning thus results in a slowdown for :code:`Elements{io}` 2^16 and 2^20 (orange line below red baseline),
but a speedup for 2^24 and 2^28 (orange line above red baseline).
In general, bigger axis values for plots for importance-ordered axes, like :code:`Elements{io}`,
should be prioritized in evaluating a given tuning, because GPUs are optimized for large problem sizes.
However, while the almost 4% slowdown for 2^16 elements at entropy 0.544 may be bearable,
a close to 7% slowdown for 2^20 elements at entropy 1 is probably too large to accept this tuning,
despite the solid 3.5-8% speedup for larger element counts.
The shown ratios are generated by fitting an equal amount of quantiles into the samples of the baseline and the variant,
and then showing the quotient for each corresponding quantile from baseline and variant.
For background information on the quantile-respectful density estimation,
we refer the reader to this `article <https://aakinshin.net/posts/qrde-hd>`_.
By default, a quantile corresponds to a percentile, and thus a ratio plot contains 100 data points
expressing the speedup of the slowest 1% in the variant over the slowest 1% in the baseline (left),
then the second slowest 1%, etc., until the speedup of the fastest 1% in the variant over the fastest 1% in the baseline (right).
The detailed analysis via variant plots is needed,
because a single aggregated score cannot represent the distribution of samples obtained from highly concurrent algorithms, such as those in CUB.
Even though NVBench reruns a benchmark many times to gain statistical confidence in the result,
the runtime of a CUB algorithm does not necessarily follow a normal distribution.
For example, the concurrent nature of some algorithms may result in bimodal or even more complex distributions,
as a consequence of how the hardware schedules and executes threads.
Also, the kind of distribution may be different between baseline and variant.
For all these reasons, comparing the distribution of samples is the only reliable way to determine,
whether a tuning provides a consistent speedup for all runtime workloads.
Creating and extending tuning policies
--------------------------------------------------------------------------------
Once a suitable tuning result has been selected, we have to translate it into C++ code that will be picked up by CUB.
The tuning variant name shown by :code:`analyze.py` gives us all the information on the selected tuning values.
Here is an example:
.. code:: bash
$ ../benchmarks/scripts/analyze.py --top=1
cub.bench.radix_sort.keys[T{ct}=I8, OffsetT{ct}=I64]:
variant score mins means maxs
71 ipt_19.tpb_512 1.250941 1.155738 1.321665 1.647868
Assume we have determined this tuning to be the best one for sorting I8 keys using radix_sort using I64 offsets.
The ``variant`` can be decoded using the ``// %RANGE%`` comments in the C++ source code of the benchmark,
since the names of the reported parameters in the variant are derived from these:
.. code:: c++
// %RANGE% TUNE_ITEMS_PER_THREAD ipt 7:24:1
// %RANGE% TUNE_THREADS_PER_BLOCK tpb 128:1024:32
The variant ``ipt_19.tpb_512``, which stands for 19 items per thread (``ipt``) and 512 threads per block (``tpb``),
was thus compiled with ``-DTUNE_ITEMS_PER_THREAD=19 -DTUNE_THREADS_PER_BLOCK=512``.
The meaning of these values is specific to the benchmark definition,
and we have to check the benchmark's source code for how they are applied.
Equally named tuning parameters may not translate to different benchmarks (please double check).
As a user of CUB, such a new set of tuning parameters (i.e. a variant) can then be used to define a policy selector,
which is passed to the public CUB API through the environment,
as :ref:`sketched above <cub-tuning-infra-authoring-benchmarks>`:
.. code:: c++
struct policy_selector {
_CCCL_HOST_DEVICE_API constexpr auto operator()(cuda::compute_capability /*cc*/) const -> cub::AlgorithmPolicy {
return {
.threads_per_block = 512,
.items_per_thread = 19,
...
};
}
};
The default tunings defined inside CUB's source use the same infrastructure
but should only be changed and extended by the CCCL maintainers.
All default tunings are found in the :code:`cub/device/dispatch/tuning/tuning_*.cuh` headers, organized by algorithm.
CUB's policy selectors are highly parameterized on type information and traits of the input arguments to CUB algorithms
(like accumulator type, offset size, and operation kind),
which they turn into a policy for a given compute capability.
The way tuning values are selected is different for each CUB algorithm and requires studying the corresponding code.
The general principles of policy selectors and tunings are documented :ref:`here <cub-policy-selectors>`.
For example, signed and unsigned integers of the same size are often represented by the same tuning.
In general, variants for which the algorithmic behavior is expected to be the same
(same arithmetic intensity, no special instructions for one of the data types, same amount of bytes to load/store, etc.)
are covered by the same tuning.
When a better variant has been found and CUB already has a tuning for this variant,
the tuning parameter values can simply be updated in the corresponding CUB tuning header.
This is usually the case when a CUB algorithm has been reengineered and shows different performance characteristics,
or more tuning parameters are exposed (e.g., a new load algorithm is available).
For example, an existing tuning selection function may contain code like:
.. code:: c++
constexpr auto get_sm90_tuning(type_t accum_t, op_kind_t op, int offset_size, int accum_size) {
if (op == op_kind_t::plus && offset_size == 4 && accum_size == 4)
return { .threads_per_block = 256, .items_per_thread = 14 }; // tuning variant proposes: 512 and 19
...
}
Since we have found that 512 threads per block and 19 items per thread are better, we can update the value in place.
A different case is when we tune beyond what's currently supported by CUB's existing tunings.
This may be because we tune for a new GPU architecture,
in which case a new branch based on the :code:`cuda::compute_capability` passed to the :code:`policy_selector::operator()`
should be introduced, handling this new GPU's compute capability.
Or we tune for new key, value or offset types, etc.,
in which case the existing tuning functions may need additional branches.
There is no general rule on how this extension is done, though.
The implementation may be different for each CUB algorithm.
In the seldom case, that no variant outperforms the baseline,
it must be ensured that any newly added logic correctly falls back to the old tuning values.
There is again no general rule on how this is implemented.
Verification
--------------------------------------------------------------------------------
Once we have selected tunings and implemented them in CUB, we need to verify them.
This process consists of two steps.
Firstly, we need to ensure that adding new tunings and policies did not break existing tunings.
This is most relevant when tunings for new compute capabilities have been added.
To verify this, compile the corresponding benchmarks for the previous compute capabilities
(excluding the new tunings) before and after modifying any tunings,
and compare the generated SASS :code:(`cuobjdump -sass`).
It should not have changed.
Secondly, we must benchmark and compare the performance of the tuned algorithm before and after the tunings have been applied.
This extra step is needed, because the score shown during the tuning analysis is just an aggregated result.
Individual benchmarks may still have regressed for some compile-time workloads.
Fortunately, this is no different than :ref:`running <cub-benchmarking-running>` the corresponding CUB benchmark with and without the changes,
and :ref:`comparing <cub-benchmarking-comparing>` the resulting JSON files.
Such a diff should be supplied to any request to change CUB tunings.
If verification fails for some compile-time workloads (there are regressions), there are two options:
1. Discard the tuning entirely and ensure the tuning selection falls back to the baseline tuning.
2. Narrow down the tuning template specialization to only apply to the workloads where it improves performance,
and fallback where it regressed.
The latter is more complex and may not be justified, if the improvements are small or the use case too narrow.
Use your judgement. Good luck!

22
cccl_upstream/docs/cub/warp_wide.rst Normal file
View File

@@ -0,0 +1,22 @@
.. _warp-module:
Warp-Wide "Collective" Primitives
==================================================
.. toctree::
:glob:
:hidden:
:maxdepth: 2
api/warp
CUB warp-level algorithms are specialized for execution by threads in the same CUDA warp.
These algorithms may only be invoked by ``1 <= n <= 32`` *consecutive* threads in the same warp:
* :cpp:struct:`cub::WarpExchange` rearranges data partitioned across a CUDA warp
* :cpp:class:`cub::WarpLoad` loads a linear segment of items from memory into a CUDA warp
* :cpp:class:`cub::WarpMergeSort` sorts items partitioned across a CUDA warp
* :cpp:struct:`cub::WarpReduce` computes reduction of items partitioned across a CUDA warp
* :cpp:struct:`cub::WarpReduceBatched` computes reduction of multiple batches of items partitioned across a CUDA warp
* :cpp:struct:`cub::WarpScan` computes a prefix scan of items partitioned across a CUDA warp
* :cpp:class:`cub::WarpStore` stores items partitioned across a CUDA warp to a linear segment of memory

185
cccl_upstream/docs/cudax/Doxyfile Normal file
View File

@@ -0,0 +1,185 @@
# Doxyfile for CUDAX
PROJECT_NAME = CUDAX
OUTPUT_DIRECTORY = ../_build/doxygen/cudax
CREATE_SUBDIRS = NO
GENERATE_HTML = NO
GENERATE_LATEX = NO
GENERATE_XML = YES
XML_OUTPUT = xml
XML_PROGRAMLISTING = YES
# Input directory - specific paths for cudax
INPUT = ../../cudax/include/cuda/experimental/__copy_bytes \
../../cudax/include/cuda/experimental/__fill_bytes \
../../cudax/include/cuda/experimental/__container \
../../cudax/include/cuda/experimental/__device \
../../cudax/include/cuda/experimental/graph.cuh \
../../cudax/include/cuda/experimental/__graph \
../../cudax/include/cuda/experimental/__group \
../../cudax/include/cuda/experimental/__launch \
../../cudax/include/cuda/experimental/__places \
../../cudax/include/cuda/experimental/__places/exec \
../../cudax/include/cuda/experimental/__stream \
../../cudax/include/cuda/experimental/stf.cuh \
../../cudax/include/cuda/experimental/__stf \
../../cudax/include/cuda/experimental/__stf/internal \
../../cudax/include/cuda/experimental/__stf/utility \
../../cudax/include/cuda/experimental/__stf/localization \
../../cudax/include/cuda/experimental/__stf/allocators \
../../cudax/include/cuda/experimental/__stf/graph \
../../cudax/include/cuda/experimental/__stf/graph/internal \
../../cudax/include/cuda/experimental/__stf/graph/interfaces \
../../cudax/include/cuda/experimental/__stf/places \
../../cudax/include/cuda/experimental/__stf/places/exec/host \
../../cudax/include/cuda/experimental/__stf/stream \
../../cudax/include/cuda/experimental/__stf/stream/internal \
../../cudax/include/cuda/experimental/__stf/stream/interfaces
RECURSIVE = YES
# Exclude patterns from repo.toml
EXCLUDE = ../../cudax/include/cuda/experimental/__graph/fwd.cuh
EXCLUDE_PATTERNS = */detail/* */test/* */tests/* */examples/* */__detail/*
# Exclude symbols from repo.toml
EXCLUDE_SYMBOLS = *detail* *RESERVED* *reserved* *__* _A* _B* _C* _D* _E* _F* _G* _H* _I* _J* _K* _L* _M* _N* _O* _P* _Q* _R* _S* _T* _U* _V* _W* _X* _Y* _Z* UNITTEST
# Path for @snippet references to test/example files
EXAMPLE_PATH = ../../cudax/include/cuda/experimental/__stf/utility \
../../cudax/test/multi_gpu/algorithms \
../../cudax/test/coop/reduce
FILE_PATTERNS = *.h *.hpp *.cuh
EXTENSION_MAPPING = cuh=C++ cu=C++
# Documentation extraction settings
EXTRACT_ALL = YES
EXTRACT_PRIVATE = NO
EXTRACT_STATIC = YES
EXTRACT_LOCAL_CLASSES = YES
HIDE_UNDOC_MEMBERS = NO
HIDE_UNDOC_CLASSES = NO
SHOW_INCLUDE_FILES = YES
INLINE_INHERITED_MEMB = YES
FULL_PATH_NAMES = YES
STRIP_FROM_PATH = ../../cudax
SHORT_NAMES = NO
# Parsing settings
JAVADOC_AUTOBRIEF = YES
QT_AUTOBRIEF = NO
MULTILINE_CPP_IS_BRIEF = NO
INHERIT_DOCS = YES
SEPARATE_MEMBER_PAGES = NO
TAB_SIZE = 4
BUILTIN_STL_SUPPORT = YES
# Preprocessing
ENABLE_PREPROCESSING = YES
MACRO_EXPANSION = YES
EXPAND_ONLY_PREDEF = NO
SEARCH_INCLUDES = YES
SKIP_FUNCTION_MACROS = YES
# Predefined macros for CUDAX
PREDEFINED = \
"__device__=" \
"__host__=" \
"__global__=" \
"__forceinline__=" \
"__declspec(x)=" \
"__align__(x)=" \
"__cccl_lib_mdspan" \
"_CCCL_AND=&&" \
"_CCCL_CONCEPT=constexpr bool" \
"_CCCL_DOXYGEN_INVOKED" \
"_CCCL_HOST_DEVICE=" \
"_CCCL_DEVICE=" \
"_CCCL_HOST=" \
"_CCCL_FORCEINLINE=" \
"_CCCL_CONSTEXPR_CXX20=constexpr" \
"_CCCL_CONSTEXPR_CXX23=constexpr" \
"_CCCL_API=inline" \
"_CCCL_HOST_DEVICE_API=inline" \
"_CCCL_HOST_API=inline" \
"_CCCL_DEVICE_API=inline" \
"_CCCL_NODEBUG_API=inline" \
"_CCCL_NODEBUG_DEVICE_API=inline" \
"_CCCL_NODEBUG_HOST_API=inline" \
"_CCCL_TRIVIAL_API=inline" \
"_CCCL_TRIVIAL_HOST_API=inline" \
"_CCCL_TRIVIAL_DEVICE_API=inline" \
"_CCCL_PUBLIC_API=inline" \
"_CCCL_PUBLIC_DEVICE_API=inline" \
"_CCCL_PUBLIC_HOST_API=inline" \
"_CCCL_BEGIN_NAMESPACE_CUDA_STD=namespace cuda::std {" \
"_CCCL_END_NAMESPACE_CUDA_STD=}" \
"_CCCL_CATCH=catch" \
"_CCCL_CATCH_ALL=catch (...)" \
"_CCCL_CATCH_FALLTHROUGH=" \
"_CCCL_CONSTEXPR_FRIEND=friend " \
"_CCCL_CTK_AT_LEAST(x, y)=1" \
"_CCCL_CTK_BELOW(x, y)=0" \
"_CCCL_CUDACC_AT_LEAST(x, y)=1" \
"_CCCL_CUDACC_BELOW(x, y)=0" \
"_CCCL_DIAG_POP=" \
"_CCCL_DIAG_PUSH=" \
"_CCCL_DIAG_SUPPRESS_CLANG(x)=" \
"_CCCL_DIAG_SUPPRESS_GCC(x)=" \
"_CCCL_DIAG_SUPPRESS_MSVC(x)=" \
"_CCCL_DIAG_SUPPRESS_NVHPC(x)=" \
"_CCCL_EXEC_CHECK_DISABLE=" \
"_CCCL_GLOBAL_CONSTANT=inline constexpr" \
"_CCCL_HAS_CTK()=1" \
"_CCCL_HIDE_FROM_ABI=" \
"_CCCL_REQUIRES(x)= ::cuda::std::enable_if_t<x, int> = 0>" \
"_CCCL_STD_VER=2020" \
"_CCCL_SUPPRESS_DEPRECATED_POP=" \
"_CCCL_SUPPRESS_DEPRECATED_PUSH=" \
"_CCCL_TEMPLATE(x)=template<x, " \
"_CCCL_TRAILING_REQUIRES(x)=-> x requires " \
"_CCCL_TRY=try" \
"_CCCL_TYPE_VISIBILITY_DEFAULT=" \
"_CCCL_TYPE_VISIBILITY_HIDDEN=" \
"_CCCL_VISIBILITY_DEFAULT=" \
"_CCCL_VISIBILITY_HIDDEN=" \
"_CCCL_LIFETIMEBOUND=" \
"_CUDAX_CONSTEXPR_FRIEND=friend" \
"_LIBCUDACXX_HAS_SPACESHIP_OPERATOR()=1" \
"CCCL_DEPRECATED=" \
"CCCL_DEPRECATED_BECAUSE(x)=" \
"CCCL_IGNORE_DEPRECATED_CPP_DIALECT" \
"CUB_DISABLE_NAMESPACE_MAGIC" \
"CUB_IGNORE_NAMESPACE_MAGIC_ERROR" \
"CUB_NAMESPACE_BEGIN=namespace cub {" \
"CUB_NAMESPACE_END=}" \
"CUB_RUNTIME_FUNCTION=" \
"THRUST_FWD(x)=x" \
"THRUST_NAMESPACE_BEGIN=namespace thrust {" \
"THRUST_NAMESPACE_END=}" \
"THRUST_PREVENT_MACRO_SUBSTITUTION" \
"_CCCL_HOSTED()=1"
# Additional settings from repo.toml
DISTRIBUTE_GROUP_DOC = YES
DOXYFILE_ENCODING = UTF-8
INPUT_ENCODING = UTF-8
EXAMPLE_RECURSIVE = YES
EXAMPLE_PATTERNS = *.cu
AUTOLINK_SUPPORT = YES
# IMPORTANT: Aliases for custom commands
# The rst alias enables embedding reStructuredText in doxygen comments
# Using the same format as repo_docs for compatibility
ALIASES = "rst=\verbatim embed:rst:leading-asterisk"
ALIASES += "endrst=\endverbatim"
ALIASES += "rststar=\verbatim embed:rst:leading-asterisk"
ALIASES += "inlinerst=\verbatim embed:rst:inline"
# Quiet mode
QUIET = YES
WARNINGS = YES
WARN_AS_ERROR = FAIL_ON_WARNINGS
WARN_IF_UNDOCUMENTED = NO
WARN_IF_DOC_ERROR = YES
WARN_IF_INCOMPLETE_DOC = NO
WARN_NO_PARAMDOC = NO

27
cccl_upstream/docs/cudax/container.rst Normal file
View File

@@ -0,0 +1,27 @@
.. _cudax-containers:
Containers library
===================
.. toctree::
:glob:
:maxdepth: 1
api/class*uninitialized__buffer*
The headers of the container library provide facilities to store elements on the heap. They are heavily inspired by the
C++ `containers library <https://en.cppreference.com/w/cpp/container>`__ but deviate from the standard provided ones due to different requirements from
heterogeneous systems.
They build upon :ref:`memory_resources <libcudacxx-extended-api-memory-resources>` to ensure that e.g. execution space
annotations are checked by the type system.
.. _libcudacxx-containers-uninitialized-async-buffer:
.. list-table::
:widths: 25 45 30
:header-rows: 0
* - :ref:`<cuda/experimental/container.cuh> <cudax-containers-uninitialized-buffer>`
- Facilities providing uninitialized *heterogeneous* potentially stream ordered storage satisfying a set of properties
- cudax 2.7.0 / CCCL 2.7.0

52
cccl_upstream/docs/cudax/graph.rst Normal file
View File

@@ -0,0 +1,52 @@
.. _cudax-graph:
Graphs library
==============
.. toctree::
:glob:
:maxdepth: 1
api/struct*graph*
The headers of the graph library provide facilities to create and manage CUDA graphs.
This library is under construction and not yet ready for production use.
.. list-table::
:widths: 25 45 30
:header-rows: 0
* - :ref:`<cuda/experimental/graph.cuh> <cudax-graph-graph-builder>`
- Class `cuda::experimental::graph_builder`: An owning wrapper for a `cudaGraph_t` object.
- cudax 2.9.0 / CCCL 2.9.0
* - :ref:`<cuda/experimental/graph.cuh> <cudax-graph-graph-node-ref>`
- Class `cuda::experimental::graph_node_ref`: A non-owning wrapper for a `cudaGraphNode_t` object.
- cudax 2.9.0 / CCCL 2.9.0
* - :ref:`<cuda/experimental/graph.cuh> <cudax-graph-graph>`
- Class `cuda::experimental::graph`: An owning wrapper for a `cudaGraphExec_t` object.
- cudax 2.9.0 / CCCL 2.9.0
.. _cudax-graph-graph-builder:
graph_builder
-------------
``cuda::experimental::graph_builder`` is an owning wrapper for a ``cudaGraph_t`` object.
See the API reference entries in this section for details.
.. _cudax-graph-graph-node-ref:
graph_node_ref
--------------
``cuda::experimental::graph_node_ref`` is a non-owning wrapper for a ``cudaGraphNode_t`` object.
See the API reference entries in this section for details.
.. _cudax-graph-graph:
graph
-----
``cuda::experimental::graph`` is an owning wrapper for a ``cudaGraphExec_t`` object.
See the API reference entries in this section for details.

39
cccl_upstream/docs/cudax/index.rst Normal file
View File

@@ -0,0 +1,39 @@
.. _cudax-module:
CUDA Experimental
=================
.. toctree::
:hidden:
:maxdepth: 1
Overview <self>
container
graph
places
stf
API reference <api/index>
``CUDA Experimental`` (``cudax``) provides experimental new features that are still in development and subject to change.
However, any feature within this library has important use cases and we encourage users to experiment with them.
Specifically, ``cudax`` provides:
- :ref:`asynchronous host from/to device byte-wise mdspan copy <cudax-copy-bytes>`
- :ref:`mdspan byte fill <cudax-fill-bytes>`
- :ref:`uninitialized storage <libcudacxx-containers-uninitialized-async-buffer>`
- :ref:`graph functionality <cudax-graph>`
- dimensions description functionality
- :ref:`places <cudax-places>` for managing execution and data affinity across devices
- :ref:`an implementation of the STF (Sequential Task Flow) programming model <stf>`
Stability Guarantees
---------------------
There are no stability guarantees whatsoever. We reserve the right to change both the ABI and the API of any feature
within ``cudax`` at any time without notice.
Availability
-------------
Due to its experimental nature and the lack of stability guarantees, ``cudax`` is not shipped with the CUDA toolkit but
is solely available through GitHub.

778
cccl_upstream/docs/cudax/places.rst Normal file
View File

@@ -0,0 +1,778 @@
.. _cudax-places:
Places
======
.. contents::
:depth: 2
Places are abstractions that represent where code executes and where data
resides across the non-uniform memory of a CUDA system. They provide a
unified interface for managing execution affinity, stream pools, memory
allocation, and device context switching -- independently of any task-based
programming model.
Places come in two flavors:
- **Execution places** (``exec_place``) determine where code is executed.
- **Data places** (``data_place``) specify where data is located in memory.
The places API is part of the ``cuda::experimental::places`` C++ namespace
and can be used standalone via the ``cuda/experimental/places.cuh`` header,
without pulling in the full CUDASTF task-graph framework. For backward
compatibility, all places types are also available in the
``cuda::experimental::stf`` namespace.
.. _places-execution-places:
Execution places
----------------
An *execution place* describes a location where computation can occur.
The following factory methods create the most common execution places:
- ``exec_place::device(id)`` -- a specific CUDA device
- ``exec_place::host()`` -- the host CPU
- ``exec_place::current_device()`` -- the CUDA device that is currently active
- ``exec_place::cuda_context(ctx, devid)`` -- an externally-owned CUDA driver
context; the device ordinal is derived from the context when ``devid`` is
omitted
When an execution place is activated, it sets the appropriate CUDA context
(e.g. calls ``cudaSetDevice``). Each execution place also has an *affine*
data place: the memory location naturally associated with it. For a device
execution place the affine data place is the device's global memory; for
the host it is pinned host memory (RAM).
A CUDA-context execution place is non-owning. The caller must keep the
``CUcontext`` alive while the place and any streams obtained from it are in use.
.. _places-data-places:
Data places
-----------
A *data place* describes a memory location where data can reside. The
following factory methods are available:
- ``data_place::device(id)`` -- global memory of a specific CUDA device
- ``data_place::host()`` -- pinned host memory
- ``data_place::managed()`` -- CUDA managed (unified) memory
- ``data_place::affine()`` -- the data place naturally associated with the
current execution place
The *affine* data place is the default: when no data place is specified,
data is placed in the memory that is local to the execution place. For
example, a task running on device 0 will access data in device 0's global
memory by default.
Non-affine placement is also supported: data can be placed on a different
device or in host memory regardless of where the computation runs. This is
useful for sparse accesses (leveraging CUDA Unified Memory page faulting)
or for addressing memory capacity constraints. Non-affine placement assumes
the hardware and OS support such accesses (NVLINK, UVM, etc.).
.. _places-container-keys:
Places as container keys
------------------------
Both ``exec_place`` and ``data_place`` can be used as keys in standard
associative containers. The library provides the required comparison and
hash support:
- **``std::map``** and **``std::set``** use ``operator<`` (strict weak
ordering) for keys. Both place types implement ``operator<``, so they
can be used as ordered map or set keys.
- **``std::unordered_map``** and **``std::unordered_set``** require a
hash function and equality. The library specializes ``cuda::experimental::stf::hash``
for both place types (accessible from both the ``stf`` and ``places`` namespaces),
and both implement ``operator==``.
This allows, for example, maintaining per-place handles (e.g. CUBLAS or
CUSOLVER handles keyed by ``exec_place``) or per-place caches keyed by
``data_place``, using either ordered or hash-based containers as needed.
The following snippet shows lazy creation of a CUBLAS handle per execution
place using an ``std::unordered_map`` keyed by ``exec_place``:
.. code:: c++
#include <cuda/experimental/places.cuh>
#include <cublas_v2.h>
using namespace cuda::experimental::places;
cublasHandle_t& get_cublas_handle(const exec_place& ep = exec_place::current_device())
{
static std::unordered_map<exec_place, cublasHandle_t, hash<exec_place>> handles;
auto& h = handles[ep];
if (h == cublasHandle_t{})
{
exec_place_scope scope(ep);
cuda_safe_call(cublasCreate(&h));
}
return h;
}
.. _places-activate:
Setting the current device or context
--------------------------------------
The ``exec_place::activate()`` method provides a generic alternative to
``cudaSetDevice()`` that works uniformly across different execution place types.
This is useful when you want to set the current CUDA device or context without
using tasks.
The method returns an ``exec_place`` representing the previous state, which can
be used to restore the original device or context.
**Behavior by execution place type:**
- **Device places** (``exec_place::device(id)``): Calls ``cudaSetDevice(id)``
- **Green context places**: Sets the current CUDA driver context via ``cuCtxSetCurrent()``
- **Host places**: No-op
**Basic usage with devices:**
.. code:: cpp
exec_place place = exec_place::device(1);
exec_place prev = place.activate(); // Switch to device 1
// ... perform operations on device 1 ...
place.deactivate(prev); // Restore previous device
**Alternative restoration pattern:**
You can also restore by calling ``activate()`` on the returned place:
.. code:: cpp
exec_place place = exec_place::device(1);
exec_place prev = place.activate();
// ... work on device 1 ...
prev.activate(); // Equivalent to place.deactivate(prev)
**Usage with green contexts (CUDA 12.4+):**
Green contexts provide SM-level partitioning of GPU resources. The
``activate()``/``deactivate()`` methods handle the underlying driver context
management:
.. code:: cpp
// Create green contexts with 8 SMs each
green_context_helper gc(8, device_id);
auto view = gc.get_view(0);
exec_place gc_place = exec_place::green_ctx(view);
exec_place prev = gc_place.activate(); // Sets green context as current
// ... GPU work runs with SM affinity ...
gc_place.deactivate(prev); // Restore original context
**RAII scope for scoped activation:**
For exception-safe code or when you want automatic restoration, use the
``exec_place_scope`` RAII helper:
.. code:: cpp
{
exec_place_scope scope(exec_place::device(1));
// Device 1 is now active
// ... perform operations on device 1 ...
}
// Previous device is automatically restored when scope goes out of scope
The guard automatically restores the previous execution place when it goes out
of scope, making it useful for exception-safe code.
.. _places-stream-management:
Stream management with execution places
----------------------------------------
Execution places can be used independently of any task system to manage CUDA
streams in a structured way. This is useful when you want to use place
abstractions (devices, green contexts) for stream management without the full
task-based programming model.
Stream pools for pooled places (``device(N)``, ``host()``) live in an
``exec_place_resources`` registry that the caller owns. Pass the registry to
``exec_place::pick_stream`` to get a CUDA stream; the per-place pool inside the
registry is created lazily on first request and is destroyed when the registry
is destroyed.
The method accepts an optional ``for_computation`` hint (defaults to ``true``)
that may select between computation and data transfer stream pools to improve
overlapping. This is purely a performance hint, and it does not affect
correctness. Not all execution places enforce it.
.. code:: cpp
#include <cuda/experimental/places.cuh>
using namespace cuda::experimental::places;
// Standalone use: own the registry yourself.
exec_place_resources resources;
// Get a stream from the current device
exec_place place = exec_place::current_device();
cudaStream_t stream = place.pick_stream(resources);
// Use the stream for CUDA operations
myKernel<<<grid, block, 0, stream>>>(d_data);
// Get streams from specific devices (sharing the same registry)
cudaStream_t stream_dev0 = exec_place::device(0).pick_stream(resources);
cudaStream_t stream_dev1 = exec_place::device(1).pick_stream(resources);
Inside a CUDASTF context, the context's ``async_resources_handle`` already
holds an ``exec_place_resources`` registry. Convenience overloads accept the
handle directly so call sites do not have to dereference it:
.. code:: cpp
cudaStream_t stream = place.pick_stream(ctx.async_resources());
Stream pools are populated lazily -- CUDA streams are only created when first
requested via ``pick_stream(resources)`` (or ``pick_stream(ctx.async_resources())``
inside CUDASTF). Self-contained places (``exec_place::cuda_stream(s)``,
green-context places) ignore the registry and return their own embedded pool
instead, so the user-provided ``cudaStream_t`` / ``CUgreenCtx`` must outlive
any place that wraps it.
.. _places-memory-allocation:
Memory allocation with data places
------------------------------------
Data places provide a unified interface for memory allocation that works across
different memory types (host, device, managed) and place extensions (green
contexts, user-defined places). This allows you to allocate memory while
benefiting from the place abstraction.
The ``data_place::allocate()`` and ``data_place::deallocate()`` methods provide
raw memory allocation. The stream parameter defaults to ``nullptr``, which is
convenient for non-stream-ordered allocations (host, managed) where the stream
is ignored:
.. code:: cpp
#include <cuda/experimental/places.cuh>
using namespace cuda::experimental::places;
// Allocate on host (pinned memory) - stream defaults to nullptr
void* host_ptr = data_place::host().allocate(1024);
// ... use host_ptr ...
data_place::host().deallocate(host_ptr, 1024);
// Allocate on a specific device (stream-ordered)
cudaStream_t stream;
cudaStreamCreate(&stream);
void* dev_ptr = data_place::device(0).allocate(1024, stream);
// ... use dev_ptr with stream ...
data_place::device(0).deallocate(dev_ptr, 1024, stream);
cudaStreamDestroy(stream);
// Allocate managed memory - stream defaults to nullptr
void* managed_ptr = data_place::managed().allocate(1024);
// ... use managed_ptr from host or device ...
data_place::managed().deallocate(managed_ptr, 1024);
**Stream-ordered vs immediate allocations:**
Different data places have different allocation behaviors:
- **Host** (``data_place::host()``): Uses ``cudaMallocHost()`` / ``cudaFreeHost()`` - immediate, stream parameter is ignored
- **Managed** (``data_place::managed()``): Uses ``cudaMallocManaged()`` / ``cudaFree()`` - immediate, stream parameter is ignored (note: ``cudaFree`` may introduce implicit synchronization)
- **Device** (``data_place::device(id)``): Uses ``cudaMallocAsync()`` / ``cudaFreeAsync()`` - stream-ordered
- **Extensions** (green contexts, etc.): Behavior depends on the extension implementation
You can query whether a place uses stream-ordered allocation with
``allocation_is_stream_ordered()``:
.. code:: cpp
data_place place = data_place::device(0);
if (place.allocation_is_stream_ordered()) {
// Allocation is stream-ordered - synchronize via the stream
void* ptr = place.allocate(size, stream);
myKernel<<<grid, block, 0, stream>>>(ptr);
place.deallocate(ptr, size, stream);
cudaStreamSynchronize(stream);
} else {
// Allocation is immediate - stream is ignored, safe to use right away
void* ptr = place.allocate(size);
// ... use ptr ...
place.deallocate(ptr, size);
}
This abstraction is particularly useful when writing generic code that needs to
work with different types of places, including custom place extensions.
Geometry-aware allocation with allocate_nd
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Some places need to know the shape of the tensor being allocated, not just its
size: a composite data place distributes the allocation according to a
partitioner that maps *element coordinates* to places. ``allocate_nd()`` takes
the tensor extents (dimension 0 varying fastest) and the element size:
.. code:: cpp
// 2-D tensor of nx x ny doubles, distributed by the place's partitioner
void* ptr = place.allocate_nd(dim4(nx, ny), sizeof(double));
// ...
place.deallocate(ptr, nx * ny * sizeof(double));
For most places this is equivalent to ``allocate(prod(dims) * elemsize)``. For
composite places it is required: the byte-count ``allocate()`` throws there,
since a byte count alone cannot carry the geometry the partitioner needs. A
caller that genuinely has untyped bytes states that explicitly with
``allocate_nd(dim4(nbytes), 1)``, which distributes the buffer with byte
granularity. This raw-byte form applies to composite places built from
scale-free partitioners only; a composite place backed by a structured
partition (see :ref:`places-structured-partitions`) accepts exactly the
extents of the tensor the partition was built for and rejects anything else,
including a flat byte count.
.. _places-vmm:
VMM-based allocation with mem_create
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For advanced use cases involving CUDA's Virtual Memory Management (VMM) API,
``data_place`` also provides the ``mem_create()`` method. This is a lower-level
interface used internally by localized arrays (``composite_slice``) to create
physical memory segments that are then mapped into a contiguous virtual address
space.
Unlike ``allocate()``, which returns a usable pointer directly, ``mem_create()``
returns a ``CUmemGenericAllocationHandle`` that must be subsequently mapped with
``cuMemMap()`` before use:
.. code:: cpp
#include <cuda/experimental/places.cuh>
using namespace cuda::experimental::places;
// Create a physical memory handle for device 0
CUmemGenericAllocationHandle handle;
data_place::device(0).mem_create(&handle, size);
// The handle must be mapped to a virtual address before use
// (see CUDA VMM documentation for cuMemMap, cuMemSetAccess, etc.)
**When to use each method:**
- Use ``allocate()`` for most cases - it provides ready-to-use memory with
stream-ordered semantics where applicable.
- Use ``mem_create()`` only when you need explicit control over virtual memory
mapping, such as creating localized arrays that span multiple devices with a
unified virtual address space.
**Limitations of mem_create:**
- Only supports device memory and host memory (pinned)
- Managed memory is **not supported** by the VMM API
- The returned handle requires additional VMM API calls to be usable
Custom place extensions can override ``mem_create()`` to provide specialized
VMM allocation behavior (e.g., memory localization for hardware partitions).
.. _places-grid:
Grid of places
--------------
It is possible to manipulate places which are a collection of multiple places.
In particular, it is possible to define an execution place which corresponds
to multiple device execution places.
Creating grids of places
^^^^^^^^^^^^^^^^^^^^^^^^
A grid of execution places is an ``exec_place`` that contains multiple
underlying places. Grids are created with the ``make_grid`` free function:
.. code:: c++
// Create a 1D grid from a vector of places
exec_place grid = make_grid(std::vector<exec_place>{
exec_place::device(0), exec_place::device(1)
});
The ``exec_place::all_devices()`` helper creates a grid of all available
CUDA devices:
.. code:: c++
exec_place all = exec_place::all_devices();
Similarly, ``exec_place::n_devices(n)`` creates a grid from the first ``n``
devices:
.. code:: c++
exec_place first_four = exec_place::n_devices(4);
It is possible to retrieve the total number of elements in a grid using
the ``size()`` method, and individual places with ``get_place(i)``:
.. code:: c++
exec_place grid = exec_place::all_devices();
for (size_t i = 0; i < grid.size(); i++) {
exec_place dev = grid.get_place(i);
// ...
}
Shaped grids
^^^^^^^^^^^^
Grids of places need not be 1D arrays. They can be structured as a
multi-dimensional grid described with a ``dim4`` class by passing it to
``make_grid`` or ``n_devices``:
.. code:: c++
// Create a shaped grid: 8 devices arranged as a 2x2x2 cube
exec_place cube = exec_place::n_devices(8, dim4(2, 2, 2));
// Or from an explicit vector
exec_place shaped = make_grid(my_places, dim4(4, 2));
Note that the total size of the ``dim4`` must match the number of places.
It is possible to query the *shape* of the grid using ``get_dims()``,
which returns a ``dim4`` object. Individual places can be accessed by
multi-dimensional position using ``get_place(pos4)``.
Reshaping and collapsing grid axes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
An existing grid can be viewed with different dimensions using
``reshape()``. The new dimensions must contain exactly the same number of
places:
.. code:: c++
exec_place cube = make_grid(my_places, dim4(2, 3, 4));
exec_place flat = cube.reshape(dim4(24));
Reshaping changes only the grid coordinate system. It preserves dimension-0-
fastest linear order, so ``flat.get_place(i) == cube.get_place(i)`` for every
linear index ``i``. It does not reorder, replicate, or remove places.
``collapse_axes(first, last)`` is a convenience operation that combines a
contiguous inclusive range of axes. The collapsed extent is the product of
the selected extents; later axes shift left and trailing extents become one:
.. code:: c++
exec_place grid = make_grid(my_places, dim4(2, 3, 4));
exec_place grid_6x4 = grid.collapse_axes(0, 1); // dim4(6, 4)
exec_place grid_2x12 = grid.collapse_axes(1, 2); // dim4(2, 12)
exec_place grid_24 = grid.collapse_axes(0, 3); // dim4(24)
These operations are useful when a partition should consume several axes of
a processor grid as one logical axis. They are coordinate transformations,
not :ref:`places-partitioning`: the latter decomposes a place into constituent
resources.
.. _places-partitioning:
Partitioning grids
^^^^^^^^^^^^^^^^^^
The ``place_partition`` class partitions an execution place at a given
granularity. This is useful for splitting a multi-device grid into its
constituent devices, or for partitioning a device into green contexts or
CUDA streams.
The partitioning granularity is specified by ``place_partition_scope``:
- ``place_partition_scope::cuda_device`` -- partition into individual devices
- ``place_partition_scope::green_context`` -- partition into green contexts (CUDA 12.4+)
- ``place_partition_scope::cuda_stream`` -- partition into CUDA streams
.. code:: c++
exec_place grid = exec_place::all_devices();
// Partition into individual devices
place_partition devices(grid, place_partition_scope::cuda_device);
for (auto& dev : devices) {
// dev is an exec_place for a single device
}
// Convert back to an exec_place grid
exec_place new_grid = devices.to_exec_place();
The ``exec_place::partition_by_scope()`` method provides a shorthand that
returns a new ``exec_place`` grid directly:
.. code:: c++
exec_place grid = exec_place::all_devices();
exec_place by_device = grid.partition_by_scope(place_partition_scope::cuda_device);
.. _places-data-partitioning:
Data partitioning policies
^^^^^^^^^^^^^^^^^^^^^^^^^^
When using a grid of places with CUDASTF constructs such as ``parallel_for``,
*data partitioning policies* express how data and index spaces are dispatched
over the different places of a grid.
.. code:: c++
class MyPartition : public partitioner_base {
public:
template <typename S_out, typename S_in>
static const S_out apply(const S_in& in, pos4 position, dim4 grid_dims);
void get_executor(pos4* result, pos4 data_coords, dim4 data_dims, dim4 grid_dims);
};
A partitioning class must implement an ``apply`` method which takes:
- a reference to a shape of type ``S_in``
- a position within a grid of execution places, described using an object of
type ``pos4``
- the dimension of this grid expressed as a ``dim4`` object
``apply`` returns a shape which corresponds to the subset of the ``in``
shape associated to this entry of the grid. Note that the output shape
type ``S_out`` may be different from the ``S_in`` type of the input
shape.
To support different types of shapes, appropriate overloads of the
``apply`` method should be implemented.
This ``apply`` method is typically used by the ``parallel_for``
construct in order to dispatch indices over the different places.
A partitioning class must also implement the ``get_executor`` virtual
method which allows localized data allocators. This
method indicates, for each entry of a shape, on which place this entry
should *preferably* be allocated.
``get_executor`` writes a ``pos4`` coordinate in the execution place
grid into ``*result``, and its input arguments are:
- a coordinate within the shape described as a ``pos4`` object
- the dimension of the shape expressed as a ``dim4`` object
- the dimension of the execution place grid expressed as a ``dim4`` object
Defining the ``get_executor`` makes it possible to map a piece of data
over an execution place grid. The ``get_executor`` method of a partitioning
policy in an execution place grid therefore defines the *affine data
place* of a logical data accessed on that grid.
Predefined partitioning policies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are currently two policies readily available:
- ``tiled_partition<TILE_SIZE>`` dispatches entries of a shape using a
*tiled* layout. For multi-dimensional shapes, the outermost dimension is
dispatched into contiguous tiles of size ``TILE_SIZE``.
- ``blocked_partition`` dispatches entries of the shape using a *blocked*
layout, where each entry of the grid of places receives approximately
the same contiguous portion of the shape, dispatched along the outermost
dimension.
This illustrates how a 2D shape is dispatched over 3 places using the
blocked layout:
.. code:: text
__________________________________
| | | |
| | | |
| | | |
| P 0 | P 1 | P 2 |
| | | |
| | | |
|___________|___________|_________|
This illustrates how a 2D shape is dispatched over 3 places using a
tiled layout, where the dimension of the tiles is indicated by the
``TILE_SIZE`` parameter:
.. code:: text
________________________________
| | | | | | |
| | | | | | |
| | | | | | |
| P 0 | P 1 | P 2 | P 0 | P 1 |P2|
| | | | | | |
| | | | | | |
|_____|_____|_____|_____|_____|__|
.. _places-structured-partitions:
Structured partitions
---------------------
The classic partitioning policies above are *scale-free*: ``blocked_partition``
splits whatever shape it is handed, knows nothing about the tensor it will be
applied to, and always dispatches along the outermost dimension. A
*structured partition* (``cute_partition``) is the complementary tool: it
describes, dimension by dimension, how **one specific tensor** maps onto a
grid of places.
.. code:: c++
using namespace cuda::experimental::places;
// A 3-D tensor: dimension 1 blocked over the places of the grid,
// dimensions 0 and 2 not distributed
auto part = make_partition(
dim4(nx, ny, nz),
partition_spec{whole, blocked<0>, whole},
grid.get_dims());
Each entry in ``partition_spec`` selects a policy for the corresponding
tensor dimension: ``whole`` (not distributed), ``blocked<axis>``,
``cyclic<axis>``, or ``block_cyclic<axis>(block_size)``. Rank, policy,
mesh-axis, and leaf counts are preserved in the C++ type; tensor extents,
strides, and block sizes remain runtime values. This is strictly more
expressive than the classic policies -- splitting dimension 1 of a 3-D
tensor, or mixing policies across dimensions, cannot be stated with
``blocked_partition``.
The reference shape, padding, and predication
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The first argument of ``make_partition`` is the tensor's extents: unlike a
classic policy, a structured partition is **bound to one reference shape**,
and remains the authority on it. This is a deliberate trade, and the source
of most of the type's properties:
- Split dimensions are *padded up to divisibility* (a 10-element dimension
blocked over 3 places is treated as 12, in chunks of 4). Padding makes the
underlying layout exact and bijective, which is what keeps every query
closed-form: validation is a linear pass over the layout, and the owner of
a coordinate is a chain of divisions and modulos.
- Coordinates beyond the true extents (the *padding phantoms*) own no bytes
and do no work: consumers discard them by comparing coordinates against
the true extents. This is the *predication* idiom of CUTLASS/CuTe
("partition the rounded-up shape, predicate the boundary") rather than
per-place clamping, which would break the layout's uniformity.
Ownership can be queried directly, and -- more importantly -- a candidate
mapping can be **scored before any memory is committed**:
.. code:: c++
pos4 owner = part.owner(pos4(x, y, z)); // grid position owning (x,y,z)
// Dry run: same block-majority decision procedure as a real allocation
localized_stats stats = evaluate_localized_placement(grid, part, sizeof(double));
// stats.bytes_per_place, stats.accuracy() (estimated fraction of local bytes),
// stats.nallocs, ... -- tune the spec, then allocate
Placement through a structured partition
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A structured partition can back a composite data place. Because the
partition is bound to one tensor, such a place is *per-tensor* -- allocate
with the partition's exact extents (compare with the classic composite
place, which is a reusable shape-free policy):
.. code:: c++
data_place dp = make_composite_data_place(grid, part);
void* ptr = dp.allocate_nd(dim4(nx, ny, nz), sizeof(double));
// physical pages land on the place owning them, per the partition
dp.deallocate(ptr, nx * ny * nz * sizeof(double));
Two structured composite places built from equal partitions compare equal,
so they denote the same data placement wherever data places are compared.
Conventions and limits
^^^^^^^^^^^^^^^^^^^^^^
- Extents follow the **dimension-0-fastest** linearization of
``dim4::get_index()`` (the convention of STF slices). A row-major front-end
must present its *whole* description in this order -- the extents, the
per-dimension ``partition_spec``, and any coordinates passed to ``owner()``
reverse together, since reversing only the extents would silently re-target
each policy at the wrong axis.
- At most 4 tensor dimensions (the ``pos4``/``dim4`` domain).
- Typed partitions and their kernel-facing sub-shapes store exactly their
layout leaves. Runtime interfaces (including C/Python opaque handles) erase
them to a canonical descriptor only at the data-place boundary.
- The partition object is trivially copyable and its queries are host/device
callable.
The ``partitioned_axpy`` example shows the intended workflow end to end:
express the partition once, evaluate it, run tasks over data placed by it,
and perform a raw geometry-aware allocation.
Computing over structured partitions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The same ``parallel_for`` entry point that accepts the classic policies
accepts a structured partition instance, which then decides **both** the
per-place kernel decomposition and (through the task's affine data place)
the placement of the data those kernels touch -- one object, both sides:
.. code:: c++
// Every place computes exactly the coordinates it owns
ctx.parallel_for(part, grid, lX.shape(), lX.write())
->*[] __device__(size_t x, size_t y, size_t z, auto X) { ... };
The shape argument may also be a ``box`` describing a *region within the
tensor the partition was built for* (validated by containment) -- e.g. the
interior of a stencil domain. Each place still enumerates its own
coordinates; those outside the region (like the padding phantoms of uneven
extents) are skipped by a per-coordinate predicate, so iteration stays
aligned with data ownership rather than re-splitting the region:
.. code:: c++
box interior({1ul, nx - 1}, {1ul, ny - 1}, {1ul, nz - 1});
ctx.parallel_for(part, grid, interior, lX.rw())->*...;
Predication has a cost proportional to the *rejected* fraction of the
enumerated coordinates, which makes it the right tool for regions that are
dense in their bounds (interiors: the rejected boundary shell is a
surface-to-volume fraction) and the wrong tool for thin regions. For
boundary-style updates -- a face of the domain, say -- prefer one of:
- **fuse** the boundary handling into the volumetric kernel's body when the
condition is cheap (application-dependent);
- iterate the face with a **classic scale-free policy** (tight, no rejected
coordinates) while an explicit dependency keeps placement on the
partition's composite place:
.. code:: c++
auto dist = make_composite_data_place(grid, part);
box face({0ul, nx}, {0ul, ny}, {0ul, 1ul});
ctx.parallel_for(blocked_partition(), grid, face, lX.rw(dist))->*...;
The face's few remote writes (places computing parts of a face another
place owns) are typically negligible against the volumetric traffic.
The ``fdtd_mgpu`` example demonstrates the full pattern: a single
``make_partition`` call decides which dimension splits for every task --
initialization over the full shape, updates over interior boxes, a point
source -- and places the fields' data, so changing the distribution of the
whole simulation is editing one ``partition_spec`` entry.

2424
cccl_upstream/docs/cudax/stf.rst Normal file
View File

File diff suppressed because it is too large Load Diff

330
cccl_upstream/docs/cudax/stf/custom_data_interface.rst Normal file
View File

@@ -0,0 +1,330 @@
.. _stf_custom_data_interface:
CUDASTF offers an extensible API that allows users to implement their
own data interface.
Let us for example go through the different steps to implement a data
interface for a very simple simple implementation of a matrix class.
For the sake of simplicity, we here only consider the CUDA stream
backend, but adding support for the CUDA graph backend simply require
some extra steps which use the CUDA graph API.
Implementation of the ``matrix`` class
======================================
For the sake of simplicity, we consider a very simple representation of
matrix, only defined by the dimensions m and n, and by the base address
of the matrix which we assume to be contiguous.
.. code:: c++
template <typename T>
class matrix {
public:
matrix(size_t m, size_t n, T* base) : m(m), n(n), base(base) {}
__host__ __device__ T& operator()(size_t i, size_t j) { return base[i + j * m]; }
__host__ __device__ const T& operator()(size_t i, size_t j) const { return base[i + j * m]; }
size_t m, n;
T* base;
};
Defining the shape of a matrix
==============================
The first step consists in defining what is the *shape* of a matrix. The
shape of a matrix should be a class that defines all parameters which
are the same for all data instances, ``m`` and ``n``. On the other hand,
the base address should not be part of this shape class, because each
data instance will have its own base address.
To define what is the shape of a matrix, we need to specialize the
``cudastf::shape_of`` trait class.
.. code:: c++
template <typename T>
class cudastf::shape_of<matrix<T>> {
public:
/**
* @brief The default constructor.
*
* All `shape_of` specializations must define this constructor.
*/
shape_of() = default;
explicit shape_of(size_t m, size_t n) : m(m), n(n) {}
/**
* @name Copies a shape.
*
* All `shape_of` specializations must define this constructor.
*/
shape_of(const shape_of&) = default;
/**
* @brief Extracts the shape from a matrix
*
* @param M matrix to get the shape from
*
* All `shape_of` specializations must define this constructor.
*/
shape_of(const matrix<T>& M) : shape_of<matrix<T>>(M.m, M.n) {}
/// Mandatory method : defined the total number of elements in the shape
size_t size() const { return m * n; }
size_t m;
size_t n;
};
We here see that ``shape_of<matrix<T>>`` contains two ``size_t`` fields
``m`` and ``n``.
In addition, we need to define a default constructor and a copy
constructors.
To implement the ``.shape()`` member of the ``logical_data`` class, we
need to define a constructor which takes a const reference to a matrix.
Finally, if the ``ctx.parallel_for`` construct is needed, we must define
a ``size_t size() const`` method which computes the total number of
elements in a shape.
Hash of a matrix
================
For internal needs, such as using (unordered) maps of data instances,
CUDASTF need to have specialized forms of the ``std::hash`` trait class.
The ``()`` operator of this class should compute a unique identifier
associated to the description of the data instance. This typically means
computing a hash of the matrix sizes, and of the base address. Note that
this hash *does not* depend on the actual content of the matrix.
In code snippet, we are using the ``cudastf::hash_combine`` helper which
updates a hash value with another value. This function is available from
the ``cudastf/utility/hash.h`` header.
.. code:: c++
template <typename T>
struct std::hash<matrix<T>> {
std::size_t operator()(matrix<T> const& m) const noexcept {
// Combine hashes from the base address and sizes
return cudastf::hash_all(m.m, m.n, m.base);
}
};
Defining a data interface
=========================
We can now implement the actual data interface for a matrix class, which
defines the basic operations that CUDASTF need to perform on a matrix.
The ``matrix_stream_interface`` class inherits from the
``data_interface`` class, but to implement a data interface using APIs
based on CUDA streams, ``matrix_stream_interface`` inherits from
``stream_data_interface_simple<matrix<T>>`` which contains pure virtual
functions that need to be implemented.
.. code:: c++
template <typename T>
class matrix_stream_interface : public stream_data_interface_simple<matrix<T>> {
public:
using base = stream_data_interface_simple<matrix<T>>;
using base::shape_t;
/// Initialize from an existing matrix
matrix_stream_interface(matrix<T> m) : base(std::move(m)) {}
/// Initialize from a shape of matrix
matrix_stream_interface(shape_t s) : base(s) {}
/// Copy the content of an instance to another instance
///
/// This implementation assumes that we have registered memory if one of the data place is the host
void stream_data_copy(const data_place& dst_memory_node, instance_id_t dst_instance_id,
const data_place& src_memory_node, instance_id_t src_instance_id, cudaStream_t stream) override {
assert(src_memory_node != dst_memory_node);
cudaMemcpyKind kind = cudaMemcpyDeviceToDevice;
if (src_memory_node == data_place::host) {
kind = cudaMemcpyHostToDevice;
}
if (dst_memory_node == data_place::host) {
kind = cudaMemcpyDeviceToHost;
}
const matrix<T>& src_instance = this->instance(src_instance_id);
const matrix<T>& dst_instance = this->instance(dst_instance_id);
size_t sz = src_instance.m * src_instance.n * sizeof(T);
cuda_safe_call(cudaMemcpyAsync((void*) dst_instance.base, (void*) src_instance.base, sz, kind, stream));
}
/// allocate an instance on a specific data place
///
/// setting *s to a negative value informs CUDASTF that the allocation
/// failed, and that a memory reclaiming mechanism need to be performed.
void stream_data_allocate(backend_ctx_untyped& ctx, const data_place& memory_node, instance_id_t instance_id, ssize_t& s,
void** extra_args, cudaStream_t stream) override {
matrix<T>& instance = this->instance(instance_id);
size_t sz = instance.m * instance.n * sizeof(T);
T* base_ptr;
if (memory_node == data_place::host) {
// Fallback to a synchronous method as there is no asynchronous host allocation API
cuda_safe_call(cudaStreamSynchronize(stream));
cuda_safe_call(cudaHostAlloc(&base_ptr, sz, cudaHostAllocMapped));
} else {
cuda_safe_call(cudaMallocAsync(&base_ptr, sz, stream));
}
// By filling a positive number, we notify that the allocation was successful
*s = sz;
instance.base = base_ptr;
}
/// deallocate an instance
void stream_data_deallocate(backend_ctx_untyped& ctx, const data_place& memory_node, instance_id_t instance_id, void* extra_args,
cudaStream_t stream) override {
matrix<T>& instance = this->instance(instance_id);
if (memory_node == data_place::host) {
// Fallback to a synchronous method as there is no asynchronous host deallocation API
cuda_safe_call(cudaStreamSynchronize(stream));
cuda_safe_call(cudaFreeHost(instance.base));
} else {
cuda_safe_call(cudaFreeAsync(instance.base, stream));
}
}
/// Register the host memory associated to an instance of matrix
///
/// Note that this pin_host_memory method is not mandatory, but then it is
/// the responsibility of the user to only passed memory that is already
/// registered, and the allocation method on the host must allocate
/// registered memory too. Otherwise, copy methods need to be synchronous.
bool pin_host_memory(instance_id_t instance_id) override {
matrix<T>& instance = this->instance(instance_id);
if (!instance.base) {
return false;
}
cuda_safe_call(pin_memory(instance.base, instance.m * instance.n * sizeof(T)));
return true;
}
/// Unregister memory pinned by pin_host_memory
void unpin_host_memory(instance_id_t instance_id) override {
matrix<T>& instance = this->instance(instance_id);
unpin_memory(instance.base);
}
};
``matrix_stream_interface`` must meet the following requirements so that
they can be used in the CUDA stream backend : - It must provide
constructors which take either a matrix, or a shape of matrix as
arguments. - It must implement the ``stream_data_copy``,
``stream_data_allocate`` and ``stream_data_deallocate`` virtual methods,
which respectively define how to copy an instance into another instance,
how to allocate an instance, and how to deallocate an instance. - It may
implement the ``pin_host_memory`` and ``unpin_host_memory`` virtual
methods which respectively register and unregister the memory associated
to an instance allocated on the host. These two methods are not
mandatory, but it is the responsibility of the user to either only pass
and allocate registered host buffers, or to ensure that the copy method
does not require such memory pinning. Similarly, accessing an instance
located in host memory from a device typically requires to access
registered memory.
Associating a data interface with the CUDA stream backend
=========================================================
To ensure that we can initialize a logical data from a matrix, or from
the shape of a matrix with ``stream_ctx::logical_data``, we then need to
specialize the ``cudastf::streamed_interface_of`` trait class.
The resulting class must simply define a type named ``type`` which is
the type of the data interface for the CUDA stream backend.
.. code:: c++
template <typename T>
class cudastf::streamed_interface_of<matrix<T>> {
public:
using type = matrix_stream_interface<T>;
};
Once we have defined this trait class, it is for example possible to
initialize a logical data from a matrix, or from a matrix shape :
.. code:: c++
std::vector<int> v(m * n, 0);
matrix M(m, n, &v[0]);
// Initialize from a matrix
auto lM = ctx.logical_data(M);
// Initialize from a shape
auto lM2 = ctx.logical_data(shape_of<matrix<int>>(m, n));
Example of code using the ``matrix`` data interface
===================================================
We can now use the ``matrix`` class in CUDASTF, and access it from
tasks. In this code, we first initialize a matrix on the host, we then
apply a task which will update its content on the current device. We
finally check that the content is correct, by the means of the
write-back mechanism that automatically updates the reference data
instance of a logical data when calling ``ctx.sync()``.
.. code:: c++
template <typename T>
__global__ void kernel(matrix<T> M) {
int tid_x = blockIdx.x * blockDim.x + threadIdx.x;
int nthreads_x = gridDim.x * blockDim.x;
int tid_y = blockIdx.y * blockDim.y + threadIdx.y;
int nthreads_y = gridDim.y * blockDim.y;
for (int x = tid_x; x < M.m; x += nthreads_x)
for (int y = tid_y; y < M.n; y += nthreads_y) {
M(x, y) += -x + 7 * y;
}
}
int main() {
stream_ctx ctx;
const size_t m = 8;
const size_t n = 10;
std::vector<int> v(m * n);
for (size_t j = 0; j < n; j++)
for (size_t i = 0; i < m; i++) {
v[i + j * m] = 17 * i + 23 * j;
}
matrix<int> M(m, n, &v[0]);
auto lM = ctx.logical_data(M);
// M(i,j) += -i + 7*i
ctx.task(lM.rw())->*[](cudaStream_t s, auto dM) { kernel<<<dim3(8, 8), dim3(8, 8), 0, s>>>(dM); };
ctx.sync();
for (size_t j = 0; j < n; j++)
for (size_t i = 0; i < m; i++) {
assert(v[i + j * m] == (17 * i + 23 * j) + (-i + 7*i));
}
}

5
cccl_upstream/docs/cudax/stf/images/dag-sections-0.dot Normal file
View File

@@ -0,0 +1,5 @@
digraph {
"NODE_23" [style="filled" fillcolor="red" label="task fence"]
"NODE_3" -> "NODE_23"
"NODE_3" [style="filled" fillcolor="white" label="foo"]
}

BIN
cccl_upstream/docs/cudax/stf/images/dag-sections-0.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.9 KiB

14
cccl_upstream/docs/cudax/stf/images/dag-sections-1.dot Normal file
View File

@@ -0,0 +1,14 @@
digraph {
"NODE_23" [style="filled" fillcolor="red" label="task fence"]
subgraph cluster_section_1 {
color=black;
style=dashed
label="foo"
"NODE_13"
"NODE_3"
} // end subgraph cluster_section_1
"NODE_13" -> "NODE_23"
"NODE_3" -> "NODE_13"
"NODE_13" [style="filled" fillcolor="white" label="bar"]
"NODE_3" [style="filled" fillcolor="white" label="bar"]
}

BIN
cccl_upstream/docs/cudax/stf/images/dag-sections-1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

36
cccl_upstream/docs/cudax/stf/images/dag-sections-2.dot Normal file
View File

@@ -0,0 +1,36 @@
digraph {
"NODE_23" [style="filled" fillcolor="red" label="task fence"]
subgraph cluster_section_1 {
subgraph cluster_section_2 {
color=black;
style=dashed
label="bar"
"NODE_9"
"NODE_5"
"NODE_3"
} // end subgraph cluster_section_2
subgraph cluster_section_5 {
color=black;
style=dashed
label="bar"
"NODE_19"
"NODE_15"
"NODE_13"
} // end subgraph cluster_section_5
color=black;
style=dashed
label="foo"
} // end subgraph cluster_section_1
"NODE_19" -> "NODE_23"
"NODE_3" -> "NODE_5"
"NODE_5" -> "NODE_9"
"NODE_15" -> "NODE_19"
"NODE_9" -> "NODE_13"
"NODE_13" -> "NODE_15"
"NODE_19" [style="filled" fillcolor="white" label="baz"]
"NODE_15" [style="filled" fillcolor="white" label="baz"]
"NODE_13" [style="filled" fillcolor="white" label="t1\nA(read)(0) \nB(rw)(0) "]
"NODE_9" [style="filled" fillcolor="white" label="baz"]
"NODE_5" [style="filled" fillcolor="white" label="baz"]
"NODE_3" [style="filled" fillcolor="white" label="t1\nA(read)(0) \nB(rw)(0) "]
}

BIN
cccl_upstream/docs/cudax/stf/images/dag-sections-2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

69
cccl_upstream/docs/cudax/stf/images/dag-sections.dot Normal file
View File

@@ -0,0 +1,69 @@
digraph {
"NODE_23" [style="filled" fillcolor="red" label="task fence"]
subgraph cluster_section_1 {
subgraph cluster_section_2 {
subgraph cluster_section_3 {
color=black;
style=dashed
label="baz"
"NODE_7"
"NODE_5"
} // end subgraph cluster_section_3
subgraph cluster_section_4 {
color=black;
style=dashed
label="baz"
"NODE_11"
"NODE_9"
} // end subgraph cluster_section_4
color=black;
style=dashed
label="bar"
"NODE_3"
} // end subgraph cluster_section_2
subgraph cluster_section_5 {
subgraph cluster_section_6 {
color=black;
style=dashed
label="baz"
"NODE_17"
"NODE_15"
} // end subgraph cluster_section_6
subgraph cluster_section_7 {
color=black;
style=dashed
label="baz"
"NODE_21"
"NODE_19"
} // end subgraph cluster_section_7
color=black;
style=dashed
label="bar"
"NODE_13"
} // end subgraph cluster_section_5
color=black;
style=dashed
label="foo"
} // end subgraph cluster_section_1
"NODE_15" -> "NODE_17"
"NODE_9" -> "NODE_11"
"NODE_11" -> "NODE_15"
"NODE_13" -> "NODE_17"
"NODE_19" -> "NODE_21"
"NODE_11" -> "NODE_13"
"NODE_7" -> "NODE_9"
"NODE_5" -> "NODE_7"
"NODE_17" -> "NODE_19"
"NODE_3" -> "NODE_7"
"NODE_21" -> "NODE_23"
"NODE_21" [style="filled" fillcolor="white" label="t3\nA(rw)(0) \nB(read)(0) \nC(read)(0) "]
"NODE_19" [style="filled" fillcolor="white" label="t2\nA(read)(0) \nC(rw)(0) "]
"NODE_17" [style="filled" fillcolor="white" label="t3\nA(rw)(0) \nB(read)(0) \nC(read)(0) "]
"NODE_15" [style="filled" fillcolor="white" label="t2\nA(read)(0) \nC(rw)(0) "]
"NODE_13" [style="filled" fillcolor="white" label="t1\nA(read)(0) \nB(rw)(0) "]
"NODE_11" [style="filled" fillcolor="white" label="t3\nA(rw)(0) \nB(read)(0) \nC(read)(0) "]
"NODE_9" [style="filled" fillcolor="white" label="t2\nA(read)(0) \nC(rw)(0) "]
"NODE_7" [style="filled" fillcolor="white" label="t3\nA(rw)(0) \nB(read)(0) \nC(read)(0) "]
"NODE_5" [style="filled" fillcolor="white" label="t2\nA(read)(0) \nC(rw)(0) "]
"NODE_3" [style="filled" fillcolor="white" label="t1\nA(read)(0) \nB(rw)(0) "]
}

BIN
cccl_upstream/docs/cudax/stf/images/dag-sections.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 101 KiB

BIN
cccl_upstream/docs/cudax/stf/images/dot-output-axpy-annotated.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
cccl_upstream/docs/cudax/stf/images/dot-output-axpy-events.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 132 KiB

BIN
cccl_upstream/docs/cudax/stf/images/dot-output-axpy.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
cccl_upstream/docs/cudax/stf/images/dot-output-heat.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 113 KiB

10
cccl_upstream/docs/cudax/stf/images/graph_01.dot Normal file
View File

@@ -0,0 +1,10 @@
digraph {
T_1 [label="T1\nX(rw)"];
T_2 [label="T2\nX(read)\nY(rw)"];
T_3 [label="T3\nX(read)\nZ(rw)"];
T_4 [label="T4\nY(read)\nZ(rw)"];
T_1 -> T_2;
T_1 -> T_3;
T_2 -> T_4;
T_3 -> T_4;
}

BIN
cccl_upstream/docs/cudax/stf/images/graph_01.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

16
cccl_upstream/docs/cudax/stf/images/graph_02.dot Normal file
View File

@@ -0,0 +1,16 @@
digraph {
subgraph cluster_0 {
label="device 0";
T_1 [label="T_1(A^W)"];
T_2 [label="T_2(A^R, B^W)"];
}
subgraph cluster_1 {
label="device 1";
T_3 [label="T_3(A^R, C^W)"];
T_4 [label="T_4(B^R, C^R, D^W)"];
}
T_1 -> T_2 [label="A"];
T_1 -> T_3 [label="A"];
T_2 -> T_4 [label="B"];
T_3 -> T_4 [label ="C"];
}

BIN
cccl_upstream/docs/cudax/stf/images/graph_02.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

BIN
cccl_upstream/docs/cudax/stf/images/ncu-ui.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 171 KiB

26
cccl_upstream/docs/cudax/stf/images/task-sequence-user.dot Normal file
View File

@@ -0,0 +1,26 @@
digraph {
compound=true;
subgraph cluster_0 {
label="T1";
K1 [label="K1"];
K2 [label="K2"];
}
subgraph cluster_1 {
label="T2";
K3 [label="K3"];
}
subgraph cluster_2 {
label="T3";
K4 [label="K4"];
}
subgraph cluster_3 {
label="T4";
cb [label="callback"];
}
K1 -> K2;
K2 -> K3 [ltail=cluster_0,lhead=cluster_1,minlen=2];
K2 -> K4 [ltail=cluster_0,lhead=cluster_2,minlen=2];
K3 -> cb [ltail=cluster_1,lhead=cluster_3,minlen=2];
K4 -> cb [ltail=cluster_2,lhead=cluster_3,minlen=2];
}

BIN
cccl_upstream/docs/cudax/stf/images/task-sequence-user.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

68
cccl_upstream/docs/cudax/stf/images/task-sequence.dot Normal file
View File

@@ -0,0 +1,68 @@
digraph {
compound=true;
subgraph cluster_00 {
label="";
AA [label="Allocate A"];
}
subgraph cluster_01 {
label="";
CA [label="Copy A H->D"];
}
subgraph cluster_10 {
label="";
AB [label="Allocate B"];
}
subgraph cluster_11 {
label="";
CB [label="Copy B H->D"];
}
subgraph cluster_0 {
label="T1";
K1 [label="K1"];
K2 [label="K2"];
}
CA -> K1 [ltail=cluster_01,lhead=cluster_0,minlen=2];
AA -> CA [ltail=cluster_00,lhead=cluster_01,minlen=2];
CB -> K1 [ltail=cluster_11,lhead=cluster_0,minlen=2];
AB -> CB [ltail=cluster_10,lhead=cluster_11,minlen=2];
subgraph cluster_1 {
label="T2";
K3 [label="K3"];
}
subgraph cluster_2 {
label="T3";
K4 [label="K4"];
}
K1 -> K2;
K2 -> K3 [ltail=cluster_0,lhead=cluster_1,minlen=2];
K2 -> K4 [ltail=cluster_0,lhead=cluster_2,minlen=2];
subgraph cluster_02 {
label="";
CA2 [label="Copy A D->A"];
}
subgraph cluster_12 {
label="";
CB2 [label="Copy B D->A"];
}
subgraph cluster_3 {
label="T4";
cb [label="callback"];
}
K3 -> CA2 [ltail=cluster_1,lhead=cluster_02,minlen=2];
K4 -> CB2 [ltail=cluster_2,lhead=cluster_12,minlen=2];
CA2 -> cb [ltail=cluster_02,lhead=cluster_3,minlen=2]
CB2 -> cb [ltail=cluster_12,lhead=cluster_3,minlen=2]
}

Some files were not shown because too many files have changed in this diff Show More