Compile-time benchmark reference ================================ The compile-time benchmark builds generated one-include CUDA translation units (TUs) and summarizes NVCC ``--fdevice-time-trace`` output. It is a TU compile-time benchmark, even when the current input set comes from public include-check targets. Entry point ----------- The top-level entry point is: .. code-block:: bash ci/build_compile_time_bench.sh The wrapper configures a caller-selected CMake preset with compile-time instrumentation, builds selected targets, prepares Perfetto-friendly trace copies, writes a generated-TU summary CSV, and emits one or more event summary CSVs. Generated outputs ----------------- By default, outputs are written under: - ``build///compile_time/tu_summary.csv`` - ``build///compile_time/event_reports/`` - ``build///compile_time/perfetto_traces/`` Raw NVCC traces are generated under: - ``build///compile_time/raw_traces/`` When ``-baseline-ref`` is used, baseline raw traces are copied out of the temporary baseline worktree before cleanup and preserved under: - ``build///compile_time/baseline_raw_traces/`` Build controls -------------- The wrapper accepts build-shape parameters so it behaves like other ``ci/build*.sh`` entry points: .. code-block:: bash PARALLEL_LEVEL=16 ci/build_compile_time_bench.sh \ -preset all-dev \ -target libcudacxx.test.public_headers \ -cmake-options "-DCMAKE_CUDA_ARCHITECTURES=native" Useful build options include: - ``-preset `` - ``-cmake-options `` - ``-target `` (repeatable; replaces the default public include-check target set) - ``-baseline-ref `` (build a temporary baseline worktree for comparison) - ``-skip-configure`` - ``-skip-build`` The default target set is the current public include-check target set: - ``cub.headers.base`` - ``thrust.cpp.cuda.headers.base`` - ``libcudacxx.test.public_headers`` The one-include source TUs are created by those CMake targets themselves. The benchmark wrapper only enables extra compile options on generated CUDA TUs: ``CCCL_COMPILE_TIME_GENERATE_DEVICE_TIME_TRACES`` writes NVCC device-time trace JSON, and ``CCCL_COMPILE_TIME_SAVE_PREPROCESSED_TUS`` preserves compiler preprocessed/temporary TU artifacts for the TU summary. Event summaries --------------- Arguments after ``--`` are forwarded to ``ci/compile_time/summarize_events.py`` after the raw trace directory. If no event-summary arguments are provided, the wrapper runs: .. code-block:: bash -f file-processing -e -n 15 For CI-style multi-slice reports, pass a JSON slice file: .. code-block:: bash ci/build_compile_time_bench.sh -baseline-ref origin/main -- \ --slices /path/to/slices.json The slice file contains a ``slices`` array. Each slice has a stable ``id``, display ``title``, ``filter``, ``timing`` (``inclusive`` or ``exclusive``), ``sort``, ``top``, and ``threshold`` in seconds. Multi-slice mode writes each slice under ``event_reports//`` and writes a normalized ``event_reports/summary.json`` manifest for PR comment rendering. Empty slices are represented in the manifest and CSVs. Slices that match no events, have no matching trace files, or have no comparable event keys record warnings so they are visible in PR comments instead of looking like ordinary no-change results. Empty slices with no warnings are omitted recursively by the PR comment renderer. Single-slice examples: .. code-block:: bash ci/build_compile_time_bench.sh -skip-build -- -f scanning-function-body -i -n 20 ci/build_compile_time_bench.sh -skip-build -- -f template-instantiation -e -n 15 --tag templates ci/build_compile_time_bench.sh -skip-build -- -f 'Scanning|Instantiating' -i -n 25 ci/build_compile_time_bench.sh -skip-build -- -f code-generation -i --scope-filter "" Baseline comparisons -------------------- Pass ``-baseline-ref `` to compare the current tree state against a baseline commit. The wrapper creates a temporary detached worktree for the baseline, builds both the current tree and the baseline with the same preset, targets, and common build options, then runs the requested event slice as a baseline/current comparison: .. code-block:: bash ci/build_compile_time_bench.sh \ -baseline-ref origin/main \ -- -f file-processing -e --sort total -n 25 --threshold 0.2 Comparison mode writes three subdirectories under ``/compile_time/event_reports/``: - ``baseline/``: the normal report for the baseline traces - ``current/``: the normal report for the current traces - ``comparison/``: ``worse`` and ``better`` CSVs for the requested filter, timing, exclusivity, sort, and top-N slice In multi-slice comparison mode, the same layout appears under each ``event_reports//`` directory. Trace files are matched by relative path. Delta CSVs only compare event keys that appear in both sides of the same matched trace file. Unmatched child events are still counted in their matched parent event's exclusive cost, so disappearing or newly appearing nested work remains visible as a parent cost change instead of being subtracted away. If there are no comparable event keys, the comparison CSVs are still written with headers and no rows. Pass ``--threshold `` after the wrapper's ``--`` separator in comparison mode to omit ``worse`` / ``better`` rows whose total impact change is not greater than that threshold. Baseline/current reports use ``--sort`` for their own top-N ordering, but comparison reports are ranked by total impact across all matched traces so repeated small movements outrank a larger movement in only one trace. When ``-baseline-ref`` is used, the wrapper treats the invocation as an event comparison and skips the generated-TU CSV unless ``-tu-csv`` is provided explicitly. To compare arbitrary trace directories outside the wrapper layout, run ``ci/compile_time/summarize_events.py`` directly. Pull-request reporting ---------------------- Compile-time PR reporting is configured in ``ci/matrix.yaml`` under ``compile_time.pull_request``. Each config selects the GPU runner, devcontainer launch arguments, baseline ref, preset, targets, wrapper arguments, and report slices. ``ci/compile_time/parse_matrix.py`` validates that section and emits the GitHub Actions matrix for the reusable compile-time benchmark workflow. The reusable workflow uploads: - event report CSVs and ``summary.json`` - current raw traces - baseline raw traces - Perfetto-friendly traces - the rendered PR comment body ``ci/compile_time/render_pr_comment.py`` renders the comment from ``summary.json``. Regressions and improvements are rendered in separate ``
`` sections and are never mixed in one table. Slice warnings are rendered separately. Empty sections with no warnings are omitted recursively. Sticky comments are keyed by ``compile-time-bench-``; previous comments for the same config are archived as outdated when a new one is posted. This reporting is informational and is not part of the aggregate branch protection ``CI`` job. Commit messages containing ``[skip-compile-time-bench]`` skip compile-time benchmark dispatch. Filters and scope filtering --------------------------- Built-in filter names include: - ``file-processing`` - ``scanning-function-body`` - ``template-instantiation`` - ``template-class-instantiation`` - ``template-function-instantiation`` - ``pending-instantiations`` - ``frontend`` - ``host-compiler`` - ``code-generation`` - ``optimizer`` - ``total-compilation`` - ``all`` Unknown filters are interpreted as case-insensitive regular expressions over event names and event details. Symbol-like events, such as function parsing, template instantiation, function IR generation, and optimizer-function events, are scope-filtered by default to top-level CCCL-owned namespaces: - ``cuda::`` - ``thrust::`` - ``cub::`` - ``cccl::`` This keeps reports focused on CCCL symbols instead of system library symbols pulled into the same generated TU. The filter applies to demangled trace details and to decoded namespace prefixes from Itanium-mangled symbols in trace details. It does not filter path/phase events such as file processing, host compiler phases, or total compilation time. Pass ``--scope-filter `` after the wrapper's ``--`` separator to choose a different case-sensitive symbol-scope regex. Pass an empty string to disable symbol-scope filtering: .. code-block:: bash ci/build_compile_time_bench.sh -skip-build -- \ -f template-instantiation -i --scope-filter "" ``host-compiler`` matches the host compiler preprocessing / compiling events that appear in the device-time-trace output. ``total-compilation`` is a synthetic per-trace event whose inclusive time is the wall-clock span from the first timed trace event to the last timed trace event. This includes host compiler, cudafe, NVVM, fatbinary, and gaps visible inside the trace timeline, but it is not a separate external wall-clock measurement of untraced driver, Ninja, or process-launch overhead. Compared to Ninja log timings, ``total-compilation`` generally undercounts each TU by a small, consistent amount because the trace span starts at the first timed event and ends at the last timed event rather than at process launch/exit. That makes it a good relative-comparison and ranking metric, but not an exact replacement for external wall-clock command duration. Use ``--sort`` to choose the selected ranking metric: - ``total`` - ``avg`` - ``avg-root-tu`` - ``max`` Perfetto trace preparation -------------------------- The wrapper prepares Perfetto-friendly trace copies by default. To prepare an existing trace directory manually: .. code-block:: bash ci/compile_time/prepare_traces.py \ --input build///compile_time/raw_traces \ --output /tmp/compile_time_perfetto CSV output ---------- ``summarize_tus.py`` writes: - ``tu_input`` - ``transitive_loc`` - ``tu_source`` - ``preprocessed_tu`` This CSV is a generated-TU input/LOC summary. Use the ``total-compilation`` event filter when you need per-TU compile-time rankings from trace data. ``summarize_events.py`` writes stable event keys and both inclusive and exclusive metrics, including: - ``event_name`` - ``event_key`` (repo-root-relative path for project file-processing events) - ``selected_total_s`` - ``selected_avg_per_event_s`` - ``selected_avg_per_root_tu_s`` - ``total_inclusive_s`` / ``total_exclusive_s`` - ``event_count`` - ``trace_count`` - ``root_tu_count`` Comparison CSVs additionally include total-impact columns (``baseline_impact_s``, ``current_impact_s``, ``impact_delta_s``, ``impact_magnitude_s``) plus selected-metric columns (``baseline_selected_s``, ``current_selected_s``, ``selected_delta_s``, ``selected_magnitude_s``). Notebook workflow ----------------- For exploratory analysis, use ``ci/compile_time/analytics.ipynb``.