dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix: critical config + tuning corrections from CCCL source analysis

Browse Source 
computility-run.yaml:
  max-num-seqs 1→256: benchmark sweeps [128,256] concurrent seqs,
    current config processes 1 while 127 queue. KV cache budget:
    256 seqs × 2048 tokens × 80KB/token = 41.9GB < 45GB available.
  max-num-batched-tokens 8192→32768: support 256 concurrent prefills.
  gpu-memory-utilization 0.9→0.95: provide KV cache headroom.

Dockerfile:
  Deploy paged_attention_v2_triton.py to vllm package path so
  try-triton-first logic in _custom_ops.py can find it. Falls back
  to PyTorch V2 automatically if Triton V2 fails (SMEM/runtime).

muh/tuning/common.cuh:
  scale_mem_bound max_smem now a parameter (default 48KB). Allows
  policy_selectors to pass hw.max_shared_memory_per_block if actual
  SMEM differs from CCCL 48KB assumption.

muh/tuning/tuning_transform.cuh:
  bytes_in_flight 16KB→32KB. Old derivation used 900/50=18 GB/s/SM
  (wrong, SM=16 confirmed). Actual per-SM BW = 56 GB/s.
  32KB is estimate pending benchmark sweep.

SM count 50→16 corrections across all affected files.
This commit is contained in:
Claude
2026-08-03 06:45:54 +00:00
parent 071fa361a3
commit cdc01bbc6a
7 changed files with 33 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
muh/include/muh/tuning/common.cuh
View File

@@ -146,8 +146,11 @@ struct scaling_result {
/// b) Upper clamp was nominal*1 — should be nominal*2
/// c) No SMEM cap on threads — CCCL caps threads to prevent SMEM overflow
constexpr scaling_result scale_mem_bound(
int nominal_4B_threads, int nominal_4B_items, int target_type_size) {
constexpr int max_smem = 48 * 1024; // 49152 bytes
int nominal_4B_threads, int nominal_4B_items, int target_type_size,
int max_smem = 48 * 1024) {
// max_smem default 48KB matches CCCL (util_arch.cuh:116).
// Pass hw.max_shared_memory_per_block from policy_selector to override
// if BI-V100 actual SMEM differs (_custom_ops.py claims 32KB).
// Step 1+2: scale items, clamp to [1, nominal*2]
int items = nominal_4B_items * 4 / target_type_size;

3
muh/include/muh/tuning/tuning_reduce.cuh
View File

@@ -61,7 +61,8 @@ enum class determinism_t {
struct bi100_float32_plus_o4 {
// accum_size=4, tile = 512*16*4 = 32768 ≤ 49152 ✓
// SM100 ref: ipt_16.tpb_512.ipv_2 1.061 1.000 1.065 1.167
// Derivation: SMEM OK, threads=512 for occupancy on 50 SMs. Keep.
// Derivation: SMEM OK, threads=512. SM=16 (not 50 from spec sheet).
// At 16 SMs, fewer concurrent CTAs → consider larger tiles. Pending benchmark.
static constexpr int items = 16;
static constexpr int threads = 512;
static constexpr int items_per_vec_load = 2;

11
muh/include/muh/tuning/tuning_transform.cuh
View File

@@ -92,10 +92,13 @@ struct policy_selector {
if (items_for_vec < 1) items_for_vec = 1;
// items_for_latency: enough items to hide memory latency
// CCCL cc_to_min_bytes_in_flight: B200=64KB, H100=48KB, A100=16KB, V100=12KB
// BI-V100 per-SM BW = 900/50 = 18 GB/s ≈ A100 (2000/108 = 18.5 GB/s)
// → Use 16KB (A100-level), not 48-64KB
int bytes_in_flight = 16 * 1024;
// CCCL cc_to_min_bytes_in_flight: B200=64KB(54GB/s/SM), H100=48KB(25GB/s/SM),
// A100=16KB(18.5GB/s/SM), V100=12KB(14GB/s/SM)
// BI-V100: SM=16 (confirmed), per-SM BW = 900/16 = 56 GB/s
// bytes_in_flight = BW_per_SM × HBM_latency. BI-V100 HBM latency unknown.
// 56 GB/s per SM is B200-level BW, but latency likely differs (not NVIDIA arch).
// Estimate 32KB pending benchmark: %RANGE% bytes_in_flight 12288:65536:4096
int bytes_in_flight = 32 * 1024;
int items_for_latency = bytes_in_flight / (256 * min_elem_size);
if (items_for_latency < 1) items_for_latency = 1;