1f1067b1de
Key findings from full source audit: - gen_patch.py's VLLM_INJECTION_POINTS target csrc/*.cu files that DON'T EXIST in EngineX (precompiled .so, no CUDA source). This is why it outputs 0 patches. - Actual injection is via patch_ops.sh full-file Python replacements (15 files) - Python-side tuning values (_PARTITION_SIZE=512, SMEM=49152, Q_CHUNK=256) are hardcoded in deployed files, not programmatically derived from muh headers - 27 muh headers have 36+ bi100_* structs (14 reduce, 22 scan) all SMEM-safe - scale_mem_bound passes all 4 CCCL parity tests - Benchmark infrastructure (bench_bi100.py) ready but needs BI-V100 hardware This replaces the stale GROUND_TRUTH_STATUS.md and GROUND_TRUTH_STATUS_v2.md.
3.9 KiB
3.9 KiB
muh Pipeline Status — Ground Truth
Last verified: 2026-08-07T01:45:31Z by automated analysis
Architecture Summary
CCCL policy_selector(compute_capability) → ReducePolicy{threads, items, vec, algo, load_mod}
↕ mirrors
muh policy_selector(hardware_capability) → same struct types, BI-V100 values
↕ gen_patch.py extracts bi100_* values
vllm patch_ops.sh → full-file Python replacements with tuning values baked in
Injection Reality
What gen_patch.py THINKS (csrc/*.cu — DEAD)
tuning_reduce.cuh → csrc/attention/attention_kernels.cu NUM_THREADS ← NO .cu SOURCE
tuning_scan.cuh → csrc/attention/paged_attention_v1.cu SCAN_BLOCK_SIZE ← NO .cu SOURCE
tuning_topk.cuh → csrc/sampling/sampling_kernels.cu SAMPLING_BLOCK_SIZE ← NO .cu SOURCE
What ACTUALLY happens (Python runtime — ALIVE)
_custom_ops.py → SMEM 49152 (was 32768) ← DEPLOYED ✓
paged_attn.py → _PARTITION_SIZE=512 ← DEPLOYED ✓ (V2 partition, NOT CTA tile)
xformers.py → _Q_CHUNK=256, sdpa_fallback ← DEPLOYED ✓
sampler.py → torch.topk fast path ← DEPLOYED ✓
prefix_prefill.py → Triton BLOCK_M/N/warps ← DEPLOYED ✓ (but Triton not available)
computility-run.yaml → vllm server args ← DEPLOYED ✓
The Gap
muh C++ headers define precise per-type-per-op tuning values (14 reduce structs, 22 scan structs). But the vllm engine on BI-V100 runs ixformer .so (precompiled, not tunable) + Python fallbacks. The C++ headers' values cannot be injected into the precompiled .so. They CAN inform:
- Python fallback implementations (paged_attn.py, xformers.py) — tile sizes, chunk sizes
- Triton JIT configs — if Triton were available (it's not on BI-V100 base image)
- Future EngineX releases that expose tuning knobs
Asset Inventory
| Asset | Count | Status |
|---|---|---|
| CCCL tuning headers (upstream) | 27 | Complete |
| muh BI-V100 headers | 27 | Complete (14 reduce + 22 scan + others) |
| muh schema YAMLs | 27 | Complete |
| CUB benchmarks | 91 | Synced to NVIDIA/cccl main |
| CUB tests | 243 | Complete |
| CUB examples | 18 | Complete |
| Thrust examples | 60 | Complete |
| Deployed patches | 15 files | Via patch_ops.sh full replacement |
| bench_bi100.py search spaces | 5 algos | Defined, needs BI-V100 hardware to run |
Tool Chain Status
| Tool | Input | Output | Status |
|---|---|---|---|
| parse.py | baseline.muh | JSON config | ✓ Working |
| gen_patch.py | tuning_*.cuh | Patch report | ⚠ Reports structs but generates 0 patches (injection mapping mismatch) |
| gen_yaml.py | baseline.muh | computility-run.yaml | ✓ Working |
| bench_bi100.py | algo+dtype | CCCL-format speedup data | Needs BI-V100 hardware |
| patch_ops.sh | qwen3_6_scripts/ | Docker vllm patches | ✓ Working |
| muh_dispatch.py | hw+dtype+head_dim | AttentionConfig | ✓ Working (needs torch) |
| scale_mem_bound | (threads, items, type_size) | (items, threads) | ✓ CCCL parity verified |
Critical Numbers
| Metric | Competition Threshold | Current Status |
|---|---|---|
| Functional tests | 50+ pass | 13 items In Progress (all FEA) |
| Effect deviation | ≤ ±4% | Untested (needs hardware) |
| Token throughput weighted | ≥ 8000 | Untested |
| Output TPS weight | 83% (×16.796) | Reduce/scan/topk optimization focus |
| SMEM limit | 49152 bytes | All 36 scan+reduce structs verified ✓ |
| SM count | 16 (confirmed) | All headers updated |
Next Actions (Ranked by Competition Impact)
- Run bench_bi100.py on BI-V100 → get real speedup data for reduce/scan/topk
- Backfill speedup data to muh headers → replace TBD/theoretical values
- Optimize Python fallback tile sizes → paged_attn.py, xformers.py Q_CHUNK
- Tune computility-run.yaml → max-num-seqs, max-batched-tokens, gpu-mem-util
- Enable prefix caching benchmark → cached_tokens > 0 for repeat prompts