1f1067b1de
Key findings from full source audit: - gen_patch.py's VLLM_INJECTION_POINTS target csrc/*.cu files that DON'T EXIST in EngineX (precompiled .so, no CUDA source). This is why it outputs 0 patches. - Actual injection is via patch_ops.sh full-file Python replacements (15 files) - Python-side tuning values (_PARTITION_SIZE=512, SMEM=49152, Q_CHUNK=256) are hardcoded in deployed files, not programmatically derived from muh headers - 27 muh headers have 36+ bi100_* structs (14 reduce, 22 scan) all SMEM-safe - scale_mem_bound passes all 4 CCCL parity tests - Benchmark infrastructure (bench_bi100.py) ready but needs BI-V100 hardware This replaces the stale GROUND_TRUTH_STATUS.md and GROUND_TRUTH_STATUS_v2.md.
87 lines
3.9 KiB
Markdown
87 lines
3.9 KiB
Markdown
# muh Pipeline Status — Ground Truth
|
||
|
||
**Last verified**: 2026-08-07T01:45:31Z by automated analysis
|
||
|
||
## Architecture Summary
|
||
|
||
```
|
||
CCCL policy_selector(compute_capability) → ReducePolicy{threads, items, vec, algo, load_mod}
|
||
↕ mirrors
|
||
muh policy_selector(hardware_capability) → same struct types, BI-V100 values
|
||
↕ gen_patch.py extracts bi100_* values
|
||
vllm patch_ops.sh → full-file Python replacements with tuning values baked in
|
||
```
|
||
|
||
## Injection Reality
|
||
|
||
### What gen_patch.py THINKS (csrc/*.cu — DEAD)
|
||
```
|
||
tuning_reduce.cuh → csrc/attention/attention_kernels.cu NUM_THREADS ← NO .cu SOURCE
|
||
tuning_scan.cuh → csrc/attention/paged_attention_v1.cu SCAN_BLOCK_SIZE ← NO .cu SOURCE
|
||
tuning_topk.cuh → csrc/sampling/sampling_kernels.cu SAMPLING_BLOCK_SIZE ← NO .cu SOURCE
|
||
```
|
||
|
||
### What ACTUALLY happens (Python runtime — ALIVE)
|
||
```
|
||
_custom_ops.py → SMEM 49152 (was 32768) ← DEPLOYED ✓
|
||
paged_attn.py → _PARTITION_SIZE=512 ← DEPLOYED ✓ (V2 partition, NOT CTA tile)
|
||
xformers.py → _Q_CHUNK=256, sdpa_fallback ← DEPLOYED ✓
|
||
sampler.py → torch.topk fast path ← DEPLOYED ✓
|
||
prefix_prefill.py → Triton BLOCK_M/N/warps ← DEPLOYED ✓ (but Triton not available)
|
||
computility-run.yaml → vllm server args ← DEPLOYED ✓
|
||
```
|
||
|
||
### The Gap
|
||
muh C++ headers define precise per-type-per-op tuning values (14 reduce structs, 22 scan structs).
|
||
But the vllm engine on BI-V100 runs ixformer .so (precompiled, not tunable) + Python fallbacks.
|
||
The C++ headers' values cannot be injected into the precompiled .so.
|
||
They CAN inform:
|
||
1. Python fallback implementations (paged_attn.py, xformers.py) — tile sizes, chunk sizes
|
||
2. Triton JIT configs — if Triton were available (it's not on BI-V100 base image)
|
||
3. Future EngineX releases that expose tuning knobs
|
||
|
||
## Asset Inventory
|
||
|
||
| Asset | Count | Status |
|
||
|-------|-------|--------|
|
||
| CCCL tuning headers (upstream) | 27 | Complete |
|
||
| muh BI-V100 headers | 27 | Complete (14 reduce + 22 scan + others) |
|
||
| muh schema YAMLs | 27 | Complete |
|
||
| CUB benchmarks | 91 | Synced to NVIDIA/cccl main |
|
||
| CUB tests | 243 | Complete |
|
||
| CUB examples | 18 | Complete |
|
||
| Thrust examples | 60 | Complete |
|
||
| Deployed patches | 15 files | Via patch_ops.sh full replacement |
|
||
| bench_bi100.py search spaces | 5 algos | Defined, needs BI-V100 hardware to run |
|
||
|
||
## Tool Chain Status
|
||
|
||
| Tool | Input | Output | Status |
|
||
|------|-------|--------|--------|
|
||
| parse.py | baseline.muh | JSON config | ✓ Working |
|
||
| gen_patch.py | tuning_*.cuh | Patch report | ⚠ Reports structs but generates 0 patches (injection mapping mismatch) |
|
||
| gen_yaml.py | baseline.muh | computility-run.yaml | ✓ Working |
|
||
| bench_bi100.py | algo+dtype | CCCL-format speedup data | Needs BI-V100 hardware |
|
||
| patch_ops.sh | qwen3_6_scripts/ | Docker vllm patches | ✓ Working |
|
||
| muh_dispatch.py | hw+dtype+head_dim | AttentionConfig | ✓ Working (needs torch) |
|
||
| scale_mem_bound | (threads, items, type_size) | (items, threads) | ✓ CCCL parity verified |
|
||
|
||
## Critical Numbers
|
||
|
||
| Metric | Competition Threshold | Current Status |
|
||
|--------|----------------------|----------------|
|
||
| Functional tests | 50+ pass | 13 items In Progress (all FEA) |
|
||
| Effect deviation | ≤ ±4% | Untested (needs hardware) |
|
||
| Token throughput weighted | ≥ 8000 | Untested |
|
||
| Output TPS weight | 83% (×16.796) | Reduce/scan/topk optimization focus |
|
||
| SMEM limit | 49152 bytes | All 36 scan+reduce structs verified ✓ |
|
||
| SM count | 16 (confirmed) | All headers updated |
|
||
|
||
## Next Actions (Ranked by Competition Impact)
|
||
|
||
1. **Run bench_bi100.py on BI-V100** → get real speedup data for reduce/scan/topk
|
||
2. **Backfill speedup data to muh headers** → replace TBD/theoretical values
|
||
3. **Optimize Python fallback tile sizes** → paged_attn.py, xformers.py Q_CHUNK
|
||
4. **Tune computility-run.yaml** → max-num-seqs, max-batched-tokens, gpu-mem-util
|
||
5. **Enable prefix caching benchmark** → cached_tokens > 0 for repeat prompts
|