8.4 KiB
8.4 KiB
System Design
Architecture
Docker Image (FROM bi100-3.2.3-x86-ubuntu20.04-py3.10-poc-llm-infer:v1.2.3)
│
├── /workspace/
│ ├── computility-run.yaml # vLLM launch args
│ └── qwen3_6_scripts/
│ ├── patch_ops.sh # Build-time: deploy all patches
│ ├── precompile_gdn.py # Build-time: compile .cu → .so
│ ├── qwen3_5.py # Model: GDN + MoE + Attention
│ ├── flash_qla_sm70/
│ │ ├── csrc/gdn_forward.cu # SM70 fused GDN CUDA kernel (1919 lines)
│ │ ├── fused_fwd.py # Python wrapper, loads .so
│ │ ├── naive_gdn.py # PyTorch reference fallback
│ │ └── __init__.py
│ ├── serving_chat.py # OpenAI API handler
│ ├── protocol.py # Request/response models
│ ├── chat_utils.py # Tool call handling
│ ├── api_server.py # FastAPI app
│ ├── cli_args.py # CLI argument extensions
│ ├── registry.py # Model registry (adds Qwen3_5)
│ ├── paged_attn.py # Paged attention PyTorch fallback
│ ├── mamba_cache.py # GDN state cache manager
│ ├── sequence.py # Token count fix
│ ├── scheduler.py # Chunked prefill fix
│ ├── xformers.py # SDPA fallback patches
│ ├── patch_xformers_*.py # xformers monkey-patches
│ ├── patch_model_runner.py # prefix_cache_hit fix
│ ├── patch_numerical_stability.py
│ ├── patch_transformers_qwen3_5.py
│ ├── patch_vllm_tool_parser.py
│ ├── qwen3coder_tool_parser.py # Tool call parser
│ └── tool_parsers_init.py
│
├── /usr/local/corex/ # Base image SDK
│ ├── lib64/
│ │ ├── libcublas.so
│ │ ├── libcudart.so
│ │ ├── libcudnn.so
│ │ ├── libcutlass.so
│ │ ├── libixattn.so
│ │ └── clang/16/ # CUDA compiler
│ └── lib/python3/dist-packages/
│ ├── torch/
│ ├── vllm/ # Base vLLM 0.6.3
│ └── ixformer/ # Hardware acceleration ops
│
└── /model/ # Qwen3.5-27B weights (16 shards)
Build Pipeline
Dockerfile
│
├── COPY qwen3_6_scripts/ → /workspace/qwen3_6_scripts/
├── COPY computility-run.yaml → /workspace/
│
└── RUN patch_ops.sh
│
├── 1. Find vllm install path ($VLLM)
├── 2. apt install ninja-build
├── 3. pip install transformers==4.55.3
├── 4. Shell probe (ls corex .so, ls corex .py, ls native qwen3_5.py)
├── 5. Deploy qwen3_5.py → $VLLM/model_executor/models/
├── 6. Deploy registry.py (add Qwen3_5ForCausalLM)
├── 7. Deploy flash_qla_sm70/ → $VLLM/model_executor/models/
├── 8. Run precompile_gdn.py → flash_qla_sm70/build/*.so
├── 9. Deploy paged_attn.py, mamba_cache.py, sequence.py, scheduler.py
├── 10. Deploy xformers patches (monkey-patch SDPA)
├── 11. Deploy tool parser + reasoning parser
├── 12. Deploy serving_chat.py, protocol.py, api_server.py, chat_utils.py
└── 13. Mirror all to $VLLM2 if second vllm install exists
Runtime Data Flow
HTTP Request (OpenAI format)
│
▼
api_server.py → serving_chat.py
│
├── protocol.py: validate request, handle max_completion_tokens
├── chat_utils.py: format messages, handle tool_calls
│
▼
vLLM AsyncLLMEngine
│
├── scheduler.py → batch requests
├── model_runner.py → execute_model()
│
▼
qwen3_5.py: Qwen3_5ForCausalLM.forward()
│
├── Embedding → token embeddings
│
├── 64 Decoder Layers (loop):
│ │
│ ├── Layers with GatedDeltaNet (4 of 36 attention layers):
│ │ │
│ │ ├── Projections: in_proj_qkv, in_proj_z, in_proj_b, in_proj_a
│ │ ├── Conv1d (depthwise causal)
│ │ ├── L2 normalize q, k
│ │ │
│ │ ├── DISPATCH:
│ │ │ ├── 1st: CoreX fused kernel (if corex_gdn.py packaged)
│ │ │ ├── 2nd: FlashQLA SM70 kernel (prefill only, gdn_forward.cu)
│ │ │ └── 3rd: PyTorch _torch_chunk_gated_delta_rule (with NaN clamp)
│ │ │
│ │ ├── Gated RMSNorm
│ │ └── out_proj
│ │
│ ├── Layers with Full Attention (32 of 36):
│ │ └── xformers SDPA (patched fallback for BI-V100)
│ │
│ ├── MoE (all 36 layers):
│ │ ├── Gate → router logits → topk
│ │ ├── DISPATCH:
│ │ │ ├── 1st: CoreX fused MoE (if corex_moe.py packaged)
│ │ │ └── 2nd: PyTorch loop over experts
│ │ ├── Shared expert (with sigmoid gate)
│ │ └── All-reduce (TP)
│ │
│ └── RMSNorm (pre/post)
│
├── Final RMSNorm
├── LM Head → logits
└── Sampler → tokens
GDN Kernel Dispatch Detail
GatedDeltaNet.forward(hidden_states, attn_metadata, conv_state, temporal_state)
│
├── is_prefill? (attn_metadata.num_prefill_tokens > 0)
│ │
│ ├── YES (prefill):
│ │ ├── Try FlashQLA SM70:
│ │ │ ├── Project q,k,v,gate,beta
│ │ │ ├── Conv1d
│ │ │ ├── L2norm
│ │ │ ├── Reshape to [1, L, H, 128]
│ │ │ ├── chunk_gated_delta_rule_fwd_sm70(q,k,v,g,beta,state)
│ │ │ │ └── gdn_forward.cu → flash_qla_sm70_gdn_strided.so
│ │ │ ├── Update temporal_state
│ │ │ ├── Gated RMSNorm + out_proj
│ │ │ └── Return
│ │ │
│ │ └── Fallback: _torch_chunk_gated_delta_rule (PyTorch, chunked)
│ │
│ └── NO (decode):
│ └── PyTorch single-step recurrent update
│ ├── Conv1d state update
│ ├── temporal_state decay + delta write
│ ├── Query @ state → output
│ └── Return
│
└── Both paths end with: Gated RMSNorm → out_proj → all_reduce
computility-run.yaml Key Args
max_model_len: 80000 # Must be < KV cache capacity (88112)
gpu_memory_utilization: 0.9
max_num_seqs: 1
tensor_parallel_size: 4
enforce_eager: true # No CUDA graphs (BI-V100 compatibility)
enable_prefix_caching: true
max_seq_len_to_capture: 8192
tool_call_parser: qwen3_coder
reasoning_parser: qwen3
File Dependencies
qwen3_5.py imports:
├── vllm.attention (Attention, AttentionMetadata)
├── vllm.model_executor.layers.* (linear, norm, sampler, etc.)
├── vllm.model_executor.models.mamba_cache (MambaCacheManager)
├── vllm.model_executor.models.flash_qla_sm70 (SM70 kernel)
├── ixformer (optional, hardware-accelerated ops)
└── vllm.model_executor.models.corex_gdn (optional, if packaged)
flash_qla_sm70/fused_fwd.py imports:
├── torch.utils.cpp_extension.load (JIT compile .cu → .so)
└── gdn_forward.cu (CUDA source, compiled to .so)
serving_chat.py imports:
├── vllm.entrypoints.openai.protocol (request validation)
├── vllm.entrypoints.chat_utils
└── vllm engine client
Scoring Modules (competition)
Module 1: functional_acceptance (52 tests)
├── d01-d10: basic, stream, tools, reasoning, multimodal, thinking
├── t1-t16: auth, n=2, max_tokens, stop, system, temperature, etc.
└── 4 skipped: d08, t11a, t11b, t16b
Module 2: case_truncation
└── Output truncation correctness
Module 3: replay_tencent
└── 881 real requests, throughput scoring
└── Output TPS weight: 83%
Module 4: opencompass
└── Model quality benchmarks