# System Design

## Architecture

```
Docker Image (FROM bi100-3.2.3-x86-ubuntu20.04-py3.10-poc-llm-infer:v1.2.3)
│
├── /workspace/
│   ├── computility-run.yaml          # vLLM launch args
│   └── qwen3_6_scripts/
│       ├── patch_ops.sh              # Build-time: deploy all patches
│       ├── precompile_gdn.py         # Build-time: compile .cu → .so
│       ├── qwen3_5.py                # Model: GDN + MoE + Attention
│       ├── flash_qla_sm70/
│       │   ├── csrc/gdn_forward.cu   # SM70 fused GDN CUDA kernel (1919 lines)
│       │   ├── fused_fwd.py          # Python wrapper, loads .so
│       │   ├── naive_gdn.py          # PyTorch reference fallback
│       │   └── __init__.py
│       ├── serving_chat.py           # OpenAI API handler
│       ├── protocol.py               # Request/response models
│       ├── chat_utils.py             # Tool call handling
│       ├── api_server.py             # FastAPI app
│       ├── cli_args.py               # CLI argument extensions
│       ├── registry.py               # Model registry (adds Qwen3_5)
│       ├── paged_attn.py             # Paged attention PyTorch fallback
│       ├── mamba_cache.py            # GDN state cache manager
│       ├── sequence.py               # Token count fix
│       ├── scheduler.py              # Chunked prefill fix
│       ├── xformers.py               # SDPA fallback patches
│       ├── patch_xformers_*.py       # xformers monkey-patches
│       ├── patch_model_runner.py     # prefix_cache_hit fix
│       ├── patch_numerical_stability.py
│       ├── patch_transformers_qwen3_5.py
│       ├── patch_vllm_tool_parser.py
│       ├── qwen3coder_tool_parser.py # Tool call parser
│       └── tool_parsers_init.py
│
├── /usr/local/corex/                 # Base image SDK
│   ├── lib64/
│   │   ├── libcublas.so
│   │   ├── libcudart.so
│   │   ├── libcudnn.so
│   │   ├── libcutlass.so
│   │   ├── libixattn.so
│   │   └── clang/16/                # CUDA compiler
│   └── lib/python3/dist-packages/
│       ├── torch/
│       ├── vllm/                     # Base vLLM 0.6.3
│       └── ixformer/                 # Hardware acceleration ops
│
└── /model/                           # Qwen3.5-27B weights (16 shards)
```

## Build Pipeline

```
Dockerfile
    │
    ├── COPY qwen3_6_scripts/ → /workspace/qwen3_6_scripts/
    ├── COPY computility-run.yaml → /workspace/
    │
    └── RUN patch_ops.sh
         │
         ├── 1. Find vllm install path ($VLLM)
         ├── 2. apt install ninja-build
         ├── 3. pip install transformers==4.55.3
         ├── 4. Shell probe (ls corex .so, ls corex .py, ls native qwen3_5.py)
         ├── 5. Deploy qwen3_5.py → $VLLM/model_executor/models/
         ├── 6. Deploy registry.py (add Qwen3_5ForCausalLM)
         ├── 7. Deploy flash_qla_sm70/ → $VLLM/model_executor/models/
         ├── 8. Run precompile_gdn.py → flash_qla_sm70/build/*.so
         ├── 9. Deploy paged_attn.py, mamba_cache.py, sequence.py, scheduler.py
         ├── 10. Deploy xformers patches (monkey-patch SDPA)
         ├── 11. Deploy tool parser + reasoning parser
         ├── 12. Deploy serving_chat.py, protocol.py, api_server.py, chat_utils.py
         └── 13. Mirror all to $VLLM2 if second vllm install exists
```

## Runtime Data Flow

```
HTTP Request (OpenAI format)
    │
    ▼
api_server.py → serving_chat.py
    │
    ├── protocol.py: validate request, handle max_completion_tokens
    ├── chat_utils.py: format messages, handle tool_calls
    │
    ▼
vLLM AsyncLLMEngine
    │
    ├── scheduler.py → batch requests
    ├── model_runner.py → execute_model()
    │
    ▼
qwen3_5.py: Qwen3_5ForCausalLM.forward()
    │
    ├── Embedding → token embeddings
    │
    ├── 64 Decoder Layers (loop):
    │   │
    │   ├── Layers with GatedDeltaNet (4 of 36 attention layers):
    │   │   │
    │   │   ├── Projections: in_proj_qkv, in_proj_z, in_proj_b, in_proj_a
    │   │   ├── Conv1d (depthwise causal)
    │   │   ├── L2 normalize q, k
    │   │   │
    │   │   ├── DISPATCH:
    │   │   │   ├── 1st: CoreX fused kernel (if corex_gdn.py packaged)
    │   │   │   ├── 2nd: FlashQLA SM70 kernel (prefill only, gdn_forward.cu)
    │   │   │   └── 3rd: PyTorch _torch_chunk_gated_delta_rule (with NaN clamp)
    │   │   │
    │   │   ├── Gated RMSNorm
    │   │   └── out_proj
    │   │
    │   ├── Layers with Full Attention (32 of 36):
    │   │   └── xformers SDPA (patched fallback for BI-V100)
    │   │
    │   ├── MoE (all 36 layers):
    │   │   ├── Gate → router logits → topk
    │   │   ├── DISPATCH:
    │   │   │   ├── 1st: CoreX fused MoE (if corex_moe.py packaged)
    │   │   │   └── 2nd: PyTorch loop over experts
    │   │   ├── Shared expert (with sigmoid gate)
    │   │   └── All-reduce (TP)
    │   │
    │   └── RMSNorm (pre/post)
    │
    ├── Final RMSNorm
    ├── LM Head → logits
    └── Sampler → tokens
```

## GDN Kernel Dispatch Detail

```
GatedDeltaNet.forward(hidden_states, attn_metadata, conv_state, temporal_state)
    │
    ├── is_prefill? (attn_metadata.num_prefill_tokens > 0)
    │   │
    │   ├── YES (prefill):
    │   │   ├── Try FlashQLA SM70:
    │   │   │   ├── Project q,k,v,gate,beta
    │   │   │   ├── Conv1d
    │   │   │   ├── L2norm
    │   │   │   ├── Reshape to [1, L, H, 128]
    │   │   │   ├── chunk_gated_delta_rule_fwd_sm70(q,k,v,g,beta,state)
    │   │   │   │   └── gdn_forward.cu → flash_qla_sm70_gdn_strided.so
    │   │   │   ├── Update temporal_state
    │   │   │   ├── Gated RMSNorm + out_proj
    │   │   │   └── Return
    │   │   │
    │   │   └── Fallback: _torch_chunk_gated_delta_rule (PyTorch, chunked)
    │   │
    │   └── NO (decode):
    │       └── PyTorch single-step recurrent update
    │           ├── Conv1d state update
    │           ├── temporal_state decay + delta write
    │           ├── Query @ state → output
    │           └── Return
    │
    └── Both paths end with: Gated RMSNorm → out_proj → all_reduce
```

## computility-run.yaml Key Args

```yaml
max_model_len: 80000        # Must be < KV cache capacity (88112)
gpu_memory_utilization: 0.9
max_num_seqs: 1
tensor_parallel_size: 4
enforce_eager: true          # No CUDA graphs (BI-V100 compatibility)
enable_prefix_caching: true
max_seq_len_to_capture: 8192
tool_call_parser: qwen3_coder
reasoning_parser: qwen3
```

## File Dependencies

```
qwen3_5.py imports:
    ├── vllm.attention (Attention, AttentionMetadata)
    ├── vllm.model_executor.layers.* (linear, norm, sampler, etc.)
    ├── vllm.model_executor.models.mamba_cache (MambaCacheManager)
    ├── vllm.model_executor.models.flash_qla_sm70 (SM70 kernel)
    ├── ixformer (optional, hardware-accelerated ops)
    └── vllm.model_executor.models.corex_gdn (optional, if packaged)

flash_qla_sm70/fused_fwd.py imports:
    ├── torch.utils.cpp_extension.load (JIT compile .cu → .so)
    └── gdn_forward.cu (CUDA source, compiled to .so)

serving_chat.py imports:
    ├── vllm.entrypoints.openai.protocol (request validation)
    ├── vllm.entrypoints.chat_utils
    └── vllm engine client
```

## Scoring Modules (competition)

```
Module 1: functional_acceptance (52 tests)
    ├── d01-d10: basic, stream, tools, reasoning, multimodal, thinking
    ├── t1-t16: auth, n=2, max_tokens, stop, system, temperature, etc.
    └── 4 skipped: d08, t11a, t11b, t16b

Module 2: case_truncation
    └── Output truncation correctness

Module 3: replay_tencent
    └── 881 real requests, throughput scoring
    └── Output TPS weight: 83%

Module 4: opencompass
    └── Model quality benchmarks
```