under test, not sure no errors
This commit is contained in:
42
docs/server_recon/machine_profile.md
Normal file
42
docs/server_recon/machine_profile.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# Competition Server Profile
|
||||
**Captured**: 2026-08-01
|
||||
|
||||
## Hardware
|
||||
- **GPU**: 4× Iluvatar BI-V100 32GB HBM each (128GB total)
|
||||
- Clock: SM 1500MHz / Mem 1200MHz
|
||||
- Driver: 3.2.1, COREX 10.2
|
||||
- Power: 250W TDP per card
|
||||
- **CPU**: Intel Xeon Gold 6530
|
||||
- **RAM**: 503GB DDR
|
||||
- **Disk**: 3.5TB overlay, 100GB JuiceFS (public-storage)
|
||||
|
||||
## Software
|
||||
- **OS**: Ubuntu 20.04.6 LTS, kernel 5.15.0-119
|
||||
- **COREX**: 3.2.3 at `/usr/local/corex`
|
||||
- **torch**: 2.1.0+corex.3.2.3
|
||||
- **vllm**: 0.6.3+corex.3.2.3
|
||||
- **transformers**: 4.51.3
|
||||
|
||||
## Model
|
||||
- **Path**: `/root/public-storage/models/Qwen/Qwen3.6-35B-A3B/`
|
||||
- **Name**: Qwen3.6-35B-A3B (MoE, 35B total, 3B active)
|
||||
- **Note**: 4 cards × 32GB = 128GB total, model fits
|
||||
|
||||
## Key Paths
|
||||
- `/root/llm-infer/` — benchmark scripts, README
|
||||
- `/root/public-storage/models/Qwen/` — model weights
|
||||
- `/root/apps/llm-modelzoo/benchmark/vllm/` — benchmark tools
|
||||
- `/share/fshare/common/models/` — shared model storage (NFS)
|
||||
|
||||
## Benchmark Tools
|
||||
- `benchmark_server_v0.5.0.py` — automated server benchmark
|
||||
- Sweeps: max-num-seqs=[128,256] × num-prompts=[1,128] × input=[128,1024] × output=[128,1024]
|
||||
- `benchmark_server_v0.5.0.sh` — launches vllm server + benchmark client
|
||||
- Sets `NCCL_FORCESYNC_DISABLE=1`
|
||||
- Auto-cleanup of vllm processes
|
||||
- `benchmark_serving_tokens.py` — online serving benchmark client
|
||||
|
||||
## Scoring Formula
|
||||
`Output TPS × 16.796 + Input TPS × 2.799 + Cache TPS × 0.56`
|
||||
- Threshold: ≥ 8000 weighted score
|
||||
- Output TPS weight: 83% of total score
|
||||
45
docs/server_recon/qwen36_bootstrap_issue.md
Normal file
45
docs/server_recon/qwen36_bootstrap_issue.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# Qwen3.6-35B-A3B Bootstrap Issue
|
||||
|
||||
## Problem
|
||||
vllm 0.6.3+corex.3.2.3 does not recognize `qwen3_5_moe` model type.
|
||||
|
||||
```
|
||||
ValueError: The checkpoint you are trying to load has model type `qwen3_5_moe`
|
||||
but Transformers does not recognize this architecture.
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
- Model `config.json` specifies `"model_type": "qwen3_5_moe"` and `"architectures": ["Qwen3_5MoeForCausalLM"]`
|
||||
- Server transformers version: 4.51.3 (needs ≥ 4.57.1)
|
||||
- Server vllm version: 0.6.3+corex.3.2.3
|
||||
|
||||
## Model Architecture (from config.json)
|
||||
- **Type**: Qwen3_5MoeForCausalLM (MoE with linear attention)
|
||||
- **Total params**: ~35B
|
||||
- **Active params per token**: ~3B (8 of 256 experts)
|
||||
- **Hidden size**: 2048
|
||||
- **Layers**: 40 (30 linear_attention + 10 full_attention, every 4th is full)
|
||||
- **Experts**: 256 total, 8 per token
|
||||
- **Expert intermediate**: 512
|
||||
- **Shared expert intermediate**: 512
|
||||
- **Head dim**: 256
|
||||
- **KV heads**: 2 (GQA ratio 8:1)
|
||||
- **Max position**: 262144
|
||||
- **Vocab**: 248320
|
||||
- **Precision**: bfloat16
|
||||
- **Linear attention**: conv kernel dim=4, 16 key heads (dim128), 32 value heads (dim128)
|
||||
- **MTP**: 1 hidden layer (multi-token prediction)
|
||||
- **Vision**: yes (patch16, depth27, hidden1152)
|
||||
|
||||
## Key Architecture Features
|
||||
1. **Hybrid attention**: 3 linear_attention + 1 full_attention pattern (30+10=40 layers)
|
||||
2. **MoE**: 256 experts, top-8 routing = very sparse
|
||||
3. **Linear attention with conv**: NOT standard transformer — uses conv kernel dim=4
|
||||
4. **Multi-token prediction (MTP)**: 1 extra hidden layer for speculative prediction
|
||||
5. **Multimodal**: has vision encoder (but competition likely tests text only)
|
||||
|
||||
## Solution Paths
|
||||
1. **EngineX route**: Check if the competition's enginex-vllm package already supports this model
|
||||
- The repo has `enginex-vllm-bi100-qwen36-main.zip` (96MB) — THIS is likely the answer
|
||||
2. **Upgrade transformers**: `pip install transformers>=4.57.1` (may break corex compatibility)
|
||||
3. **Custom model registration**: Register Qwen3_5MoeForCausalLM in vllm's model registry
|
||||
Reference in New Issue
Block a user