dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

fix(CRITICAL): max_model_len 256000→80000 + topk_softmax silent fallback + deploy _custom_ops

Browse Source 
Three fixes from comp 168 log analysis:

1. computility-run.yaml: max_model_len 256000→80000
   - 256000 causes OOM (comp 168: CUDA OOM at 31.72GB)
   - BI-V100 KV cache capacity ~88112 blocks

2. _custom_ops.py: topk_softmax silent fallback
   - ixf_F.vllm_moe_topk_softmax missing in base image
   - New: try ixformer._C.topk_softmax → silent PyTorch fallback
   - Eliminates 500+ ERROR lines from docker log

3. patch_ops.sh: deploy _custom_ops.py
   - Previously excluded; now deployed to fix topk_softmax issue

Ref: upstream_ref/xllm/core/kernels/ilu/ixformer.h
This commit is contained in:
project6-dev
2026-08-10 06:56:23 +00:00
parent 4a91c31ffc
commit af08856d5c
3 changed files with 37 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
computility-run.yaml
View File

@@ -8,7 +8,7 @@ command:
- --served-model-name
- llm
- --max-model-len
- '256000'
- '80000'
- --gpu-memory-utilization
- '0.95'
- --trust-remote-code