fix(precision): guard all corex .so outputs with nan_to_num + reduce max-model-len
MoE kernels: - topk_softmax: add .contiguous() + nan_to_num + re-normalize weights - direct_routed: nan_to_num on w2_reduce output - exact_reduce: nan_to_num on serial_float output GDN kernels: - packed_decode: nan_to_num on core_out BI-V100 CUB may produce non-finite values in fp16 softmax/reduce. These guards prevent garbage propagation without disabling the kernels. max-model-len: 256000 → 131072 (4x32GB BI-V100 OOM prevention) Dockerfile: unchanged (no force push needed)
This commit is contained in:
@@ -8,7 +8,7 @@ command:
|
||||
- --served-model-name
|
||||
- llm
|
||||
- --max-model-len
|
||||
- '256000'
|
||||
- '131072'
|
||||
- --gpu-memory-utilization
|
||||
- '0.95'
|
||||
- --trust-remote-code
|
||||
|
||||
Reference in New Issue
Block a user