dylanyunlong/project_6
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

Compare commits

base: dylanyunlong:451bdc8204b9b91bedf352a346bfd9c06684b69b
Branches Tags
dylanyunlong:main
...
compare: dylanyunlong:71644e15302db067c5b8442209455c84730cdfbf
Branches Tags
dylanyunlong:main

3 Commits
451bdc8204 ... 71644e1530

Author SHA1 Message Date
project6-dev
71644e1530 fix: num-gpu-blocks-override 4000→5500 — 4000 blocks can't fit a 70K prompt 
5500 blocks × 16 tokens = 88K token capacity.
Covers max-model-len=80000 + output generation headroom.
 2026-08-13 13:25:23 +00:00
project6-dev
cf7824313f fix(OOM): add --num-gpu-blocks-override=4000 to skip profiling 
Profiling OOM: flash_attn_varlen_func allocates large temp buffers during
profile_run with 8192 dummy tokens. patch_worker_profile_override.py already
has skip logic when num_gpu_blocks_override is set.

4000 blocks × 16 tokens/block = 64K token KV capacity.
With max-model-len=80000 and prefix caching, this is sufficient.
Sub 168 reference: ran 262K context on 0.95 util without override because
base image profiling doesn't use flash_attn_varlen (uses native xformers).
 2026-08-13 13:24:36 +00:00
Claude
8d2f30f065 fix(critical): remove LD_PRELOAD libcccl_allocator.so — crashes service if .so missing 
The CCCL preload allocator build frequently fails (CUB headers conflict
with CoreX CUDA 10.2). When the .so doesn't exist, LD_PRELOAD causes
the Python process to abort immediately, resulting in Connection Refused
for all 881 replay requests (0 score).

This was the root cause of sub 655's failure: service never started.
 2026-08-13 13:15:13 +00:00
1 changed files with 3 additions and 2 deletions
Expand all files Collapse all files

5
computility-run.yaml
View File

@@ -32,6 +32,8 @@ command:
- --enforce-eager - --enforce-eager
- --dtype - --dtype
- half - half
- --num-gpu-blocks-override
- '5500'
env: env:
- name: VLLM_ENGINE_ITERATION_TIMEOUT_S - name: VLLM_ENGINE_ITERATION_TIMEOUT_S
value: 3600 value: 3600
@@ -49,5 +51,4 @@ env:
value: '1' value: '1'
- name: PYTORCH_CUDA_ALLOC_CONF - name: PYTORCH_CUDA_ALLOC_CONF
value: max_split_size_mb:512 value: max_split_size_mb:512
- name: LD_PRELOAD
value: /workspace/qwen3_6_scripts/cccl_preload/libcccl_allocator.so