fix: lower gpu-memory-utilization to 0.85 — prevent profiling OOM
CCCL CachingDeviceAllocator reserves some memory for its bin cache. With 0.90 utilization + max-model-len=80000, profiling stage OOMs. 0.85 leaves ~1.6GB headroom per GPU for profiling + allocator cache.
This commit is contained in:
@@ -10,7 +10,7 @@ command:
|
||||
- --max-model-len
|
||||
- '80000'
|
||||
- --gpu-memory-utilization
|
||||
- '0.90'
|
||||
- '0.85'
|
||||
- --trust-remote-code
|
||||
- -tp
|
||||
- '4'
|
||||
|
||||
Reference in New Issue
Block a user