EngineX-Hygon/sglang
5
0
Fork 0
Code Issues Pull Requests Actions 7 Projects Releases Wiki Activity

misc: speedup load safetensors (#1319)

Browse Source 
Co-authored-by: ispobock <ISPObaoke@163.com>
This commit is contained in:
Yineng Zhang
2024-09-04 04:29:53 +10:00
committed by GitHub
parent 1e495e0847
commit dc67d97693
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
python/sglang/srt/model_executor/model_runner.py
View File

@@ -162,6 +162,7 @@ class ModelRunner:
return min_per_gpu_memory return min_per_gpu_memory
def load_model(self): def load_model(self):
torch.set_num_threads(1)
logger.info( logger.info(
f"Load weight begin. avail mem={get_available_gpu_memory(self.gpu_id):.2f} GB" f"Load weight begin. avail mem={get_available_gpu_memory(self.gpu_id):.2f} GB"
) )