From c69291eefc810489f5f31d3594308d6eafa0d5f9 Mon Sep 17 00:00:00 2001 From: bazingazhou233-hub Date: Sat, 14 Mar 2026 22:41:02 +0800 Subject: [PATCH] [Doc] Add USE_MODELSCOPE_HUB=0 to lm-eval guide (#7279) ## Summary - Add `USE_MODELSCOPE_HUB=0` to both Online and Offline lm-eval sections - Add explanatory notes about Docker containers launching with `VLLM_USE_MODELSCOPE=True` The Docker containers set `VLLM_USE_MODELSCOPE=True`, which causes lm-eval to download datasets from ModelScope instead of HuggingFace, resulting in "Repo not exists" errors. Setting `USE_MODELSCOPE_HUB=0` disables this behavior. Fixes #607 - vLLM version: v0.17.0 - vLLM main: https://github.com/vllm-project/vllm/commit/4034c3d32e30d01639459edd3ab486f56993876d Signed-off-by: bazingazhou233-hub Co-authored-by: bazingazhou233-hub --- .../developer_guide/evaluation/using_lm_eval.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/source/developer_guide/evaluation/using_lm_eval.md b/docs/source/developer_guide/evaluation/using_lm_eval.md index 2e5453c3..fad9c26d 100644 --- a/docs/source/developer_guide/evaluation/using_lm_eval.md +++ b/docs/source/developer_guide/evaluation/using_lm_eval.md @@ -106,9 +106,17 @@ Install lm-eval in the container: ```bash export HF_ENDPOINT="https://hf-mirror.com" +export USE_MODELSCOPE_HUB=0 pip install lm-eval[api] ``` +:::{note} +The Docker container is launched with `VLLM_USE_MODELSCOPE=True`, which may +cause lm-eval to download datasets from ModelScope instead of HuggingFace. +Setting `USE_MODELSCOPE_HUB=0` disables this behavior so that lm-eval can +fetch datasets from HuggingFace correctly. +::: + Run the following command: ```shell @@ -170,9 +178,17 @@ Install lm-eval in the container: ```bash export HF_ENDPOINT="https://hf-mirror.com" +export USE_MODELSCOPE_HUB=0 pip install lm-eval ``` +:::{note} +The Docker container is launched with `VLLM_USE_MODELSCOPE=True`, which may +cause lm-eval to download datasets from ModelScope instead of HuggingFace. +Setting `USE_MODELSCOPE_HUB=0` disables this behavior so that lm-eval can +fetch datasets from HuggingFace correctly. +::: + Run the following command: ```shell