Switch to bridge networking with fixed container port 80 and served name llm
- run.sh: bridge + -p ${HOST_PORT:-24396}:80, health check uses host port
- entrypoint.sh: PORT defaults to 80, SERVED_NAME defaults to llm
detected port/served name become log hints only
- verify.sh: read host port from port bindings, expect model name llm
- docker-compose.yml: ports mapping instead of network_mode host
- README: update tables, commands and rationale
This commit is contained in:
52
README.md
52
README.md
@@ -2,10 +2,12 @@
|
|||||||
|
|
||||||
一个镜像同时支持两个 Gemma 4 家族模型,启动时读模型 `config.json` 自动选择运行参数:
|
一个镜像同时支持两个 Gemma 4 家族模型,启动时读模型 `config.json` 自动选择运行参数:
|
||||||
|
|
||||||
| 模型 | 类型 | 权重 | TP | 卡数 | 端口 | served name | eager | 显存 | seqs |
|
| 模型 | 类型 | 权重 | TP | 卡数 | eager | 显存 | seqs |
|
||||||
|---|---|---:|---:|---:|---:|---|---:|---:|---:|
|
|---|---|---:|---:|---:|---:|---:|---:|
|
||||||
| `gemma-4-31B-it` | dense,60 层 | 58.25 GiB | **4** | 4 | **8095** | `gemma-4-31B-it` | 0 | 0.90 | 4 |
|
| `gemma-4-31B-it` | dense,60 层 | 58.25 GiB | **4** | 4 | 0 | 0.90 | 4 |
|
||||||
| `GEV-26B-Decide` | MoE,30 层 / 128 experts | 48.07 GiB | **2** | 2 | **8094** | `GEV-26B-Decide` | 1 | 0.95 | 2 |
|
| `GEV-26B-Decide` | MoE,30 层 / 128 experts | 48.07 GiB | **2** | 2 | 1 | 0.95 | 2 |
|
||||||
|
|
||||||
|
两个变体共用:容器内端口 **80**、模型名 **`llm`**、宿主端口 **24396**(由 `-p` 决定)。
|
||||||
|
|
||||||
> 本仓库由原先各自独立的 `gemma4-31b-deploy` 和 `gev-26b-decide-deploy` 两套目录合并而来,
|
> 本仓库由原先各自独立的 `gemma4-31b-deploy` 和 `gev-26b-decide-deploy` 两套目录合并而来,
|
||||||
> 合并依据见文末「为什么可以合并」。
|
> 合并依据见文末「为什么可以合并」。
|
||||||
@@ -87,8 +89,8 @@ experts = num_experts / num_local_experts / n_routed_experts
|
|||||||
|
|
||||||
| 条件 | 变体 | 运行参数 |
|
| 条件 | 变体 | 运行参数 |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `experts > 1` | GEV-26B-Decide(MoE) | TP2 / 8094 / eager1 / 0.95 / 2 seqs |
|
| `experts > 1` | GEV-26B-Decide(MoE) | TP2 / eager1 / 0.95 / 2 seqs |
|
||||||
| `experts` 缺失或 `== 1` | Gemma-4-31B(dense) | TP4 / 8095 / eager0 / 0.90 / 4 seqs |
|
| `experts` 缺失或 `== 1` | Gemma-4-31B(dense) | TP4 / eager0 / 0.90 / 4 seqs |
|
||||||
| 无 experts 且目录名含 `gev` | GEV | 同上 |
|
| 无 experts 且目录名含 `gev` | GEV | 同上 |
|
||||||
| 无 experts 且目录名含 `gemma` | Gemma-4-31B | 同上 |
|
| 无 experts 且目录名含 `gemma` | Gemma-4-31B | 同上 |
|
||||||
| 其它 / 架构不是 Gemma4 | 直接报错 | 不启动 |
|
| 其它 / 架构不是 Gemma4 | 直接报错 | 不启动 |
|
||||||
@@ -122,8 +124,8 @@ TP=4 ./run.sh .../GEV-26B-Decide # 704/4=176,k % 32 != 0
|
|||||||
$ python3 detect_model.py /mnt/disk0/models/autotrust/GEV-26B-Decide
|
$ python3 detect_model.py /mnt/disk0/models/autotrust/GEV-26B-Decide
|
||||||
DETECTED_MODEL_KIND=gev-26b-decide
|
DETECTED_MODEL_KIND=gev-26b-decide
|
||||||
DETECTED_TP=2
|
DETECTED_TP=2
|
||||||
DETECTED_PORT=8094
|
DETECTED_PORT=8094 # 仅提示,不决定实际端口
|
||||||
DETECTED_SERVED_NAME=GEV-26B-Decide
|
DETECTED_SERVED_NAME=GEV-26B-Decide # 仅提示,模型名固定 llm
|
||||||
DETECTED_GPU_COUNT=2
|
DETECTED_GPU_COUNT=2
|
||||||
DETECTED_MAX_NUM_SEQS=2
|
DETECTED_MAX_NUM_SEQS=2
|
||||||
DETECTED_GPU_MEM_UTIL=0.95
|
DETECTED_GPU_MEM_UTIL=0.95
|
||||||
@@ -147,7 +149,7 @@ python3 detect_model.py <dir> --tp 4 # 按 TP4 校验可行性
|
|||||||
GPUS=12,13,14,15 ./run.sh /mnt/disk0/models/google/gemma-4-31B-it
|
GPUS=12,13,14,15 ./run.sh /mnt/disk0/models/google/gemma-4-31B-it
|
||||||
|
|
||||||
# 换端口和模型名
|
# 换端口和模型名
|
||||||
PORT=9001 SERVED_NAME=my-gemma4 ./run.sh /mnt/disk0/models/google/gemma-4-31B-it
|
HOST_PORT=24397 ./run.sh /mnt/disk0/models/google/gemma-4-31B-it # 换宿主端口
|
||||||
|
|
||||||
# 加 vLLM 参数
|
# 加 vLLM 参数
|
||||||
EXTRA_ARGS="--max-model-len 16384" ./run.sh /mnt/disk0/models/google/gemma-4-31B-it
|
EXTRA_ARGS="--max-model-len 16384" ./run.sh /mnt/disk0/models/google/gemma-4-31B-it
|
||||||
@@ -183,7 +185,6 @@ GEMMA4_SKIP_TP_CHECK
|
|||||||
| 参数 | Gemma-4-31B | GEV-26B-Decide |
|
| 参数 | Gemma-4-31B | GEV-26B-Decide |
|
||||||
|---|---:|---:|
|
|---|---:|---:|
|
||||||
| TP | 4 | 2 |
|
| TP | 4 | 2 |
|
||||||
| 端口 | 8095 | 8094 |
|
|
||||||
| eager | 0 | 1 |
|
| eager | 0 | 1 |
|
||||||
| 显存 | 0.90 | 0.95 |
|
| 显存 | 0.90 | 0.95 |
|
||||||
| seqs | 4 | 2 |
|
| seqs | 4 | 2 |
|
||||||
@@ -195,8 +196,10 @@ GEMMA4_SKIP_TP_CHECK
|
|||||||
docker run ... \
|
docker run ... \
|
||||||
--entrypoint /usr/local/bin/gemma4-entrypoint.sh \
|
--entrypoint /usr/local/bin/gemma4-entrypoint.sh \
|
||||||
--workdir /opt/gemma4 \
|
--workdir /opt/gemma4 \
|
||||||
--expose 8095 \
|
-p 24396:80 \
|
||||||
-e MODEL_PATH=/models/gemma4 \
|
-e MODEL_PATH=/models/gemma4 \
|
||||||
|
-e PORT=80 \
|
||||||
|
-e SERVED_NAME=llm \
|
||||||
-e AUTO_DETECT=1 \
|
-e AUTO_DETECT=1 \
|
||||||
<image>
|
<image>
|
||||||
```
|
```
|
||||||
@@ -242,7 +245,7 @@ vLLM 0.17.0+corex.20260612070235
|
|||||||
./verify.sh --model-dir /mnt/disk0/models/google/gemma-4-31B-it --with-reasoning
|
./verify.sh --model-dir /mnt/disk0/models/google/gemma-4-31B-it --with-reasoning
|
||||||
|
|
||||||
# 直接指定服务地址
|
# 直接指定服务地址
|
||||||
./verify.sh --base-url http://127.0.0.1:8095 --served-name gemma-4-31B-it
|
./verify.sh --base-url http://127.0.0.1:24396 --served-name llm
|
||||||
```
|
```
|
||||||
|
|
||||||
检查项:`/health`、`/v1/models`、`/v1/chat/completions`,可选推理题和图片识别。
|
检查项:`/health`、`/v1/models`、`/v1/chat/completions`,可选推理题和图片识别。
|
||||||
@@ -373,47 +376,54 @@ CI 的 `naming to` 行与本表 digest 一致,可作为复核依据。
|
|||||||
|
|
||||||
镜像不声明 `ENTRYPOINT`,`--entrypoint` 必须显式给。以下命令可直接粘贴。
|
镜像不声明 `ENTRYPOINT`,`--entrypoint` 必须显式给。以下命令可直接粘贴。
|
||||||
|
|
||||||
**Gemma-4-31B-it(TP4,4 卡,8095)**
|
**Gemma-4-31B-it(TP4,4 卡,宿主 24396 → 容器 80)**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run -d --name gemma4-31b \
|
docker run -d --name gemma4-31b \
|
||||||
--entrypoint /usr/local/bin/gemma4-entrypoint.sh \
|
--entrypoint /usr/local/bin/gemma4-entrypoint.sh \
|
||||||
--workdir /opt/gemma4 \
|
--workdir /opt/gemma4 \
|
||||||
--privileged --pid host --network host --ipc host \
|
--privileged --pid host --ipc host \
|
||||||
-v /dev:/dev \
|
-v /dev:/dev \
|
||||||
-v /mnt/disk0/models/google/gemma-4-31B-it:/models/gemma4:ro \
|
-v /mnt/disk0/models/google/gemma-4-31B-it:/models/gemma4:ro \
|
||||||
--ulimit memlock=-1 --ulimit stack=67108864 \
|
--ulimit memlock=-1 --ulimit stack=67108864 \
|
||||||
|
-p 24396:80 \
|
||||||
-e CUDA_VISIBLE_DEVICES=15,14,13,12 \
|
-e CUDA_VISIBLE_DEVICES=15,14,13,12 \
|
||||||
-e MODEL_PATH=/models/gemma4 \
|
-e MODEL_PATH=/models/gemma4 \
|
||||||
|
-e PORT=80 \
|
||||||
|
-e SERVED_NAME=llm \
|
||||||
-e AUTO_DETECT=1 \
|
-e AUTO_DETECT=1 \
|
||||||
-e PYTHONUNBUFFERED=1 \
|
-e PYTHONUNBUFFERED=1 \
|
||||||
--restart unless-stopped \
|
--restart unless-stopped \
|
||||||
harbor.4pd.io/modelhubxc/xiejiahao/gemma4-bi150:v0.1.0
|
harbor.4pd.io/modelhubxc/xiejiahao/gemma4-bi150:v0.1.0
|
||||||
```
|
```
|
||||||
|
|
||||||
**GEV-26B-Decide(TP2,2 卡,8094)**
|
**GEV-26B-Decide(TP2,2 卡,宿主 24396 → 容器 80)**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker run -d --name gev-26b-decide \
|
docker run -d --name gev-26b-decide \
|
||||||
--entrypoint /usr/local/bin/gemma4-entrypoint.sh \
|
--entrypoint /usr/local/bin/gemma4-entrypoint.sh \
|
||||||
--workdir /opt/gemma4 \
|
--workdir /opt/gemma4 \
|
||||||
--privileged --pid host --network host --ipc host \
|
--privileged --pid host --ipc host \
|
||||||
-v /dev:/dev \
|
-v /dev:/dev \
|
||||||
-v /mnt/disk0/models/autotrust/GEV-26B-Decide:/models/gemma4:ro \
|
-v /mnt/disk0/models/autotrust/GEV-26B-Decide:/models/gemma4:ro \
|
||||||
--ulimit memlock=-1 --ulimit stack=67108864 \
|
--ulimit memlock=-1 --ulimit stack=67108864 \
|
||||||
|
-p 24396:80 \
|
||||||
-e CUDA_VISIBLE_DEVICES=14,15 \
|
-e CUDA_VISIBLE_DEVICES=14,15 \
|
||||||
-e MODEL_PATH=/models/gemma4 \
|
-e MODEL_PATH=/models/gemma4 \
|
||||||
|
-e PORT=80 \
|
||||||
|
-e SERVED_NAME=llm \
|
||||||
-e AUTO_DETECT=1 \
|
-e AUTO_DETECT=1 \
|
||||||
-e PYTHONUNBUFFERED=1 \
|
-e PYTHONUNBUFFERED=1 \
|
||||||
--restart unless-stopped \
|
--restart unless-stopped \
|
||||||
harbor.4pd.io/modelhubxc/xiejiahao/gemma4-bi150:v0.1.0
|
harbor.4pd.io/modelhubxc/xiejiahao/gemma4-bi150:v0.1.0
|
||||||
```
|
```
|
||||||
|
|
||||||
两个变体共用挂载点 `/models/gemma4`,靠 `config.json` 区分;TP / 端口 / eager / 显存 / 并发数都不用传,入口脚本会自动判定(见第 3 节)。
|
两个变体共用挂载点 `/models/gemma4`,靠 `config.json` 区分;TP / eager / 显存 / 并发数都不用传,入口脚本会自动判定(见第 3 节)。
|
||||||
|
|
||||||
需要覆盖时加同名 `-e`,例如 `-e MAX_MODEL_LEN=16384`、`-e PORT=24396`。
|
容器内端口恒为 80、模型名恒为 `llm`,所以两个变体可以复用同一个宿主端口 24396,串行切换时客户端不用改地址也不用改 `model` 字段。
|
||||||
|
需要覆盖时加同名 `-e`,例如 `-e MAX_MODEL_LEN=16384`。
|
||||||
|
|
||||||
**注意**:`--network host` 下不要写 `-p`,端口映射会被丢弃;要换对外端口就传 `-e PORT=<port>`。
|
**注意**:并行跑两个变体时必须给不同的 `-p` 左侧端口,例如 `-p 24397:80`。
|
||||||
|
|
||||||
### 11.3 用 run.sh 启动 CI 镜像
|
### 11.3 用 run.sh 启动 CI 镜像
|
||||||
|
|
||||||
@@ -437,10 +447,10 @@ BUILD=0 IMAGE=harbor.4pd.io/modelhubxc/xiejiahao/gemma4-bi150:v0.1.0 \
|
|||||||
| vLLM 补丁 | 两个 | 同样两个 | 不需要 |
|
| vLLM 补丁 | 两个 | 同样两个 | 不需要 |
|
||||||
| docker run 参数 | 相同 | 相同 | — |
|
| docker run 参数 | 相同 | 相同 | — |
|
||||||
| TP | 4 | 2 | ✅ config 可判 |
|
| TP | 4 | 2 | ✅ config 可判 |
|
||||||
| 端口 | 8095 | 8094 | ✅ config 可判 |
|
|
||||||
| served name | `gemma-4-31B-it` | `GEV-26B-Decide` | ✅ config 可判 |
|
|
||||||
| eager | 0 | 1 | ✅ config 可判 |
|
| eager | 0 | 1 | ✅ config 可判 |
|
||||||
| 显存 / seqs | 0.90 / 4 | 0.95 / 2 | ✅ config 可判 |
|
| 显存 / seqs | 0.90 / 4 | 0.95 / 2 | ✅ config 可判 |
|
||||||
|
| 端口 | 80 | 80 | 统一,无需判定 |
|
||||||
|
| served name | `llm` | `llm` | 统一,无需判定 |
|
||||||
|
|
||||||
差异项全部可由 `config.json` 推断,和 Llama 4 用 `num_local_experts` 分 Scout/Maverick 是同一形态,
|
差异项全部可由 `config.json` 推断,和 Llama 4 用 `num_local_experts` 分 Scout/Maverick 是同一形态,
|
||||||
因此按"一个模型家族一个仓库"的原则合并。
|
因此按"一个模型家族一个仓库"的原则合并。
|
||||||
|
|||||||
@@ -9,12 +9,14 @@ services:
|
|||||||
# 镜像本身不设置启动命令,启动方式全部由这里决定。
|
# 镜像本身不设置启动命令,启动方式全部由这里决定。
|
||||||
entrypoint: ["/usr/local/bin/gemma4-entrypoint.sh"]
|
entrypoint: ["/usr/local/bin/gemma4-entrypoint.sh"]
|
||||||
working_dir: /opt/gemma4
|
working_dir: /opt/gemma4
|
||||||
expose:
|
|
||||||
- "8094"
|
|
||||||
- "8095"
|
|
||||||
|
|
||||||
# vLLM 多卡 NCCL / 端口直通
|
# 容器内固定监听 80,宿主机端口由 HOST_PORT 决定(默认 24396)。
|
||||||
network_mode: host
|
expose:
|
||||||
|
- "80"
|
||||||
|
ports:
|
||||||
|
- "${HOST_PORT:-24396}:80"
|
||||||
|
|
||||||
|
# bridge 网络 + 端口映射;ipc host 供 vLLM/PyTorch 多卡共享内存。
|
||||||
ipc: host
|
ipc: host
|
||||||
|
|
||||||
# Iluvatar 运行时需要 privileged + /dev 绑定。
|
# Iluvatar 运行时需要 privileged + /dev 绑定。
|
||||||
@@ -36,11 +38,13 @@ services:
|
|||||||
# entrypoint 按检测出的 TP 使用前 TP 张。
|
# entrypoint 按检测出的 TP 使用前 TP 张。
|
||||||
CUDA_VISIBLE_DEVICES: "${GPUS:-15,14,13,12}"
|
CUDA_VISIBLE_DEVICES: "${GPUS:-15,14,13,12}"
|
||||||
|
|
||||||
|
# 容器内端口固定 80,模型名固定 llm(AGENTS.md 5.2 / 5.5)。
|
||||||
|
PORT: "80"
|
||||||
|
SERVED_NAME: "${SERVED_NAME:-llm}"
|
||||||
|
|
||||||
# 留空时由 entrypoint 根据 config.json 自动判断。
|
# 留空时由 entrypoint 根据 config.json 自动判断。
|
||||||
# 需要手动覆盖时直接设置同名环境变量即可。
|
# 需要手动覆盖时直接设置同名环境变量即可。
|
||||||
TP: "${TP:-}"
|
TP: "${TP:-}"
|
||||||
PORT: "${PORT:-}"
|
|
||||||
SERVED_NAME: "${SERVED_NAME:-}"
|
|
||||||
MAX_NUM_SEQS: "${MAX_NUM_SEQS:-}"
|
MAX_NUM_SEQS: "${MAX_NUM_SEQS:-}"
|
||||||
GPU_MEM_UTIL: "${GPU_MEM_UTIL:-}"
|
GPU_MEM_UTIL: "${GPU_MEM_UTIL:-}"
|
||||||
ENFORCE_EAGER: "${ENFORCE_EAGER:-}"
|
ENFORCE_EAGER: "${ENFORCE_EAGER:-}"
|
||||||
|
|||||||
@@ -38,20 +38,24 @@ if [[ "${AUTO_DETECT}" == "1" ]]; then
|
|||||||
eval "${DETECT_OUTPUT}"
|
eval "${DETECT_OUTPUT}"
|
||||||
MODEL_KIND="${MODEL_KIND:-${DETECTED_MODEL_KIND}}"
|
MODEL_KIND="${MODEL_KIND:-${DETECTED_MODEL_KIND}}"
|
||||||
TP="${TP:-${DETECTED_TP}}"
|
TP="${TP:-${DETECTED_TP}}"
|
||||||
PORT="${PORT:-${DETECTED_PORT}}"
|
|
||||||
SERVED_NAME="${SERVED_NAME:-${DETECTED_SERVED_NAME}}"
|
|
||||||
MAX_NUM_SEQS="${MAX_NUM_SEQS:-${DETECTED_MAX_NUM_SEQS}}"
|
MAX_NUM_SEQS="${MAX_NUM_SEQS:-${DETECTED_MAX_NUM_SEQS}}"
|
||||||
GPU_MEM_UTIL="${GPU_MEM_UTIL:-${DETECTED_GPU_MEM_UTIL}}"
|
GPU_MEM_UTIL="${GPU_MEM_UTIL:-${DETECTED_GPU_MEM_UTIL}}"
|
||||||
ENFORCE_EAGER="${ENFORCE_EAGER:-${DETECTED_ENFORCE_EAGER}}"
|
ENFORCE_EAGER="${ENFORCE_EAGER:-${DETECTED_ENFORCE_EAGER}}"
|
||||||
MODEL_DESCRIPTION="${DETECTED_DESCRIPTION:-}"
|
MODEL_DESCRIPTION="${DETECTED_DESCRIPTION:-}"
|
||||||
|
# 检测出的 port / served_name 仅用于日志提示,不再决定容器内监听端口和模型名:
|
||||||
|
# 容器内端口固定 80,模型名固定 llm(见 AGENTS.md 5.2 / 5.5)。
|
||||||
|
DETECTED_HOST_PORT_HINT="${DETECTED_PORT:-}"
|
||||||
|
DETECTED_NAME_HINT="${DETECTED_SERVED_NAME:-}"
|
||||||
else
|
else
|
||||||
MODEL_KIND="${MODEL_KIND:-unknown}"
|
MODEL_KIND="${MODEL_KIND:-unknown}"
|
||||||
|
DETECTED_HOST_PORT_HINT=""
|
||||||
|
DETECTED_NAME_HINT=""
|
||||||
fi
|
fi
|
||||||
|
|
||||||
HOST="${HOST:-0.0.0.0}"
|
HOST="${HOST:-0.0.0.0}"
|
||||||
TP="${TP:-4}"
|
TP="${TP:-4}"
|
||||||
PORT="${PORT:-8095}"
|
PORT="${PORT:-80}"
|
||||||
SERVED_NAME="${SERVED_NAME:-gemma4}"
|
SERVED_NAME="${SERVED_NAME:-llm}"
|
||||||
MAX_MODEL_LEN="${MAX_MODEL_LEN:-8192}"
|
MAX_MODEL_LEN="${MAX_MODEL_LEN:-8192}"
|
||||||
MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}"
|
MAX_NUM_SEQS="${MAX_NUM_SEQS:-4}"
|
||||||
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}"
|
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}"
|
||||||
@@ -81,6 +85,10 @@ echo " extra : ${EXTRA_ARGS}"
|
|||||||
if [[ -n "${MODEL_DESCRIPTION}" ]]; then
|
if [[ -n "${MODEL_DESCRIPTION}" ]]; then
|
||||||
echo " detected : ${MODEL_DESCRIPTION}"
|
echo " detected : ${MODEL_DESCRIPTION}"
|
||||||
fi
|
fi
|
||||||
|
if [[ -n "${DETECTED_HOST_PORT_HINT}" ]]; then
|
||||||
|
echo " hint : 变体默认宿主端口 ${DETECTED_HOST_PORT_HINT},模型名曾用 ${DETECTED_NAME_HINT}"
|
||||||
|
echo " (容器内固定 80 / served name 固定 llm,实际端口由 docker run -p 决定)"
|
||||||
|
fi
|
||||||
echo "============================================================"
|
echo "============================================================"
|
||||||
|
|
||||||
# shellcheck disable=SC2086
|
# shellcheck disable=SC2086
|
||||||
|
|||||||
29
run.sh
29
run.sh
@@ -16,7 +16,8 @@
|
|||||||
# GPUS=12,13,14,15 ./run.sh /path/to/gemma4
|
# GPUS=12,13,14,15 ./run.sh /path/to/gemma4
|
||||||
# EXTRA_DOCKER_ARGS="-v /mnt/disk1:/data:ro" ./run.sh /path/to/model
|
# EXTRA_DOCKER_ARGS="-v /mnt/disk1:/data:ro" ./run.sh /path/to/model
|
||||||
# EXTRA_ARGS="--max-model-len 16384" ./run.sh /path/to/model
|
# EXTRA_ARGS="--max-model-len 16384" ./run.sh /path/to/model
|
||||||
# TP=4 PORT=8099 SERVED_NAME=my-gemma4 ./run.sh /path/to/model
|
# HOST_PORT=24397 ./run.sh /path/to/model # 换宿主机端口(默认 24396)
|
||||||
|
# TP=4 SERVED_NAME=llm ./run.sh /path/to/model
|
||||||
# BUILD=0 IMAGE=<ci-image> ./run.sh /path/to/model # 使用已有镜像
|
# BUILD=0 IMAGE=<ci-image> ./run.sh /path/to/model # 使用已有镜像
|
||||||
# DRY_RUN=1 ./run.sh /path/to/model # 只打印 docker run 命令
|
# DRY_RUN=1 ./run.sh /path/to/model # 只打印 docker run 命令
|
||||||
#
|
#
|
||||||
@@ -35,7 +36,8 @@ usage() {
|
|||||||
模型类型由 MODEL_DIR/config.json 自动判断,不需要 gemma4-31b / gev-26b-decide 参数。
|
模型类型由 MODEL_DIR/config.json 自动判断,不需要 gemma4-31b / gev-26b-decide 参数。
|
||||||
|
|
||||||
环境变量覆盖:
|
环境变量覆盖:
|
||||||
IMAGE NAME MODEL_HOST_DIR MODEL_MOUNT PORT SERVED_NAME TP GPUS
|
IMAGE NAME MODEL_HOST_DIR MODEL_MOUNT SERVED_NAME TP GPUS
|
||||||
|
HOST_PORT(宿主机端口,默认 24396;容器内固定 80)
|
||||||
MAX_MODEL_LEN MAX_NUM_SEQS GPU_MEM_UTIL MAX_IMAGES ENFORCE_EAGER
|
MAX_MODEL_LEN MAX_NUM_SEQS GPU_MEM_UTIL MAX_IMAGES ENFORCE_EAGER
|
||||||
AUTO_DETECT EXTRA_ARGS EXTRA_DOCKER_ARGS BUILD=0/1 DRY_RUN=0/1
|
AUTO_DETECT EXTRA_ARGS EXTRA_DOCKER_ARGS BUILD=0/1 DRY_RUN=0/1
|
||||||
EOF
|
EOF
|
||||||
@@ -81,9 +83,12 @@ MODEL_KIND="${MODEL_KIND:-${DETECTED_MODEL_KIND}}"
|
|||||||
IMAGE="${IMAGE:-gemma4-bi150:2026-10-27}"
|
IMAGE="${IMAGE:-gemma4-bi150:2026-10-27}"
|
||||||
NAME="${NAME:-${MODEL_KIND}}"
|
NAME="${NAME:-${MODEL_KIND}}"
|
||||||
HOST="${HOST:-0.0.0.0}"
|
HOST="${HOST:-0.0.0.0}"
|
||||||
PORT="${PORT:-${DETECTED_PORT}}"
|
# 容器内端口固定 80;宿主机端口是 HOST_PORT,默认 24396(串行部署复用同一端口)。
|
||||||
|
# PORT 保留为兼容别名,若显式给 PORT 则视作 HOST_PORT。
|
||||||
|
CONTAINER_PORT=80
|
||||||
|
HOST_PORT="${HOST_PORT:-${PORT:-24396}}"
|
||||||
MODEL_MOUNT="${MODEL_MOUNT:-/models/gemma4}"
|
MODEL_MOUNT="${MODEL_MOUNT:-/models/gemma4}"
|
||||||
SERVED_NAME="${SERVED_NAME:-${DETECTED_SERVED_NAME}}"
|
SERVED_NAME="${SERVED_NAME:-llm}"
|
||||||
TP="${TP:-${DETECTED_TP}}"
|
TP="${TP:-${DETECTED_TP}}"
|
||||||
MAX_MODEL_LEN="${MAX_MODEL_LEN:-8192}"
|
MAX_MODEL_LEN="${MAX_MODEL_LEN:-8192}"
|
||||||
MAX_NUM_SEQS="${MAX_NUM_SEQS:-${DETECTED_MAX_NUM_SEQS}}"
|
MAX_NUM_SEQS="${MAX_NUM_SEQS:-${DETECTED_MAX_NUM_SEQS}}"
|
||||||
@@ -129,11 +134,11 @@ DOCKER_RUN=(
|
|||||||
docker run -d --name "${NAME}"
|
docker run -d --name "${NAME}"
|
||||||
--entrypoint /usr/local/bin/gemma4-entrypoint.sh
|
--entrypoint /usr/local/bin/gemma4-entrypoint.sh
|
||||||
--workdir /opt/gemma4
|
--workdir /opt/gemma4
|
||||||
--expose "${PORT}"
|
--expose "${CONTAINER_PORT}"
|
||||||
--privileged
|
--privileged
|
||||||
--pid host
|
--pid host
|
||||||
--network host
|
|
||||||
--ipc host
|
--ipc host
|
||||||
|
-p "${HOST_PORT}:${CONTAINER_PORT}"
|
||||||
-v /dev:/dev
|
-v /dev:/dev
|
||||||
-v "${MODEL_HOST_DIR}:${MODEL_MOUNT}:ro"
|
-v "${MODEL_HOST_DIR}:${MODEL_MOUNT}:ro"
|
||||||
--ulimit memlock=-1
|
--ulimit memlock=-1
|
||||||
@@ -144,7 +149,7 @@ DOCKER_RUN=(
|
|||||||
-e MODEL_KIND="${MODEL_KIND}"
|
-e MODEL_KIND="${MODEL_KIND}"
|
||||||
-e HOST="${HOST}"
|
-e HOST="${HOST}"
|
||||||
-e SERVED_NAME="${SERVED_NAME}"
|
-e SERVED_NAME="${SERVED_NAME}"
|
||||||
-e PORT="${PORT}"
|
-e PORT="${CONTAINER_PORT}"
|
||||||
-e TP="${TP}"
|
-e TP="${TP}"
|
||||||
-e MAX_MODEL_LEN="${MAX_MODEL_LEN}"
|
-e MAX_MODEL_LEN="${MAX_MODEL_LEN}"
|
||||||
-e MAX_NUM_SEQS="${MAX_NUM_SEQS}"
|
-e MAX_NUM_SEQS="${MAX_NUM_SEQS}"
|
||||||
@@ -160,7 +165,9 @@ DOCKER_RUN=(
|
|||||||
|
|
||||||
echo "==> detected ${MODEL_KIND}: ${DETECTED_DESCRIPTION}"
|
echo "==> detected ${MODEL_KIND}: ${DETECTED_DESCRIPTION}"
|
||||||
echo " model : ${MODEL_HOST_DIR}"
|
echo " model : ${MODEL_HOST_DIR}"
|
||||||
echo " TP : ${TP} (${GPU_COUNT} GPUs), port=${PORT}, name=${SERVED_NAME}"
|
echo " TP : ${TP} (${GPU_COUNT} GPUs)"
|
||||||
|
echo " port : 127.0.0.1:${HOST_PORT} -> container:${CONTAINER_PORT}"
|
||||||
|
echo " name : ${SERVED_NAME}"
|
||||||
echo " GPUs : ${GPUS}"
|
echo " GPUs : ${GPUS}"
|
||||||
echo " eager : ${ENFORCE_EAGER}, util=${GPU_MEM_UTIL}, seqs=${MAX_NUM_SEQS}"
|
echo " eager : ${ENFORCE_EAGER}, util=${GPU_MEM_UTIL}, seqs=${MAX_NUM_SEQS}"
|
||||||
|
|
||||||
@@ -185,11 +192,11 @@ echo "==> run ${MODEL_KIND}"
|
|||||||
|
|
||||||
# 冷启动:Gemma-4-31B 约 59 GB、GEV 约 49 GB,首次读盘实测 5-8 分钟。
|
# 冷启动:Gemma-4-31B 约 59 GB、GEV 约 49 GB,首次读盘实测 5-8 分钟。
|
||||||
HEALTH_TIMEOUT="${HEALTH_TIMEOUT:-900}"
|
HEALTH_TIMEOUT="${HEALTH_TIMEOUT:-900}"
|
||||||
echo "==> wait for http://127.0.0.1:${PORT}/health (up to ${HEALTH_TIMEOUT}s)"
|
echo "==> wait for http://127.0.0.1:${HOST_PORT}/health (up to ${HEALTH_TIMEOUT}s)"
|
||||||
DEADLINE=$((SECONDS + HEALTH_TIMEOUT))
|
DEADLINE=$((SECONDS + HEALTH_TIMEOUT))
|
||||||
while (( SECONDS < DEADLINE )); do
|
while (( SECONDS < DEADLINE )); do
|
||||||
if curl -sf "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
|
if curl -sf "http://127.0.0.1:${HOST_PORT}/health" >/dev/null 2>&1; then
|
||||||
echo "ready: http://127.0.0.1:${PORT}/v1/models"
|
echo "ready: http://127.0.0.1:${HOST_PORT}/v1/models"
|
||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
sleep 5
|
sleep 5
|
||||||
|
|||||||
40
verify.sh
40
verify.sh
@@ -21,7 +21,7 @@ cd "$(dirname "$0")"
|
|||||||
|
|
||||||
MODEL_DIR="${MODEL_HOST_DIR:-}"
|
MODEL_DIR="${MODEL_HOST_DIR:-}"
|
||||||
CONTAINER="${CONTAINER:-}"
|
CONTAINER="${CONTAINER:-}"
|
||||||
PORT="${PORT:-}"
|
PORT="${PORT:-${HOST_PORT:-}}"
|
||||||
SERVED_NAME="${SERVED_NAME:-}"
|
SERVED_NAME="${SERVED_NAME:-}"
|
||||||
BASE_URL="${BASE_URL:-}"
|
BASE_URL="${BASE_URL:-}"
|
||||||
TIMEOUT="${TIMEOUT:-300}"
|
TIMEOUT="${TIMEOUT:-300}"
|
||||||
@@ -101,7 +101,21 @@ for m in mounts:
|
|||||||
break
|
break
|
||||||
' || true)"
|
' || true)"
|
||||||
fi
|
fi
|
||||||
[[ -z "${PORT}" ]] && PORT="$(container_env "${CONTAINER}" PORT)"
|
# 宿主端口来自端口映射(-p),不是容器内的 PORT(现在恒为 80)
|
||||||
|
if [[ -z "${PORT}" ]]; then
|
||||||
|
PORT="$(docker inspect "${CONTAINER}" --format '{{json .HostConfig.PortBindings}}' 2>/dev/null \
|
||||||
|
| python3 -c '
|
||||||
|
import json, sys
|
||||||
|
try:
|
||||||
|
binds = json.load(sys.stdin) or {}
|
||||||
|
except Exception:
|
||||||
|
sys.exit(0)
|
||||||
|
for container_port, mapping in binds.items():
|
||||||
|
if mapping:
|
||||||
|
print(mapping[0].get("HostPort", ""))
|
||||||
|
break
|
||||||
|
' || true)"
|
||||||
|
fi
|
||||||
[[ -z "${SERVED_NAME}" ]] && SERVED_NAME="$(container_env "${CONTAINER}" SERVED_NAME)"
|
[[ -z "${SERVED_NAME}" ]] && SERVED_NAME="$(container_env "${CONTAINER}" SERVED_NAME)"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -111,7 +125,7 @@ GEV_DEFAULT="/mnt/disk0/models/autotrust/GEV-26B-Decide"
|
|||||||
|
|
||||||
# 没有 docker 时(例如在 ugly_hopper 内运行),按端口探测正在运行的服务。
|
# 没有 docker 时(例如在 ugly_hopper 内运行),按端口探测正在运行的服务。
|
||||||
if [[ -z "${CONTAINER}" && -z "${PORT}" && -z "${MODEL_DIR}" && -z "${BASE_URL}" ]]; then
|
if [[ -z "${CONTAINER}" && -z "${PORT}" && -z "${MODEL_DIR}" && -z "${BASE_URL}" ]]; then
|
||||||
for candidate_port in 8095 8094; do
|
for candidate_port in 24396 8095 8094; do
|
||||||
if curl -fsS --max-time 2 "http://127.0.0.1:${candidate_port}/health" >/dev/null 2>&1; then
|
if curl -fsS --max-time 2 "http://127.0.0.1:${candidate_port}/health" >/dev/null 2>&1; then
|
||||||
PORT="${candidate_port}"
|
PORT="${candidate_port}"
|
||||||
break
|
break
|
||||||
@@ -119,8 +133,9 @@ if [[ -z "${CONTAINER}" && -z "${PORT}" && -z "${MODEL_DIR}" && -z "${BASE_URL}"
|
|||||||
done
|
done
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# 从 served name 反推模型目录(方便 --base-url + --served-name 的用法)。
|
# 模型名现在恒为 llm(AGENTS.md 5.5),不能再靠它反推变体。
|
||||||
if [[ -z "${MODEL_DIR}" && -n "${SERVED_NAME}" ]]; then
|
# 兼容旧用法:如果显式传了带变体的 served name,仍然尝试反推目录。
|
||||||
|
if [[ -z "${MODEL_DIR}" && -n "${SERVED_NAME}" && "${SERVED_NAME}" != "llm" ]]; then
|
||||||
case "${SERVED_NAME}" in
|
case "${SERVED_NAME}" in
|
||||||
*GEV*|*gev*)
|
*GEV*|*gev*)
|
||||||
[[ -f "${GEV_DEFAULT}/config.json" ]] && MODEL_DIR="${GEV_DEFAULT}"
|
[[ -f "${GEV_DEFAULT}/config.json" ]] && MODEL_DIR="${GEV_DEFAULT}"
|
||||||
@@ -131,10 +146,9 @@ if [[ -z "${MODEL_DIR}" && -n "${SERVED_NAME}" ]]; then
|
|||||||
esac
|
esac
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# 端口也不再区分变体(宿主端口默认统一 24396),只按目录存在性取默认值。
|
||||||
if [[ -z "${MODEL_DIR}" && -z "${BASE_URL}" ]]; then
|
if [[ -z "${MODEL_DIR}" && -z "${BASE_URL}" ]]; then
|
||||||
if [[ "${PORT}" == "8094" && -f "${GEV_DEFAULT}/config.json" ]]; then
|
if [[ -f "${GEMMA4_DEFAULT}/config.json" ]]; then
|
||||||
MODEL_DIR="${GEV_DEFAULT}"
|
|
||||||
elif [[ -f "${GEMMA4_DEFAULT}/config.json" ]]; then
|
|
||||||
MODEL_DIR="${GEMMA4_DEFAULT}"
|
MODEL_DIR="${GEMMA4_DEFAULT}"
|
||||||
elif [[ -f "${GEV_DEFAULT}/config.json" ]]; then
|
elif [[ -f "${GEV_DEFAULT}/config.json" ]]; then
|
||||||
MODEL_DIR="${GEV_DEFAULT}"
|
MODEL_DIR="${GEV_DEFAULT}"
|
||||||
@@ -159,13 +173,13 @@ fi
|
|||||||
if [[ -n "${DETECT_JSON}" ]]; then
|
if [[ -n "${DETECT_JSON}" ]]; then
|
||||||
KIND="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["kind"])' <<< "${DETECT_JSON}")"
|
KIND="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["kind"])' <<< "${DETECT_JSON}")"
|
||||||
DETECTED_PORT="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["port"])' <<< "${DETECT_JSON}")"
|
DETECTED_PORT="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["port"])' <<< "${DETECT_JSON}")"
|
||||||
DETECTED_NAME="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["served_name"])' <<< "${DETECT_JSON}")"
|
# DETECTED_PORT / served_name 只是变体信息,不作为实际端口和模型名:
|
||||||
[[ -z "${PORT}" ]] && PORT="${DETECTED_PORT}"
|
# 容器内恒 80,宿主端口默认 24396,模型名恒 llm(AGENTS.md 5.2 / 5.4 / 5.5)。
|
||||||
[[ -z "${SERVED_NAME}" ]] && SERVED_NAME="${DETECTED_NAME}"
|
DETECTED_HOST_PORT_HINT="${DETECTED_PORT:-}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
PORT="${PORT:-8095}"
|
PORT="${PORT:-24396}"
|
||||||
SERVED_NAME="${SERVED_NAME:-gemma4}"
|
SERVED_NAME="${SERVED_NAME:-llm}"
|
||||||
BASE_URL="${BASE_URL:-http://127.0.0.1:${PORT}}"
|
BASE_URL="${BASE_URL:-http://127.0.0.1:${PORT}}"
|
||||||
BASE_URL="${BASE_URL%/}"
|
BASE_URL="${BASE_URL%/}"
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user