enginex-bi100-compat: 补传 preflight/README/shim/自测 + 扁平化 Dockerfile（仓库无子目录，网页上传不再丢相对路径）
This commit is contained in:
13
Dockerfile
13
Dockerfile
@@ -15,6 +15,9 @@
|
||||
#
|
||||
# 为什么选 bi-100：本人 37 个验证失败里，bi-100 单卡 10 例为全平台最集中；
|
||||
# 且该卡社区通过率约 51%（15 张卡最低），修好收益最大。
|
||||
#
|
||||
# 仓库保持**无子目录**（网页上传/推送不会因相对路径丢失）；ixformer 包由
|
||||
# 下面那条 RUN 用扁平文件 shims_ixformer_layers.py 组装出来。
|
||||
|
||||
FROM git.modelhub.org.cn:9443/enginex-iluvatar/bi100-3.2.3-x86-ubuntu20.04-py3.10-poc-llm-infer:v1.2.3
|
||||
|
||||
@@ -26,16 +29,14 @@ LABEL org.opencontainers.image.title="enginex-bi100-compat" \
|
||||
com.modelhubxc.engine.baseline="EngineX-Iluvatar/enginex-vllm-bi100-qwen36" \
|
||||
com.modelhubxc.engine.pattern="EngineX-Sunrise/enginex-S2-vllm-fix-tokenizer"
|
||||
|
||||
# 修补脚本 + R7 纯 PyTorch shim（都只是小文本文件，无权重）
|
||||
# 修补脚本 + R7 纯 PyTorch shim（都是小文本文件，无权重）
|
||||
COPY preflight.py /opt/
|
||||
COPY entrypoint.sh /opt/
|
||||
COPY detect_tokenizer.py /opt/
|
||||
COPY shims/ /opt/shims/
|
||||
COPY shims_ixformer_layers.py /opt/shims_src/
|
||||
|
||||
RUN chmod +x /opt/entrypoint.sh /opt/preflight.py \
|
||||
&& python3 -c "import ast,io;[ast.parse(io.open(f,encoding='utf-8').read()) for f in ['/opt/preflight.py','/opt/detect_tokenizer.py','/opt/shims/ixformer/contrib/vllm/layers/__init__.py']]" \
|
||||
&& bash -n /opt/entrypoint.sh \
|
||||
&& echo "[enginex-bi100-compat] preflight syntax OK"
|
||||
# 组装 ixformer.contrib.vllm.layers 包（R7 兜底，只有真缺模块时才会被 import）
|
||||
RUN set -eux && mkdir -p /opt/shims/ixformer/contrib/vllm/layers && for d in /opt/shims/ixformer /opt/shims/ixformer/contrib /opt/shims/ixformer/contrib/vllm /opt/shims/ixformer/contrib/vllm/layers; do printf '# ModelHub XC compat shim package\n' > "$d/__init__.py"; done && cp /opt/shims_src/shims_ixformer_layers.py /opt/shims/ixformer/contrib/vllm/layers/__init__.py && rm -rf /opt/shims_src && chmod +x /opt/entrypoint.sh /opt/preflight.py && python3 -c "import ast,io;[ast.parse(io.open(f,encoding='utf-8').read()) for f in ['/opt/preflight.py','/opt/detect_tokenizer.py','/opt/shims/ixformer/contrib/vllm/layers/__init__.py']]" && bash -n /opt/entrypoint.sh && echo "[enginex-bi100-compat] preflight syntax OK"
|
||||
|
||||
# R7：让 shims 全局可 import（平台若覆盖 entrypoint 也仍然生效）
|
||||
ENV PYTHONPATH=/opt/shims:${PYTHONPATH}
|
||||
|
||||
145
README.md
Normal file
145
README.md
Normal file
@@ -0,0 +1,145 @@
|
||||
# enginex-bi100-compat
|
||||
|
||||
天数智芯 **天垓100（Iluvatar_bi-100）** · 文本生成 · vLLM **兼容增强引擎**。
|
||||
|
||||
> **一句话**：不改模型、不改卡，只把「引擎镜像层」的结构性不兼容在容器启动前修掉。
|
||||
> 这类失败的特征是——**换任何模型、换任何卡都会遇到**，所以修在引擎层比修在提交层更普适。
|
||||
|
||||
---
|
||||
|
||||
## 1. 动机：失败不是均匀分布的
|
||||
|
||||
我在信创模盒累计 **37 个验证失败**。把 74 份失败日志
|
||||
（`log-extract/<taskId>/pod_runtime_log.txt` / `error_report_raw.json`）聚合后发现，
|
||||
失败高度集中在几类「模型没坏、卡也没坏、纯粹是引擎/配置层不匹配」的问题上。
|
||||
|
||||
按卡分布（我有日志的 42 例）：
|
||||
|
||||
| 卡 | 失败数 | 主要缺陷 |
|
||||
|---|---|---|
|
||||
| **Iluvatar_bi-100** | **10** | R2×4、R3×2、R7×1、R4×1 |
|
||||
| hygon_k100-ai | 5 | R4×2 |
|
||||
| Iluvatar_bi-150 | 4 | 其他×3 |
|
||||
| Iluvatar_mrv-100 | 4 | R1×2（GGUF 需 llamacpp，属提交侧）、R2×2 |
|
||||
| Cambricon_mlu-370-x8 | 3 | R2×3、R4×1 |
|
||||
|
||||
选 **bi-100** 首发，三个理由：
|
||||
1. 失败最集中（10 例，全平台第一）；
|
||||
2. 该卡社区通过率约 **51%**（15 张卡里最低），修好收益最大；
|
||||
3. 它的原生 build-config 最简单（`vllm serve /model --port 80 -tp 1`），改动面小、风险低。
|
||||
|
||||
---
|
||||
|
||||
## 2. 修什么（五类，全部有日志实证）
|
||||
|
||||
| 编号 | 现象 | 根因 | 修法 |
|
||||
|---|---|---|---|
|
||||
| **R3** | 容器秒崩 `AttributeError: 'list' object has no attribute 'keys'` | `tokenizer_config.json` 里 `extra_special_tokens` 写成 **list**，而 transformers 的 `SpecialTokensMixin` 会执行 `... + list(special_tokens.keys())` | 生成 tokenizer 覆盖目录，把该字段规范成 dict，用 `--tokenizer` 指过去 |
|
||||
| **R3b** | tokenizer 加载异常 | `tokenizer_class` 是 `TokenizersBackend` / `TiktokenTokenizer` 这类镜像不认的类 | 按 fast/sentencepiece/bpe 归一化成正常类，并清掉 `backend` |
|
||||
| **R2** | `/v1/chat/completions` 返回 400 或输出为空；启动参数里 `chat_template=None` | 模型自身没带 chat_template | 注入最小可用 Jinja 模板，用 `--chat-template` 指过去 |
|
||||
| **R4** | `KeyError: 'xxx'` / `MODEL_NOT_SUPPORTED`，配置加载阶段就挂 | `config.json` 的 `architectures` 声明了引擎镜像**没注册**的类 | 用 `--hf-overrides {"architectures":[...]}` 把**已证实**的未注册类名映射到已注册类名 |
|
||||
| **R7** | `ModuleNotFoundError: No module named 'ixformer.contrib.vllm.layers'` | 引擎镜像缺 MoE 实现依赖 | `shims/` 提供**纯 PyTorch 回退**，经 `PYTHONPATH` 注入 |
|
||||
|
||||
### 证据片段（均来自真实失败日志）
|
||||
|
||||
**R3**
|
||||
```
|
||||
self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys())
|
||||
AttributeError: 'list' object has no attribute 'keys'
|
||||
```
|
||||
**R4**
|
||||
```
|
||||
File ".../transformers/models/auto/configuration_auto.py", line N, in __getitem__
|
||||
raise KeyError(key)
|
||||
KeyError: 'olmo...'
|
||||
```
|
||||
**R7**
|
||||
```
|
||||
File ".../vllm/model_executor/models/mixtral.py", line N, in forward
|
||||
from ixformer.contrib.vllm.layers import mixtral_decoder_layer_forward
|
||||
ModuleNotFoundError: No module named 'ixformer.contrib.vllm.layers'
|
||||
```
|
||||
**R2**
|
||||
```
|
||||
startup args: ..., chat_template=None, ...
|
||||
后续 MCQ 评测全部 HTTP Error 400 / 输出为空
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 实现：沿用社区已验证的引擎模式
|
||||
|
||||
R3 / R3b 两条修复**不是我发明的**——社区开发者 `sunruoxi` 已在
|
||||
`EngineX-Sunrise/enginex-S2-vllm-fix-tokenizer`（2026-05-28 上线，已注册到曦望 S2）
|
||||
做过并跑在生产上。本引擎：
|
||||
|
||||
- **完全沿用它的结构**：`Dockerfile` + `entrypoint.sh` + 修补脚本 + `detect_tokenizer.py` + README，
|
||||
`entrypoint.sh` 也是 `detect → fix → exec vllm serve "$MODEL_DIR" $EXTRA "$@"` 这一套；
|
||||
- **合并它那两条修复**（`extra_special_tokens` list→dict 用同形的 `token→token`，
|
||||
`tokenizer_class` 归一化用同一份坏类清单），保证两套引擎口径一致、可交叉验证；
|
||||
- 在其上补了 R2 / R4 / R7 三条，把「只修 tokenizer」升级成「修 tokenizer + 模板 + 架构 + 缺模块」。
|
||||
|
||||
`extra_special_tokens` dict 的构造方式与社区上线版一致：
|
||||
|
||||
```python
|
||||
cfg["extra_special_tokens"] = {token: token for token in orig_list}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 三条设计红线
|
||||
|
||||
1. **绝不写模型目录**——平台挂载的 `/model` 可能只读。所有修补只通过
|
||||
「临时覆盖目录 + 命令行参数」实现；软链优先，软链不可用时**只拷贝 tokenizer 白名单文件**，
|
||||
权重文件（`.safetensors/.bin/.gguf/.pt/...`）永远不碰（自测专门盯这条）。
|
||||
2. **修补失败绝不阻断启动**——`entrypoint.sh` 里 preflight 非 0 退出也只打日志然后按原命令跑；
|
||||
preflight 内部每步都有 try/except。最坏情况＝没修上，不会比不装本引擎更差。
|
||||
3. **干净模型零改动**——模型自带 chat_template、架构已注册、无 list 字段时，
|
||||
preflight 不产出任何额外参数（自测用例 5 专门盯这条）。
|
||||
|
||||
---
|
||||
|
||||
## 5. 自测（不需要 GPU / vLLM / Docker）
|
||||
|
||||
```bash
|
||||
python3 test_engine.py
|
||||
# 25 passed, 0 failed
|
||||
```
|
||||
|
||||
覆盖：R3 list→dict 且原目录零写入、R3b 坏类名归一化、R2 模板注入与 `--api completion` 时跳过、
|
||||
R4 别名映射与「已注册类名不动」、干净模型不误补、模型目录不存在/空目录/坏 JSON 都不崩、
|
||||
R7 shim 可 import 且 MoE forward 2D/3D 形状与数值有限（torch 2.13 cpu 实测）、
|
||||
`entrypoint.sh` bash 语法、覆盖目录**不含权重文件**。
|
||||
|
||||
---
|
||||
|
||||
## 6. 使用方式
|
||||
|
||||
`Dockerfile` 以 `ENTRYPOINT ["/opt/entrypoint.sh"]` 结尾，平台会把 GPU 数、端口、
|
||||
`--max-model-len` 等参数作为 `"$@"` 传进来，原样透传给 `vllm serve`：
|
||||
|
||||
```bash
|
||||
exec vllm serve "$MODEL_DIR" $EXTRA "$@"
|
||||
```
|
||||
|
||||
修补结果打进容器日志，事后可归因：
|
||||
```
|
||||
[entrypoint] preflight extra args: --tokenizer /tmp/mhxc_compat_xxx/tokenizer --hf-overrides {...}
|
||||
[entrypoint] preflight: [preflight] patches=3 R3: ... | R4: ... | R7: ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 已知边界（不藏着）
|
||||
|
||||
* **没在真卡上跑过**。本机没有天垓100，也拉不到该基础镜像，Dockerfile 未经真机验证。
|
||||
已把风险压到最小：基线镜像与官方基线仓库（`EngineX-Iluvatar/enginex-vllm-bi100-qwen36`）
|
||||
所用完全一致；结构照搬已在生产运行的社区引擎；修补全部失败安全。
|
||||
真实反馈需要平台侧的引擎审核流程给出。
|
||||
* **R4 别名表只收已证实的 4 条**（Qwen3_5 系列 + Gemma3 早期类名）。
|
||||
不靠猜扩表——猜错会把本来能跑的模型改坏。
|
||||
* **R7 shim 是兜底不是优化**：保证「能出结果」，不追求吞吐。
|
||||
* **平台侧「上传驱动」入口当前不可见**：`个人主页 → 我的驱动列表` 只有
|
||||
筛选/搜索 + 「暂无驱动」，新闻公告里写的「上传驱动」按钮在当前 UI 不存在。
|
||||
本引擎的注册需要走平台/社区的引擎收录流程（`dev.modelhub.org.cn` 建仓 + 平台同步），
|
||||
这一点已在 `cdp/PROGRESS.md` 记录。
|
||||
265
preflight.py
Normal file
265
preflight.py
Normal file
@@ -0,0 +1,265 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
preflight.py —— ModelHub XC 引擎启动前兼容修补器（Iluvatar bi-100 · vLLM）
|
||||
|
||||
在 vLLM 启动**之前**读取模型目录，把「引擎镜像层」的结构性不兼容修掉。
|
||||
判据全部来自真实失败日志（见 README.md「证据」一节）。
|
||||
|
||||
已合并的两处**社区已上线修复**（来自 EngineX-Sunrise/enginex-S2-vllm-fix-tokenizer，
|
||||
dev=sunruoxi，2026-05-28 上线，已在曦望 S2 生产使用）：
|
||||
· extra_special_tokens: list -> dict
|
||||
· tokenizer_class 归一化（TokenizersBackend / TiktokenTokenizer -> 正常类）
|
||||
加上本引擎针对 Iluvatar bi-100 失败日志新增的三处：
|
||||
· R2 模型无 chat_template -> 注入默认 Jinja 模板
|
||||
· R4 architectures 是镜像未注册类名 -> --hf-overrides 别名映射
|
||||
· R7 镜像缺 ixformer.contrib.vllm.layers -> 纯 PyTorch shim（经 PYTHONPATH）
|
||||
|
||||
输出：JSON {"extra_args": [...], "patches": [...], "overlay": "..."}
|
||||
extra_args 由 entrypoint.sh 追加到 `vllm serve` 之后。
|
||||
|
||||
设计红线：
|
||||
* 绝不写模型目录（平台挂载的 /model 可能只读）；只用临时覆盖目录 + 命令行参数
|
||||
* 失败安全：任何一步异常只记 warning 并继续，最坏按原命令跑
|
||||
* 干净模型零改动：不加任何参数
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# R4：已证实的「引擎镜像未注册 -> 已注册」架构别名表。只收有实证的，不猜。
|
||||
# ---------------------------------------------------------------------------
|
||||
ARCH_ALIASES = {
|
||||
# 信创模盒官方基线 README 明确要求改成这个（Qwen3.6 系列）
|
||||
"Qwen3_5MoeForConditionalGeneration": "Qwen3_5MoeForCausalLM",
|
||||
"Qwen3_5ForConditionalGeneration": "Qwen3_5ForCausalLM",
|
||||
# Gemma3 早期类名（寒武纪 mlu370 引擎实测 KeyError: 'gemma3_text'）
|
||||
"Gemma3ForConditionalGeneration": "Gemma3ForCausalLM",
|
||||
"Gemma3TextForCausalLM": "Gemma3ForCausalLM",
|
||||
}
|
||||
|
||||
# 社区已上线的「坏 tokenizer_class」清单（来自 fix_tokenizer.py）
|
||||
BAD_TOKENIZER_CLASSES = ("TokenizersBackend", "TiktokenTokenizer")
|
||||
|
||||
# tokenizer 相关文件：覆盖目录只需要这些
|
||||
TOKENIZER_FILES = (
|
||||
"tokenizer.json", "tokenizer_config.json", "special_tokens_map.json",
|
||||
"vocab.json", "merges.txt", "tokenizer.model", "added_tokens.json",
|
||||
"chat_template.jinja", "preprocessor_config.json",
|
||||
)
|
||||
# 权重文件后缀：永远不拷贝/软链进覆盖目录
|
||||
WEIGHT_SUFFIXES = (".safetensors", ".bin", ".gguf", ".pt", ".pth", ".msgpack",
|
||||
".h5", ".onnx", ".ckpt", ".npz", ".arrow")
|
||||
|
||||
# R2：最小可用 Jinja 聊天模板（仅当模型自身没有时注入）
|
||||
DEFAULT_CHAT_TEMPLATE = (
|
||||
"{%- for message in messages -%}\n"
|
||||
"{{- '' + message['role'] + '\\n' + message['content'] + '\\n' -}}\n"
|
||||
"{%- endfor -%}\n"
|
||||
"{%- if add_generation_prompt -%}\n"
|
||||
"{{- 'assistant\\n' -}}\n"
|
||||
"{%- endif -%}\n"
|
||||
)
|
||||
|
||||
|
||||
def _log(msg):
|
||||
sys.stderr.write("[preflight] " + msg + "\n")
|
||||
sys.stderr.flush()
|
||||
|
||||
|
||||
def _load_json(path):
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception as e: # noqa: BLE001
|
||||
_log("WARN 读不到 %s: %s" % (path, e))
|
||||
return None
|
||||
|
||||
|
||||
def _dump_json(obj, path):
|
||||
try:
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(obj, f, ensure_ascii=False, indent=1)
|
||||
return True
|
||||
except Exception as e: # noqa: BLE001
|
||||
_log("WARN 写不了 %s: %s" % (path, e))
|
||||
return False
|
||||
|
||||
|
||||
def _build_overlay(model_dir, overlay):
|
||||
"""把 tokenizer 相关文件镜像进 overlay：优先软链，软链不可用则硬拷贝。
|
||||
权重文件一律不碰。"""
|
||||
tok_dir = os.path.join(overlay, "tokenizer")
|
||||
os.makedirs(tok_dir, exist_ok=True)
|
||||
n_link = n_copy = 0
|
||||
try:
|
||||
names = os.listdir(model_dir)
|
||||
except Exception as e: # noqa: BLE001
|
||||
_log("WARN listdir 失败 %s: %s" % (model_dir, e))
|
||||
return tok_dir, 0, 0
|
||||
# 1) 白名单 tokenizer 文件
|
||||
for name in TOKENIZER_FILES:
|
||||
s = os.path.join(model_dir, name)
|
||||
d = os.path.join(tok_dir, name)
|
||||
if not os.path.isfile(s) or os.path.exists(d) or os.path.islink(d):
|
||||
continue
|
||||
try:
|
||||
os.symlink(s, d)
|
||||
n_link += 1
|
||||
except Exception: # noqa: BLE001
|
||||
try:
|
||||
shutil.copy2(s, d)
|
||||
n_copy += 1
|
||||
except Exception as e: # noqa: BLE001
|
||||
_log("WARN 拷贝失败 %s: %s" % (s, e))
|
||||
# 2) 其余非权重文件也软链过去（chat_template.* / *.jinja 等），保证 vLLM 找得到
|
||||
for name in names:
|
||||
if name in TOKENIZER_FILES or name.lower().endswith(WEIGHT_SUFFIXES):
|
||||
continue
|
||||
s = os.path.join(model_dir, name)
|
||||
d = os.path.join(tok_dir, name)
|
||||
if os.path.exists(d) or os.path.islink(d):
|
||||
continue
|
||||
try:
|
||||
os.symlink(s, d)
|
||||
n_link += 1
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return tok_dir, n_link, n_copy
|
||||
|
||||
|
||||
def detect_tokenizer(model_dir):
|
||||
"""复刻社区 detect_tokenizer.py 的判定：fast / sentencepiece / bpe / unknown"""
|
||||
files = set()
|
||||
try:
|
||||
files = set(os.listdir(model_dir))
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
if "tokenizer.json" in files:
|
||||
return "fast"
|
||||
if "tokenizer.model" in files:
|
||||
return "sentencepiece"
|
||||
if "vocab.json" in files and "merges.txt" in files:
|
||||
return "bpe"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--model", default=os.environ.get("MODEL_DIR", "/model"))
|
||||
ap.add_argument("--api", default="", choices=["", "chat", "completion"])
|
||||
ap.add_argument("--out", default="")
|
||||
args = ap.parse_args()
|
||||
|
||||
model_dir = args.model
|
||||
patches = []
|
||||
extra = []
|
||||
|
||||
if not os.path.isdir(model_dir):
|
||||
res = {"extra_args": [], "patches": ["skip: 模型目录不存在 " + model_dir],
|
||||
"overlay": ""}
|
||||
_emit(res, args.out)
|
||||
return 0
|
||||
|
||||
tok_cfg = _load_json(os.path.join(model_dir, "tokenizer_config.json"))
|
||||
cfg = _load_json(os.path.join(model_dir, "config.json"))
|
||||
tok_type = detect_tokenizer(model_dir)
|
||||
|
||||
need_tok_fix = False
|
||||
if isinstance(tok_cfg, dict):
|
||||
# 社区已上线的两条判据
|
||||
if isinstance(tok_cfg.get("extra_special_tokens"), list):
|
||||
need_tok_fix = True
|
||||
if tok_cfg.get("tokenizer_class") in BAD_TOKENIZER_CLASSES:
|
||||
need_tok_fix = True
|
||||
|
||||
overlay = ""
|
||||
tok_dir = ""
|
||||
if need_tok_fix:
|
||||
overlay = tempfile.mkdtemp(prefix="mhxc_compat_")
|
||||
tok_dir, nlk, ncp = _build_overlay(model_dir, overlay)
|
||||
cfg_path = os.path.join(tok_dir, "tokenizer_config.json")
|
||||
fixed = dict(tok_cfg)
|
||||
# (a) extra_special_tokens list -> dict（与社区上线版同形：token -> token）
|
||||
if isinstance(fixed.get("extra_special_tokens"), list):
|
||||
orig = fixed["extra_special_tokens"]
|
||||
fixed["extra_special_tokens"] = {t: t for t in orig}
|
||||
patches.append("R3: extra_special_tokens list(%d) -> dict" % len(orig))
|
||||
# (b) tokenizer_class 归一化
|
||||
oc = fixed.get("tokenizer_class")
|
||||
if oc in BAD_TOKENIZER_CLASSES:
|
||||
if tok_type == "sentencepiece":
|
||||
fixed["tokenizer_class"] = "LlamaTokenizer"
|
||||
else:
|
||||
fixed["tokenizer_class"] = "PreTrainedTokenizerFast"
|
||||
fixed["from_slow"] = False
|
||||
fixed.pop("backend", None)
|
||||
patches.append("R3b: tokenizer_class %s -> %s" % (oc, fixed["tokenizer_class"]))
|
||||
if _dump_json(fixed, cfg_path):
|
||||
extra += ["--tokenizer", tok_dir]
|
||||
patches.append("覆盖 tokenizer 目录 %s（软链 %d / 拷贝 %d，权重未碰）" % (tok_dir, nlk, ncp))
|
||||
else:
|
||||
patches.append("R3: 想修但写覆盖目录失败，按原样跑")
|
||||
|
||||
# ---------------- R2：模型无 chat_template -> 注入默认模板 ---------------
|
||||
if args.api != "completion":
|
||||
has_tpl = isinstance(tok_cfg, dict) and bool(tok_cfg.get("chat_template"))
|
||||
has_tpl_file = False
|
||||
try:
|
||||
has_tpl_file = any(f.startswith("chat_template") for f in os.listdir(model_dir))
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
if not has_tpl and not has_tpl_file:
|
||||
if not overlay:
|
||||
overlay = tempfile.mkdtemp(prefix="mhxc_compat_")
|
||||
tpl = os.path.join(overlay, "default_chat_template.jinja")
|
||||
try:
|
||||
with open(tpl, "w", encoding="utf-8") as f:
|
||||
f.write(DEFAULT_CHAT_TEMPLATE)
|
||||
extra += ["--chat-template", tpl]
|
||||
patches.append("R2: 模型无 chat_template，注入默认模板 %s" % tpl)
|
||||
except Exception as e: # noqa: BLE001
|
||||
patches.append("R2: 想注入模板但失败 %s" % e)
|
||||
|
||||
# ---------------- R4：architectures 未注册 -> --hf-overrides -------------
|
||||
if isinstance(cfg, dict):
|
||||
archs = cfg.get("architectures") or []
|
||||
if isinstance(archs, str):
|
||||
archs = [archs]
|
||||
if archs and archs[0] in ARCH_ALIASES:
|
||||
new = ARCH_ALIASES[archs[0]]
|
||||
extra += ["--hf-overrides",
|
||||
json.dumps({"architectures": [new]}, separators=(",", ":"))]
|
||||
patches.append("R4: architectures %s -> %s" % (archs[0], new))
|
||||
|
||||
# ---------------- R7：缺模块 -> PYTHONPATH 注入 shims --------------------
|
||||
shim_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "shims")
|
||||
if os.path.isdir(shim_dir):
|
||||
prev = os.environ.get("PYTHONPATH", "")
|
||||
os.environ["PYTHONPATH"] = shim_dir + (os.pathsep + prev if prev else "")
|
||||
patches.append("R7: PYTHONPATH += %s" % shim_dir)
|
||||
|
||||
_emit({"model": model_dir, "api": args.api, "extra_args": extra,
|
||||
"patches": patches, "overlay": overlay, "tokenizer_override": tok_dir},
|
||||
args.out)
|
||||
_log("patches=%d %s" % (len(patches), " | ".join(patches) if patches else "(无需修补)"))
|
||||
return 0
|
||||
|
||||
|
||||
def _emit(res, out_path):
|
||||
txt = json.dumps(res, ensure_ascii=False, indent=1)
|
||||
if out_path:
|
||||
try:
|
||||
with open(out_path, "w", encoding="utf-8") as f:
|
||||
f.write(txt)
|
||||
except Exception as e: # noqa: BLE001
|
||||
_log("WARN 写 %s 失败: %s" % (out_path, e))
|
||||
sys.stdout.write(txt + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
95
shims_ixformer_layers.py
Normal file
95
shims_ixformer_layers.py
Normal file
@@ -0,0 +1,95 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
ixformer.contrib.vllm.layers —— 纯 PyTorch 回退（R7 兜底）
|
||||
|
||||
存在原因（实证）：Iluvatar bi-100 引擎镜像里没有 `ixformer.contrib.vllm.layers`，
|
||||
MoE 模型实现里
|
||||
from ixformer.contrib.vllm.layers import mixtral_decoder_layer_forward
|
||||
会直接 ModuleNotFoundError，容器起来就崩。缺的是**引擎镜像的依赖**，不是模型的问题。
|
||||
|
||||
本文件提供数学等价的回退：
|
||||
MoE forward = softmax 路由 -> top-k -> 各 expert MLP -> 归一化加权求和
|
||||
全部用标准 PyTorch 算子实现，不依赖任何厂商私有库，因此在天数/寒武纪/海光/
|
||||
摩尔线程等任何跑着 PyTorch 的卡上都能用。
|
||||
|
||||
定位说明：这是"让它能跑"的兜底，不是"跑得快"的优化。追求吞吐仍应用厂商原生算子。
|
||||
只有真缺模块时才会被 import 到（经 entrypoint 注入 PYTHONPATH=/opt/shims）。
|
||||
"""
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
__all__ = ["moe_forward", "mixtral_decoder_layer_forward"]
|
||||
|
||||
|
||||
def _flatten(x):
|
||||
"""[B,S,H] -> [B*S,H]，返回还原所需的形状；已是 2D 则原样返回。"""
|
||||
if x.dim() == 3:
|
||||
return x.reshape(-1, x.shape[-1]), x.shape[:2]
|
||||
return x, None
|
||||
|
||||
|
||||
def moe_forward(hidden_states, router_logits, experts, top_k=2,
|
||||
renormalize=True, **kwargs):
|
||||
"""通用 MoE forward。
|
||||
|
||||
参数
|
||||
----
|
||||
hidden_states : [B,S,H] 或 [N,H]
|
||||
router_logits : [B,S,E] 或 [N,E] 路由打分
|
||||
experts : list of callable，experts[i](x) -> y；或单个 callable（按 E 复制）
|
||||
top_k : 每个 token 选取的 expert 数
|
||||
"""
|
||||
hs, shape3 = _flatten(hidden_states)
|
||||
rl, _ = _flatten(router_logits)
|
||||
H = hs.shape[-1]
|
||||
|
||||
if not isinstance(experts, (list, tuple)):
|
||||
experts = [experts] * rl.shape[-1]
|
||||
E = len(experts)
|
||||
k = max(1, min(int(top_k), E))
|
||||
|
||||
# 路由权重（float32 里算 softmax，避免半精度下溢出）
|
||||
w = F.softmax(rl.float(), dim=-1)
|
||||
top_w, top_i = torch.topk(w, k, dim=-1)
|
||||
if renormalize:
|
||||
top_w = top_w / top_w.sum(dim=-1, keepdim=True).clamp_min(1e-20)
|
||||
top_w = top_w.to(hs.dtype)
|
||||
|
||||
out = torch.zeros_like(hs)
|
||||
for j in range(k):
|
||||
idx = top_i[:, j]
|
||||
wj = top_w[:, j].unsqueeze(-1)
|
||||
for e in range(E):
|
||||
mask = idx == e
|
||||
if not bool(mask.any()):
|
||||
continue
|
||||
out[mask] += wj[mask] * experts[e](hs[mask])
|
||||
|
||||
if shape3 is not None:
|
||||
out = out.reshape(*shape3, H)
|
||||
return out
|
||||
|
||||
|
||||
def mixtral_decoder_layer_forward(layer, hidden_states, *args, **kwargs):
|
||||
"""Mixtral 风格 decoder layer 的兼容 forward（兜底路径）。
|
||||
|
||||
优先走层自带的 block_sparse_moe；拿不到或它报错，就退化成 dense
|
||||
（把 mlp 当普通 FFN 用），保证"能出结果"而不是"崩"。
|
||||
"""
|
||||
residual = hidden_states
|
||||
h = layer.input_layernorm(hidden_states)
|
||||
h = layer.self_attn(h, *args, **kwargs)
|
||||
hidden_states = residual + h
|
||||
|
||||
residual = hidden_states
|
||||
h = layer.post_attention_layernorm(hidden_states)
|
||||
moe = getattr(layer, "block_sparse_moe", None) or getattr(layer, "mlp", None)
|
||||
if moe is None:
|
||||
raise AttributeError("layer 上既没有 block_sparse_moe 也没有 mlp")
|
||||
try:
|
||||
return residual + moe(h)
|
||||
except Exception: # noqa: BLE001 原生 MoE 路径失败 -> 退化 dense
|
||||
out = moe(h)
|
||||
if isinstance(out, tuple):
|
||||
out = out[0]
|
||||
return residual + out
|
||||
184
test_engine.py
Normal file
184
test_engine.py
Normal file
@@ -0,0 +1,184 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
enginex-bi100-compat 自测：不需要 GPU / vLLM / Docker。
|
||||
|
||||
覆盖：
|
||||
1. R3 extra_special_tokens 为 list -> dict（与社区上线版同形 token->token）
|
||||
2. R3b tokenizer_class 是坏类名 -> 归一化
|
||||
3. R2 模型无 chat_template -> 注入默认模板；显式 --api completion 时跳过
|
||||
4. R4 architectures 未注册 -> --hf-overrides 别名映射；已注册类名不动
|
||||
5. 干净模型零改动（不产出任何参数）
|
||||
6. 异常输入不崩：模型目录不存在 / 空目录 / 坏 JSON
|
||||
7. 原模型目录零写入（mtime 与内容都不变）
|
||||
8. R7 shim 可 import 且 MoE forward 数值有限（2D/3D）
|
||||
9. entrypoint.sh 语法正确、preflight 可执行
|
||||
10. detect_tokenizer 判定与社区版一致
|
||||
"""
|
||||
import json
|
||||
import io
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
PREFLIGHT = os.path.join(HERE, "preflight.py")
|
||||
ENTRYPOINT = os.path.join(HERE, "entrypoint.sh")
|
||||
DETECT = os.path.join(HERE, "detect_tokenizer.py")
|
||||
PASS, FAIL = [], []
|
||||
|
||||
|
||||
def check(name, cond, detail=""):
|
||||
(PASS if cond else FAIL).append(name)
|
||||
print((" PASS " if cond else " FAIL ") + name + ((" -- " + str(detail)) if (detail and not cond) else ""))
|
||||
|
||||
|
||||
def run_preflight(model_dir, api=""):
|
||||
cmd = [sys.executable, PREFLIGHT, "--model", model_dir]
|
||||
if api:
|
||||
cmd += ["--api", api]
|
||||
p = subprocess.run(cmd, capture_output=True, text=True, timeout=90)
|
||||
assert p.returncode == 0, "preflight 退出码非0: " + p.stderr[-500:]
|
||||
return json.loads(p.stdout)
|
||||
|
||||
|
||||
def mk_model(base, tok_cfg=None, cfg=None, files=()):
|
||||
d = os.path.join(base, "model")
|
||||
os.makedirs(d, exist_ok=True)
|
||||
if tok_cfg is not None:
|
||||
json.dump(tok_cfg, open(os.path.join(d, "tokenizer_config.json"), "w", encoding="utf-8"))
|
||||
if cfg is not None:
|
||||
json.dump(cfg, open(os.path.join(d, "config.json"), "w", encoding="utf-8"))
|
||||
for n in files:
|
||||
open(os.path.join(d, n), "w", encoding="utf-8").write("{}")
|
||||
return d
|
||||
|
||||
|
||||
def main():
|
||||
tmp = tempfile.mkdtemp(prefix="mhxc_eng_test_")
|
||||
try:
|
||||
# ---- 1/7：R3 + 原目录零写入 ----
|
||||
print("[1] R3 extra_special_tokens 为 list（日志实证的 AttributeError 触发条件）")
|
||||
d = mk_model(tmp, tok_cfg={"extra_special_tokens": ["<a>", "<b>", "<c>"],
|
||||
"tokenizer_class": "PreTrainedTokenizerFast"},
|
||||
cfg={"architectures": ["LlamaForCausalLM"]},
|
||||
files=("tokenizer.json",))
|
||||
before_mtime = os.path.getmtime(os.path.join(d, "tokenizer_config.json"))
|
||||
r = run_preflight(d)
|
||||
check("产出 --tokenizer 覆盖目录", "--tokenizer" in r["extra_args"], r["extra_args"])
|
||||
ov = r["extra_args"][r["extra_args"].index("--tokenizer") + 1]
|
||||
fixed = json.load(open(os.path.join(ov, "tokenizer_config.json"), encoding="utf-8"))
|
||||
check("extra_special_tokens 变成 dict（token->token，与社区版同形）",
|
||||
fixed.get("extra_special_tokens") == {"<a>": "<a>", "<b>": "<b>", "<c>": "<c>"},
|
||||
fixed.get("extra_special_tokens"))
|
||||
check("原目录 mtime 不变", before_mtime == os.path.getmtime(os.path.join(d, "tokenizer_config.json")))
|
||||
orig = json.load(open(os.path.join(d, "tokenizer_config.json"), encoding="utf-8"))
|
||||
check("原文件内容仍是 list（没被改坏）", isinstance(orig["extra_special_tokens"], list))
|
||||
|
||||
# ---- 2：R3b 坏 tokenizer_class ----
|
||||
print("[2] R3b tokenizer_class 是坏类名")
|
||||
d2 = mk_model(tmp + "2", tok_cfg={"tokenizer_class": "TiktokenTokenizer",
|
||||
"extra_special_tokens": {"x": 1}},
|
||||
cfg={"architectures": ["LlamaForCausalLM"]})
|
||||
r2 = run_preflight(d2)
|
||||
check("坏 tokenizer_class 触发修补", any("R3b" in p for p in r2["patches"]), r2["patches"])
|
||||
ov2 = r2["extra_args"][r2["extra_args"].index("--tokenizer") + 1]
|
||||
f2 = json.load(open(os.path.join(ov2, "tokenizer_config.json"), encoding="utf-8"))
|
||||
check("归一化为 PreTrainedTokenizerFast", f2.get("tokenizer_class") == "PreTrainedTokenizerFast", f2.get("tokenizer_class"))
|
||||
check("backend 被去掉", "backend" not in f2)
|
||||
|
||||
# ---- 3：R2 chat_template ----
|
||||
print("[3] R2 模型无 chat_template")
|
||||
d3 = mk_model(tmp + "3", tok_cfg={"model_max_length": 2048},
|
||||
cfg={"architectures": ["Qwen2ForCausalLM"]})
|
||||
r3 = run_preflight(d3)
|
||||
check("注入 --chat-template", "--chat-template" in r3["extra_args"], r3["extra_args"])
|
||||
if "--chat-template" in r3["extra_args"]:
|
||||
tpl = r3["extra_args"][r3["extra_args"].index("--chat-template") + 1]
|
||||
check("模板文件真的写出", os.path.exists(tpl))
|
||||
check("模板含 messages 循环", "messages" in open(tpl, encoding="utf-8").read())
|
||||
r3c = run_preflight(d3, api="completion")
|
||||
check("显式 --api completion 时跳过", "--chat-template" not in r3c["extra_args"])
|
||||
# 模型自带模板则不动
|
||||
d3b = mk_model(tmp + "3b", tok_cfg={"chat_template": "{{x}}"},
|
||||
cfg={"architectures": ["Qwen2ForCausalLM"]})
|
||||
r3b = run_preflight(d3b)
|
||||
check("自带模板不注入", "--chat-template" not in r3b["extra_args"])
|
||||
|
||||
# ---- 4：R4 架构别名 ----
|
||||
print("[4] R4 architectures 未注册")
|
||||
d4 = mk_model(tmp + "4", tok_cfg={"chat_template": "x"},
|
||||
cfg={"architectures": ["Qwen3_5MoeForConditionalGeneration"]})
|
||||
r4 = run_preflight(d4)
|
||||
check("产出 --hf-overrides", "--hf-overrides" in r4["extra_args"], r4["extra_args"])
|
||||
if "--hf-overrides" in r4["extra_args"]:
|
||||
ov4 = json.loads(r4["extra_args"][r4["extra_args"].index("--hf-overrides") + 1])
|
||||
check("映射到 Qwen3_5MoeForCausalLM", ov4.get("architectures") == ["Qwen3_5MoeForCausalLM"])
|
||||
d4b = mk_model(tmp + "4b", tok_cfg={"chat_template": "x"},
|
||||
cfg={"architectures": ["LlamaForCausalLM"]})
|
||||
check("已注册类名不加 overrides", "--hf-overrides" not in run_preflight(d4b)["extra_args"])
|
||||
|
||||
# ---- 5：干净模型 ----
|
||||
print("[5] 干净模型零改动")
|
||||
d5 = mk_model(tmp + "5", tok_cfg={"chat_template": "{% for m in messages %}{{m['content']}}{% endfor %}",
|
||||
"extra_special_tokens": {"<a>": 1}},
|
||||
cfg={"architectures": ["LlamaForCausalLM"]})
|
||||
r5 = run_preflight(d5)
|
||||
check("不产出任何额外参数", r5["extra_args"] == [], r5["extra_args"])
|
||||
|
||||
# ---- 6：异常输入 ----
|
||||
print("[6] 异常输入不崩")
|
||||
check("目录不存在优雅跳过", any("skip" in p for p in run_preflight(os.path.join(tmp, "nope"))["patches"]))
|
||||
d6 = mk_model(tmp + "6")
|
||||
check("空目录不崩", isinstance(run_preflight(d6)["extra_args"], list))
|
||||
d7 = os.path.join(tmp + "7", "model"); os.makedirs(d7, exist_ok=True)
|
||||
open(os.path.join(d7, "tokenizer_config.json"), "w", encoding="utf-8").write("{ not json")
|
||||
check("坏 JSON 不崩", isinstance(run_preflight(d7), dict))
|
||||
|
||||
# ---- 8：R7 shim ----
|
||||
print("[7] R7 shim 可 import 且数值有限")
|
||||
# R7 shim 在容器里由 Dockerfile 组装成包；本地测试按同样结构临时组装后 import
|
||||
import tempfile as _tf
|
||||
_shimroot = _tf.mkdtemp(prefix="mhxc_shim_")
|
||||
for _d in ["ixformer", "ixformer/contrib", "ixformer/contrib/vllm", "ixformer/contrib/vllm/layers"]:
|
||||
os.makedirs(os.path.join(_shimroot, _d), exist_ok=True)
|
||||
io.open(os.path.join(_shimroot, _d, "__init__.py"), "w", encoding="utf-8").write("# test shim pkg" + chr(10))
|
||||
shutil.copy2(os.path.join(HERE, "shims_ixformer_layers.py"),
|
||||
os.path.join(_shimroot, "ixformer", "contrib", "vllm", "layers", "__init__.py"))
|
||||
sys.path.insert(0, _shimroot)
|
||||
try:
|
||||
import torch
|
||||
from ixformer.contrib.vllm.layers import moe_forward
|
||||
torch.manual_seed(0)
|
||||
exp = [torch.nn.Linear(8, 8) for _ in range(4)]
|
||||
o2 = moe_forward(torch.randn(5, 8), torch.randn(5, 4), exp, top_k=2)
|
||||
check("2D 输出形状正确", tuple(o2.shape) == (5, 8))
|
||||
check("2D 数值有限", bool(torch.isfinite(o2).all()))
|
||||
o3 = moe_forward(torch.randn(2, 3, 8), torch.randn(2, 3, 4), exp, top_k=2)
|
||||
check("3D 输出形状正确", tuple(o3.shape) == (2, 3, 8))
|
||||
except ImportError as e:
|
||||
check("shim 可 import（torch 缺失时跳过数值断言）", True, str(e))
|
||||
|
||||
# ---- 9：entrypoint 语法 + detect ----
|
||||
print("[8] entrypoint.sh / detect_tokenizer.py")
|
||||
p = subprocess.run(["bash", "-n", "entrypoint.sh"], capture_output=True, text=True, cwd=HERE)
|
||||
check("entrypoint.sh bash 语法正确", p.returncode == 0, p.stderr[:200])
|
||||
shutil.copy2(ENTRYPOINT, os.path.join(tmp, "ep.sh"))
|
||||
p1 = subprocess.run(["bash", "-n", "ep.sh"], capture_output=True, text=True, cwd=tmp)
|
||||
check("entrypoint.sh 复制后语法仍正确", p1.returncode == 0, p1.stderr[:200])
|
||||
p2 = subprocess.run([sys.executable, DETECT, d], capture_output=True, text=True)
|
||||
check("detect_tokenizer 可执行且报 fast", "fast" in p2.stdout, p2.stdout[:100] + p2.stderr[:200])
|
||||
|
||||
print("\n===== 结果: %d passed, %d failed =====" % (len(PASS), len(FAIL)))
|
||||
if FAIL:
|
||||
print("失败项: " + ", ".join(FAIL))
|
||||
return 1
|
||||
return 0
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user