49 lines
2.1 KiB
Markdown
49 lines
2.1 KiB
Markdown
|
# AgenticASR-Refiner ONNX (INT4)
|
|||
|
|
|
|||
|
|
INT4 weight-only quantization of the Optimum ONNX export of
|
|||
|
|
`Andrew0425/AgenticASR-Refiner` (Llama-based ASR transcript refiner, 24 layers,
|
|||
|
|
hidden 1536, GQA 16/2, head_dim 128, vocab 130560).
|
|||
|
|
|
|||
|
|
- `model.onnx` + `model.onnx.data` — INT4 (MatMulNBits, block size 32, symmetric,
|
|||
|
|
accuracy level 4), ~1.3 GB. Graph I/O is identical to the fp32 export
|
|||
|
|
(`input_ids` / `attention_mask` / `position_ids` / `past_key_values.*`).
|
|||
|
|
- Quantized with `onnxruntime 1.24.4` `MatMulNBitsQuantizer`
|
|||
|
|
(`onnxruntime.quantization.matmul_nbits_quantizer`).
|
|||
|
|
- Requires ONNX Runtime >= 1.20 (CPU EP supports `MatMulNBits`).
|
|||
|
|
|
|||
|
|
Tokenization uses the original repo tokenizer (`tokenizer.json` at the repo root).
|
|||
|
|
|
|||
|
|
## Verified generation (ONNX Runtime 1.28, CPU)
|
|||
|
|
|
|||
|
|
| Input | Output |
|
|||
|
|
|---|---|
|
|||
|
|
| 我今天去了公司然后然后开了个会，明天再去见张总 | 我今天去了公司然后开了个会，明天再去见张总 |
|
|||
|
|
| 你好你好你好我是那个小李啊 电话是13800138000 | 你好我是小李，电话是13800138000 |
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import numpy as np
|
|||
|
|
import onnxruntime as ort
|
|||
|
|
from transformers import AutoTokenizer
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained("Andrew0425/AgenticASR-Refiner")
|
|||
|
|
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
|
|||
|
|
input_names = [i.name for i in session.get_inputs()]
|
|||
|
|
kv_names = [n for n in input_names if n.startswith("past_key_values")]
|
|||
|
|
|
|||
|
|
prompt = tokenizer.apply_chat_template(
|
|||
|
|
[{"role": "system", "content": "你是 ASR 文本纠错助手。保留原意，最小修改。"},
|
|||
|
|
{"role": "user", "content": "我今天去了公司然后然后开了个会"}],
|
|||
|
|
tokenize=False, add_generation_prompt=True,
|
|||
|
|
)
|
|||
|
|
ids = tokenizer(prompt).input_ids
|
|||
|
|
pasts = [np.zeros((1, 2, 0, 128), dtype=np.float32) for _ in kv_names]
|
|||
|
|
|
|||
|
|
# prefill, then loop decode: feed input_ids/attention_mask/position_ids + pasts
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Note: `optimum-onnx 0.1.0` cannot yet run this checkpoint (dummy KV-cache shape
|
|||
|
|
uses `hidden_size // num_heads` = 96 instead of `head_dim` = 128); use the raw
|
|||
|
|
ONNX Runtime loop above, or a future fixed version of optimum-onnx.
|