49 lines
2.1 KiB
Markdown
49 lines
2.1 KiB
Markdown
# AgenticASR-Refiner ONNX (INT4)
|
||
|
||
INT4 weight-only quantization of the Optimum ONNX export of
|
||
`Andrew0425/AgenticASR-Refiner` (Llama-based ASR transcript refiner, 24 layers,
|
||
hidden 1536, GQA 16/2, head_dim 128, vocab 130560).
|
||
|
||
- `model.onnx` + `model.onnx.data` — INT4 (MatMulNBits, block size 32, symmetric,
|
||
accuracy level 4), ~1.3 GB. Graph I/O is identical to the fp32 export
|
||
(`input_ids` / `attention_mask` / `position_ids` / `past_key_values.*`).
|
||
- Quantized with `onnxruntime 1.24.4` `MatMulNBitsQuantizer`
|
||
(`onnxruntime.quantization.matmul_nbits_quantizer`).
|
||
- Requires ONNX Runtime >= 1.20 (CPU EP supports `MatMulNBits`).
|
||
|
||
Tokenization uses the original repo tokenizer (`tokenizer.json` at the repo root).
|
||
|
||
## Verified generation (ONNX Runtime 1.28, CPU)
|
||
|
||
| Input | Output |
|
||
|---|---|
|
||
| 我今天去了公司然后然后开了个会，明天再去见张总 | 我今天去了公司然后开了个会，明天再去见张总 |
|
||
| 你好你好你好我是那个小李啊 电话是13800138000 | 你好我是小李，电话是13800138000 |
|
||
|
||
## Usage
|
||
|
||
```python
|
||
import numpy as np
|
||
import onnxruntime as ort
|
||
from transformers import AutoTokenizer
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained("Andrew0425/AgenticASR-Refiner")
|
||
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
|
||
input_names = [i.name for i in session.get_inputs()]
|
||
kv_names = [n for n in input_names if n.startswith("past_key_values")]
|
||
|
||
prompt = tokenizer.apply_chat_template(
|
||
[{"role": "system", "content": "你是 ASR 文本纠错助手。保留原意，最小修改。"},
|
||
{"role": "user", "content": "我今天去了公司然后然后开了个会"}],
|
||
tokenize=False, add_generation_prompt=True,
|
||
)
|
||
ids = tokenizer(prompt).input_ids
|
||
pasts = [np.zeros((1, 2, 0, 128), dtype=np.float32) for _ in kv_names]
|
||
|
||
# prefill, then loop decode: feed input_ids/attention_mask/position_ids + pasts
|
||
```
|
||
|
||
Note: `optimum-onnx 0.1.0` cannot yet run this checkpoint (dummy KV-cache shape
|
||
uses `hidden_size // num_heads` = 96 instead of `head_dim` = 128); use the raw
|
||
ONNX Runtime loop above, or a future fixed version of optimum-onnx.
|