初始化项目,由ModelHub XC社区提供模型
Model: Andrew0425/AgenticASR-Refiner Source: Original Platform
This commit is contained in:
48
onnx-int4/README.md
Normal file
48
onnx-int4/README.md
Normal file
@@ -0,0 +1,48 @@
|
||||
# AgenticASR-Refiner ONNX (INT4)
|
||||
|
||||
INT4 weight-only quantization of the Optimum ONNX export of
|
||||
`Andrew0425/AgenticASR-Refiner` (Llama-based ASR transcript refiner, 24 layers,
|
||||
hidden 1536, GQA 16/2, head_dim 128, vocab 130560).
|
||||
|
||||
- `model.onnx` + `model.onnx.data` — INT4 (MatMulNBits, block size 32, symmetric,
|
||||
accuracy level 4), ~1.3 GB. Graph I/O is identical to the fp32 export
|
||||
(`input_ids` / `attention_mask` / `position_ids` / `past_key_values.*`).
|
||||
- Quantized with `onnxruntime 1.24.4` `MatMulNBitsQuantizer`
|
||||
(`onnxruntime.quantization.matmul_nbits_quantizer`).
|
||||
- Requires ONNX Runtime >= 1.20 (CPU EP supports `MatMulNBits`).
|
||||
|
||||
Tokenization uses the original repo tokenizer (`tokenizer.json` at the repo root).
|
||||
|
||||
## Verified generation (ONNX Runtime 1.28, CPU)
|
||||
|
||||
| Input | Output |
|
||||
|---|---|
|
||||
| 我今天去了公司然后然后开了个会,明天再去见张总 | 我今天去了公司然后开了个会,明天再去见张总 |
|
||||
| 你好你好你好我是那个小李啊 电话是13800138000 | 你好我是小李,电话是13800138000 |
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import onnxruntime as ort
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("Andrew0425/AgenticASR-Refiner")
|
||||
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
|
||||
input_names = [i.name for i in session.get_inputs()]
|
||||
kv_names = [n for n in input_names if n.startswith("past_key_values")]
|
||||
|
||||
prompt = tokenizer.apply_chat_template(
|
||||
[{"role": "system", "content": "你是 ASR 文本纠错助手。保留原意,最小修改。"},
|
||||
{"role": "user", "content": "我今天去了公司然后然后开了个会"}],
|
||||
tokenize=False, add_generation_prompt=True,
|
||||
)
|
||||
ids = tokenizer(prompt).input_ids
|
||||
pasts = [np.zeros((1, 2, 0, 128), dtype=np.float32) for _ in kv_names]
|
||||
|
||||
# prefill, then loop decode: feed input_ids/attention_mask/position_ids + pasts
|
||||
```
|
||||
|
||||
Note: `optimum-onnx 0.1.0` cannot yet run this checkpoint (dummy KV-cache shape
|
||||
uses `hidden_size // num_heads` = 96 instead of `head_dim` = 128); use the raw
|
||||
ONNX Runtime loop above, or a future fixed version of optimum-onnx.
|
||||
Reference in New Issue
Block a user