初始化项目，由ModelHub XC社区提供模型
Model: RedHatAI/Qwen3-0.6B-quantized.w4a16 Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
461
README.md
Normal file
461
README.md
Normal file
@@ -0,0 +1,461 @@
|
||||
---
|
||||
library_name: transformers
|
||||
license: apache-2.0
|
||||
pipeline_tag: text-generation
|
||||
base_model:
|
||||
- Qwen/Qwen3-0.6B
|
||||
tags:
|
||||
- neuralmagic
|
||||
- redhat
|
||||
- llmcompressor
|
||||
- quantized
|
||||
- INT4
|
||||
---
|
||||
|
||||
# Qwen3-0.6B-quantized.w4a16
|
||||
|
||||
## Model Overview
|
||||
- **Model Architecture:** Qwen3ForCausalLM
|
||||
- **Input:** Text
|
||||
- **Output:** Text
|
||||
- **Model Optimizations:**
|
||||
- **Weight quantization:** INT4
|
||||
- **Intended Use Cases:**
|
||||
- Reasoning.
|
||||
- Function calling.
|
||||
- Subject matter experts via fine-tuning.
|
||||
- Multilingual instruction following.
|
||||
- Translation.
|
||||
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws).
|
||||
- **Release Date:** 05/05/2025
|
||||
- **Version:** 1.0
|
||||
- **Model Developers:** RedHat (Neural Magic)
|
||||
|
||||
### Model Optimizations
|
||||
|
||||
This model was obtained by quantizing the weights of [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) to INT4 data type.
|
||||
This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%.
|
||||
|
||||
Only the weights of the linear operators within transformers blocks are quantized.
|
||||
Weights are quantized using a asymmetric per-group scheme, with group size 64.
|
||||
The [GPTQ](https://arxiv.org/abs/2210.17323) algorithm is applied for quantization, as implemented in the [llm-compressor](https://github.com/vllm-project/llm-compressor) library.
|
||||
|
||||
|
||||
## Deployment
|
||||
|
||||
This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
|
||||
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
model_id = "RedHatAI/Qwen3-0.6B-quantized.w4a16"
|
||||
number_gpus = 1
|
||||
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=20, min_p=0, max_tokens=256)
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": prompt}
|
||||
]
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
|
||||
messages = [{"role": "user", "content": "Give me a short introduction to large language model."}]
|
||||
|
||||
prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
||||
|
||||
llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
generated_text = outputs[0].outputs[0].text
|
||||
print(generated_text)
|
||||
```
|
||||
|
||||
vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
|
||||
|
||||
## Creation
|
||||
|
||||
<details>
|
||||
<summary>Creation details</summary>
|
||||
This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below.
|
||||
|
||||
|
||||
```python
|
||||
from llmcompressor.modifiers.quantization import GPTQModifier
|
||||
from llmcompressor.transformers import oneshot
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
# Load model
|
||||
model_stub = "Qwen/Qwen3-0.6B"
|
||||
model_name = model_stub.split("/")[-1]
|
||||
|
||||
num_samples = 1024
|
||||
max_seq_len = 8192
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(model_stub)
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_stub)
|
||||
|
||||
def preprocess_fn(example):
|
||||
return {"text": tokenizer.apply_chat_template(example["messages"], add_generation_prompt=False, tokenize=False)}
|
||||
|
||||
ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train")
|
||||
ds = ds.map(preprocess_fn)
|
||||
|
||||
# Configure the quantization algorithm and scheme
|
||||
recipe = GPTQModifier(
|
||||
ignore=["lm_head"],
|
||||
sequential_targets=["Qwen3DecoderLayer"],
|
||||
targets="Linear",
|
||||
dampening_frac=0.01,
|
||||
config_groups={
|
||||
"group0": {
|
||||
"targets": ["Linear"]
|
||||
"weights": {
|
||||
"num_bits": 4,
|
||||
"type": "int",
|
||||
"strategy": "group",
|
||||
"group_size": 64,
|
||||
"symmetric": False,
|
||||
"actorder": "weight",
|
||||
"observer": "mse",
|
||||
}
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
# Apply quantization
|
||||
oneshot(
|
||||
model=model,
|
||||
dataset=ds,
|
||||
recipe=recipe,
|
||||
max_seq_length=max_seq_len,
|
||||
num_calibration_samples=num_samples,
|
||||
)
|
||||
|
||||
# Save to disk in compressed-tensors format
|
||||
save_path = model_name + "-quantized.w4a16"
|
||||
model.save_pretrained(save_path)
|
||||
tokenizer.save_pretrained(save_path)
|
||||
print(f"Model and tokenizer saved to: {save_path}")
|
||||
```
|
||||
</details>
|
||||
|
||||
|
||||
|
||||
## Evaluation
|
||||
|
||||
The model was evaluated on the OpenLLM leaderboard tasks (versions 1 and 2), using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), and on reasoning tasks using [lighteval](https://github.com/neuralmagic/lighteval/tree/reasoning).
|
||||
[vLLM](https://docs.vllm.ai/en/stable/) was used for all evaluations.
|
||||
|
||||
<details>
|
||||
<summary>Evaluation details</summary>
|
||||
|
||||
**lm-evaluation-harness**
|
||||
```
|
||||
lm_eval \
|
||||
--model vllm \
|
||||
--model_args pretrained="RedHatAI/Qwen3-0.6B-quantized.w4a16",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=1 \
|
||||
--tasks openllm \
|
||||
--apply_chat_template\
|
||||
--fewshot_as_multiturn \
|
||||
--batch_size auto
|
||||
```
|
||||
|
||||
```
|
||||
lm_eval \
|
||||
--model vllm \
|
||||
--model_args pretrained="RedHatAI/Qwen3-0.6B-quantized.w4a16",dtype=auto,gpu_memory_utilization=0.5,max_model_len=8192,enable_chunk_prefill=True,tensor_parallel_size=1 \
|
||||
--tasks mgsm \
|
||||
--apply_chat_template\
|
||||
--batch_size auto
|
||||
```
|
||||
|
||||
```
|
||||
lm_eval \
|
||||
--model vllm \
|
||||
--model_args pretrained="RedHatAI/Qwen3-0.6B-quantized.w4a16",dtype=auto,gpu_memory_utilization=0.5,max_model_len=16384,enable_chunk_prefill=True,tensor_parallel_size=1 \
|
||||
--tasks leaderboard \
|
||||
--apply_chat_template\
|
||||
--fewshot_as_multiturn \
|
||||
--batch_size auto
|
||||
```
|
||||
|
||||
**lighteval**
|
||||
|
||||
lighteval_model_arguments.yaml
|
||||
```yaml
|
||||
model_parameters:
|
||||
model_name: RedHatAI/Qwen3-0.6B-quantized.w4a16
|
||||
dtype: auto
|
||||
gpu_memory_utilization: 0.9
|
||||
max_model_length: 40960
|
||||
generation_parameters:
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
min_p: 0.0
|
||||
top_p: 0.95
|
||||
max_new_tokens: 32768
|
||||
```
|
||||
|
||||
```
|
||||
lighteval vllm \
|
||||
--model_args lighteval_model_arguments.yaml \
|
||||
--tasks lighteval|aime24|0|0 \
|
||||
--use_chat_template = true
|
||||
```
|
||||
|
||||
```
|
||||
lighteval vllm \
|
||||
--model_args lighteval_model_arguments.yaml \
|
||||
--tasks lighteval|aime25|0|0 \
|
||||
--use_chat_template = true
|
||||
```
|
||||
|
||||
```
|
||||
lighteval vllm \
|
||||
--model_args lighteval_model_arguments.yaml \
|
||||
--tasks lighteval|math_500|0|0 \
|
||||
--use_chat_template = true
|
||||
```
|
||||
|
||||
```
|
||||
lighteval vllm \
|
||||
--model_args lighteval_model_arguments.yaml \
|
||||
--tasks lighteval|gpqa:diamond|0|0 \
|
||||
--use_chat_template = true
|
||||
```
|
||||
|
||||
```
|
||||
lighteval vllm \
|
||||
--model_args lighteval_model_arguments.yaml \
|
||||
--tasks extended|lcb:codegeneration \
|
||||
--use_chat_template = true
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Accuracy
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<th>Category
|
||||
</th>
|
||||
<th>Benchmark
|
||||
</th>
|
||||
<th>Qwen3-0.6B
|
||||
</th>
|
||||
<th>Qwen3-0.6B-quantized.w4a16<br>(this model)
|
||||
</th>
|
||||
<th>Recovery
|
||||
</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="7" ><strong>OpenLLM v1</strong>
|
||||
</td>
|
||||
<td>MMLU (5-shot)
|
||||
</td>
|
||||
<td>42.82
|
||||
</td>
|
||||
<td>39.80
|
||||
</td>
|
||||
<td>93.00%
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ARC Challenge (25-shot)
|
||||
</td>
|
||||
<td>32.85
|
||||
</td>
|
||||
<td>30.72
|
||||
</td>
|
||||
<td>93.5%
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>GSM-8K (5-shot, strict-match)
|
||||
</td>
|
||||
<td>1.82
|
||||
</td>
|
||||
<td>2.20
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Hellaswag (10-shot)
|
||||
</td>
|
||||
<td>43.04
|
||||
</td>
|
||||
<td>41.02
|
||||
</td>
|
||||
<td>95.3%
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Winogrande (5-shot)
|
||||
</td>
|
||||
<td>54.54
|
||||
</td>
|
||||
<td>54.62
|
||||
</td>
|
||||
<td>100.1%
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>TruthfulQA (0-shot, mc2)
|
||||
</td>
|
||||
<td>51.61
|
||||
</td>
|
||||
<td>48.77
|
||||
</td>
|
||||
<td>94.5%
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Average</strong>
|
||||
</td>
|
||||
<td><strong>37.78</strong>
|
||||
</td>
|
||||
<td><strong>36.19</strong>
|
||||
</td>
|
||||
<td><strong>95.8%</strong>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="7" ><strong>OpenLLM v2</strong>
|
||||
</td>
|
||||
<td>MMLU-Pro (5-shot)
|
||||
</td>
|
||||
<td>17.25
|
||||
</td>
|
||||
<td>14.27
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>IFEval (0-shot)
|
||||
</td>
|
||||
<td>62.83
|
||||
</td>
|
||||
<td>55.81
|
||||
</td>
|
||||
<td>88.8%
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>BBH (3-shot)
|
||||
</td>
|
||||
<td>4.23
|
||||
</td>
|
||||
<td>1.63
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Math-lvl-5 (4-shot)
|
||||
</td>
|
||||
<td>18.26
|
||||
</td>
|
||||
<td>10.26
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>GPQA (0-shot)
|
||||
</td>
|
||||
<td>0.00
|
||||
</td>
|
||||
<td>0.00
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MuSR (0-shot)
|
||||
</td>
|
||||
<td>0.00
|
||||
</td>
|
||||
<td>0.00
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Average</strong>
|
||||
</td>
|
||||
<td><strong>17.10</strong>
|
||||
</td>
|
||||
<td><strong>13.66</strong>
|
||||
</td>
|
||||
<td><strong>---</strong>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Multilingual</strong>
|
||||
</td>
|
||||
<td>MGSM (0-shot)
|
||||
</td>
|
||||
<td>19.70
|
||||
</td>
|
||||
<td>19.90
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="6" ><strong>Reasoning<br>(generation)</strong>
|
||||
</td>
|
||||
<td>AIME 2024
|
||||
</td>
|
||||
<td>9.69
|
||||
</td>
|
||||
<td>3.44
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>AIME 2025
|
||||
</td>
|
||||
<td>13.13
|
||||
</td>
|
||||
<td>6.98
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>GPQA diamond
|
||||
</td>
|
||||
<td>29.29
|
||||
</td>
|
||||
<td>27.78
|
||||
</td>
|
||||
<td>94.8%
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Math-lvl-5
|
||||
</td>
|
||||
<td>71.60
|
||||
</td>
|
||||
<td>70.60
|
||||
</td>
|
||||
<td>98.6%
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>LiveCodeBench
|
||||
</td>
|
||||
<td>12.83
|
||||
</td>
|
||||
<td>8.35
|
||||
</td>
|
||||
<td>---
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
28
added_tokens.json
Normal file
28
added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"</think>": 151668,
|
||||
"</tool_call>": 151658,
|
||||
"</tool_response>": 151666,
|
||||
"<think>": 151667,
|
||||
"<tool_call>": 151657,
|
||||
"<tool_response>": 151665,
|
||||
"<|box_end|>": 151649,
|
||||
"<|box_start|>": 151648,
|
||||
"<|endoftext|>": 151643,
|
||||
"<|file_sep|>": 151664,
|
||||
"<|fim_middle|>": 151660,
|
||||
"<|fim_pad|>": 151662,
|
||||
"<|fim_prefix|>": 151659,
|
||||
"<|fim_suffix|>": 151661,
|
||||
"<|im_end|>": 151645,
|
||||
"<|im_start|>": 151644,
|
||||
"<|image_pad|>": 151655,
|
||||
"<|object_ref_end|>": 151647,
|
||||
"<|object_ref_start|>": 151646,
|
||||
"<|quad_end|>": 151651,
|
||||
"<|quad_start|>": 151650,
|
||||
"<|repo_name|>": 151663,
|
||||
"<|video_pad|>": 151656,
|
||||
"<|vision_end|>": 151653,
|
||||
"<|vision_pad|>": 151654,
|
||||
"<|vision_start|>": 151652
|
||||
}
|
||||
3
config.json
Normal file
3
config.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:edf28e5cce3752072471330339297b488a23655d7fa23e869ea21dd786701d52
|
||||
size 1483
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||
13
generation_config.json
Normal file
13
generation_config.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"bos_token_id": 151643,
|
||||
"do_sample": true,
|
||||
"eos_token_id": [
|
||||
151645,
|
||||
151643
|
||||
],
|
||||
"pad_token_id": 151643,
|
||||
"temperature": 0.6,
|
||||
"top_k": 20,
|
||||
"top_p": 0.95,
|
||||
"transformers_version": "4.51.3"
|
||||
}
|
||||
BIN
merges.txt
(Stored with Git LFS)
Normal file
BIN
merges.txt
(Stored with Git LFS)
Normal file
Binary file not shown.
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fb4c3fd9e5e62e482521130f86682ee73394049fb06d4d2a6f6321ab55d91567
|
||||
size 859970584
|
||||
11
recipe.yaml
Normal file
11
recipe.yaml
Normal file
@@ -0,0 +1,11 @@
|
||||
quant_stage:
|
||||
quant_modifiers:
|
||||
GPTQModifier:
|
||||
ignore: [lm_head]
|
||||
sequential_targets: [Qwen3DecoderLayer]
|
||||
dampening_frac: 0.01
|
||||
config_groups:
|
||||
group0:
|
||||
targets: [Linear]
|
||||
weights: {num_bits: 4, type: int, strategy: group, group_size: 64, symmetric: false,
|
||||
actorder: weight, observer: mse}
|
||||
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"eos_token": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b7a6a993d40b42d517297bb247ff66679e5bc9dd7a5143be0620faf210b42861
|
||||
size 11422753
|
||||
3
tokenizer_config.json
Normal file
3
tokenizer_config.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:253153d0738ceb4c668d2eff957714dd2bea0b56de772a9fdccd96cbf517e6a0
|
||||
size 9706
|
||||
BIN
vocab.json
(Stored with Git LFS)
Normal file
BIN
vocab.json
(Stored with Git LFS)
Normal file
Binary file not shown.
Reference in New Issue
Block a user