RedHatAI/starcoder2-3b-FP8
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: RedHatAI/starcoder2-3b-FP8
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-10-23 20:16:13 +08:00
commit 577b19aba0
12 changed files with 373 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

233
README.md Normal file
View File

@@ -0,0 +1,233 @@
---
tags:
- fp8
- vllm
license: other
license_name: bigcode-openrail-m
license_link: https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement
---
# starcoder2-3b-FP8
## Model Overview
- **Model Architecture:** starcoder2-3b
- **Input:** Text
- **Output:** Text
- **Model Optimizations:**
- **Weight quantization:** FP8
- **Activation quantization:** FP8
- **Intended Use Cases:** Intended for commercial and research use in English.
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
- **Release Date:** 8/1/2024
- **Version:** 1.0
- **License(s):** [bigcode-openrail-m](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement)
- **Model Developers:** Neural Magic
Quantized version of [starcoder2-3b](https://huggingface.co/bigcode/starcoder2-3b).
<!-- It achieves an average score of 73.19 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 73.48. -->
It achieves an average score of 35.53 on the [HumanEval+](https://github.com/openai/human-eval?tab=readme-ov-file) benchmark, whereas the unquantized model achieves 35.35.
### Model Optimizations
This model was obtained by quantizing the weights and activations of [starcoder2-3b](https://huggingface.co/bigcode/starcoder2-3b) to FP8 data type, ready for inference with vLLM >= 0.5.2.
This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a single linear scaling maps the FP8 representations of the quantized weights and activations.
[AutoFP8](https://github.com/neuralmagic/AutoFP8) is used for quantization with 512 sequences of UltraChat.
<!-- ## Deployment
### Use with vLLM
This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
```python
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
model_id = "neuralmagic/starcoder2-3b-FP8"
sampling_params = SamplingParams(temperature=0.6, top_p=0.9, max_tokens=256)
tokenizer = AutoTokenizer.from_pretrained(model_id)
messages = [
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
{"role": "user", "content": "Who are you?"},
]
prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
llm = LLM(model=model_id, trust_remote_code=True, max_model_len=4096)
outputs = llm.generate(prompts, sampling_params)
generated_text = outputs[0].outputs[0].text
print(generated_text)
```
vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details. -->
## Creation
This model was created by applying [LLM Compressor with calibration samples from UltraChat](https://github.com/vllm-project/llm-compressor/blob/sa/big_model_support/examples/big_model_offloading/big_model_w8a8_calibrate.py), as presented in the code snipet below.
A slight modification to the code was made due to the parameters of the model. Running the below code will throw an index error, and simply replacing the erroneous line with ```max_quant_shape = param.shape[0]``` resolves the issue.
```python
import torch
from datasets import load_dataset
from transformers import AutoTokenizer
from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
from llmcompressor.transformers.compression.helpers import (
calculate_offload_device_map,
custom_offload_device_map,
)
recipe = """
quant_stage:
quant_modifiers:
QuantizationModifier:
ignore: ["lm_head"]
config_groups:
group_0:
weights:
num_bits: 8
type: float
strategy: tensor
dynamic: false
symmetric: true
input_activations:
num_bits: 8
type: float
strategy: tensor
dynamic: false
symmetric: true
targets: ["Linear"]
"""
model_stub = "bigcode/starcoder2-3b"
model_name = model_stub.split("/")[-1]
device_map = calculate_offload_device_map(
model_stub, reserve_for_hessians=False, num_gpus=8, torch_dtype=torch.float16
)
model = SparseAutoModelForCausalLM.from_pretrained(
model_stub, torch_dtype=torch.float16, device_map=device_map
)
tokenizer = AutoTokenizer.from_pretrained(model_stub)
output_dir = f"./{model_name}-FP8"
DATASET_ID = "HuggingFaceH4/ultrachat_200k"
DATASET_SPLIT = "train_sft"
NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 4096
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
def preprocess(example):
return {
"text": " ".join([msg["content"] for msg in example["messages"]])
}
ds = ds.map(preprocess)
def tokenize(sample):
return tokenizer(
sample["text"],
padding=False,
max_length=MAX_SEQUENCE_LENGTH,
truncation=True,
add_special_tokens=False,
)
ds = ds.map(tokenize, remove_columns=ds.column_names)
oneshot(
model=model,
output_dir=output_dir,
dataset=ds,
recipe=recipe,
max_seq_length=MAX_SEQUENCE_LENGTH,
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
save_compressed=True,
)
```
## Evaluation
The model was evaluated on the [HumanEval+](https://github.com/openai/human-eval?tab=readme-ov-file) benchmark with the [Neural Magic fork](https://github.com/neuralmagic/evalplus) of the [EvalPlus implementation of HumanEval+](https://github.com/evalplus/evalplus) and the [vLLM](https://docs.vllm.ai/en/stable/) engine, using the following command:
```
python codegen/generate.py --model neuralmagic/starcoder2-3b-FP8 --temperature 0.2 --n_samples 50 --resume --root ~ --dataset humaneval
python evalplus/sanitize.py ~/humaneval/neuralmagic--starcoder2-3b-FP8_vllm_temp_0.2
evalplus.evaluate --dataset humaneval --samples ~/humaneval/neuralmagic--starcoder2-3b-FP8_vllm_temp_0.2-sanitized
```
### Accuracy
#### HumanEval+ evaluation scores
<table>
<tr>
<td><strong>Benchmark</strong>
</td>
<td><strong>starcoder2-3b</strong>
</td>
<td><strong>starcoder2-3b-FP8(this model)</strong>
</td>
<td><strong>Recovery</strong>
</td>
</tr>
<tr>
<td>base pass@1
</td>
<td>30.7
</td>
<td>30.8
</td>
<td>100.3%
</td>
</tr>
<tr>
<td>base pass@10
</td>
<td>44.9
</td>
<td>45.4
</td>
<td>101.1%
</td>
</tr>
<tr>
<td>base+extra pass@1
</td>
<td>26.6
</td>
<td>26.5
</td>
<td>99.62%
</td>
</tr>
<tr>
<td>base+extra pass@10
</td>
<td>39.2
</td>
<td>39.4
</td>
<td>100.5%
</td>
</tr>
<tr>
<td><strong>Average</strong>
</td>
<td><strong>35.35</strong>
</td>
<td><strong>35.53</strong>
</td>
<td><strong>100.3%</strong>
</td>
</tr>
</table>

70
config.json Normal file
View File

@@ -0,0 +1,70 @@
{
"_name_or_path": "/home/linghao/.cache/huggingface/hub/models--bigcode--starcoder2-3b/snapshots/733247c55e3f73af49ce8e9c7949bf14af205928",
"architectures": [
"Starcoder2ForCausalLM"
],
"attention_dropout": 0.1,
"bos_token_id": 0,
"embedding_dropout": 0.1,
"eos_token_id": 0,
"hidden_act": "gelu_pytorch_tanh",
"hidden_size": 3072,
"initializer_range": 0.018042,
"intermediate_size": 12288,
"max_position_embeddings": 16384,
"mlp_type": "default",
"model_type": "starcoder2",
"norm_epsilon": 1e-05,
"norm_type": "layer_norm",
"num_attention_heads": 24,
"num_hidden_layers": 30,
"num_key_value_heads": 2,
"residual_dropout": 0.1,
"rope_theta": 999999.4420358813,
"sliding_window": 4096,
"torch_dtype": "float16",
"transformers_version": "4.44.0.dev0",
"use_bias": true,
"use_cache": true,
"vocab_size": 49152,
"quantization_config": {
"config_groups": {
"group_0": {
"input_activations": {
"block_structure": null,
"dynamic": false,
"group_size": null,
"num_bits": 8,
"observer": "minmax",
"observer_kwargs": {},
"strategy": "tensor",
"symmetric": true,
"type": "float"
},
"output_activations": null,
"targets": [
"Linear"
],
"weights": {
"block_structure": null,
"dynamic": false,
"group_size": null,
"num_bits": 8,
"observer": "minmax",
"observer_kwargs": {},
"strategy": "tensor",
"symmetric": true,
"type": "float"
}
}
},
"format": "naive-quantized",
"global_compression_ratio": 1.4225351755647873,
"ignore": [
"lm_head"
],
"kv_cache_scheme": null,
"quant_method": "compressed-tensors",
"quantization_status": "compressed"
}
}

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 0,
"eos_token_id": 0,
"transformers_version": "4.44.0.dev0"
}

3
merges.txt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ac9dfb2c5bd3b10bd46aa7fdaa8c54e8e1bcfe1618d9288320ebc82c518442fc
size 441705

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:afffc343654caa0afece610ab84d61a322ee6850b29841b24e4bc801556ada70
size 3484485728

10
recipe.yaml Normal file
View File

@@ -0,0 +1,10 @@
quant_stage:
quant_modifiers:
QuantizationModifier:
ignore: [lm_head]
config_groups:
group_0:
weights: {num_bits: 8, type: float, strategy: tensor, dynamic: false, symmetric: true}
input_activations: {num_bits: 8, type: float, strategy: tensor, dynamic: false,
symmetric: true}
targets: [Linear]

3
special_tokens_map.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7b8d9e1e1efd775a22a19ee45b85a31b524766c0db89de00ed249a7433794762
size 1332

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:17fa145258b20c18287f1e3bd804e074cc13333f11984a2f5a2f11c5110437aa
size 2060947

3
tokenizer_config.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:967231a56db74a08ced2afea94492b831e581da94a411436cce9ec40a0541f00
size 7909

3
vocab.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:542ee3f8b27b3a825754728b4dc59eff31065fecce843a93f09b98f13dbd7de9
size 777202