初始化项目，由ModelHub XC社区提供模型
Model: neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
model.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
454
README.md
Normal file
454
README.md
Normal file
@@ -0,0 +1,454 @@
|
||||
---
|
||||
tags:
|
||||
- w4a16
|
||||
- int4
|
||||
- vllm
|
||||
license: apache-2.0
|
||||
license_link: https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache-2.0.md
|
||||
language:
|
||||
- en
|
||||
base_model: ibm-granite/granite-3.1-2b-instruct
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# granite-3.1-2b-instruct-quantized.w4a16
|
||||
|
||||
## Model Overview
|
||||
- **Model Architecture:** granite-3.1-2b-instruct
|
||||
- **Input:** Text
|
||||
- **Output:** Text
|
||||
- **Model Optimizations:**
|
||||
- **Weight quantization:** INT4
|
||||
- **Activation quantization:** INT4
|
||||
- **Release Date:** 1/8/2025
|
||||
- **Version:** 1.0
|
||||
- **Model Developers:** Neural Magic
|
||||
|
||||
Quantized version of [ibm-granite/granite-3.1-2b-instruct](https://huggingface.co/ibm-granite/granite-3.1-2b-instruct).
|
||||
It achieves an average score of 61.54 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 61.98.
|
||||
|
||||
### Model Optimizations
|
||||
|
||||
This model was obtained by quantizing the weights of [ibm-granite/granite-3.1-2b-instruct](https://huggingface.co/ibm-granite/granite-3.1-2b-instruct) to INT4 data type, ready for inference with vLLM >= 0.5.2.
|
||||
This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights of the linear operators within transformers blocks are quantized.
|
||||
|
||||
## Deployment
|
||||
|
||||
### Use with vLLM
|
||||
|
||||
This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer
|
||||
from vllm import LLM, SamplingParams
|
||||
|
||||
max_model_len, tp_size = 4096, 1
|
||||
model_name = "neuralmagic/granite-3.1-2b-instruct-quantized.w4a16"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True)
|
||||
sampling_params = SamplingParams(temperature=0.3, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])
|
||||
|
||||
messages_list = [
|
||||
[{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],
|
||||
]
|
||||
|
||||
prompt_token_ids = [tokenizer.apply_chat_template(messages, add_generation_prompt=True) for messages in messages_list]
|
||||
|
||||
outputs = llm.generate(prompt_token_ids=prompt_token_ids, sampling_params=sampling_params)
|
||||
|
||||
generated_text = [output.outputs[0].text for output in outputs]
|
||||
print(generated_text)
|
||||
```
|
||||
|
||||
vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
|
||||
|
||||
## Creation
|
||||
|
||||
This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below.
|
||||
|
||||
<details>
|
||||
<summary>Model Creation Code</summary>
|
||||
|
||||
```bash
|
||||
python quantize.py --model_path ibm-granite/granite-3.1-2b-instruct --quant_path "output_dir/granite-3.1-2b-instruct-quantized.w4a16" --calib_size 1024 --dampening_frac 0.01 --observer mse --group_size 64
|
||||
```
|
||||
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
from llmcompressor.modifiers.quantization import GPTQModifier
|
||||
from llmcompressor.transformers import oneshot, apply
|
||||
import argparse
|
||||
from compressed_tensors.quantization import QuantizationScheme, QuantizationArgs, QuantizationType, QuantizationStrategy
|
||||
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--model_path', type=str)
|
||||
parser.add_argument('--quant_path', type=str)
|
||||
parser.add_argument('--calib_size', type=int, default=256)
|
||||
parser.add_argument('--dampening_frac', type=float, default=0.1)
|
||||
parser.add_argument('--observer', type=str, default="minmax")
|
||||
parser.add_argument('--group_size', type=int, default="128")
|
||||
args = parser.parse_args()
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
args.model_path,
|
||||
device_map="auto",
|
||||
torch_dtype="auto",
|
||||
use_cache=False,
|
||||
trust_remote_code=True,
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(args.model_path)
|
||||
|
||||
|
||||
NUM_CALIBRATION_SAMPLES = args.calib_size
|
||||
DATASET_ID = "neuralmagic/LLM_compression_calibration"
|
||||
DATASET_SPLIT = "train"
|
||||
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
|
||||
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
|
||||
|
||||
def preprocess(example):
|
||||
return {"text": example["text"]}
|
||||
|
||||
ds = ds.map(preprocess)
|
||||
|
||||
def tokenize(sample):
|
||||
return tokenizer(
|
||||
sample["text"],
|
||||
padding=False,
|
||||
truncation=False,
|
||||
add_special_tokens=True,
|
||||
)
|
||||
|
||||
|
||||
ds = ds.map(tokenize, remove_columns=ds.column_names)
|
||||
|
||||
recipe = [
|
||||
GPTQModifier(
|
||||
targets=["Linear"],
|
||||
ignore=["lm_head"],
|
||||
scheme="w4a16",
|
||||
dampening_frac=args.dampening_frac,
|
||||
observer=args.observer,
|
||||
group_size=args.group_size
|
||||
)
|
||||
]
|
||||
oneshot(
|
||||
model=model,
|
||||
dataset=ds,
|
||||
recipe=recipe,
|
||||
num_calibration_samples=args.calib_size,
|
||||
max_seq_length=8196,
|
||||
)
|
||||
|
||||
# Save to disk compressed.
|
||||
model.save_pretrained(quant_path, save_compressed=True)
|
||||
tokenizer.save_pretrained(quant_path)
|
||||
```
|
||||
</details>
|
||||
|
||||
## Evaluation
|
||||
|
||||
The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard), OpenLLM Leaderboard [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands:
|
||||
|
||||
<details>
|
||||
<summary>Evaluation Commands</summary>
|
||||
|
||||
OpenLLM Leaderboard V1:
|
||||
```
|
||||
lm_eval \
|
||||
--model vllm \
|
||||
--model_args pretrained="neuralmagic/granite-3.1-2b-instruct-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
|
||||
--tasks openllm \
|
||||
--write_out \
|
||||
--batch_size auto \
|
||||
--output_path output_dir \
|
||||
--show_config
|
||||
```
|
||||
|
||||
OpenLLM Leaderboard V2:
|
||||
```
|
||||
lm_eval \
|
||||
--model vllm \
|
||||
--model_args pretrained="neuralmagic/granite-3.1-2b-instruct-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
|
||||
--tasks leaderboard \
|
||||
--write_out \
|
||||
--batch_size auto \
|
||||
--output_path output_dir \
|
||||
--show_config
|
||||
```
|
||||
|
||||
#### HumanEval
|
||||
##### Generation
|
||||
```
|
||||
python3 codegen/generate.py \
|
||||
--model neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 \
|
||||
--bs 16 \
|
||||
--temperature 0.2 \
|
||||
--n_samples 50 \
|
||||
--root "." \
|
||||
--dataset humaneval
|
||||
```
|
||||
##### Sanitization
|
||||
```
|
||||
python3 evalplus/sanitize.py \
|
||||
humaneval/neuralmagic--granite-3.1-2b-instruct-quantized.w4a16_vllm_temp_0.2
|
||||
```
|
||||
##### Evaluation
|
||||
```
|
||||
evalplus.evaluate \
|
||||
--dataset humaneval \
|
||||
--samples humaneval/neuralmagic--granite-3.1-2b-instruct-quantized.w4a16_vllm_temp_0.2-sanitized
|
||||
```
|
||||
</details>
|
||||
|
||||
### Accuracy
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Category</th>
|
||||
<th>Metric</th>
|
||||
<th>ibm-granite/granite-3.1-2b-instruct</th>
|
||||
<th>neuralmagic/granite-3.1-2b-instruct-quantized.w4a16</th>
|
||||
<th>Recovery (%)</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td rowspan="7"><b>OpenLLM V1</b></td>
|
||||
<td>ARC-Challenge (Acc-Norm, 25-shot)</td>
|
||||
<td>55.63</td>
|
||||
<td>54.18</td>
|
||||
<td>97.39</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>GSM8K (Strict-Match, 5-shot)</td>
|
||||
<td>60.96</td>
|
||||
<td>62.85</td>
|
||||
<td>103.10</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>HellaSwag (Acc-Norm, 10-shot)</td>
|
||||
<td>75.21</td>
|
||||
<td>73.36</td>
|
||||
<td>97.54</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MMLU (Acc, 5-shot)</td>
|
||||
<td>54.38</td>
|
||||
<td>52.17</td>
|
||||
<td>95.93</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>TruthfulQA (MC2, 0-shot)</td>
|
||||
<td>55.93</td>
|
||||
<td>56.83</td>
|
||||
<td>101.61</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Winogrande (Acc, 5-shot)</td>
|
||||
<td>69.67</td>
|
||||
<td>69.85</td>
|
||||
<td>100.26</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>Average Score</b></td>
|
||||
<td><b>61.98</b></td>
|
||||
<td><b>61.54</b></td>
|
||||
<td><b>99.29</b></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="7"><b>OpenLLM V2</b></td>
|
||||
<td>IFEval (Inst Level Strict Acc, 0-shot)</td>
|
||||
<td>67.99</td>
|
||||
<td>67.63</td>
|
||||
<td>99.47</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>BBH (Acc-Norm, 3-shot)</td>
|
||||
<td>44.11</td>
|
||||
<td>43.22</td>
|
||||
<td>97.98</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Math-Hard (Exact-Match, 4-shot)</td>
|
||||
<td>8.66</td>
|
||||
<td>8.77</td>
|
||||
<td>101.27</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>GPQA (Acc-Norm, 0-shot)</td>
|
||||
<td>28.30</td>
|
||||
<td>28.56</td>
|
||||
<td>100.92</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MUSR (Acc-Norm, 0-shot)</td>
|
||||
<td>35.12</td>
|
||||
<td>35.26</td>
|
||||
<td>100.40</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>MMLU-Pro (Acc, 5-shot)</td>
|
||||
<td>26.87</td>
|
||||
<td>27.27</td>
|
||||
<td>101.49</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>Average Score</b></td>
|
||||
<td><b>35.17</b></td>
|
||||
<td><b>35.12</b></td>
|
||||
<td><b>99.84</b></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="2"><b>Coding</b></td>
|
||||
<td>HumanEval Pass@1</td>
|
||||
<td>53.40</td>
|
||||
<td>52.30</td>
|
||||
<td><b>97.94</b></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
|
||||
## Inference Performance
|
||||
|
||||
|
||||
This model achieves up to 1.9x speedup in single-stream deployment, depending on hardware and use-case scenario.
|
||||
The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.6.6.post1, and [GuideLLM](https://github.com/neuralmagic/guidellm).
|
||||
|
||||
<details>
|
||||
<summary>Benchmarking Command</summary>
|
||||
|
||||
```
|
||||
guidellm --model neuralmagic/granite-3.1-2b-instruct-quantized.w4a16 --target "http://localhost:8000/v1" --data-type emulated --data "prompt_tokens=<prompt_tokens>,generated_tokens=<generated_tokens>" --max seconds 360 --backend aiohttp_server
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Single-stream performance (measured with vLLM version 0.6.6.post1)
|
||||
<table>
|
||||
<tr>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<td></td>
|
||||
<th style="text-align: center;" colspan="7" >Latency (s)</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th>GPU class</th>
|
||||
<th>Model</th>
|
||||
<th>Speedup</th>
|
||||
<th>Code Completion<br>prefill: 256 tokens<br>decode: 1024 tokens</th>
|
||||
<th>Docstring Generation<br>prefill: 768 tokens<br>decode: 128 tokens</th>
|
||||
<th>Code Fixing<br>prefill: 1024 tokens<br>decode: 1024 tokens</th>
|
||||
<th>RAG<br>prefill: 1024 tokens<br>decode: 128 tokens</th>
|
||||
<th>Instruction Following<br>prefill: 256 tokens<br>decode: 128 tokens</th>
|
||||
<th>Multi-turn Chat<br>prefill: 512 tokens<br>decode: 256 tokens</th>
|
||||
<th>Large Summarization<br>prefill: 4096 tokens<br>decode: 512 tokens</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="vertical-align: middle;" rowspan="3" >A5000</td>
|
||||
<td>granite-3.1-2b-instruct</td>
|
||||
<td></td>
|
||||
<td>10.9</td>
|
||||
<td>1.4</td>
|
||||
<td>11.0</td>
|
||||
<td>1.5</td>
|
||||
<td>1.4</td>
|
||||
<td>2.8</td>
|
||||
<td>6.1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>granite-3.1-2b-instruct-quantized.w8a8</td>
|
||||
<td>1.37</td>
|
||||
<td>7.9</td>
|
||||
<td>1.0</td>
|
||||
<td>8.0</td>
|
||||
<td>1.1</td>
|
||||
<td>1.0</td>
|
||||
<td>2.0</td>
|
||||
<td>4.7</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>granite-3.1-2b-instruct-quantized.w4a16<br>(this model)</td>
|
||||
<td>1.94</td>
|
||||
<td>5.4</td>
|
||||
<td>0.7</td>
|
||||
<td>5.5</td>
|
||||
<td>0.8</td>
|
||||
<td>0.7</td>
|
||||
<td>1.4</td>
|
||||
<td>3.4</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="vertical-align: middle;" rowspan="3" >A6000</td>
|
||||
<td>granite-3.1-2b-instruct</td>
|
||||
<td></td>
|
||||
<td>9.8</td>
|
||||
<td>1.3</td>
|
||||
<td>10.0</td>
|
||||
<td>1.3</td>
|
||||
<td>1.3</td>
|
||||
<td>2.6</td>
|
||||
<td>5.4</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>granite-3.1-2b-instruct-quantized.w8a8</td>
|
||||
<td>1.31</td>
|
||||
<td>7.8</td>
|
||||
<td>1.0</td>
|
||||
<td>7.6</td>
|
||||
<td>1.0</td>
|
||||
<td>0.9</td>
|
||||
<td>1.9</td>
|
||||
<td>4.5</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>granite-3.1-2b-instruct-quantized.w4a16<br>(this model)</td>
|
||||
<td>1.87</td>
|
||||
<td>5.1</td>
|
||||
<td>0.7</td>
|
||||
<td>5.2</td>
|
||||
<td>0.7</td>
|
||||
<td>0.7</td>
|
||||
<td>1.3</td>
|
||||
<td>3.1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="vertical-align: middle;" rowspan="3" >L40</td>
|
||||
<td>granite-3.1-2b-instruct</td>
|
||||
<td></td>
|
||||
<td>9.3</td>
|
||||
<td>1.2</td>
|
||||
<td>9.4</td>
|
||||
<td>1.2</td>
|
||||
<td>1.2</td>
|
||||
<td>2.3</td>
|
||||
<td>5.0</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>granite-3.1-2b-instruct-FP8-dynamic</td>
|
||||
<td>1.26</td>
|
||||
<td>7.3</td>
|
||||
<td>0.9</td>
|
||||
<td>7.4</td>
|
||||
<td>1.0</td>
|
||||
<td>0.9</td>
|
||||
<td>1.8</td>
|
||||
<td>4.1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>granite-3.1-2b-instruct-quantized.w4a16<br>(this model)</td>
|
||||
<td>1.88</td>
|
||||
<td>4.8</td>
|
||||
<td>0.6</td>
|
||||
<td>4.9</td>
|
||||
<td>0.6</td>
|
||||
<td>0.6</td>
|
||||
<td>1.2</td>
|
||||
<td>2.8</td>
|
||||
</tr>
|
||||
</table>
|
||||
5
added_tokens.json
Normal file
5
added_tokens.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<|end_of_role|>": 49153,
|
||||
"<|start_of_role|>": 49152,
|
||||
"<|tool_call|>": 49154
|
||||
}
|
||||
65
config.json
Normal file
65
config.json
Normal file
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"_name_or_path": "ibm-granite/granite-3.1-2b-instruct",
|
||||
"architectures": [
|
||||
"GraniteForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.1,
|
||||
"attention_multiplier": 0.015625,
|
||||
"bos_token_id": 0,
|
||||
"embedding_multiplier": 12.0,
|
||||
"eos_token_id": 0,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2048,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 8192,
|
||||
"logits_scaling": 8.0,
|
||||
"max_position_embeddings": 131072,
|
||||
"mlp_bias": false,
|
||||
"model_type": "granite",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 40,
|
||||
"num_key_value_heads": 8,
|
||||
"pad_token_id": 0,
|
||||
"quantization_config": {
|
||||
"config_groups": {
|
||||
"group_0": {
|
||||
"input_activations": null,
|
||||
"output_activations": null,
|
||||
"targets": [
|
||||
"Linear"
|
||||
],
|
||||
"weights": {
|
||||
"actorder": null,
|
||||
"block_structure": null,
|
||||
"dynamic": false,
|
||||
"group_size": 64,
|
||||
"num_bits": 4,
|
||||
"observer": "mse",
|
||||
"observer_kwargs": {},
|
||||
"strategy": "group",
|
||||
"symmetric": true,
|
||||
"type": "int"
|
||||
}
|
||||
}
|
||||
},
|
||||
"format": "pack-quantized",
|
||||
"global_compression_ratio": 2.0771812517233883,
|
||||
"ignore": [
|
||||
"lm_head"
|
||||
],
|
||||
"kv_cache_scheme": null,
|
||||
"quant_method": "compressed-tensors",
|
||||
"quantization_status": "compressed",
|
||||
"sparsity_config": {}
|
||||
},
|
||||
"residual_multiplier": 0.22,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 5000000.0,
|
||||
"tie_word_embeddings": true,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.47.1",
|
||||
"use_cache": true,
|
||||
"vocab_size": 49155
|
||||
}
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 0,
|
||||
"pad_token_id": 0,
|
||||
"transformers_version": "4.47.1"
|
||||
}
|
||||
48892
merges.txt
Normal file
48892
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:80eacfccfdee065bac8252fc7e806717b506678e7e35090eb0b3246380e9b4fb
|
||||
size 1695488752
|
||||
11
recipe.yaml
Normal file
11
recipe.yaml
Normal file
@@ -0,0 +1,11 @@
|
||||
quant_stage:
|
||||
quant_modifiers:
|
||||
GPTQModifier:
|
||||
sequential_update: true
|
||||
dampening_frac: 0.01
|
||||
ignore: [lm_head]
|
||||
config_groups:
|
||||
group_0:
|
||||
weights: {num_bits: 4, type: int, symmetric: true, strategy: group, group_size: 64,
|
||||
observer: mse}
|
||||
targets: [Linear]
|
||||
35
special_tokens_map.json
Normal file
35
special_tokens_map.json
Normal file
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|start_of_role|>",
|
||||
"<|end_of_role|>",
|
||||
"<|tool_call|>"
|
||||
],
|
||||
"bos_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
244954
tokenizer.json
Normal file
244954
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
199
tokenizer_config.json
Normal file
199
tokenizer_config.json
Normal file
@@ -0,0 +1,199 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<fim_prefix>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "<fim_middle>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"3": {
|
||||
"content": "<fim_suffix>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"4": {
|
||||
"content": "<fim_pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"5": {
|
||||
"content": "<filename>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"6": {
|
||||
"content": "<gh_stars>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"7": {
|
||||
"content": "<issue_start>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"8": {
|
||||
"content": "<issue_comment>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"9": {
|
||||
"content": "<issue_closed>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"10": {
|
||||
"content": "<jupyter_start>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"11": {
|
||||
"content": "<jupyter_text>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"12": {
|
||||
"content": "<jupyter_code>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"13": {
|
||||
"content": "<jupyter_output>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"14": {
|
||||
"content": "<empty_output>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"15": {
|
||||
"content": "<commit_before>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"16": {
|
||||
"content": "<commit_msg>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"17": {
|
||||
"content": "<commit_after>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"18": {
|
||||
"content": "<reponame>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49152": {
|
||||
"content": "<|start_of_role|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49153": {
|
||||
"content": "<|end_of_role|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49154": {
|
||||
"content": "<|tool_call|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"<|start_of_role|>",
|
||||
"<|end_of_role|>",
|
||||
"<|tool_call|>"
|
||||
],
|
||||
"bos_token": "<|end_of_text|>",
|
||||
"chat_template": "{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content'] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"Knowledge Cutoff Date: April 2024.\nToday's Date: \" + strftime_now('%B %d, %Y') + \".\nYou are Granite, developed by IBM.\" %}\n {%- if tools and documents %}\n {%- set system_message = system_message + \" You are a helpful AI assistant with access to the following tools. When a tool is required to answer the user's query, respond with <|tool_call|> followed by a JSON list of tools used. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.\n\nWrite the response to the user's input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data.\" %}\n {%- elif tools %}\n {%- set system_message = system_message + \" You are a helpful AI assistant with access to the following tools. When a tool is required to answer the user's query, respond with <|tool_call|> followed by a JSON list of tools used. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request.\" %}\n {%- elif documents %}\n {%- set system_message = system_message + \" Write the response to the user's input by strictly aligning with the facts in the provided documents. If the information needed to answer the question is not available in the documents, inform the user that the question cannot be answered based on the available data.\" %}\n {%- else %}\n {%- set system_message = system_message + \" You are a helpful AI assistant.\" %} \n {%- endif %}\n {%- if 'citations' in controls and documents %}\n {%- set system_message = system_message + '\n\nIn your response, use the symbols <co> and </co> to indicate when a fact comes from a document in the search result, e.g <co>0</co> for a fact from document 0. Afterwards, list all the citations with their corresponding documents in an ordered list.' %}\n {%- endif %}\n {%- if 'hallucinations' in controls and documents %}\n {%- set system_message = system_message + '\n\nFinally, after the response is written, include a numbered list of sentences from the response that are potentially hallucinated and not based in the documents.' %}\n {%- endif %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{{- '<|start_of_role|>system<|end_of_role|>' + system_message + '<|end_of_text|>\n' }}\n{%- if tools %}\n {{- '<|start_of_role|>tools<|end_of_role|>' }}\n {{- tools | tojson(indent=4) }}\n {{- '<|end_of_text|>\n' }}\n{%- endif %}\n{%- if documents %}\n {{- '<|start_of_role|>documents<|end_of_role|>' }}\n {%- for document in documents %}\n {{- 'Document ' + loop.index0 | string + '\n' }}\n {{- document['text'] }}\n {%- if not loop.last %}\n {{- '\n\n'}}\n {%- endif%}\n {%- endfor %}\n {{- '<|end_of_text|>\n' }}\n{%- endif %}\n{%- for message in loop_messages %}\n {{- '<|start_of_role|>' + message['role'] + '<|end_of_role|>' + message['content'] + '<|end_of_text|>\n' }}\n {%- if loop.last and add_generation_prompt %}\n {{- '<|start_of_role|>assistant' }}\n {%- if controls %}\n {{- ' ' + controls | tojson()}}\n {%- endif %}\n {{- '<|end_of_role|>' }}\n {%- endif %}\n{%- endfor %}",
|
||||
"clean_up_tokenization_spaces": true,
|
||||
"eos_token": "<|end_of_text|>",
|
||||
"errors": "replace",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 9223372036854775807,
|
||||
"pad_token": "<|end_of_text|>",
|
||||
"padding_side": "left",
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": "<|end_of_text|>",
|
||||
"vocab_size": 49152
|
||||
}
|
||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user