初始化项目，由ModelHub XC社区提供模型
Model: neuralmagic/Llama-2-7b-chat-quantized.w4a16 Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
model.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
259
README.md
Normal file
259
README.md
Normal file
@@ -0,0 +1,259 @@
|
|||||||
|
---
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
license: llama2
|
||||||
|
---
|
||||||
|
|
||||||
|
# Llama-2-7b-chat-quantized.w4a16
|
||||||
|
|
||||||
|
## Model Overview
|
||||||
|
- **Model Architecture:** Llama-2
|
||||||
|
- **Input:** Text
|
||||||
|
- **Output:** Text
|
||||||
|
- **Model Optimizations:**
|
||||||
|
- **Weight quantization:** INT4
|
||||||
|
- **Intended Use Cases:** Intended for commercial and research use in English. Similarly to [Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat), this models is intended for assistant-like chat.
|
||||||
|
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
|
||||||
|
- **Release Date:** 7/11/2024
|
||||||
|
- **Version:** 1.0
|
||||||
|
- **License(s)**: [LLama2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/blob/main/LICENSE.txt)
|
||||||
|
- **Model Developers:** Neural Magic
|
||||||
|
|
||||||
|
Quantized version of [Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat).
|
||||||
|
It achieves an average score of 52.64 on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) benchmark (version 1), whereas the unquantized model achieves 53.40.
|
||||||
|
|
||||||
|
### Model Optimizations
|
||||||
|
|
||||||
|
This model was obtained by quantizing the weights of [Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat) to INT4 data type.
|
||||||
|
This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 25%.
|
||||||
|
|
||||||
|
Only the weights of the linear operators within transformers blocks are quantized. Symmetric group-wise quantization is applied, in which a linear scaling per group maps the INT4 and floating point representations of the quantized weights.
|
||||||
|
[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) is used for quantization with 10% damping factor, group-size as 128 and 512 sequences sampled from [Open-Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus).
|
||||||
|
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
### Use with vLLM
|
||||||
|
|
||||||
|
This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from vllm import LLM, SamplingParams
|
||||||
|
from transformers import AutoTokenizer
|
||||||
|
|
||||||
|
model_id = "neuralmagic/Llama-2-7b-chat-quantized.w4a16"
|
||||||
|
|
||||||
|
sampling_params = SamplingParams(temperature=0.6, top_p=0.9, max_tokens=256)
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
|
||||||
|
{"role": "user", "content": "Who are you?"},
|
||||||
|
]
|
||||||
|
|
||||||
|
prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||||
|
|
||||||
|
llm = LLM(model=model_id, tensor_parallel_size=2)
|
||||||
|
|
||||||
|
outputs = llm.generate(prompts, sampling_params)
|
||||||
|
|
||||||
|
generated_text = outputs[0].outputs[0].text
|
||||||
|
print(generated_text)
|
||||||
|
```
|
||||||
|
|
||||||
|
vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
|
||||||
|
|
||||||
|
### Use with transformers
|
||||||
|
|
||||||
|
This model is supported by Transformers leveraging the integration with the [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) data format.
|
||||||
|
The following example contemplates how the model can be used using the `generate()` function.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||||
|
|
||||||
|
model_id = "neuralmagic/Llama-2-7b-chat-quantized.w4a16"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
model_id,
|
||||||
|
torch_dtype="auto",
|
||||||
|
device_map="auto",
|
||||||
|
)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
|
||||||
|
{"role": "user", "content": "Who are you?"},
|
||||||
|
]
|
||||||
|
|
||||||
|
input_ids = tokenizer.apply_chat_template(
|
||||||
|
messages,
|
||||||
|
add_generation_prompt=True,
|
||||||
|
return_tensors="pt"
|
||||||
|
).to(model.device)
|
||||||
|
|
||||||
|
terminators = [
|
||||||
|
tokenizer.eos_token_id,
|
||||||
|
tokenizer.convert_tokens_to_ids("<|eot_id|>")
|
||||||
|
]
|
||||||
|
|
||||||
|
outputs = model.generate(
|
||||||
|
input_ids,
|
||||||
|
max_new_tokens=256,
|
||||||
|
eos_token_id=terminators,
|
||||||
|
do_sample=True,
|
||||||
|
temperature=0.6,
|
||||||
|
top_p=0.9,
|
||||||
|
)
|
||||||
|
response = outputs[0][input_ids.shape[-1]:]
|
||||||
|
print(tokenizer.decode(response, skip_special_tokens=True))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Creation
|
||||||
|
|
||||||
|
This model was created by applying the [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) library as presented in the code snipet below.
|
||||||
|
Although AutoGPTQ was used for this particular model, Neural Magic is transitioning to using [llm-compressor](https://github.com/vllm-project/llm-compressor) which supports several quantization schemes and models not supported by AutoGPTQ.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import AutoTokenizer
|
||||||
|
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
|
||||||
|
from datasets import load_dataset
|
||||||
|
import random
|
||||||
|
|
||||||
|
model_id = "meta-llama/Llama-2-7b-chat"
|
||||||
|
|
||||||
|
num_samples = 512
|
||||||
|
max_seq_len = 4096
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||||
|
|
||||||
|
preprocess_fn = lambda example: {"text": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n{text}".format_map(example)}
|
||||||
|
|
||||||
|
dataset_name = "neuralmagic/LLM_compression_calibration"
|
||||||
|
dataset = load_dataset(dataset_name, split="train")
|
||||||
|
ds = dataset.shuffle().select(range(num_samples))
|
||||||
|
ds = ds.map(preprocess_fn)
|
||||||
|
|
||||||
|
examples = [
|
||||||
|
tokenizer(
|
||||||
|
example["text"], padding=False, max_length=max_seq_len, truncation=True,
|
||||||
|
) for example in ds
|
||||||
|
]
|
||||||
|
|
||||||
|
quantize_config = BaseQuantizeConfig(
|
||||||
|
bits=4,
|
||||||
|
group_size=128,
|
||||||
|
desc_act=True,
|
||||||
|
model_file_base_name="model",
|
||||||
|
damp_percent=0.1,
|
||||||
|
)
|
||||||
|
|
||||||
|
model = AutoGPTQForCausalLM.from_pretrained(
|
||||||
|
model_id,
|
||||||
|
quantize_config,
|
||||||
|
device_map="auto",
|
||||||
|
)
|
||||||
|
|
||||||
|
model.quantize(examples)
|
||||||
|
model.save_pretrained("Llama-2-7b-chat-quantized.w4a16")
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Evaluation
|
||||||
|
|
||||||
|
The model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) leaderboard tasks (version 1) with the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/383bbd54bc621086e05aa1b030d8d4d5635b25e6) (commit 383bbd54bc621086e05aa1b030d8d4d5635b25e6) and the [vLLM](https://docs.vllm.ai/en/stable/) engine, using the following command:
|
||||||
|
```
|
||||||
|
lm_eval \
|
||||||
|
--model vllm \
|
||||||
|
--model_args pretrained="neuralmagic/Llama-2-7b-chat-quantized.w4a16",dtype=auto,tensor_parallel_size=2,gpu_memory_utilization=0.4,add_bos_token=True,max_model_len=4096 \
|
||||||
|
--tasks openllm \
|
||||||
|
--batch_size auto
|
||||||
|
```
|
||||||
|
|
||||||
|
### Accuracy
|
||||||
|
|
||||||
|
#### Open LLM Leaderboard evaluation scores
|
||||||
|
<table>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Benchmark</strong>
|
||||||
|
</td>
|
||||||
|
<td><strong>Llama-2-7b-chat </strong>
|
||||||
|
</td>
|
||||||
|
<td><strong>Llama-2-7b-chat-quantized.w4a16(this model)</strong>
|
||||||
|
</td>
|
||||||
|
<td><strong>Recovery</strong>
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>MMLU (5-shot)
|
||||||
|
</td>
|
||||||
|
<td>47.33
|
||||||
|
</td>
|
||||||
|
<td>46.7
|
||||||
|
</td>
|
||||||
|
<td>98.66%
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>ARC Challenge (25-shot)
|
||||||
|
</td>
|
||||||
|
<td>53.24
|
||||||
|
</td>
|
||||||
|
<td>53.75
|
||||||
|
</td>
|
||||||
|
<td>100.95%
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>GSM-8K (5-shot, strict-match)
|
||||||
|
</td>
|
||||||
|
<td>23.19
|
||||||
|
</td>
|
||||||
|
<td>21.75
|
||||||
|
</td>
|
||||||
|
<td>93.79%
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Hellaswag (10-shot)
|
||||||
|
</td>
|
||||||
|
<td>78.64
|
||||||
|
</td>
|
||||||
|
<td>77.35
|
||||||
|
</td>
|
||||||
|
<td>98.36%
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Winogrande (5-shot)
|
||||||
|
</td>
|
||||||
|
<td>72.45
|
||||||
|
</td>
|
||||||
|
<td>70.4
|
||||||
|
</td>
|
||||||
|
<td>98.66%
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>TruthfulQA (0-shot)
|
||||||
|
</td>
|
||||||
|
<td>45.58
|
||||||
|
</td>
|
||||||
|
<td>45.94
|
||||||
|
</td>
|
||||||
|
<td>100.78%
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Average</strong>
|
||||||
|
</td>
|
||||||
|
<td><strong>53.40</strong>
|
||||||
|
</td>
|
||||||
|
<td><strong>52.64</strong>
|
||||||
|
</td>
|
||||||
|
<td><strong>98.58%</strong>
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
</table>
|
||||||
42
config.json
Normal file
42
config.json
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
{
|
||||||
|
"_name_or_path": "meta-llama/Llama-2-7b-chat-hf",
|
||||||
|
"architectures": [
|
||||||
|
"LlamaForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 1,
|
||||||
|
"eos_token_id": 2,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 11008,
|
||||||
|
"max_position_embeddings": 4096,
|
||||||
|
"mlp_bias": false,
|
||||||
|
"model_type": "llama",
|
||||||
|
"num_attention_heads": 32,
|
||||||
|
"num_hidden_layers": 32,
|
||||||
|
"num_key_value_heads": 32,
|
||||||
|
"pretraining_tp": 1,
|
||||||
|
"quantization_config": {
|
||||||
|
"bits": 4,
|
||||||
|
"checkpoint_format": "gptq",
|
||||||
|
"damp_percent": 0.1,
|
||||||
|
"desc_act": true,
|
||||||
|
"group_size": 128,
|
||||||
|
"model_file_base_name": "model",
|
||||||
|
"model_name_or_path": null,
|
||||||
|
"quant_method": "gptq",
|
||||||
|
"static_groups": false,
|
||||||
|
"sym": true,
|
||||||
|
"true_sequential": true
|
||||||
|
},
|
||||||
|
"rms_norm_eps": 1e-05,
|
||||||
|
"rope_scaling": null,
|
||||||
|
"rope_theta": 10000.0,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"torch_dtype": "float16",
|
||||||
|
"transformers_version": "4.41.2",
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 32000
|
||||||
|
}
|
||||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
|||||||
|
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:8e8b5b99101a9a2ff5b03e353393d5e3febcc1ba7fc972db0378d8d3bb174818
|
||||||
|
size 3893970760
|
||||||
13
quantize_config.json
Normal file
13
quantize_config.json
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"bits": 4,
|
||||||
|
"group_size": 128,
|
||||||
|
"damp_percent": 0.1,
|
||||||
|
"desc_act": true,
|
||||||
|
"static_groups": false,
|
||||||
|
"sym": true,
|
||||||
|
"true_sequential": true,
|
||||||
|
"model_name_or_path": null,
|
||||||
|
"model_file_base_name": "model",
|
||||||
|
"quant_method": "gptq",
|
||||||
|
"checkpoint_format": "gptq"
|
||||||
|
}
|
||||||
23
special_tokens_map.json
Normal file
23
special_tokens_map.json
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"bos_token": {
|
||||||
|
"content": "<s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"eos_token": {
|
||||||
|
"content": "</s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
"unk_token": {
|
||||||
|
"content": "<unk>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
93397
tokenizer.json
Normal file
93397
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
Binary file not shown.
42
tokenizer_config.json
Normal file
42
tokenizer_config.json
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
{
|
||||||
|
"add_bos_token": true,
|
||||||
|
"add_eos_token": false,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"0": {
|
||||||
|
"content": "<unk>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"1": {
|
||||||
|
"content": "<s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"2": {
|
||||||
|
"content": "</s>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"bos_token": "<s>",
|
||||||
|
"chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\n' + system_message + '\\n<</SYS>>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + '[INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}",
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "</s>",
|
||||||
|
"legacy": false,
|
||||||
|
"model_max_length": 1000000000000000019884624838656,
|
||||||
|
"pad_token": null,
|
||||||
|
"padding_side": "right",
|
||||||
|
"sp_model_kwargs": {},
|
||||||
|
"tokenizer_class": "LlamaTokenizer",
|
||||||
|
"unk_token": "<unk>",
|
||||||
|
"use_default_system_prompt": false
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user