Intel/gemma-2b-int4-inc
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Intel/gemma-2b-int4-inc
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-09 14:27:13 +08:00
commit 8ee7a4803f
10 changed files with 333 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

49
.gitattributes vendored Normal file
View File

@@ -0,0 +1,49 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

126
README.md Normal file
View File

@@ -0,0 +1,126 @@
---
license: apache-2.0
datasets:
- NeelNanda/pile-10k
---
## Model Details
This model is an int4 model with group_size 128 of [google/gemma-2b](https://huggingface.co/google/gemma-2b) generated by [intel/auto-round](https://github.com/intel/auto-round).
### Use the model
### INT4 Inference with ITREX on CPU
Install the latest [intel-extension-for-transformers](
https://github.com/intel/intel-extension-for-transformers)
```python
from intel_extension_for_transformers.transformers import AutoModelForCausalLM
from transformers import AutoTokenizer
quantized_model_dir = "Intel/gemma-2b-int4-inc"
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
device_map="auto",
trust_remote_code=False,
use_neural_speed=False,
)
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, use_fast=True)
print(tokenizer.decode(model.generate(**tokenizer("There is a girl who likes adventure,", return_tensors="pt").to(model.device),max_new_tokens=50)[0]))
"""
<bos>There is a girl who likes adventure, and she is a girl who likes to travel. She is a girl who likes to explore the world and see new things. She is a girl who likes to meet new people and learn about their cultures. She is a girl who likes to take risks
"""
```
### INT4 Inference with AutoGPTQ's kernel
```python
##pip install auto-gptq
from transformers import AutoModelForCausalLM, AutoTokenizer
quantized_model_dir = "Intel/gemma-2b-int4-inc"
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
device_map="auto",
trust_remote_code=False,
)
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, use_fast=True)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
##<bos>There is a girl who likes adventure, and she is a girl who likes to travel. She is a girl who likes to explore the world and see new things. She is a girl who likes to meet new people and learn about their cultures. She is a girl who likes to take risks
```
### Evaluate the model
pip3 install lm-eval==0.4.2
pip install auto-gptq
Please note that there is a discrepancy between the baseline result and the official data, which is a known issue within the official model card community.
```bash
lm_eval --model hf --model_args pretrained="Intel/gemma-2b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu --batch_size 16
```
| Metric | BF16 | FP16 | AutoRound v0.1 | AutoRound v0.2 |
| -------------- | ------ | ------ | -------------- | -------------- |
| Avg. | 0.5263 | 0.5277 | 0.5235 | 0.5248 |
| mmlu | 0.3287 | 0.3287 | 0.3297 | 0.3309 |
| lambada_openai | 0.6344 | 0.6375 | 0.6307 | 0.6379 |
| hellaswag | 0.5273 | 0.5281 | 0.5159 | 0.5184 |
| winogrande | 0.6504 | 0.6488 | 0.6543 | 0.6575 |
| piqa | 0.7671 | 0.7720 | 0.7612 | 0.7606 |
| truthfulqa_mc1 | 0.2203 | 0.2203 | 0.2203 | 0.2191 |
| openbookqa | 0.2980 | 0.3020 | 0.3000 | 0.3060 |
| boolq | 0.6927 | 0.6936 | 0.6939 | 0.6966 |
| arc_easy | 0.7420 | 0.7403 | 0.7353 | 0.7357 |
| arc_challenge | 0.4019 | 0.4061 | 0.3933 | 0.3857 |
Here is the sample command to reproduce the model
```bash
git clone https://github.com/intel/auto-round
cd auto-round/examples/language-modeling
pip install -r requirements.txt
python3 main.py \
--model_name google/gemma-2b \
--device 0 \
--group_size 128 \
--bits 4 \
--iters 400 \
--model_dtype "float16" \
--deployment_device 'gpu' \
--output_dir "./tmp_autoround"
```
## Ethical Considerations and Limitations
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
Therefore, before deploying any applications of the model, developers should perform safety testing.
## Caveats and Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
Here are a couple of useful links to learn more about Intel's AI software:
* Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
* Intel Extension for Transformers [link](https://github.com/intel/intel-extension-for-transformers)
## Disclaimer
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

49
config.json Normal file
View File

@@ -0,0 +1,49 @@
{
"_name_or_path": "/models/gemma-2b",
"architectures": [
"GemmaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 2,
"eos_token_id": 1,
"head_dim": 256,
"hidden_act": "gelu",
"hidden_activation": null,
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 16384,
"max_position_embeddings": 8192,
"model_type": "gemma",
"num_attention_heads": 8,
"num_hidden_layers": 18,
"num_key_value_heads": 1,
"pad_token_id": 0,
"quantization_config": {
"autoround_version": "0.2.0.dev",
"bits": 4,
"damp_percent": 0.01,
"desc_act": false,
"enable_minmax_tuning": true,
"enable_quanted_input": true,
"group_size": 128,
"is_marlin_format": false,
"iters": 400,
"lr": 0.0025,
"minmax_lr": 0.0025,
"model_file_base_name": "model",
"model_name_or_path": null,
"quant_method": "gptq",
"scale_dtype": "float16",
"static_groups": false,
"sym": false,
"true_sequential": false
},
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 10000.0,
"torch_dtype": "float16",
"transformers_version": "4.40.2",
"use_cache": true,
"vocab_size": 256000
}

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcbf563b9667464d9217348e712763af0ae6acd26c5b53dc15483c18d51e910d
size 3130472744

20
quantize_config.json Normal file
View File

@@ -0,0 +1,20 @@
{
"bits": 4,
"group_size": 128,
"damp_percent": 0.01,
"desc_act": false,
"static_groups": false,
"sym": false,
"true_sequential": false,
"model_name_or_path": null,
"model_file_base_name": "model",
"is_marlin_format": false,
"quant_method": "intel/auto-round",
"autoround_version": "0.2.0.dev",
"iters": 400,
"lr": 0.0025,
"minmax_lr": 0.0025,
"enable_minmax_tuning": true,
"enable_quanted_input": true,
"scale_dtype": "float16"
}

30
special_tokens_map.json Normal file
View File

@@ -0,0 +1,30 @@
{
"bos_token": {
"content": "<bos>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<eos>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<pad>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4db21bfaffa1fd75fd741df2d95dc51e539d5cc38b07934bae0d7d129db90662
size 17477581

3
tokenizer.model Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:61a7b147390c64585d6c3543dd6fc636906c9af3865a5548f27f31aee1d4c8e2
size 4241003

49
tokenizer_config.json Normal file
View File

@@ -0,0 +1,49 @@
{
"add_bos_token": true,
"add_eos_token": false,
"added_tokens_decoder": {
"0": {
"content": "<pad>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<eos>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "<bos>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"3": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"bos_token": "<bos>",
"clean_up_tokenization_spaces": false,
"eos_token": "<eos>",
"legacy": null,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<pad>",
"sp_model_kwargs": {},
"spaces_between_special_tokens": false,
"tokenizer_class": "GemmaTokenizer",
"unk_token": "<unk>",
"use_default_system_prompt": false
}