96 lines
2.9 KiB
Markdown
96 lines
2.9 KiB
Markdown
---
|
|
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
|
inference: true
|
|
model_type: Llama
|
|
hardware_tag:
|
|
- Intel Xeon
|
|
license: apache-2.0
|
|
license_name: apache-2.0
|
|
license_link: >-
|
|
https://choosealicense.com/licenses/apache-2.0/
|
|
name: RedHatAI/TinyLlama-1.1B-Chat-v1.0-pruned2.4
|
|
description: >-
|
|
Pruned TinyLlama model.
|
|
readme: >-
|
|
https://huggingface.co/RedHatAI/TinyLlama-1.1B-Chat-v1.0-pruned2.4/blob/main/README.md
|
|
tags:
|
|
- nm-vllm
|
|
- sparse
|
|
---
|
|
## TinyLlama-1.1B-Chat-v1.0-pruned2.4
|
|
This repo contains model files for [TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) optimized for [NM-vLLM](https://github.com/neuralmagic/nm-vllm), a high-throughput serving engine for compressed LLMs.
|
|
|
|
This model was pruned with [SparseGPT](https://arxiv.org/abs/2301.00774), using [SparseML](https://github.com/neuralmagic/sparseml).
|
|
|
|
## Inference
|
|
Install [NM-vLLM](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage:
|
|
```bash
|
|
pip install nm-vllm[sparse]
|
|
```
|
|
Run in a Python pipeline for local inference:
|
|
```python
|
|
from vllm import LLM, SamplingParams
|
|
|
|
model = LLM("nm-testing/TinyLlama-1.1B-Chat-v1.0-pruned2.4", sparsity="semi_structured_sparse_w16a16")
|
|
prompt = "How to make banana bread?"
|
|
formatted_prompt = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
|
|
|
|
sampling_params = SamplingParams(max_tokens=100,temperature=0,repetition_penalty=1.3)
|
|
outputs = model.generate(formatted_prompt, sampling_params=sampling_params)
|
|
print(outputs[0].outputs[0].text)
|
|
"""
|
|
Banana bread is a delicious dessert that is made with bananas. Here is how to make banana bread:
|
|
|
|
1. Firstly, you need to cut bananas into small pieces.
|
|
2. Then, you need to slice the bananas into small pieces
|
|
"""
|
|
```
|
|
|
|
## Prompt template
|
|
|
|
```
|
|
<|im_start|>user
|
|
{prompt}<|im_end|>
|
|
<|im_start|>assistant
|
|
|
|
```
|
|
|
|
## Sparsification
|
|
For details on how this model was sparsified, see the `recipe.yaml` in this repo and follow the instructions below.
|
|
|
|
Install [SparseML](https://github.com/neuralmagic/sparseml):
|
|
```bash
|
|
git clone https://github.com/neuralmagic/sparseml
|
|
pip install -e "sparseml[transformers]"
|
|
```
|
|
|
|
Replace the recipe as you like and run this one-shot compression script to apply SparseGPT:
|
|
```python
|
|
import sparseml.transformers
|
|
|
|
original_model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
|
|
calibration_dataset = "open_platypus"
|
|
output_directory = "output/"
|
|
|
|
recipe = """
|
|
test_stage:
|
|
obcq_modifiers:
|
|
SparseGPTModifier:
|
|
sparsity: 0.5
|
|
sequential_update: true
|
|
mask_structure: '2:4'
|
|
targets: ['re:model.layers.\d*$']
|
|
"""
|
|
|
|
# Apply SparseGPT to the model
|
|
sparseml.transformers.oneshot(
|
|
model=original_model_name,
|
|
dataset=calibration_dataset,
|
|
recipe=recipe,
|
|
output_dir=output_directory,
|
|
)
|
|
```
|
|
|
|
## Slack
|
|
|
|
For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ) |