90 lines
3.0 KiB
Markdown
90 lines
3.0 KiB
Markdown
---
|
|
library_name: transformers
|
|
base_model: nvidia/Llama-3.1-Minitron-4B-Width-Base
|
|
datasets:
|
|
- teknium/OpenHermes-2.5
|
|
pipeline_tag: text-generation
|
|
license: other
|
|
license_name: nvidia-open-model-license
|
|
license_link: >-
|
|
https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
|
|
---
|
|
|
|
# Llama-3.1-Minitron-4B-Chat
|
|
|
|
This is an instruction-tuned version of [nvidia/Llama-3.1-Minitron-4B-Width-Base](https://huggingface.co/nvidia/Llama-3.1-Minitron-4B-Width-Base) that has underwent **supervised fine-tuning** with 64k instruction-response pairs from the [teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) dataset on a single **A100 40GB** GPU.
|
|
|
|
## How to use
|
|
### Chat Format
|
|
|
|
Given the nature of the training data, the Llama-3.1-Minitron-4B chat model is best suited for prompts using the chat format as follows.
|
|
You can provide the prompt as a question with a generic template as follows:
|
|
```markdown
|
|
<|im_start|>system
|
|
You are a helpful assistant.<|im_end|>
|
|
<|im_start|>user
|
|
Question?<|im_end|>
|
|
<|im_start|>assistant
|
|
```
|
|
|
|
For example:
|
|
```markdown
|
|
<|im_start|>system
|
|
You are a helpful assistant.<|im_end|>
|
|
<|im_start|>user
|
|
How to explain Internet for a medieval knight?<|im_end|>
|
|
<|im_start|>assistant
|
|
```
|
|
where the model generates the text after `<|im_start|>assistant` .
|
|
|
|
### Sample inference code
|
|
|
|
Support for this model will be added in the upcoming transformers release. In the meantime, please **install the library from source**:
|
|
|
|
```
|
|
pip install git+https://github.com/huggingface/transformers
|
|
```
|
|
|
|
This code snippets show how to get quickly started with running the model on a GPU:
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
|
|
|
torch.random.manual_seed(0)
|
|
|
|
model_id = "rasyosef/Llama-3.1-Minitron-4B-Chat"
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
device_map="auto",
|
|
torch_dtype=torch.bfloat16
|
|
)
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
|
|
messages = [
|
|
{"role": "system", "content": "You are a helpful AI assistant."},
|
|
{"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"},
|
|
{"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."},
|
|
{"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"},
|
|
]
|
|
|
|
pipe = pipeline(
|
|
"text-generation",
|
|
model=model,
|
|
tokenizer=tokenizer,
|
|
)
|
|
|
|
generation_args = {
|
|
"max_new_tokens": 256,
|
|
"return_full_text": False,
|
|
"temperature": 0.0,
|
|
"do_sample": False,
|
|
}
|
|
|
|
output = pipe(messages, **generation_args)
|
|
print(output[0]['generated_text'])
|
|
```
|
|
|
|
Note: If you want to use flash attention, call _AutoModelForCausalLM.from_pretrained()_ with _attn_implementation="flash_attention_2"_
|