63 lines
1.7 KiB
Markdown
63 lines
1.7 KiB
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- text-generation
|
||
|
|
- casual-lm
|
||
|
|
- sft
|
||
|
|
- trl
|
||
|
|
base_model: Qwen/Qwen2.5-3B
|
||
|
|
datasets:
|
||
|
|
- HuggingFaceTB/smoltalk
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen-2.5-3B Smoltalk SFT
|
||
|
|
|
||
|
|
This is a fine-tuned version of the 3-billion parameter **Qwen/Qwen2.5-3B** base model. It has been instruction fine-tuned via Low-Rank Adaptation (LoRA) and fully merged.
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
- **Base Model:** Qwen/Qwen2.5-3B
|
||
|
|
- **Fine-tuning Dataset:** HuggingFaceTB/smoltalk (everyday-conversations subset)
|
||
|
|
- **Methodology:** Supervised Fine-Tuning (SFT) using TRL
|
||
|
|
- **Hardware Used:** 1 x NVIDIA L4 GPU (24GB VRAM)
|
||
|
|
|
||
|
|
## How to Get Started
|
||
|
|
|
||
|
|
You can load and use this model directly with the Hugging Face `pipeline` API.
|
||
|
|
|
||
|
|
```python
|
||
|
|
import torch
|
||
|
|
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
|
||
|
|
|
||
|
|
MODEL_ID = "Kerassy/qwen-2.5-3b-smoltalk-sft"
|
||
|
|
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
||
|
|
MODEL_ID,
|
||
|
|
torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
|
||
|
|
device_map="auto"
|
||
|
|
)
|
||
|
|
|
||
|
|
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
|
||
|
|
|
||
|
|
messages = [
|
||
|
|
{"role": "user", "content": "Why is the sky blue?"}
|
||
|
|
]
|
||
|
|
|
||
|
|
formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
|
|
|
||
|
|
outputs = pipe(
|
||
|
|
formatted_prompt,
|
||
|
|
max_new_tokens=128,
|
||
|
|
do_sample=True,
|
||
|
|
temperature=0.7,
|
||
|
|
top_k=40,
|
||
|
|
clean_up_tokenization_spaces=False,
|
||
|
|
pad_token_id=tokenizer.pad_token_id,
|
||
|
|
eos_token_id=tokenizer.encode("<|end|>")[0] if "<|end|>" in tokenizer.get_vocab() else tokenizer.eos_token_id
|
||
|
|
)
|
||
|
|
|
||
|
|
print(outputs[0]['generated_text'])
|