49 lines
953 B
Markdown
49 lines
953 B
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
base_model:
|
||
|
|
- Qwen/Qwen3-1.7B-Base
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen3 1.7B Smoltalk
|
||
|
|
|
||
|
|
Qwen3 1.7B just SFT'd on [smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk).
|
||
|
|
|
||
|
|
Trained using TRL with:
|
||
|
|
|
||
|
|
- 4x H100 from lambda labs
|
||
|
|
- epochs: 2 (i.e. 3,300 steps using our batch size)
|
||
|
|
- effective batch size: 128 (4 per gpu, 8 grad accumulation steps)
|
||
|
|
- warmup ratio: 0.03
|
||
|
|
- weight decay: 0.01
|
||
|
|
- learning rate: Forgot, will come back to add later.
|
||
|
|
- learning rate scheduler: cosine
|
||
|
|
- final training loss: 0.6432
|
||
|
|
|
||
|
|
|
||
|
|
|Benchmark|Score|
|
||
|
|
|---------|-----|
|
||
|
|
|AIME25|0%|
|
||
|
|
|GPQA|24.8%|
|
||
|
|
|GSM8K|54.2%|
|
||
|
|
|IFBench|18.3%|
|
||
|
|
|IFEval|55%|
|
||
|
|
|MMLU-Pro|22.8%|
|
||
|
|
|Multi-IF|32.5%|
|
||
|
|
|
||
|
|
## Quick Start
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import pipeline
|
||
|
|
|
||
|
|
pipe = pipeline(
|
||
|
|
"text-generation",
|
||
|
|
model="lino-levan/qwen-3-1.7b-smoltalk",
|
||
|
|
)
|
||
|
|
messages = [
|
||
|
|
{"role": "user", "content": "What is 3 * 8?"},
|
||
|
|
]
|
||
|
|
output = pipe(messages)
|
||
|
|
print(output[0]["generated_text"][-1]["content"])
|
||
|
|
```
|
||
|
|
|