70 lines
1.4 KiB
Markdown
70 lines
1.4 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
base_model: HuggingFaceTB/SmolLM2-360M
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- smollm2
|
||
|
|
- instruction-tuning
|
||
|
|
- supervised-fine-tuning
|
||
|
|
- alpaca
|
||
|
|
- math
|
||
|
|
- reasoning
|
||
|
|
- gpteacher
|
||
|
|
- metamath
|
||
|
|
- small-language-models
|
||
|
|
---
|
||
|
|
|
||
|
|
# SmolLM2-360M Math Instruct
|
||
|
|
|
||
|
|
This is an instruction-tuned version of `HuggingFaceTB/SmolLM2-360M`.
|
||
|
|
|
||
|
|
The model was fine-tuned on a small mixed instruction dataset containing general instruction-following examples and math reasoning examples.
|
||
|
|
## Quick start
|
||
|
|
|
||
|
|
```python
|
||
|
|
import torch
|
||
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
|
|
||
|
|
model_id = "srmty/smolLM2-360M-math-instruct"
|
||
|
|
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
|
|
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
||
|
|
model_id,
|
||
|
|
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
|
||
|
|
device_map="auto" if torch.cuda.is_available() else None,
|
||
|
|
)
|
||
|
|
|
||
|
|
if tokenizer.pad_token is None:
|
||
|
|
tokenizer.pad_token = tokenizer.eos_token
|
||
|
|
|
||
|
|
model.eval()
|
||
|
|
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
## Training Data
|
||
|
|
|
||
|
|
The training mix used:
|
||
|
|
|
||
|
|
- `teknium/GPTeacher-General-Instruct`
|
||
|
|
- `meta-math/MetaMathQA` subset
|
||
|
|
|
||
|
|
The data was formatted using Alpaca-style prompts.
|
||
|
|
|
||
|
|
## Prompt Format
|
||
|
|
|
||
|
|
Use this format during inference:
|
||
|
|
|
||
|
|
```text
|
||
|
|
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
|
||
|
|
|
||
|
|
### Instruction:
|
||
|
|
{instruction}
|
||
|
|
|
||
|
|
### Input:
|
||
|
|
{input}
|
||
|
|
|
||
|
|
### Response:
|