96 lines
4.9 KiB
Markdown
96 lines
4.9 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
language:
|
||
- en
|
||
base_model: timothywong731/tim-360m-base
|
||
datasets:
|
||
- HuggingFaceTB/smol-smoltalk
|
||
- HuggingFaceH4/ultrafeedback_binarized
|
||
tags:
|
||
- tim
|
||
- qwen3
|
||
- from-scratch
|
||
- sft
|
||
- dpo
|
||
- chat
|
||
model-index:
|
||
- name: tim-360m-instruct
|
||
results:
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: hellaswag, name: HellaSwag (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.395, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: ai2_arc, name: ARC-Easy (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.501, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: ai2_arc, name: ARC-Challenge (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.262, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: piqa, name: PIQA (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.652, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: winogrande, name: WinoGrande (0-shot)}
|
||
metrics: [{type: acc, value: 0.524, name: acc}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: openbookqa, name: OpenBookQA (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.302, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: social_i_qa, name: SocialIQA (0-shot)}
|
||
metrics: [{type: acc, value: 0.408, name: acc}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: sciq, name: SciQ (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.714, name: acc_norm}]
|
||
---
|
||
|
||
# TIM-360M-instruct
|
||
|
||
The **instruction-tuned** variant of [TIM-360M-base](https://huggingface.co/timothywong731/tim-360m-base) — a **361.8M-parameter** decoder-only transformer trained **from scratch** on 2× RTX 4090. Post-trained with **SFT → DPO**, no TRL.
|
||
|
||
Exported as **`Qwen3ForCausalLM`** — loads with `transformers` and **no `trust_remote_code`**.
|
||
|
||
## Chat format
|
||
|
||
ChatML. Turns are wrapped `<|im_start|>{role}\n{content}<|im_end|>`; the assistant turn ends on `<|im_end|>` (token id 2), which is the model's generation stop token. Use the tokenizer's chat template — don't hand-format.
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
tok = AutoTokenizer.from_pretrained("timothywong731/tim-360m-instruct")
|
||
m = AutoModelForCausalLM.from_pretrained("timothywong731/tim-360m-instruct", torch_dtype=torch.bfloat16)
|
||
|
||
msgs = [{"role": "user", "content": "Explain what a transformer is in two sentences."}]
|
||
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
|
||
out = m.generate(ids, max_new_tokens=200, do_sample=True, temperature=0.7, top_p=0.9)
|
||
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
## Architecture
|
||
|
||
Identical to the base model: SmolLM2-360M skeleton, pre-LN RMSNorm, RoPE (θ=100000), SwiGLU, **grouped-query attention** (15 query / 5 KV heads) with **per-head QK-norm**, tied embeddings, no biases. 32 layers, hidden 960, head_dim 64, FFN 2560, vocab 49,152. Context: `max_position_embeddings` 4096, trained at seq_len 2048.
|
||
|
||
## Post-training
|
||
|
||
1. **SFT** on [smol-smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smol-smoltalk) — multi-turn instruction following, loss on assistant turns only.
|
||
2. **DPO** on [ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) — a frozen reference policy, sigmoid preference loss (β=0.1, lr 5e-7), scored over the final assistant turn.
|
||
|
||
## Evaluation
|
||
|
||
lm-evaluation-harness (0.4.12), **0-shot**, `acc_norm` where applicable:
|
||
|
||
| HellaSwag | ARC-Easy | ARC-Challenge | PIQA | WinoGrande | OpenBookQA | SciQ | SocialIQA | CommonsenseQA | WikiText (bpb ↓) |
|
||
|---|---|---|---|---|---|---|---|---|---|
|
||
| 0.395 | 0.501 | 0.262 | 0.652 | 0.524 | 0.302 | 0.714 | 0.408 | 0.203 | 1.130 |
|
||
|
||
Post-training **preserved base capability** — most tasks move within noise of [the base model](https://huggingface.co/timothywong731/tim-360m-base), with a real gain on **SocialIQA (+3.0)** and small gains on HellaSwag / OpenBookQA. WikiText bits-per-byte rises (1.06 → 1.13), as expected: a chat-tuned model is no longer a pure next-token predictor of raw web text.
|
||
|
||
## Limitations
|
||
|
||
Small (360M) and lightly trained (~30B pretraining tokens, then light post-training). Instruction-following is **basic** — expect short, sometimes shallow answers, weak arithmetic and multi-step reasoning, and confident hallucination. English-only. **DPO is not safety alignment** — this model has no dedicated safety tuning; filter before exposing to end users.
|
||
|
||
## Provenance & license
|
||
|
||
**Apache-2.0.** Pretraining data (all named, public): FineWeb-Edu (ODC-BY), DCLM-Baseline (CC-BY-4.0), Stack-Edu, FineMath (ODC-BY). Post-training data: smol-smoltalk and ultrafeedback_binarized (see their dataset cards for licenses). No gated or scraped-private data.
|