84 lines
3.9 KiB
Markdown
84 lines
3.9 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
language:
|
||
- en
|
||
datasets:
|
||
- HuggingFaceFW/fineweb-edu
|
||
- mlfoundations/dclm-baseline-1.0
|
||
- HuggingFaceTB/finemath
|
||
tags:
|
||
- tim
|
||
- qwen3
|
||
- from-scratch
|
||
model-index:
|
||
- name: tim-360m-base
|
||
results:
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: hellaswag, name: HellaSwag (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.388, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: ai2_arc, name: ARC-Easy (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.515, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: ai2_arc, name: ARC-Challenge (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.259, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: piqa, name: PIQA (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.658, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: winogrande, name: WinoGrande (0-shot)}
|
||
metrics: [{type: acc, value: 0.525, name: acc}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: openbookqa, name: OpenBookQA (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.294, name: acc_norm}]
|
||
- task: {type: text-generation, name: Text Generation}
|
||
dataset: {type: sciq, name: SciQ (0-shot)}
|
||
metrics: [{type: acc_norm, value: 0.722, name: acc_norm}]
|
||
---
|
||
|
||
# TIM-360M-base
|
||
|
||
A **361.8M-parameter** decoder-only transformer, trained **from scratch** on ~30B tokens on 2× RTX 4090. Part of [TIM](https://github.com/) (Transformer Inference Model) — an end-to-end, from-scratch LLM built to be legible and honestly evaluated.
|
||
|
||
Exported as **`Qwen3ForCausalLM`** — it loads with `transformers` and **no `trust_remote_code`**.
|
||
|
||
## Architecture
|
||
|
||
SmolLM2-360M skeleton with modern choices: pre-LN RMSNorm, RoPE (θ=100000), SwiGLU, **grouped-query attention** (15 query / 5 KV heads) with **per-head QK-norm**, tied embeddings, no biases. 32 layers, hidden 960, head_dim 64, FFN 2560, vocab 49,152 (byte-level BPE). Context: `max_position_embeddings` 4096, **trained at seq_len 2048**.
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
tok = AutoTokenizer.from_pretrained("timothywong731/tim-360m-base")
|
||
m = AutoModelForCausalLM.from_pretrained("timothywong731/tim-360m-base", torch_dtype=torch.bfloat16)
|
||
ids = tok("The capital of France is", return_tensors="pt")
|
||
print(tok.decode(m.generate(**ids, max_new_tokens=20)[0]))
|
||
```
|
||
|
||
## Training
|
||
|
||
~30B tokens, mix: **70% web** (FineWeb-Edu `sample-100BT` + DCLM-Baseline), **15%** Stack-Edu (code), **10%** FineMath-3+, **5%** misc. Muon + AdamW, WSD (warmup–stable–decay) schedule, bf16, 2× RTX 4090 (DDP, no NVLink).
|
||
|
||
## Evaluation
|
||
|
||
lm-evaluation-harness (0.4.12), **0-shot**, `acc_norm` where applicable:
|
||
|
||
| HellaSwag | ARC-Easy | ARC-Challenge | PIQA | WinoGrande | OpenBookQA | SciQ | SocialIQA | CommonsenseQA | WikiText (bpb ↓) |
|
||
|---|---|---|---|---|---|---|---|---|---|
|
||
| 0.388 | 0.515 | 0.259 | 0.658 | 0.525 | 0.294 | 0.722 | 0.378 | 0.196 | 1.058 |
|
||
|
||
Healthy for the scale: at ~30B tokens this is **~130× less training data** than SmolLM2-360M (4T). Near-parity on early-saturating tasks (PIQA, SciQ), a real gap on token-hungry ones (HellaSwag, ARC) — as the token budget predicts.
|
||
|
||
## Limitations
|
||
|
||
Small (360M) and lightly trained (~30B tokens). **Not instruction-tuned** — this is the base model. Weak at arithmetic and multi-step reasoning; will hallucinate facts; English-only; **no safety tuning** — do not deploy to end users without your own filtering.
|
||
|
||
## Provenance & license
|
||
|
||
**Apache-2.0.** Trained only on named, public datasets: FineWeb-Edu (ODC-BY), DCLM-Baseline (CC-BY-4.0), Stack-Edu (derived from The Stack v2), FineMath (ODC-BY). No gated or scraped-private data. Attribution and removal requests: open an issue on the source repository.
|