187 lines
7.7 KiB
Markdown
187 lines
7.7 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
- fr
|
||
- it
|
||
- de
|
||
- es
|
||
- pl
|
||
license: apache-2.0
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
tags:
|
||
- transformers
|
||
- reasoning
|
||
- synthetic-data
|
||
datasets:
|
||
- PleIAs/SYNTH
|
||
---
|
||
|
||
# 🥖 Baguettotron-600M
|
||
|
||
<div align="center">
|
||
<img src="https://huggingface.co/PleIAs/Baguettotron/resolve/main/figures/pleias.jpg" width="60%" alt="Pleias" />
|
||
</div>
|
||
|
||
<p align="center">
|
||
<a href="https://arxiv.org/abs/2609.37891"><b>Paper (NeurIPS 2026)</b></a> ·
|
||
<a href="https://huggingface.co/datasets/PleIAs/SYNTH"><b>SYNTH dataset</b></a> ·
|
||
<a href="https://huggingface.co/PleIAs/Baguettotron"><b>Baguettotron-350M</b></a> ·
|
||
<a href="https://huggingface.co/PleIAs/baguettotron-moe"><b>Baguettotron-MoE</b></a>
|
||
</p>
|
||
|
||
**Baguettotron-600M** is a 0.6B-parameter reasoning model trained from scratch on 158B tokens of [SYNTH](https://huggingface.co/datasets/PleIAs/SYNTH), a fully synthetic corpus amplified from 58,698 Wikipedia articles. It is the main reference model of our NeurIPS 2026 paper, [*It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs*](https://arxiv.org/abs/2609.37891).
|
||
|
||
- **Single training stage:** no separate mid-training, SFT or RLHF. The model follows instructions, reasons in `<think>` traces and recalls facts because SYNTH trains those capabilities directly.
|
||
- **Token-efficient:** within 4.7 points of Qwen3-0.6B on 22 multiple-choice tasks and 0.7 points on 8 open-ended tasks, trained on ~230× fewer tokens.
|
||
- **Factually precise:** highest FActScore in its parameter tier (41.7% macro vs. 31.6% for Qwen3-0.6B).
|
||
|
||
## Model design
|
||
|
||
Baguettotron-600M is a standard dense decoder (Llama/Qwen-style) and loads natively in `transformers` and vLLM (`LlamaForCausalLM`, no remote code).
|
||
|
||
<p align="center">
|
||
<img width="80%" src="figures/architecture.png" alt="Baguettotron-600M architecture">
|
||
</p>
|
||
|
||
| | |
|
||
|---|---|
|
||
| Parameters | 608M (incl. 67M tied embeddings) |
|
||
| Layers | 48 |
|
||
| Hidden size | 1024 |
|
||
| Attention | GQA, 16 query / 4 KV heads, head dim 64 |
|
||
| MLP | SwiGLU, intermediate size 2816 |
|
||
| Position encoding | RoPE, θ = 10,000 |
|
||
| Norm | RMSNorm (pre-norm), ε = 1e-6 |
|
||
| Embeddings | Tied input/output |
|
||
| Vocabulary | 65,536 (Pleias tokenizer) |
|
||
| Context length | 2,048 tokens |
|
||
|
||
## Training
|
||
|
||
- **Data:** SYNTH, 158.3B tokens (~2 passes over the ~75B-token corpus)
|
||
- **Steps:** 151,000 at global batch 512 × 2,048 tokens (~1.05M tokens/step)
|
||
- **Optimizer:** AdamW, peak LR 1.5e-3, 10k-step warmup, linear decay over the final 16.6% of steps to 0.2% of peak; weight decay 0.01, grad clip 1.0
|
||
- **Hardware:** 16× H100 (4 nodes × 4 GPUs) on MareNostrum 5 (BSC), `torchtitan` with FSDP, ~52.6 h, ~30% MFU
|
||
|
||
## Evaluation
|
||
|
||
All scores are from the paper. Each model is prompted in its native format; Baguettotron models get no system prompt and have `<think>\n` seeded after the assistant turn.
|
||
|
||
<p align="center">
|
||
<img width="90%" src="figures/benchmarks_pareto.png" alt="Accuracy vs training tokens">
|
||
</p>
|
||
|
||
| Model | Training tokens | MCQ (22 tasks) | Open-ended (8 tasks) | FActScore (macro) |
|
||
|---|---|---|---|---|
|
||
| **Baguettotron-600M** | 158B | 42.2 | 24.3 | **41.7** |
|
||
| Baguettotron-MoE (13B / 1B active) | 50B | 39.6 | 23.8 | 46.3 |
|
||
| Baguettotron-350M | 200B | 36.3 | 16.0 | 32.4 |
|
||
| Qwen3-0.6B | ~36T | **46.9** | **25.0** | 31.6 |
|
||
| LFM2.5-350M | 28T | 42.3 | 20.8 | 22.0 |
|
||
| SmolLM2-360M-Instruct | 4T | 22.8 | 21.6 | 26.6 |
|
||
| Gemma-3-270M-IT | 2T | 22.5 | 14.9 | 19.4 |
|
||
|
||
- **Ties or beats Qwen3-0.6B** on TruthfulQA (42.6 vs. 32.7), ESGenius (62.2 vs. 55.1) and FormationEval (62.8 vs. 58.0)
|
||
- **Weak:** benchmarks that reward broad web knowledge (ARC-Challenge, GeoBench), since SYNTH covers only its seed articles
|
||
- **Data, not architecture:** the same architecture, tokenizer and step count trained on FineWiki or FinePDFs-Edu reaches only 26.1 / 25.1 on MCQ, even after SmolTalk post-training
|
||
- **Reasoning traces matter:** retraining without traces costs 9.1 points on TruthfulQA and 10.0 on NuclearQA
|
||
|
||
## Prompt format
|
||
|
||
The model was trained on ChatML with a `<think>` block and no system prompt. The bundled chat template opens the assistant turn with `<think>\n`:
|
||
|
||
```xml
|
||
<|im_start|>user
|
||
What do you know about the Treaty of Westphalia?<|im_end|>
|
||
<|im_start|>assistant
|
||
<think>
|
||
```
|
||
|
||
The model writes its reasoning, closes it with `</think>`, answers, and ends the turn with `<|im_end|>`.
|
||
|
||
**RAG.** Pass sources inside the user turn. The answer then cites them with `<ref>[quote]</ref>`:
|
||
|
||
```xml
|
||
<|im_start|>user
|
||
{question}
|
||
|
||
<source_1>[…]</source_1>
|
||
<source_2>[…]</source_2><|im_end|>
|
||
<|im_start|>assistant
|
||
<think>
|
||
```
|
||
|
||
**Reasoning notation.** Traces use SYNTH's compact stenographic style (`→` derivation, `↺` backtracking, `∴` conclusion, …). The confidence markers `●` (high), `◐` (partial) and `○` (low) are informative: on traces dominated by uncertain markers, FActScore drops from 0.44 to 0.40, and the model commits to ~15–20% fewer facts. See the [Baguettotron-350M card](https://huggingface.co/PleIAs/Baguettotron) for the full notation.
|
||
|
||
## Inference
|
||
|
||
### vLLM
|
||
|
||
```bash
|
||
vllm serve PleIAs/baguettotron-600m --reasoning-parser deepseek_r1
|
||
```
|
||
|
||
`--reasoning-parser deepseek_r1` moves the `<think>` trace into a separate `reasoning` field. Tested with vLLM 0.24.0.
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||
resp = client.chat.completions.create(
|
||
model="PleIAs/baguettotron-600m",
|
||
messages=[{"role": "user", "content": "What do you know about the Treaty of Westphalia?"}],
|
||
temperature=0.1,
|
||
top_p=0.95,
|
||
presence_penalty=0.1,
|
||
max_tokens=1536,
|
||
)
|
||
print(resp.choices[0].message.reasoning) # <think> trace
|
||
print(resp.choices[0].message.content) # answer
|
||
```
|
||
|
||
Recommended sampling: `temperature=0.1`, `top_p=0.95`, `presence_penalty=0.1`. Prompt and generation share the 2,048-token context, so keep `max_tokens` below 2,048 minus the prompt length.
|
||
|
||
### Transformers
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
repo = "PleIAs/baguettotron-600m"
|
||
tokenizer = AutoTokenizer.from_pretrained(repo)
|
||
model = AutoModelForCausalLM.from_pretrained(repo, dtype="bfloat16", device_map="auto")
|
||
|
||
messages = [{"role": "user", "content": "What do you know about the Treaty of Westphalia?"}]
|
||
inputs = tokenizer.apply_chat_template(
|
||
messages, add_generation_prompt=True, return_tensors="pt"
|
||
).to(model.device)
|
||
out = model.generate(inputs, max_new_tokens=1536, do_sample=True, temperature=0.1, top_p=0.95)
|
||
print(tokenizer.decode(out[0, inputs.shape[1]:]))
|
||
```
|
||
|
||
## Limitations
|
||
|
||
- **2,048-token context** shared by prompt, reasoning and answer
|
||
- **Knowledge is bounded by the seed corpus** (~58.7k Wikipedia articles): recall is precise on seed topics and weaker outside them
|
||
- **Reasoning traces are English-only**, even when the prompt and answer are in another language
|
||
- **No preference tuning or safety alignment**; not intended for high-stakes use without further evaluation
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@inproceedings{langlais2026itsalltraining,
|
||
title = {It's All Training: A Fully Synthetic Single-Stage Recipe for {LLMs}},
|
||
author = {Langlais, Pierre-Carl and Delobelle, Pieter and Detrois, Yannick and Chizhov, Pavel and Rosas-Hinostroza, Carlos and Si Smail, Neil and Burtin, Benjamin and Shcharbakova, Hanna and Yamshchikov, Ivan and Stasenko, Anastasia},
|
||
booktitle = {Advances in Neural Information Processing Systems},
|
||
year = {2026},
|
||
eprint = {2609.37891},
|
||
archivePrefix = {arXiv},
|
||
primaryClass = {cs.CL},
|
||
url = {https://arxiv.org/abs/2609.37891}
|
||
}
|
||
```
|
||
|
||
## Acknowledgements
|
||
|
||
Trained on MareNostrum 5 (BSC) through the EuroHPC Extreme Scale Access call (EHPC-EXT-2025E01-092, *JULIP: Powerful LLMs made in Europe*). Parts of this research received funding from SPRIN-D, the German Federal Agency for Breakthrough Innovation.
|