42 lines
1.5 KiB
Markdown
42 lines
1.5 KiB
Markdown
|
---
|
||
|
|
license: mit
|
||
|
|
base_model: meta-llama/Llama-3.2-3B-Instruct
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags: [latent-reasoning, looped-transformer, gsm8k, lotus]
|
||
|
|
---
|
||
|
|
|
||
|
|
# gsm-lotus-llama3b
|
||
|
|
|
||
|
|
**LOTUS** (direct LM-head supervision, no CODI) checkpoint fine-tuned from `meta-llama/Llama-3.2-3B-Instruct` on GSM8k-Aug, from the paper
|
||
|
|
[Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers](https://arxiv.org/abs/2606.31779).
|
||
|
|
This is the LOTUS model.
|
||
|
|
|
||
|
|
- **GSM8K (GSM8k-Aug) test accuracy:** 70.05% (924/1319)
|
||
|
|
- **Latent config:** K = 6 blocks, c_thought = 25 tokens/block, n_looped_iters = 6
|
||
|
|
- **Base:** meta-llama/Llama-3.2-3B-Instruct (vocab 128256 -> 128259 for 3 latent tokens)
|
||
|
|
|
||
|
|
## Loading
|
||
|
|
`from_pretrained` loads the weights only — the looped padded architecture needs the LOTUS code
|
||
|
|
([code repo](https://github.com/yingfan-bot/lotus)).
|
||
|
|
|
||
|
|
## Reproduce the number above
|
||
|
|
Run in the pinned env (torch 2.7 / transformers 4.46.2). This loads the safetensors straight from this
|
||
|
|
repo and runs the latent loop — no separate checkpoint file needed:
|
||
|
|
```bash
|
||
|
|
python scripts/eval.py \
|
||
|
|
--model_id yingfanbot/gsm-lotus-llama3b \
|
||
|
|
--datasets gsm8k --n_looped_iters 6 --c_thought 25 --bf16
|
||
|
|
```
|
||
|
|
Yields 70.05% on GSM8k-Aug (verified by loading this repo's safetensors directly).
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
```bibtex
|
||
|
|
@article{fan2026bridging,
|
||
|
|
title={Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers},
|
||
|
|
author={Fan, Ying and Svete, Anej and Lee, Kangwook},
|
||
|
|
journal={arXiv preprint arXiv:2606.31779},
|
||
|
|
year={2026}
|
||
|
|
}
|
||
|
|
```
|