tinycomma-1.8b-llama3-tokenizer/README.md

---
datasets:
- common-pile/comma_v0.1_training_dataset
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
---

# TinyComma 1.8B 

TinyComma 1.8B is a 1.8B parameter, decoder-only base LM trained entirely on permissively licensed data from the [Common Pile](https://huggingface.co/collections/common-pile/common-pile-v01). Different from the official Comma model series, TinyComma 1.8B uses the 128K-vocabulary [Llama3](https://huggingface.co/collections/meta-llama/llama-31) tokenizer to ensure compatibility with two-model decoding setups.
We trained TinyComma 1.8B to support our research on inference-time copyright mitigation. 

- **Paper:** [Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model](https://arxiv.org/abs/2602.07120)
- **Repository:** [jacqueline-he/anchored-decoding](https://github.com/jacqueline-he/anchored-decoding)
- **Project Page:** [Interactive Demo](https://tinyurl.com/anchored-decoding-demo)

## Benchmarking TinyComma 1.8B

We benchmarked TinyComma 1.8B and several other permissively trained base models on several common natural language understanding tasks from the [OLMES](https://github.com/allenai/olmes) evaluation suite. 

<p align="center">
  <img src="https://huggingface.co/datasets/jacquelinehe/tinycomma-assets/resolve/main/pretraining_benchmark.png" width="800"><br>
  <em>Benchmarking results using OLMES. TinyComma 1.8B outperforms other models of its size range.</em>
</p>

## Training details 
We trained TinyComma 1.8B using the [lingua](https://github.com/facebookresearch/lingua/) training framework. Pre-training consists of two stages: (1) a 156B-token generation training stage over the entire Common Pile, following original domain weights specified by [Kandpal et al., 2025](https://arxiv.org/pdf/2506.05209#page=49.20),
and (2) a 13.5B-token cooldown stage on a weighted mixture of three high-quality domains (70% Wikimedia, 15% DOAB, and 15% Data Provenance Initiative data). Our hardware is a single node of 8 140 GiB H200 GPUs. Model configuration and pre-training hyperparameter details are below:

<div style="text-align: center;">
  <table style="margin: 0 auto;">
    <caption>TinyComma 1.8B model configuration.</caption>
    <thead>
      <tr>
        <th>Params</th>
        <th>Head Dim.</th>
        <th>Hidden Size</th>
        <th>Attn. Heads</th>
        <th>Hidden Layers</th>
        <th>KV Heads</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>1,758,562,304</td>
        <td>64</td>
        <td>2048</td>
        <td>32</td>
        <td>24</td>
        <td>32</td>
      </tr>
    </tbody>
  </table>
</div>
<br><br>
<div style="text-align: center;">
  <table style="margin: 0 auto;">
    <caption>TinyComma 1.8B pretraining configuration.</caption>
    <thead>
      <tr>
        <th>Hyperparameters</th>
        <th>Values</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>Optimizer</td>
        <td>AdamW (<i>&beta;</i><sub>1</sub>=0.9, <i>&beta;</i><sub>2</sub>=0.95)</td>
      </tr>
      <tr>
        <td>Learning rate</td>
        <td>3e<sup>&minus;3</sup> for Stage 1, 1e<sup>&minus;3</sup> for Stage 2</td>
      </tr>
      <tr>
        <td>Weight decay</td>
        <td>0.033 for Stage 1</td>
      </tr>
      <tr>
        <td>Batch size</td>
        <td>4M tokens</td>
      </tr>
      <tr>
        <td>Warmup</td>
        <td>1000 steps for Stage 1, none for Stage 2</td>
      </tr>
      <tr>
        <td>Schedule</td>
        <td>Cosine schedule for Stage 1, linear schedule for Stage 2</td>
      </tr>
      <tr>
        <td>Sequence length</td>
        <td>Pack to 2048 tokens</td>
      </tr>
    </tbody>
  </table>
</div>


## Citation
```bibtex
@article{he2026anchored,
  title={{Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model}},
  author={Jacqueline He and Jonathan Hayase and Wen-tau Yih and Sewoong Oh and Luke Zettlemoyer and Pang Wei Koh},
  journal={arXiv preprint},
  year={2026}
}
```