初始化项目，由ModelHub XC社区提供模型

Model: jacquelinehe/tinycomma-1.8b-llama3-tokenizer
Source: Original Platform

README.md

@@ -0,0 +1,110 @@
+---
+datasets:
+- common-pile/comma_v0.1_training_dataset
+language:
+- en
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# TinyComma 1.8B 
+
+TinyComma 1.8B is a 1.8B parameter, decoder-only base LM trained entirely on permissively licensed data from the [Common Pile](https://huggingface.co/collections/common-pile/common-pile-v01). Different from the official Comma model series, TinyComma 1.8B uses the 128K-vocabulary [Llama3](https://huggingface.co/collections/meta-llama/llama-31) tokenizer to ensure compatibility with two-model decoding setups.
+We trained TinyComma 1.8B to support our research on inference-time copyright mitigation. 
+
+- **Paper:** [Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model](https://arxiv.org/abs/2602.07120)
+- **Repository:** [jacqueline-he/anchored-decoding](https://github.com/jacqueline-he/anchored-decoding)
+- **Project Page:** [Interactive Demo](https://tinyurl.com/anchored-decoding-demo)
+
+## Benchmarking TinyComma 1.8B
+
+We benchmarked TinyComma 1.8B and several other permissively trained base models on several common natural language understanding tasks from the [OLMES](https://github.com/allenai/olmes) evaluation suite. 
+
+<p align="center">
+  <img src="https://huggingface.co/datasets/jacquelinehe/tinycomma-assets/resolve/main/pretraining_benchmark.png" width="800"><br>
+  <em>Benchmarking results using OLMES. TinyComma 1.8B outperforms other models of its size range.</em>
+</p>
+
+## Training details 
+We trained TinyComma 1.8B using the [lingua](https://github.com/facebookresearch/lingua/) training framework. Pre-training consists of two stages: (1) a 156B-token generation training stage over the entire Common Pile, following original domain weights specified by [Kandpal et al., 2025](https://arxiv.org/pdf/2506.05209#page=49.20),
+and (2) a 13.5B-token cooldown stage on a weighted mixture of three high-quality domains (70% Wikimedia, 15% DOAB, and 15% Data Provenance Initiative data). Our hardware is a single node of 8 140 GiB H200 GPUs. Model configuration and pre-training hyperparameter details are below:
+
+<div style="text-align: center;">
+  <table style="margin: 0 auto;">
+    <caption>TinyComma 1.8B model configuration.</caption>
+    <thead>
+      <tr>
+        <th>Params</th>
+        <th>Head Dim.</th>
+        <th>Hidden Size</th>
+        <th>Attn. Heads</th>
+        <th>Hidden Layers</th>
+        <th>KV Heads</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td>1,758,562,304</td>
+        <td>64</td>
+        <td>2048</td>
+        <td>32</td>
+        <td>24</td>
+        <td>32</td>
+      </tr>
+    </tbody>
+  </table>
+</div>
+<br><br>
+<div style="text-align: center;">
+  <table style="margin: 0 auto;">
+    <caption>TinyComma 1.8B pretraining configuration.</caption>
+    <thead>
+      <tr>
+        <th>Hyperparameters</th>
+        <th>Values</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td>Optimizer</td>
+        <td>AdamW (<i>&beta;</i><sub>1</sub>=0.9, <i>&beta;</i><sub>2</sub>=0.95)</td>
+      </tr>
+      <tr>
+        <td>Learning rate</td>
+        <td>3e<sup>&minus;3</sup> for Stage 1, 1e<sup>&minus;3</sup> for Stage 2</td>
+      </tr>
+      <tr>
+        <td>Weight decay</td>
+        <td>0.033 for Stage 1</td>
+      </tr>
+      <tr>
+        <td>Batch size</td>
+        <td>4M tokens</td>
+      </tr>
+      <tr>
+        <td>Warmup</td>
+        <td>1000 steps for Stage 1, none for Stage 2</td>
+      </tr>
+      <tr>
+        <td>Schedule</td>
+        <td>Cosine schedule for Stage 1, linear schedule for Stage 2</td>
+      </tr>
+      <tr>
+        <td>Sequence length</td>
+        <td>Pack to 2048 tokens</td>
+      </tr>
+    </tbody>
+  </table>
+</div>
+
+
+## Citation
+```bibtex
+@article{he2026anchored,
+  title={{Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model}},
+  author={Jacqueline He and Jonathan Hayase and Wen-tau Yih and Sewoong Oh and Luke Zettlemoyer and Pang Wei Koh},
+  journal={arXiv preprint},
+  year={2026}
+}
+```