初始化项目，由ModelHub XC社区提供模型

Model: jacquelinehe/tinycomma-1.8b-llama3-tokenizer
Source: Original Platform

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,110 @@
+---
+datasets:
+- common-pile/comma_v0.1_training_dataset
+language:
+- en
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# TinyComma 1.8B 
+
+TinyComma 1.8B is a 1.8B parameter, decoder-only base LM trained entirely on permissively licensed data from the [Common Pile](https://huggingface.co/collections/common-pile/common-pile-v01). Different from the official Comma model series, TinyComma 1.8B uses the 128K-vocabulary [Llama3](https://huggingface.co/collections/meta-llama/llama-31) tokenizer to ensure compatibility with two-model decoding setups.
+We trained TinyComma 1.8B to support our research on inference-time copyright mitigation. 
+
+- **Paper:** [Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model](https://arxiv.org/abs/2602.07120)
+- **Repository:** [jacqueline-he/anchored-decoding](https://github.com/jacqueline-he/anchored-decoding)
+- **Project Page:** [Interactive Demo](https://tinyurl.com/anchored-decoding-demo)
+
+## Benchmarking TinyComma 1.8B
+
+We benchmarked TinyComma 1.8B and several other permissively trained base models on several common natural language understanding tasks from the [OLMES](https://github.com/allenai/olmes) evaluation suite. 
+
+<p align="center">
+  <img src="https://huggingface.co/datasets/jacquelinehe/tinycomma-assets/resolve/main/pretraining_benchmark.png" width="800"><br>
+  <em>Benchmarking results using OLMES. TinyComma 1.8B outperforms other models of its size range.</em>
+</p>
+
+## Training details 
+We trained TinyComma 1.8B using the [lingua](https://github.com/facebookresearch/lingua/) training framework. Pre-training consists of two stages: (1) a 156B-token generation training stage over the entire Common Pile, following original domain weights specified by [Kandpal et al., 2025](https://arxiv.org/pdf/2506.05209#page=49.20),
+and (2) a 13.5B-token cooldown stage on a weighted mixture of three high-quality domains (70% Wikimedia, 15% DOAB, and 15% Data Provenance Initiative data). Our hardware is a single node of 8 140 GiB H200 GPUs. Model configuration and pre-training hyperparameter details are below:
+
+<div style="text-align: center;">
+  <table style="margin: 0 auto;">
+    <caption>TinyComma 1.8B model configuration.</caption>
+    <thead>
+      <tr>
+        <th>Params</th>
+        <th>Head Dim.</th>
+        <th>Hidden Size</th>
+        <th>Attn. Heads</th>
+        <th>Hidden Layers</th>
+        <th>KV Heads</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td>1,758,562,304</td>
+        <td>64</td>
+        <td>2048</td>
+        <td>32</td>
+        <td>24</td>
+        <td>32</td>
+      </tr>
+    </tbody>
+  </table>
+</div>
+<br><br>
+<div style="text-align: center;">
+  <table style="margin: 0 auto;">
+    <caption>TinyComma 1.8B pretraining configuration.</caption>
+    <thead>
+      <tr>
+        <th>Hyperparameters</th>
+        <th>Values</th>
+      </tr>
+    </thead>
+    <tbody>
+      <tr>
+        <td>Optimizer</td>
+        <td>AdamW (<i>&beta;</i><sub>1</sub>=0.9, <i>&beta;</i><sub>2</sub>=0.95)</td>
+      </tr>
+      <tr>
+        <td>Learning rate</td>
+        <td>3e<sup>&minus;3</sup> for Stage 1, 1e<sup>&minus;3</sup> for Stage 2</td>
+      </tr>
+      <tr>
+        <td>Weight decay</td>
+        <td>0.033 for Stage 1</td>
+      </tr>
+      <tr>
+        <td>Batch size</td>
+        <td>4M tokens</td>
+      </tr>
+      <tr>
+        <td>Warmup</td>
+        <td>1000 steps for Stage 1, none for Stage 2</td>
+      </tr>
+      <tr>
+        <td>Schedule</td>
+        <td>Cosine schedule for Stage 1, linear schedule for Stage 2</td>
+      </tr>
+      <tr>
+        <td>Sequence length</td>
+        <td>Pack to 2048 tokens</td>
+      </tr>
+    </tbody>
+  </table>
+</div>
+
+
+## Citation
+```bibtex
+@article{he2026anchored,
+  title={{Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model}},
+  author={Jacqueline He and Jonathan Hayase and Wen-tau Yih and Sewoong Oh and Luke Zettlemoyer and Pang Wei Koh},
+  journal={arXiv preprint},
+  year={2026}
+}
+```

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "eos_token_id": 128001,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 5632,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 32,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.52.4",
+  "use_cache": true,
+  "vocab_size": 128256
+}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "bos_token_id": 128000,
+  "do_sample": true,
+  "eos_token_id": 128001,
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "4.52.4"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:816c096e975c21b3c2e971d32e3aef2a03152d094c636fe705d9efbaf80bc3e1
+size 3517149992

special_tokens_map.json

@@ -0,0 +1,16 @@
+{
+  "bos_token": {
+    "content": "<|begin_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|end_of_text|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c5cf44023714fb39b05e71e425f8d7b92805ff73f7988b083b8c87f0bf87393
+size 17209961