初始化项目，由ModelHub XC社区提供模型

Model: codelion/SmolLM2-70M Source: Original Platform
2026-05-17 11:19:26 +08:00
commit fa8abef9c8
10 changed files with 294388 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,254 @@
+---
+language:
+- en
+license: apache-2.0
+library_name: transformers
+tags:
+- pretraining
+- educational
+- pedagogical
+- sutra
+- smollm2
+- llama
+pipeline_tag: text-generation
+model-index:
+- name: SmolLM2-70M
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: ai2_arc
+      name: ARC-Easy
+      config: ARC-Easy
+    metrics:
+    - type: acc_norm
+      value: 33.00
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: ai2_arc
+      name: ARC-Challenge
+      config: ARC-Challenge
+    metrics:
+    - type: acc_norm
+      value: 22.35
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: boolq
+      name: BoolQ
+    metrics:
+    - type: acc
+      value: 39.66
+      name: Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: hellaswag
+      name: HellaSwag
+    metrics:
+    - type: acc_norm
+      value: 26.14
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: piqa
+      name: PIQA
+    metrics:
+    - type: acc_norm
+      value: 54.84
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: sciq
+      name: SciQ
+    metrics:
+    - type: acc_norm
+      value: 45.20
+      name: Normalized Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: winogrande
+      name: WinoGrande
+    metrics:
+    - type: acc
+      value: 50.04
+      name: Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: truthful_qa
+      name: TruthfulQA MC2
+    metrics:
+    - type: acc
+      value: 48.02
+      name: Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: gsm8k
+      name: GSM8K
+    metrics:
+    - type: exact_match
+      value: 0.53
+      name: Exact Match (5-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: cais/mmlu
+      name: MMLU
+    metrics:
+    - type: acc
+      value: 22.96
+      name: Accuracy (0-shot)
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      type: openbookqa
+      name: OpenBookQA
+    metrics:
+    - type: acc_norm
+      value: 27.60
+      name: Normalized Accuracy (0-shot)
+base_model: HuggingFaceTB/SmolLM2-70M
+datasets:
+- codelion/sutra-10B
+---
+
+# SmolLM2-70M
+
+A SmolLM2-70M model pretrained on the [Sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) pedagogical dataset for 3 epochs (~30.6B tokens total). This model demonstrates that a 69M parameter model can be trained to near-capacity performance using dense, curated educational data.
+
+## Model Details
+
+| Property | Value |
+|----------|-------|
+| Architecture | LlamaForCausalLM |
+| Parameters | 69.2M |
+| Hidden Size | 384 |
+| Layers | 32 |
+| Attention Heads | 6 (2 KV heads) |
+| Context Length | 8,192 |
+| Vocabulary | 49,152 |
+| Precision | bfloat16 |
+| Base Model | [SmolLM2-70M](https://huggingface.co/HuggingFaceTB/SmolLM2-70M) |
+| Training Dataset | [Sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) (10.2B tokens) |
+
+## Training
+
+The model was trained for 3 epochs on the Sutra-10B dataset using a single NVIDIA L40S GPU (46GB). This checkpoint is the best perplexity checkpoint from epoch 3.
+
+| Epoch | Tokens | Training Time | Learning Rate | Best Perplexity |
+|-------|--------|---------------|---------------|-----------------|
+| 1 | 10.2B | 25.82h | 3e-4 → 3e-5 | 39.50 |
+| 2 | 10.2B | 25.78h | 1e-4 → 1e-5 | 37.81 |
+| 3 | 10.2B | 26.16h | 3e-5 → 3e-6 | 37.72 |
+| **Total** | **30.6B** | **77.76h** | — | **37.72** |
+
+Training configuration:
+- Optimizer: AdamW (fused), weight decay 0.1
+- Schedule: Cosine with warmup
+- Batch size: 4 per device, gradient accumulation 8 (effective ~262K tokens/step)
+- Sequence length: 8,192
+- Flash Attention 2, TF32 matmul, torch.compile
+- Throughput: ~110K tokens/sec
+
+## Benchmark Results
+
+All benchmarks evaluated using [lm-evaluation-harness](https://github.com/EleutherAI/lm-eval) v0.4.11. All tasks are 0-shot except GSM8K (5-shot).
+
+### This Model vs Training Progression
+
+| Benchmark | **E3-best** | E3-final | E2-best | E2-final | E1-final |
+|-----------|:-----------:|:--------:|:-------:|:--------:|:--------:|
+| ARC-Easy | **33.00** | 33.16 | 32.83 | 33.12 | 33.46 |
+| ARC-Challenge | **22.35** | 21.67 | 22.61 | 22.44 | 22.44 |
+| BoolQ | **39.66** | 39.66 | 39.79 | 39.54 | 39.79 |
+| HellaSwag | **26.14** | 26.03 | 26.08 | 25.91 | 26.03 |
+| PIQA | **54.84** | 55.01 | 54.24 | 54.13 | 54.62 |
+| SciQ | **45.20** | 46.30 | 44.10 | 45.50 | 43.60 |
+| WinoGrande | **50.04** | 49.33 | 50.51 | 48.70 | 48.78 |
+| TruthfulQA | **48.02** | 47.93 | 48.30 | 48.14 | 48.30 |
+| GSM8K | **0.53** | 0.61 | 0.68 | 0.83 | 0.15 |
+| MMLU | **22.96** | 22.87 | 23.00 | 22.98 | 22.99 |
+| OpenBookQA | **27.60** | 27.60 | — | — | — |
+| **Average (10)** | **34.27** | 34.26 | 34.21 | 34.13 | 34.02 |
+
+### Comparison with 1B Token Baselines (SmolLM2-70M)
+
+These are results from training the same SmolLM2-70M model on various 1B-token datasets from the [Pre-training Dataset Samples](https://huggingface.co/collections/codelion/pre-training-dataset-samples-686bd760abf1a43b0ce32829) collection for 1 epoch, showing that Sutra-10B at 3 epochs achieves the highest performance for this model size.
+
+| Dataset (1B tokens) | HellaSwag | PIQA | WinoGrande | ARC-C | MMLU | TruthfulQA | GSM8K | Avg |
+|---------------------|-----------|------|------------|-------|------|------------|-------|-----|
+| **Sutra-10B (3 epochs)** | **26.14** | **54.84** | **50.04** | **22.35** | 22.96 | **48.02** | 0.53 | **34.27** |
+| [Sutra-1B](https://huggingface.co/datasets/codelion/sutra-1B) | 25.43 | 53.86 | 49.41 | 23.04 | 22.91 | 49.09 | 1.14 | 32.13 |
+| [FineWiki-1B](https://huggingface.co/datasets/HuggingFaceFW/finewiki) | 25.56 | 51.69 | 48.86 | 24.15 | **23.34** | 51.16 | 0.91 | 32.24 |
+| [FinePDFs-1B](https://huggingface.co/datasets/HuggingFaceFW/FinePDFs) | 25.58 | 52.56 | 50.51 | 22.44 | 22.95 | 51.41 | 1.21 | 32.38 |
+| [DCLM-Baseline-1B](https://huggingface.co/datasets/codelion/dclm-baseline-1B) | 25.85 | 55.17 | 50.20 | 21.08 | 22.97 | 49.21 | 0.68 | 32.16 |
+| [FineWeb-Edu-1B](https://huggingface.co/datasets/codelion/fineweb-edu-1B) | 25.72 | 55.11 | 50.36 | 21.25 | 22.96 | 48.11 | 1.21 | 32.10 |
+| [Essential-Web-1B](https://huggingface.co/datasets/sumukshashidhar-archive/essential-web-v1.0-sample-1B) | 26.02 | 55.44 | 48.30 | 20.99 | 22.95 | 49.59 | 1.29 | 32.08 |
+| [Synth-1B](https://huggingface.co/datasets/codelion/synth-1B) | 26.63 | 50.98 | 48.78 | 21.93 | 23.24 | 47.10 | 1.29 | 31.42 |
+
+## Key Findings
+
+1. **Capacity ceiling**: The 70M parameter model reaches its capacity ceiling at approximately 10B tokens. Additional epochs (up to 30.6B total tokens) yield only marginal improvements in benchmark scores (+0.25 average from epoch 1 to epoch 3), despite continued perplexity improvement (39.50 → 37.72).
+
+2. **Perplexity vs benchmarks**: Perplexity continues to decrease across epochs, but downstream benchmark performance plateaus, suggesting the model's representational capacity is the bottleneck rather than data exposure.
+
+3. **Data quality matters**: Even at 1B tokens, Sutra outperforms or matches larger web-crawled datasets (DCLM, FineWeb-Edu, Essential-Web) on average, demonstrating the value of curated pedagogical content.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("codelion/SmolLM2-70M", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("codelion/SmolLM2-70M")
+
+input_text = "The theory of relativity states that"
+inputs = tokenizer(input_text, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=100)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Limitations
+
+- This is a 69M parameter base model (not instruction-tuned) — it generates completions, not conversational responses
+- Performance is at the capacity ceiling for this model size; larger models would benefit more from the Sutra-10B dataset
+- The model was trained primarily on English educational content
+
+## Related Resources
+
+- **Dataset**: [codelion/sutra-10B](https://huggingface.co/datasets/codelion/sutra-10B) — 10B token pedagogical pretraining dataset
+- **Sutra Framework**: Generates structured educational content optimized for LLM pretraining
+
+## Citation
+
+```bibtex
+@article{sharma2026sutra,
+  title={Scaling Pedagogical Pretraining: From Optimal Mixing to 10 Billion Tokens},
+  author={Sharma, Asankhaya},
+  year={2026},
+  url={https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens}
+}
+```
+
+## License
+
+Apache 2.0