初始化项目,由ModelHub XC社区提供模型
Model: atenareply/smollm2-135m-noval-instruct Source: Original Platform
This commit is contained in:
79
README.md
Normal file
79
README.md
Normal file
@@ -0,0 +1,79 @@
|
||||
---
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
language:
|
||||
- en
|
||||
base_model:
|
||||
- atenareply/smollm2-135m-noval
|
||||
- HuggingFaceTB/SmolLM2-135M
|
||||
datasets:
|
||||
- atenareply/noval-corp-sft-small
|
||||
- HuggingFaceTB/smol-smoltalk
|
||||
tags:
|
||||
- smollm2
|
||||
- sft
|
||||
- instruction-tuning
|
||||
- chatml
|
||||
license: apache-2.0
|
||||
---
|
||||
|
||||
# SmolLM2-135M Noval — Instruct (SFT)
|
||||
|
||||
Instruction-tuned domain assistant: SFT of the 135M CPT model on grounded domain instruction pairs blended with general instructions.
|
||||
|
||||
## Overview
|
||||
|
||||
- **Stage**: SFT (TRL SFTTrainer, ChatML, assistant-only loss)
|
||||
- **Lineage**: SmolLM2-135M → CPT (noval) → **SFT** (this model)
|
||||
- **Method**: SFT from the CPT'd checkpoint. ChatML with a `{% generation %}` block + assistant-only loss; grounded instruction pairs (teacher-distilled, generator≠judge, verbatim-number grounding filter [[genie]](#citations)) blended 60/40 with smol-smoltalk [[collapse]](#citations).
|
||||
- **Domain**: fictional — Orbital Mining Corporation (OMC) technical docs + Mars Express telemetry.
|
||||
|
||||
## Training
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Dataset | noval-corp-sft-small — train 1,388 / val 43 (chat `messages`) |
|
||||
| LR / epochs | 2e-5 cosine, warmup 0.03, 3 epochs, max_len 2048, eff_batch 32 |
|
||||
|
||||
## Evaluation
|
||||
|
||||
| Metric | Value | Note |
|
||||
|---|---|---|
|
||||
| train_loss | **2.257** | healthy curve, no overfit |
|
||||
| eval_loss | **1.946** | per-epoch 2.048 → 1.959 → 1.946 |
|
||||
| token accuracy | **0.57** | 0.47 → 0.57 |
|
||||
|
||||
_Eval is a held-out, deterministic verifiable harness (synthetic tasks); baseline = the pre-SFT ParamΔ model. See `noval-corp/scripts/eval_agentic.py`._
|
||||
|
||||
## Intended use & limitations
|
||||
|
||||
Domain Q&A / summarization / code-explanation over OMC / Mars-Express, in chat (ChatML) format.
|
||||
|
||||
**Limitations:**
|
||||
- 135M instruct is intrinsically limited.
|
||||
- Instruction-following eval was handed off externally (no in-repo task metrics).
|
||||
- Fictional domain; sparse one-off entities hallucination-prone.
|
||||
|
||||
## Innovations tested
|
||||
|
||||
- **Local-subscription teacher distillation** — grounded pairs generated by Claude Code (generator ≠ judge), with a rule-based verbatim-number grounding gate [[genie]](#citations).
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
tok = AutoTokenizer.from_pretrained("atenareply/smollm2-135m-noval-instruct")
|
||||
model = AutoModelForCausalLM.from_pretrained("atenareply/smollm2-135m-noval-instruct")
|
||||
msgs = [{"role": "user", "content": "What does the Orbital Mining Corporation do?"}]
|
||||
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
|
||||
print(tok.decode(model.generate(**inputs, max_new_tokens=256)[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Citations
|
||||
|
||||
- GENIE (grounded synthesis) — Mirza et al., ICLR 2024. arXiv:2401.14367
|
||||
- Model collapse (≥50% real data) — Shumailov et al., 2024. arXiv:2404.01413
|
||||
|
||||
---
|
||||
_Card generated by `noval-corp/scripts/gen_model_cards.py` (standardized across the noval-corp model family)._
|
||||
Reference in New Issue
Block a user