100 lines
3.2 KiB
Markdown
100 lines
3.2 KiB
Markdown
|
---
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
license: apache-2.0
|
|||
|
|
library_name: transformers
|
|||
|
|
base_model: HuggingFaceTB/SmolLM2-1.7B
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- quantization
|
|||
|
|
- qat
|
|||
|
|
- quantization-aware-training
|
|||
|
|
- scheduled-qat
|
|||
|
|
- smollm2
|
|||
|
|
- edge-deployment
|
|||
|
|
model_name: SmolLM2-1.7B Scheduled QAT Linear INT4
|
|||
|
|
datasets:
|
|||
|
|
- wikitext
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# SmolLM2-1.7B — Scheduled QAT (Linear Schedule)
|
|||
|
|
|
|||
|
|
This model was produced by **Scheduled Quantization-Aware Training** with a linear precision reduction schedule, targeting INT4 deployment on edge devices (Android, iOS, Raspberry Pi).
|
|||
|
|
|
|||
|
|
## Important
|
|||
|
|
|
|||
|
|
**This model is in bfloat16 — it is NOT quantized.** QAT trains weights to be robust to quantization noise, but the actual quantization happens at export time. For the quantized GGUF versions ready for deployment, see:
|
|||
|
|
|
|||
|
|
**[jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF](https://huggingface.co/jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF)**
|
|||
|
|
|
|||
|
|
## Training Details
|
|||
|
|
|
|||
|
|
| Parameter | Value |
|
|||
|
|
|-----------|-------|
|
|||
|
|
| **Base model** | [HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B) |
|
|||
|
|
| **Method** | Scheduled QAT (Linear bit-width reduction) |
|
|||
|
|
| **Training data** | WikiText-103 (4000 sequences × 512 tokens) |
|
|||
|
|
| **Hardware** | Kaggle TPU v5e-8 (8 cores) |
|
|||
|
|
| **Epochs** | 1 |
|
|||
|
|
| **Effective batch size** | 64 (4 per-core × 2 grad accum × 8 cores) |
|
|||
|
|
| **Learning rate** | 2e-5 (cosine decay) |
|
|||
|
|
| **Optimizer** | AdamW (weight_decay=0.01) |
|
|||
|
|
| **Training precision** | bfloat16 |
|
|||
|
|
| **Training time** | ~1150 seconds |
|
|||
|
|
|
|||
|
|
### Bit-Width Schedule
|
|||
|
|
|
|||
|
|
| Phase | Epoch Range | Bit-width |
|
|||
|
|
|-------|------------|-----------|
|
|||
|
|
| Warmup | 0.0 → 0.1 | FP32 (no quantization noise) |
|
|||
|
|
| Linear reduction | 0.1 → 0.9 | 32 → 16 → 8 → 4 (gradual) |
|
|||
|
|
| Stabilization | 0.9 → 1.0 | INT4 (final fine-tuning) |
|
|||
|
|
|
|||
|
|
### Results (WikiText-103 Test)
|
|||
|
|
|
|||
|
|
| Metric | Value |
|
|||
|
|
|--------|-------|
|
|||
|
|
| Test loss | 3.0392 |
|
|||
|
|
| Test perplexity | 20.89 |
|
|||
|
|
|
|||
|
|
## How to Use
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
"jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-INT4",
|
|||
|
|
torch_dtype=torch.bfloat16,
|
|||
|
|
device_map="auto",
|
|||
|
|
)
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained("jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-INT4")
|
|||
|
|
|
|||
|
|
inputs = tokenizer("The future of AI is", return_tensors="pt").to(model.device)
|
|||
|
|
outputs = model.generate(**inputs, max_new_tokens=100)
|
|||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Files
|
|||
|
|
|
|||
|
|
| File | Description |
|
|||
|
|
|------|-------------|
|
|||
|
|
| `model.safetensors` | Model weights (bfloat16) |
|
|||
|
|
| `config.json` | Model architecture config |
|
|||
|
|
| `tokenizer.json` | Tokenizer |
|
|||
|
|
| `results.json` | Training results (loss, perplexity) |
|
|||
|
|
| `training_log.json` | Step-by-step training log |
|
|||
|
|
|
|||
|
|
## Related
|
|||
|
|
|
|||
|
|
- **GGUF (quantized, for deployment):** [jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF](https://huggingface.co/jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF)
|
|||
|
|
- **Base model:** [HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)
|
|||
|
|
|
|||
|
|
## Citation
|
|||
|
|
|
|||
|
|
This model is part of a thesis on Scheduled Quantization-Aware Training for Small Language Models targeting edge deployment.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache 2.0 (same as base model)
|