100 lines
3.2 KiB
Markdown
100 lines
3.2 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
base_model: HuggingFaceTB/SmolLM2-1.7B
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- quantization
|
||
- qat
|
||
- quantization-aware-training
|
||
- scheduled-qat
|
||
- smollm2
|
||
- edge-deployment
|
||
model_name: SmolLM2-1.7B Scheduled QAT Linear INT4
|
||
datasets:
|
||
- wikitext
|
||
---
|
||
|
||
# SmolLM2-1.7B — Scheduled QAT (Linear Schedule)
|
||
|
||
This model was produced by **Scheduled Quantization-Aware Training** with a linear precision reduction schedule, targeting INT4 deployment on edge devices (Android, iOS, Raspberry Pi).
|
||
|
||
## Important
|
||
|
||
**This model is in bfloat16 — it is NOT quantized.** QAT trains weights to be robust to quantization noise, but the actual quantization happens at export time. For the quantized GGUF versions ready for deployment, see:
|
||
|
||
**[jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF](https://huggingface.co/jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF)**
|
||
|
||
## Training Details
|
||
|
||
| Parameter | Value |
|
||
|-----------|-------|
|
||
| **Base model** | [HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B) |
|
||
| **Method** | Scheduled QAT (Linear bit-width reduction) |
|
||
| **Training data** | WikiText-103 (4000 sequences × 512 tokens) |
|
||
| **Hardware** | Kaggle TPU v5e-8 (8 cores) |
|
||
| **Epochs** | 1 |
|
||
| **Effective batch size** | 64 (4 per-core × 2 grad accum × 8 cores) |
|
||
| **Learning rate** | 2e-5 (cosine decay) |
|
||
| **Optimizer** | AdamW (weight_decay=0.01) |
|
||
| **Training precision** | bfloat16 |
|
||
| **Training time** | ~1150 seconds |
|
||
|
||
### Bit-Width Schedule
|
||
|
||
| Phase | Epoch Range | Bit-width |
|
||
|-------|------------|-----------|
|
||
| Warmup | 0.0 → 0.1 | FP32 (no quantization noise) |
|
||
| Linear reduction | 0.1 → 0.9 | 32 → 16 → 8 → 4 (gradual) |
|
||
| Stabilization | 0.9 → 1.0 | INT4 (final fine-tuning) |
|
||
|
||
### Results (WikiText-103 Test)
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Test loss | 3.0392 |
|
||
| Test perplexity | 20.89 |
|
||
|
||
## How to Use
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-INT4",
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
tokenizer = AutoTokenizer.from_pretrained("jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-INT4")
|
||
|
||
inputs = tokenizer("The future of AI is", return_tensors="pt").to(model.device)
|
||
outputs = model.generate(**inputs, max_new_tokens=100)
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
```
|
||
|
||
## Files
|
||
|
||
| File | Description |
|
||
|------|-------------|
|
||
| `model.safetensors` | Model weights (bfloat16) |
|
||
| `config.json` | Model architecture config |
|
||
| `tokenizer.json` | Tokenizer |
|
||
| `results.json` | Training results (loss, perplexity) |
|
||
| `training_log.json` | Step-by-step training log |
|
||
|
||
## Related
|
||
|
||
- **GGUF (quantized, for deployment):** [jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF](https://huggingface.co/jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF)
|
||
- **Base model:** [HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)
|
||
|
||
## Citation
|
||
|
||
This model is part of a thesis on Scheduled Quantization-Aware Training for Small Language Models targeting edge deployment.
|
||
|
||
## License
|
||
|
||
Apache 2.0 (same as base model)
|