---
language:
  - en
license: apache-2.0
library_name: transformers
base_model: HuggingFaceTB/SmolLM2-1.7B
pipeline_tag: text-generation
tags:
  - quantization
  - qat
  - quantization-aware-training
  - scheduled-qat
  - smollm2
  - edge-deployment
model_name: SmolLM2-1.7B Scheduled QAT Linear INT4
datasets:
  - wikitext
---

# SmolLM2-1.7B — Scheduled QAT (Linear Schedule)

This model was produced by **Scheduled Quantization-Aware Training** with a linear precision reduction schedule, targeting INT4 deployment on edge devices (Android, iOS, Raspberry Pi).

## Important

**This model is in bfloat16 — it is NOT quantized.** QAT trains weights to be robust to quantization noise, but the actual quantization happens at export time. For the quantized GGUF versions ready for deployment, see:

**[jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF](https://huggingface.co/jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF)**

## Training Details

| Parameter | Value |
|-----------|-------|
| **Base model** | [HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B) |
| **Method** | Scheduled QAT (Linear bit-width reduction) |
| **Training data** | WikiText-103 (4000 sequences × 512 tokens) |
| **Hardware** | Kaggle TPU v5e-8 (8 cores) |
| **Epochs** | 1 |
| **Effective batch size** | 64 (4 per-core × 2 grad accum × 8 cores) |
| **Learning rate** | 2e-5 (cosine decay) |
| **Optimizer** | AdamW (weight_decay=0.01) |
| **Training precision** | bfloat16 |
| **Training time** | ~1150 seconds |

### Bit-Width Schedule

| Phase | Epoch Range | Bit-width |
|-------|------------|-----------|
| Warmup | 0.0 → 0.1 | FP32 (no quantization noise) |
| Linear reduction | 0.1 → 0.9 | 32 → 16 → 8 → 4 (gradual) |
| Stabilization | 0.9 → 1.0 | INT4 (final fine-tuning) |

### Results (WikiText-103 Test)

| Metric | Value |
|--------|-------|
| Test loss | 3.0392 |
| Test perplexity | 20.89 |

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-INT4",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-INT4")

inputs = tokenizer("The future of AI is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Files

| File | Description |
|------|-------------|
| `model.safetensors` | Model weights (bfloat16) |
| `config.json` | Model architecture config |
| `tokenizer.json` | Tokenizer |
| `results.json` | Training results (loss, perplexity) |
| `training_log.json` | Step-by-step training log |

## Related

- **GGUF (quantized, for deployment):** [jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF](https://huggingface.co/jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF)
- **Base model:** [HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)

## Citation

This model is part of a thesis on Scheduled Quantization-Aware Training for Small Language Models targeting edge deployment.

## License

Apache 2.0 (same as base model)