初始化项目，由ModelHub XC社区提供模型
Model: jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF Source: Original Platform
This commit is contained in:
ModelHub XC
37
.gitattributes
vendored
Normal file
37
.gitattributes
vendored
Normal file
@@ -0,0 +1,37 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
smollm2-1.7b-sched-qat-linear-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
smollm2-1.7b-sched-qat-linear-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
115
README.md
Normal file
115
README.md
Normal file
@@ -0,0 +1,115 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
library_name: gguf
|
||||
base_model: HuggingFaceTB/SmolLM2-1.7B
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- gguf
|
||||
- llama-cpp
|
||||
- quantization
|
||||
- qat
|
||||
- quantization-aware-training
|
||||
- scheduled-qat
|
||||
- smollm2
|
||||
- edge-deployment
|
||||
- int4
|
||||
- int8
|
||||
model_name: SmolLM2-1.7B Scheduled QAT Linear GGUF
|
||||
datasets:
|
||||
- wikitext
|
||||
quantized_by: jpcurada
|
||||
---
|
||||
|
||||
# SmolLM2-1.7B — Scheduled QAT (Linear Schedule) — GGUF
|
||||
|
||||
GGUF quantized versions of [SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B), trained with **Scheduled Quantization-Aware Training** (linear bit-width reduction schedule) before quantization.
|
||||
|
||||
> **Key insight:** Unlike naive Post-Training Quantization (PTQ), these weights were specifically trained to survive quantization. During training, precision was gradually reduced from FP32 → FP16 → INT8 → INT4 following a linear schedule, allowing the model to adapt its weights to quantization noise at each stage.
|
||||
|
||||
## Files
|
||||
|
||||
| Filename | Quant | Size | BPW | Description |
|
||||
|----------|-------|------|-----|-------------|
|
||||
| `smollm2-1.7b-sched-qat-linear-Q4_K_M.gguf` | Q4_K_M | ~1.0 GB | 4.93 | **Recommended** — best quality/size ratio for edge deployment |
|
||||
| `smollm2-1.7b-sched-qat-linear-Q8_0.gguf` | Q8_0 | ~1.7 GB | 8.50 | Higher quality, larger size |
|
||||
|
||||
## Training Details
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| **Base model** | [HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B) |
|
||||
| **Method** | Scheduled QAT (Linear bit-width reduction) |
|
||||
| **Training data** | WikiText-103 (4000 sequences × 512 tokens) |
|
||||
| **Hardware** | Kaggle TPU v5e-8 (8 cores) |
|
||||
| **Epochs** | 1 |
|
||||
| **Effective batch size** | 64 (4 per-core × 2 grad accum × 8 cores) |
|
||||
| **Learning rate** | 2e-5 (cosine decay) |
|
||||
| **Optimizer** | AdamW (weight_decay=0.01) |
|
||||
| **Training time** | ~1150 seconds |
|
||||
|
||||
### Bit-Width Schedule
|
||||
|
||||
```
|
||||
Epoch: 0.0 ──── 0.1 ──────────── 0.9 ──── 1.0
|
||||
Bits: FP32 │ Linear │ INT4
|
||||
(warmup) │ 32→16→8→4 │ (stabilize)
|
||||
```
|
||||
|
||||
| Phase | Epoch Range | Bit-width |
|
||||
|-------|------------|-----------|
|
||||
| Warmup | 0.0 → 0.1 | FP32 (no quantization noise) |
|
||||
| Linear reduction | 0.1 → 0.9 | 32 → 16 → 8 → 4 (gradual) |
|
||||
| Stabilization | 0.9 → 1.0 | INT4 (final fine-tuning) |
|
||||
|
||||
### QAT Training Results (WikiText-103 Test)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Test loss | 3.0392 |
|
||||
| Test perplexity | 20.89 |
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Training (QAT):** Model weights are trained with fake quantization nodes that simulate INT4 rounding noise in every forward pass. The gradients learn to place weights near quantization grid points.
|
||||
2. **Export (this repo):** The QAT-hardened bf16 weights are converted to GGUF format and quantized to actual INT4/INT8 using `llama.cpp`'s `llama-quantize`.
|
||||
3. **Deployment:** The GGUF files run directly on edge devices via `llama.cpp` — Android, iOS, Raspberry Pi.
|
||||
|
||||
Because QAT pre-adapted the weights for quantization, these GGUF files should retain more quality than naively quantized (PTQ) versions of the same model.
|
||||
|
||||
## Usage
|
||||
|
||||
### With llama.cpp CLI
|
||||
|
||||
```bash
|
||||
# Download
|
||||
wget https://huggingface.co/jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-GGUF/resolve/main/smollm2-1.7b-sched-qat-linear-Q4_K_M.gguf
|
||||
|
||||
# Run
|
||||
./llama-cli -m smollm2-1.7b-sched-qat-linear-Q4_K_M.gguf \
|
||||
-p "The future of artificial intelligence is" -n 100
|
||||
```
|
||||
|
||||
### With llama-cpp-python
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama
|
||||
|
||||
llm = Llama(model_path="smollm2-1.7b-sched-qat-linear-Q4_K_M.gguf")
|
||||
output = llm("The future of AI is", max_tokens=100)
|
||||
print(output["choices"][0]["text"])
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- **bf16 weights (pre-quantization):** [jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-INT4](https://huggingface.co/jpcurada/SmolLM2-1.7B-Scheduled-QAT-Linear-INT4)
|
||||
- **Base model:** [HuggingFaceTB/SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)
|
||||
|
||||
## Citation
|
||||
|
||||
This model is part of a thesis on Scheduled Quantization-Aware Training for Small Language Models targeting edge deployment.
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0 (same as base model)
|
||||
3
smollm2-1.7b-sched-qat-linear-Q4_K_M.gguf
Normal file
3
smollm2-1.7b-sched-qat-linear-Q4_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:f8c3c08f5dd5c9b07c76a95a3bb8983299ba6e99d37e00e5617709c0bcbd52cd
|
||||
size 1055609440
|
||||
3
smollm2-1.7b-sched-qat-linear-Q8_0.gguf
Normal file
3
smollm2-1.7b-sched-qat-linear-Q8_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:eaa83ff8c9280d3aac5bf01cd9fb2cc00ce52efeabd9ca3e036c9ac274c20547
|
||||
size 1820414560
|
||||
Reference in New Issue
Block a user