初始化项目，由ModelHub XC社区提供模型
Model: QuantFactory/SmallThinker-3B-Preview-GGUF Source: Original Platform
This commit is contained in:
ModelHub XC
49
.gitattributes
vendored
Normal file
49
.gitattributes
vendored
Normal file
@@ -0,0 +1,49 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
SmallThinker-3B-Preview.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
78
README.md
Normal file
78
README.md
Normal file
@@ -0,0 +1,78 @@
|
||||
|
||||
---
|
||||
|
||||
datasets:
|
||||
- PowerInfer/QWQ-LONGCOT-500K
|
||||
- PowerInfer/LONGCOT-Refine-500K
|
||||
base_model:
|
||||
- Qwen/Qwen2.5-3B-Instruct
|
||||
|
||||
---
|
||||
|
||||
[](https://hf.co/QuantFactory)
|
||||
|
||||
|
||||
# QuantFactory/SmallThinker-3B-Preview-GGUF
|
||||
This is quantized version of [PowerInfer/SmallThinker-3B-Preview](https://huggingface.co/PowerInfer/SmallThinker-3B-Preview) created using llama.cpp
|
||||
|
||||
# Original Model Card
|
||||
|
||||
# SmallThinker-3B-preview
|
||||
|
||||
We introduce **SmallThinker-3B-preview**, a new model fine-tuned from the [Qwen2.5-3b-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) model.
|
||||
|
||||
## Benchmark Performance
|
||||
|
||||
| Model | AIME24 | AMC23 | GAOKAO2024_I | GAOKAO2024_II | MMLU_STEM | AMPS_Hard | math_comp |
|
||||
|---------|--------|-------|--------------|---------------|-----------|-----------|-----------|
|
||||
| Qwen2.5-3B-Instruct | 6.67 | 45 | 50 | 35.8 | 59.8 | - | - |
|
||||
| SmallThinker | 16.667 | 57.5 | 64.2 | 57.1 | 68.2 | 70 | 46.8 |
|
||||
| GPT-4o | 9.3 | - | - | - | 64.2 | 57 | 50 |
|
||||
|
||||
Limitation: Due to SmallThinker's current limitations in instruction following, for math_comp we adopt a more lenient evaluation method where only correct answers are required, without constraining responses to follow the specified AAAAA format.
|
||||
|
||||
|
||||
## Intended Use Cases
|
||||
|
||||
SmallThinker is designed for the following use cases:
|
||||
|
||||
1. **Edge Deployment:** Its small size makes it ideal for deployment on resource-constrained devices.
|
||||
2. **Draft Model for QwQ-32B-Preview:** SmallThinker can serve as a fast and efficient draft model for the larger QwQ-32B-Preview model. From my test, in llama.cpp we can get 70% speedup (from 40 tokens/s to 70 tokens/s).
|
||||
|
||||
## Training Details
|
||||
|
||||
The model was trained using 8 H100 GPUs with a global batch size of 16. The specific configuration is as follows:
|
||||
|
||||
```
|
||||
neat_packing: true
|
||||
cutoff_len: 16384
|
||||
per_device_train_batch_size: 2
|
||||
gradient_accumulation_steps: 1
|
||||
learning_rate: 1.0e-5
|
||||
num_train_epochs: 3
|
||||
lr_scheduler_type: cosine
|
||||
warmup_ratio: 0.02
|
||||
bf16: true
|
||||
ddp_timeout: 180000000
|
||||
weight_decay: 0.0
|
||||
```
|
||||
|
||||
The SFT (Supervised Fine-Tuning) process was conducted in two phases:
|
||||
|
||||
1. First Phase:
|
||||
- Used only the PowerInfer/QWQ-LONGCOT-500K dataset
|
||||
- Trained for 1.5 epochs
|
||||
|
||||
2. Second Phase:
|
||||
- Combined training with PowerInfer/QWQ-LONGCOT-500K and PowerInfer/LONGCOT-Refine datasets
|
||||
- Continued training for an additional 2 epochs
|
||||
|
||||
|
||||
## Limitations & Disclaimer
|
||||
|
||||
Please be aware of the following limitations:
|
||||
|
||||
* **Language Limitation:** The model has only been trained on English-language datasets, hence its capabilities in other languages are still lacking.
|
||||
* **Limited Knowledge:** Due to limited SFT data and the model's relatively small scale, its reasoning capabilities are constrained by its knowledge base.
|
||||
* **Unpredictable Outputs:** The model may produce unexpected outputs due to its size and probabilistic generation paradigm. Users should exercise caution and validate the model's responses.
|
||||
* **Repetition Issue:** The model tends to repeat itself when answering high-difficulty questions. Please increase the `repetition_penalty` to mitigate this issue.
|
||||
3
SmallThinker-3B-Preview.Q2_K.gguf
Normal file
3
SmallThinker-3B-Preview.Q2_K.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1dac95fad6ca91d0ce0542e989caccea7acc2abff537b9a69b4b14b58463de20
|
||||
size 1376855232
|
||||
3
SmallThinker-3B-Preview.Q3_K_L.gguf
Normal file
3
SmallThinker-3B-Preview.Q3_K_L.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a6ab3921b7715ff046291a1b38144f9a0e2e039b4781f2e220c27e1423705b79
|
||||
size 1841093824
|
||||
3
SmallThinker-3B-Preview.Q3_K_M.gguf
Normal file
3
SmallThinker-3B-Preview.Q3_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:9458e79bc29551bb1e1d2c76f39dc067036dfe5109ceffdeb8ee524b0ee26cf0
|
||||
size 1724177600
|
||||
3
SmallThinker-3B-Preview.Q3_K_S.gguf
Normal file
3
SmallThinker-3B-Preview.Q3_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3b160476646bbbb7db0779f3d861e606d6eef9da408ebccbb5eb557cc04dec42
|
||||
size 1588059328
|
||||
3
SmallThinker-3B-Preview.Q4_0.gguf
Normal file
3
SmallThinker-3B-Preview.Q4_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:535e83d757c5d336cb3a73ec289eeb08406c648e1a3670ae37ff5ce7481a306f
|
||||
size 1997878464
|
||||
3
SmallThinker-3B-Preview.Q4_1.gguf
Normal file
3
SmallThinker-3B-Preview.Q4_1.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:5bacbeb72732c25d0cdbb69a20b83ae07ae0322f0b3827c23446acc4fe7678e1
|
||||
size 2190734528
|
||||
3
SmallThinker-3B-Preview.Q4_K_M.gguf
Normal file
3
SmallThinker-3B-Preview.Q4_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d2f1a7e40c11c6deb0cc30a34859622b55fcb869d510e8382703dc4751ed62fa
|
||||
size 2104931520
|
||||
3
SmallThinker-3B-Preview.Q4_K_S.gguf
Normal file
3
SmallThinker-3B-Preview.Q4_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:248aa54bb62dd87dd2249576f708be10b9c68d9ad56033505928b6f5a127d16d
|
||||
size 2009412800
|
||||
3
SmallThinker-3B-Preview.Q5_0.gguf
Normal file
3
SmallThinker-3B-Preview.Q5_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ea65a37a0643aabed886948c4f04f8a022e77a043670e3d9af081f7ccbf3ae7c
|
||||
size 2383590592
|
||||
3
SmallThinker-3B-Preview.Q5_1.gguf
Normal file
3
SmallThinker-3B-Preview.Q5_1.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b95cdce361f0da1cafd5e6128524b75362d30da1bd03e6640a024cebb7304eae
|
||||
size 2576446656
|
||||
3
SmallThinker-3B-Preview.Q5_K_M.gguf
Normal file
3
SmallThinker-3B-Preview.Q5_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7ec42c9148a70c99e62a141d3afb6e749bd8f197016cc517ac219abc13c5c901
|
||||
size 2438739136
|
||||
3
SmallThinker-3B-Preview.Q5_K_S.gguf
Normal file
3
SmallThinker-3B-Preview.Q5_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1d452544688b201db20d89abf2d1c3438901cc53b79b32c1eb4952d998f71411
|
||||
size 2383590592
|
||||
3
SmallThinker-3B-Preview.Q6_K.gguf
Normal file
3
SmallThinker-3B-Preview.Q6_K.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2415ad659fc542d9314ad78d27f88e43f0d715c8e84967ce8f1c0e7ff8b0a5d3
|
||||
size 2793409728
|
||||
3
SmallThinker-3B-Preview.Q8_0.gguf
Normal file
3
SmallThinker-3B-Preview.Q8_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b1948c30672e367f091898df9d8327b494f3f0f8929b9cc0c868299a805e5ea7
|
||||
size 3616087232
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "others", "allow_remote": true}
|
||||
Reference in New Issue
Block a user