commit bf4eee209109e1819c9cf36a4921b862799bf280 Author: ModelHub XC Date: Wed Oct 14 06:26:14 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: QuantFactory/SmallThinker-3B-Preview-GGUF Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..c0a8e6e --- /dev/null +++ b/.gitattributes @@ -0,0 +1,49 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q4_1.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q4_0.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q5_0.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q5_1.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text +SmallThinker-3B-Preview.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000..672135d --- /dev/null +++ b/README.md @@ -0,0 +1,78 @@ + +--- + +datasets: +- PowerInfer/QWQ-LONGCOT-500K +- PowerInfer/LONGCOT-Refine-500K +base_model: +- Qwen/Qwen2.5-3B-Instruct + +--- + +[![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory) + + +# QuantFactory/SmallThinker-3B-Preview-GGUF +This is quantized version of [PowerInfer/SmallThinker-3B-Preview](https://huggingface.co/PowerInfer/SmallThinker-3B-Preview) created using llama.cpp + +# Original Model Card + +# SmallThinker-3B-preview + +We introduce **SmallThinker-3B-preview**, a new model fine-tuned from the [Qwen2.5-3b-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) model. + +## Benchmark Performance + +| Model | AIME24 | AMC23 | GAOKAO2024_I | GAOKAO2024_II | MMLU_STEM | AMPS_Hard | math_comp | +|---------|--------|-------|--------------|---------------|-----------|-----------|-----------| +| Qwen2.5-3B-Instruct | 6.67 | 45 | 50 | 35.8 | 59.8 | - | - | +| SmallThinker | 16.667 | 57.5 | 64.2 | 57.1 | 68.2 | 70 | 46.8 | +| GPT-4o | 9.3 | - | - | - | 64.2 | 57 | 50 | + +Limitation: Due to SmallThinker's current limitations in instruction following, for math_comp we adopt a more lenient evaluation method where only correct answers are required, without constraining responses to follow the specified AAAAA format. + + +## Intended Use Cases + +SmallThinker is designed for the following use cases: + +1. **Edge Deployment:** Its small size makes it ideal for deployment on resource-constrained devices. +2. **Draft Model for QwQ-32B-Preview:** SmallThinker can serve as a fast and efficient draft model for the larger QwQ-32B-Preview model. From my test, in llama.cpp we can get 70% speedup (from 40 tokens/s to 70 tokens/s). + +## Training Details + +The model was trained using 8 H100 GPUs with a global batch size of 16. The specific configuration is as follows: + +``` +neat_packing: true +cutoff_len: 16384 +per_device_train_batch_size: 2 +gradient_accumulation_steps: 1 +learning_rate: 1.0e-5 +num_train_epochs: 3 +lr_scheduler_type: cosine +warmup_ratio: 0.02 +bf16: true +ddp_timeout: 180000000 +weight_decay: 0.0 +``` + +The SFT (Supervised Fine-Tuning) process was conducted in two phases: + +1. First Phase: + - Used only the PowerInfer/QWQ-LONGCOT-500K dataset + - Trained for 1.5 epochs + +2. Second Phase: + - Combined training with PowerInfer/QWQ-LONGCOT-500K and PowerInfer/LONGCOT-Refine datasets + - Continued training for an additional 2 epochs + + +## Limitations & Disclaimer + +Please be aware of the following limitations: + +* **Language Limitation:** The model has only been trained on English-language datasets, hence its capabilities in other languages are still lacking. +* **Limited Knowledge:** Due to limited SFT data and the model's relatively small scale, its reasoning capabilities are constrained by its knowledge base. +* **Unpredictable Outputs:** The model may produce unexpected outputs due to its size and probabilistic generation paradigm. Users should exercise caution and validate the model's responses. +* **Repetition Issue:** The model tends to repeat itself when answering high-difficulty questions. Please increase the `repetition_penalty` to mitigate this issue. diff --git a/SmallThinker-3B-Preview.Q2_K.gguf b/SmallThinker-3B-Preview.Q2_K.gguf new file mode 100644 index 0000000..2573b08 --- /dev/null +++ b/SmallThinker-3B-Preview.Q2_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1dac95fad6ca91d0ce0542e989caccea7acc2abff537b9a69b4b14b58463de20 +size 1376855232 diff --git a/SmallThinker-3B-Preview.Q3_K_L.gguf b/SmallThinker-3B-Preview.Q3_K_L.gguf new file mode 100644 index 0000000..6b06aec --- /dev/null +++ b/SmallThinker-3B-Preview.Q3_K_L.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a6ab3921b7715ff046291a1b38144f9a0e2e039b4781f2e220c27e1423705b79 +size 1841093824 diff --git a/SmallThinker-3B-Preview.Q3_K_M.gguf b/SmallThinker-3B-Preview.Q3_K_M.gguf new file mode 100644 index 0000000..41c867b --- /dev/null +++ b/SmallThinker-3B-Preview.Q3_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9458e79bc29551bb1e1d2c76f39dc067036dfe5109ceffdeb8ee524b0ee26cf0 +size 1724177600 diff --git a/SmallThinker-3B-Preview.Q3_K_S.gguf b/SmallThinker-3B-Preview.Q3_K_S.gguf new file mode 100644 index 0000000..0c8681b --- /dev/null +++ b/SmallThinker-3B-Preview.Q3_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3b160476646bbbb7db0779f3d861e606d6eef9da408ebccbb5eb557cc04dec42 +size 1588059328 diff --git a/SmallThinker-3B-Preview.Q4_0.gguf b/SmallThinker-3B-Preview.Q4_0.gguf new file mode 100644 index 0000000..994a9bd --- /dev/null +++ b/SmallThinker-3B-Preview.Q4_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:535e83d757c5d336cb3a73ec289eeb08406c648e1a3670ae37ff5ce7481a306f +size 1997878464 diff --git a/SmallThinker-3B-Preview.Q4_1.gguf b/SmallThinker-3B-Preview.Q4_1.gguf new file mode 100644 index 0000000..c731f4c --- /dev/null +++ b/SmallThinker-3B-Preview.Q4_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5bacbeb72732c25d0cdbb69a20b83ae07ae0322f0b3827c23446acc4fe7678e1 +size 2190734528 diff --git a/SmallThinker-3B-Preview.Q4_K_M.gguf b/SmallThinker-3B-Preview.Q4_K_M.gguf new file mode 100644 index 0000000..625f4f0 --- /dev/null +++ b/SmallThinker-3B-Preview.Q4_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d2f1a7e40c11c6deb0cc30a34859622b55fcb869d510e8382703dc4751ed62fa +size 2104931520 diff --git a/SmallThinker-3B-Preview.Q4_K_S.gguf b/SmallThinker-3B-Preview.Q4_K_S.gguf new file mode 100644 index 0000000..fecf664 --- /dev/null +++ b/SmallThinker-3B-Preview.Q4_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:248aa54bb62dd87dd2249576f708be10b9c68d9ad56033505928b6f5a127d16d +size 2009412800 diff --git a/SmallThinker-3B-Preview.Q5_0.gguf b/SmallThinker-3B-Preview.Q5_0.gguf new file mode 100644 index 0000000..7d0d9af --- /dev/null +++ b/SmallThinker-3B-Preview.Q5_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ea65a37a0643aabed886948c4f04f8a022e77a043670e3d9af081f7ccbf3ae7c +size 2383590592 diff --git a/SmallThinker-3B-Preview.Q5_1.gguf b/SmallThinker-3B-Preview.Q5_1.gguf new file mode 100644 index 0000000..04c05cd --- /dev/null +++ b/SmallThinker-3B-Preview.Q5_1.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b95cdce361f0da1cafd5e6128524b75362d30da1bd03e6640a024cebb7304eae +size 2576446656 diff --git a/SmallThinker-3B-Preview.Q5_K_M.gguf b/SmallThinker-3B-Preview.Q5_K_M.gguf new file mode 100644 index 0000000..1546d3c --- /dev/null +++ b/SmallThinker-3B-Preview.Q5_K_M.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7ec42c9148a70c99e62a141d3afb6e749bd8f197016cc517ac219abc13c5c901 +size 2438739136 diff --git a/SmallThinker-3B-Preview.Q5_K_S.gguf b/SmallThinker-3B-Preview.Q5_K_S.gguf new file mode 100644 index 0000000..8cc08ac --- /dev/null +++ b/SmallThinker-3B-Preview.Q5_K_S.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1d452544688b201db20d89abf2d1c3438901cc53b79b32c1eb4952d998f71411 +size 2383590592 diff --git a/SmallThinker-3B-Preview.Q6_K.gguf b/SmallThinker-3B-Preview.Q6_K.gguf new file mode 100644 index 0000000..3028f61 --- /dev/null +++ b/SmallThinker-3B-Preview.Q6_K.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2415ad659fc542d9314ad78d27f88e43f0d715c8e84967ce8f1c0e7ff8b0a5d3 +size 2793409728 diff --git a/SmallThinker-3B-Preview.Q8_0.gguf b/SmallThinker-3B-Preview.Q8_0.gguf new file mode 100644 index 0000000..3c35535 --- /dev/null +++ b/SmallThinker-3B-Preview.Q8_0.gguf @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b1948c30672e367f091898df9d8327b494f3f0f8929b9cc0c868299a805e5ea7 +size 3616087232 diff --git a/configuration.json b/configuration.json new file mode 100644 index 0000000..159097f --- /dev/null +++ b/configuration.json @@ -0,0 +1 @@ +{"framework": "pytorch", "task": "others", "allow_remote": true} \ No newline at end of file