初始化项目,由ModelHub XC社区提供模型
Model: Surpem/Supertron2.1-0.6B-GGUF Source: Original Platform
This commit is contained in:
192
README.md
Normal file
192
README.md
Normal file
@@ -0,0 +1,192 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
pipeline_tag: text-generation
|
||||
base_model:
|
||||
- Surpem/Supertron2.1-0.6B
|
||||
tags:
|
||||
- gguf
|
||||
- llama-cpp
|
||||
- qwen
|
||||
- qwen3
|
||||
- chat
|
||||
- quantized
|
||||
- q4
|
||||
- q8
|
||||
- f16
|
||||
- local-llm
|
||||
- surpem
|
||||
---
|
||||
|
||||
# Supertron2.1-0.6B-GGUF
|
||||
|
||||
**Supertron2.1-0.6B-GGUF** contains GGUF exports of **Surpem/Supertron2.1-0.6B**, a compact Qwen3-based generalist model by **Surpem**.
|
||||
|
||||
This repository is for local inference with `llama.cpp`, LM Studio, Jan, KoboldCpp, text-generation-webui, and other GGUF-compatible runtimes. The original Transformers checkpoint is available at `Surpem/Supertron2.1-0.6B`.
|
||||
|
||||
## Available Files
|
||||
|
||||
| File | Type | Size | Recommended Use |
|
||||
| :-- | :-- | --: | :-- |
|
||||
| `gguf/Supertron2.1-0.6B-F16.gguf` | F16 | ~448 MiB | Highest quality GGUF, larger memory use |
|
||||
| `gguf/Supertron2.1-0.6B-Q8_0.gguf` | 8-bit | ~610 MiB | Strong quality, efficient local use |
|
||||
| `gguf/Supertron2.1-0.6B-Q4_K_M.gguf` | 4-bit K-quants | ~378 MiB | Small, fast, best for low-memory devices |
|
||||
|
||||
## Which GGUF Should I Use?
|
||||
|
||||
### Q4_K_M
|
||||
|
||||
Use this when you want the smallest practical model.
|
||||
|
||||
Good for:
|
||||
|
||||
* laptops
|
||||
* CPU inference
|
||||
* fast testing
|
||||
* low VRAM
|
||||
* general chat
|
||||
|
||||
Tradeoff: slightly lower quality than Q8/F16.
|
||||
|
||||
### Q8_0
|
||||
|
||||
Use this when you want better quality while keeping the file smaller than full precision.
|
||||
|
||||
Good for:
|
||||
|
||||
* local coding help
|
||||
* math prompts
|
||||
* better instruction following
|
||||
* GPU offload with modest VRAM
|
||||
|
||||
Tradeoff: larger than Q4.
|
||||
|
||||
### F16
|
||||
|
||||
Use this when quality matters most and memory is available.
|
||||
|
||||
Good for:
|
||||
|
||||
* comparison testing
|
||||
* re-quantization
|
||||
* quality checks
|
||||
* development workflows
|
||||
|
||||
Tradeoff: largest runtime memory use.
|
||||
|
||||
## llama.cpp Usage
|
||||
|
||||
Install or build `llama.cpp`, then run:
|
||||
|
||||
```bash
|
||||
llama-cli \
|
||||
-m gguf/Supertron2.1-0.6B-Q4_K_M.gguf \
|
||||
-p "Write a Python function that returns the nth Fibonacci number." \
|
||||
-n 256
|
||||
```
|
||||
|
||||
For chat-style prompting:
|
||||
|
||||
```bash
|
||||
llama-cli \
|
||||
-m gguf/Supertron2.1-0.6B-Q8_0.gguf \
|
||||
-cnv \
|
||||
--color \
|
||||
-p "You are Supertron, a helpful coding and math assistant."
|
||||
```
|
||||
|
||||
With GPU offload:
|
||||
|
||||
```bash
|
||||
llama-cli \
|
||||
-m gguf/Supertron2.1-0.6B-Q4_K_M.gguf \
|
||||
-ngl 99 \
|
||||
-p "Explain binary search in simple terms." \
|
||||
-n 300
|
||||
```
|
||||
|
||||
## llama-server
|
||||
|
||||
```bash
|
||||
llama-server \
|
||||
-m gguf/Supertron2.1-0.6B-Q4_K_M.gguf \
|
||||
-c 4096 \
|
||||
-ngl 99 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8080
|
||||
```
|
||||
|
||||
Then call it with an OpenAI-compatible client.
|
||||
|
||||
## Ollama Modelfile
|
||||
|
||||
Create a file named `Modelfile`:
|
||||
|
||||
```text
|
||||
FROM ./gguf/Supertron2.1-0.6B-Q4_K_M.gguf
|
||||
|
||||
PARAMETER temperature 0.7
|
||||
PARAMETER top_p 0.8
|
||||
PARAMETER top_k 20
|
||||
PARAMETER num_ctx 4096
|
||||
|
||||
SYSTEM """
|
||||
You are Supertron, a helpful assistant focused on math, coding, and general knowledge.
|
||||
"""
|
||||
```
|
||||
|
||||
Create and run:
|
||||
|
||||
```bash
|
||||
ollama create supertron2.1-0.6b -f Modelfile
|
||||
ollama run supertron2.1-0.6b
|
||||
```
|
||||
|
||||
## Recommended Settings
|
||||
|
||||
For coding and math:
|
||||
|
||||
```text
|
||||
temperature: 0.2
|
||||
top_p: 0.8
|
||||
top_k: 20
|
||||
repeat_penalty: 1.05
|
||||
```
|
||||
|
||||
For chat:
|
||||
|
||||
```text
|
||||
temperature: 0.7
|
||||
top_p: 0.8
|
||||
top_k: 20
|
||||
repeat_penalty: 1.05
|
||||
```
|
||||
|
||||
For deterministic answers:
|
||||
|
||||
```text
|
||||
temperature: 0.0
|
||||
```
|
||||
|
||||
## Model Line
|
||||
|
||||
* Original model: `Surpem/Supertron2.1-0.6B`
|
||||
* GGUF model: `Surpem/Supertron2.1-0.6B-GGUF`
|
||||
* MLX 4-bit: `Surpem/Supertron2.1-0.6B-MLX-4Bit`
|
||||
* MLX 8-bit: `Surpem/Supertron2.1-0.6B-MLX-8Bit`
|
||||
|
||||
## Notes
|
||||
|
||||
The GGUF files were converted from the latest Supertron2.1-0.6B Transformers checkpoint using llama.cpp tooling. Quantized models are approximations of the original bf16 checkpoint, and behavior can vary by runtime, prompt format, and sampling settings.
|
||||
|
||||
## Limitations
|
||||
|
||||
* Q4 is smaller but less precise than Q8/F16.
|
||||
* The model can hallucinate or produce wrong code.
|
||||
* Human review is recommended for math, code, and factual claims.
|
||||
* Do not use this model for safety-critical decisions.
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0.
|
||||
Reference in New Issue
Block a user