初始化项目，由ModelHub XC社区提供模型

Model: gvij/qwen3-0.6b-gguf Source: Original Platform
2026-09-03 21:08:40 +08:00
commit aecd296504
7 changed files with 469 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,39 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+qwen3-0.6b-fp16.gguf filter=lfs diff=lfs merge=lfs -text
+qwen3-0.6b-q4_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+qwen3-0.6b-q5_k_m.gguf filter=lfs diff=lfs merge=lfs -text
+qwen3-0.6b-q8_0.gguf filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,237 @@
+---
+license: apache-2.0
+library_name: gguf
+tags:
+- gguf
+- llama-cpp
+- cpu-inference
+- qwen3
+- qwen
+- multilingual
+- text-generation
+- conversational
+- tool-calling
+- thinking-mode
+- quantized
+- neo
+language:
+- en
+- zh
+- es
+- fr
+- de
+- ja
+- ko
+- ar
+- ru
+- pt
+- it
+- nl
+- pl
+- tr
+- vi
+- th
+- id
+- hi
+- uk
+- cs
+- ro
+- sv
+- da
+- fi
+- no
+- hu
+- el
+- he
+- fa
+pipeline_tag: text-generation
+model_type: qwen2
+base_model: Qwen/Qwen3-0.6B
+---
+
+# Qwen3-0.6B GGUF
+
+This repository contains GGUF (GPT-Generated Unified Format) conversions of the [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) language model, optimized for efficient CPU inference using llama.cpp, Ollama, and other GGUF-compatible engines.
+
+This model was converted by [NEO](https://heyneo.so) - Fully autonomous ML Engineering Agent.
+
+## Model Overview
+
+**Qwen3-0.6B** is a compact yet powerful 0.6 billion parameter causal language model from Alibaba's Qwen series, featuring:
+- **Dual-mode inference**: Supports both thinking and non-thinking modes for flexible reasoning
+- **Enhanced reasoning**: Improved logical reasoning and problem-solving capabilities
+- **Multilingual support**: Proficient in 100+ languages
+- **Tool calling**: Native support for function calling and agent workflows
+- **Extended context**: 32,768 token context length
+- **Efficient architecture**: GQA attention mechanism for optimized inference
+
+## Architecture Details
+
+- **Parameters**: 0.6B total (0.44B non-embedding)
+- **Layers**: 28 transformer layers
+- **Attention**: Grouped Query Attention (GQA)
+  - 16 Query heads
+  - 8 Key-Value heads
+- **Context Length**: 32,768 tokens
+- **Vocabulary Size**: 151,936 tokens
+- **Original Precision**: BF16
+
+## Quantization Variants
+
+This repository provides 4 GGUF quantization variants optimized for different use cases:
+
+| Model Variant | File Size | Quantization | Use Case | Quality |
+|--------------|-----------|--------------|----------|---------|
+| **qwen3-0.6b-fp16.gguf** | 1,439 MB | FP16 | Reference quality, GPU inference | Highest |
+| **qwen3-0.6b-q8_0.gguf** | 767 MB | Q8_0 | High-quality CPU inference | Very High |
+| **qwen3-0.6b-q5_k_m.gguf** | 526 MB | Q5_K_M | Production CPU deployment | High |
+| **qwen3-0.6b-q4_k_m.gguf** | 462 MB | Q4_K_M | Edge devices, mobile, low memory | Good |
+
+### Quantization Recommendations
+
+- **FP16**: Use for reference benchmarks or when GPU memory is available
+- **Q8_0**: Best balance for CPU inference with minimal quality loss
+- **Q5_K_M**: Recommended for production CPU deployments (best quality/size ratio)
+- **Q4_K_M**: Optimal for resource-constrained environments (edge, mobile, IoT)
+
+## Usage Instructions
+
+### llama.cpp CLI
+
+Download a model variant and run inference:
+
+```bash
+# Download model (replace with your preferred variant)
+huggingface-cli download gvij/qwen3-0.6b-gguf qwen3-0.6b-q5_k_m.gguf --local-dir ./models
+
+# Run inference
+./llama-cli -m models/qwen3-0.6b-q5_k_m.gguf -p "Explain quantum computing:" -n 256 --temp 0.7
+
+# Interactive chat mode
+./llama-cli -m models/qwen3-0.6b-q5_k_m.gguf -cnv --color
+```
+
+### Ollama Integration
+
+Create a Modelfile:
+
+```dockerfile
+FROM ./qwen3-0.6b-q5_k_m.gguf
+
+PARAMETER temperature 0.7
+PARAMETER top_p 0.9
+PARAMETER stop "<|im_end|>"
+PARAMETER stop "<|endoftext|>"
+
+TEMPLATE """
+<|im_start|>system
+You are a helpful AI assistant.<|im_end|>
+<|im_start|>user
+{{ .Prompt }}<|im_end|>
+<|im_start|>assistant
+"""
+```
+
+Create and run the model:
+
+```bash
+ollama create qwen3-0.6b -f Modelfile
+ollama run qwen3-0.6b "Write a Python function to calculate factorial"
+```
+
+### Python with llama-cpp-python
+
+```python
+from llama_cpp import Llama
+
+llm = Llama(
+    model_path="qwen3-0.6b-q5_k_m.gguf",
+    n_ctx=32768,
+    n_threads=8,
+    n_gpu_layers=0
+)
+
+response = llm(
+    "Explain the theory of relativity in simple terms:",
+    max_tokens=256,
+    temperature=0.7,
+    top_p=0.9,
+    stop=["<|im_end|>", "<|endoftext|>"]
+)
+
+print(response['choices'][0]['text'])
+```
+
+### Thinking Mode
+
+Qwen3 supports explicit reasoning with thinking mode. Use the `<think>` tags:
+
+```python
+prompt = """Think through this problem step by step:
+<think>
+What is 15% of 240?
+</think>"""
+
+response = llm(prompt, max_tokens=512)
+```
+
+## Performance Characteristics
+
+**CPU Inference Speed** (approximate, on modern x86-64 CPU):
+
+- **Q4_K_M**: ~20-30 tokens/second
+- **Q5_K_M**: ~18-25 tokens/second
+- **Q8_0**: ~12-18 tokens/second
+- **FP16**: ~8-12 tokens/second (CPU) / ~50-80 tokens/second (GPU)
+
+**Memory Requirements**:
+
+- **Q4_K_M**: ~600 MB RAM
+- **Q5_K_M**: ~700 MB RAM
+- **Q8_0**: ~1 GB RAM
+- **FP16**: ~1.6 GB RAM
+
+*Note: Actual performance depends on CPU architecture, clock speed, and context length.*
+
+## Features
+
+✅ **Thinking & Non-thinking Modes**: Dynamic reasoning control  
+✅ **Multilingual**: 100+ languages supported  
+✅ **Tool Calling**: Native function calling for agent workflows  
+✅ **Extended Context**: 32K token context window  
+✅ **CPU Optimized**: GGUF format for efficient CPU inference  
+✅ **Flexible Deployment**: Compatible with llama.cpp, Ollama, LMStudio, MLX-LM
+
+## Model Source
+
+- **Original Model**: [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)
+- **Model Family**: Qwen3 Series
+- **Developer**: Alibaba Cloud
+- **Conversion**: HuggingFace → GGUF using llama.cpp conversion scripts
+
+## License
+
+This model is released under the **Apache 2.0 License**, following the original Qwen3-0.6B license terms.
+
+## Citation
+
+```bibtex
+@misc{qwen3-0.6b-gguf,
+  title={Qwen3-0.6B GGUF},
+  author={Qwen Team},
+  year={2024},
+  url={https://huggingface.co/gvij/qwen3-0.6b-gguf}
+}
+```
+
+## Acknowledgments
+
+- **Qwen Team** at Alibaba Cloud for the original model
+- **llama.cpp** community for GGUF format and conversion tools
+- Model conversion performed using llama.cpp conversion pipeline
+
+---
+
+For issues, questions, or feedback, please visit the original [Qwen3-0.6B repository](https://huggingface.co/Qwen/Qwen3-0.6B).
+
+Authored and published by [NEO](https://heyneo.so)
--- a/USAGE_GUIDE.md
+++ b/USAGE_GUIDE.md
@@ -0,0 +1,181 @@
+# Qwen3-0.6B GGUF Models
+
+This directory contains Qwen3-0.6B language model converted to GGUF format with multiple quantization levels optimized for CPU inference.
+
+## Available Model Variants
+
+| Variant | File Size | Use Case | Quality | Speed |
+|---------|-----------|----------|---------|-------|
+| **FP16** | 1.4 GB | Maximum accuracy, baseline reference | Highest | Slowest |
+| **Q8_0** | 767 MB | High-quality CPU inference | Very High | Medium |
+| **Q5_K_M** | 526 MB | Balanced quality and performance | High | Fast |
+| **Q4_K_M** | 462 MB | Edge devices, fastest inference | Good | Fastest |
+
+### Quantization Recommendations
+
+- **Q4_K_M**: Best for edge devices, mobile, or when speed is critical. Minimal quality loss for most tasks.
+- **Q5_K_M**: Recommended for production use. Excellent balance of quality and resource efficiency.
+- **Q8_0**: Use when quality is paramount and you have sufficient memory. Close to FP16 performance.
+- **FP16**: Reference model for validation and quality comparison. Use for benchmarking.
+
+## Model Specifications
+
+- **Architecture**: Qwen3 Causal Language Model
+- **Parameters**: 0.6B (Non-embedding: 0.44B)
+- **Layers**: 28
+- **Attention**: Grouped Query Attention (GQA) - 16 heads for Q, 8 heads for KV
+- **Context Length**: 32,768 tokens (40,960 in config)
+- **Vocabulary Size**: 151,936 tokens
+- **Special Features**:
+  - Thinking/non-thinking mode with `<think>...</think>` tags
+  - Multilingual support (100+ languages)
+  - Tool calling capabilities
+  - Enhanced reasoning
+
+## Usage Instructions
+
+### 1. Using llama.cpp CLI
+
+```bash
+# Basic text generation
+/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf -p "Hello, I am" -n 512 --temp 0.7
+
+# Interactive chat mode
+/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf --interactive-first --reverse-prompt "User:"
+
+# With GPU offloading (if available)
+/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf -ngl 28 -p "Explain quantum computing"
+```
+
+### 2. Using Ollama
+
+Create a `Modelfile`:
+
+```dockerfile
+FROM ./qwen3-0.6b-q5_k_m.gguf
+
+PARAMETER temperature 0.7
+PARAMETER top_k 40
+PARAMETER top_p 0.9
+
+TEMPLATE """<|im_start|>system
+You are a helpful AI assistant.<|im_end|>
+<|im_start|>user
+{{ .Prompt }}<|im_end|>
+<|im_start|>assistant
+"""
+
+PARAMETER stop "<|im_start|>"
+PARAMETER stop "<|im_end|>"
+```
+
+Import and run:
+
+```bash
+# Create the model
+ollama create qwen3-0.6b -f Modelfile
+
+# Run the model
+ollama run qwen3-0.6b "What is machine learning?"
+```
+
+### 3. Using Python (llama-cpp-python)
+
+```python
+from llama_cpp import Llama
+
+# Initialize model
+llm = Llama(
+    model_path="qwen3-0.6b-q5_k_m.gguf",
+    n_ctx=2048,
+    n_threads=4,
+    n_gpu_layers=0,
+    verbose=False
+)
+
+# Generate text
+output = llm(
+    "Write a short poem about AI:",
+    max_tokens=256,
+    temperature=0.7,
+    stop=["<|im_end|>"]
+)
+
+print(output['choices'][0]['text'])
+```
+
+## Performance Characteristics
+
+### Inference Speed (approximate, CPU-dependent)
+
+| Model | Tokens/sec (2-core) | Tokens/sec (8-core) | Memory Usage |
+|-------|---------------------|---------------------|--------------|
+| Q4_K_M | ~15-20 | ~40-60 | ~800 MB |
+| Q5_K_M | ~12-18 | ~35-50 | ~900 MB |
+| Q8_0 | ~10-15 | ~30-40 | ~1.2 GB |
+| FP16 | ~8-12 | ~25-35 | ~1.8 GB |
+
+*Note: Actual performance depends on CPU architecture, cache size, and prompt complexity.*
+
+### Recommended CPU Configurations
+
+- **Minimum**: 2 cores, 2GB RAM - Use Q4_K_M
+- **Recommended**: 4 cores, 4GB RAM - Use Q5_K_M
+- **Optimal**: 8+ cores, 8GB RAM - Use Q8_0 or FP16
+
+## Context Length Management
+
+The model supports up to 32,768 tokens but uses less memory with smaller contexts:
+
+```bash
+# Short context (faster, less memory)
+llama-cli -m qwen3-0.6b-q5_k_m.gguf --ctx-size 2048
+
+# Long context (slower, more memory)
+llama-cli -m qwen3-0.6b-q5_k_m.gguf --ctx-size 32768
+```
+
+## Multilingual Support
+
+Qwen3-0.6B supports 100+ languages including English, Chinese, Spanish, French, German, Japanese, Korean, and many more.
+
+## Troubleshooting
+
+### Model fails to load
+- **Solution**: Use smaller quantization (Q4_K_M) or reduce context size
+
+### Slow generation
+- **Solution**: Increase thread count, use Q4_K_M, or reduce batch size
+
+### Poor quality outputs
+- **Solution**: Use higher quantization (Q8_0 or FP16), adjust temperature
+
+## Conversion Details
+
+- **Source**: Qwen/Qwen3-0.6B from Hugging Face Hub
+- **Conversion Tool**: llama.cpp convert_hf_to_gguf.py
+- **Base Format**: FP16 (converted from BF16)
+- **Quantization Tool**: llama-quantize
+- **Validated**: All models tested for loading and inference
+
+## License
+
+This model follows the Apache 2.0 license from the original Qwen3-0.6B model.
+
+**Original Model**: https://huggingface.co/Qwen/Qwen3-0.6B
+
+## Support & Resources
+
+- **llama.cpp Documentation**: https://github.com/ggerganov/llama.cpp
+- **Ollama Documentation**: https://ollama.ai/
+- **Qwen3 Model Card**: https://huggingface.co/Qwen/Qwen3-0.6B
+
+## Version Information
+
+- **Conversion Date**: 2024-12-17
+- **llama.cpp Version**: Build 7451 (669696e00)
+- **GGUF Version**: V3 (latest)
+
+---
+
+**Note**: Performance metrics are approximate and will vary based on hardware. Test different quantization levels to find the optimal balance for your use case.
--- a/qwen3-0.6b-fp16.gguf
+++ b/qwen3-0.6b-fp16.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5e5338391a612235b2d22a48f74e3d6848c18ea4a2ea16c358c8cdf099323b9
+size 1509347616
--- a/qwen3-0.6b-q4_k_m.gguf
+++ b/qwen3-0.6b-q4_k_m.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3479875d3e4c726f7a20b2181f5e1536aefe9925f284f9ae9997a39a7e0d8dc9
+size 484220192
--- a/qwen3-0.6b-q5_k_m.gguf
+++ b/qwen3-0.6b-q5_k_m.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4a1b07a355cee8b5c9b2649618619f7560b031035d8aa4a350a6ce8d3f01587
+size 551378208
--- a/qwen3-0.6b-q8_0.gguf
+++ b/qwen3-0.6b-q8_0.gguf
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed405ab153351dd5932ce2681d75ca01f2741091747be8a2f95a7f95fc8fda29
+size 804753696