qwen3-0.6b-gguf/USAGE_GUIDE.md

Qwen3-0.6B GGUF Models

This directory contains Qwen3-0.6B language model converted to GGUF format with multiple quantization levels optimized for CPU inference.

Available Model Variants

Variant	File Size	Use Case	Quality	Speed
FP16	1.4 GB	Maximum accuracy, baseline reference	Highest	Slowest
Q8_0	767 MB	High-quality CPU inference	Very High	Medium
Q5_K_M	526 MB	Balanced quality and performance	High	Fast
Q4_K_M	462 MB	Edge devices, fastest inference	Good	Fastest

Quantization Recommendations

• Q4_K_M: Best for edge devices, mobile, or when speed is critical. Minimal quality loss for most tasks.
• Q5_K_M: Recommended for production use. Excellent balance of quality and resource efficiency.
• Q8_0: Use when quality is paramount and you have sufficient memory. Close to FP16 performance.
• FP16: Reference model for validation and quality comparison. Use for benchmarking.

Model Specifications

• Architecture: Qwen3 Causal Language Model
• Parameters: 0.6B (Non-embedding: 0.44B)
• Layers: 28
• Attention: Grouped Query Attention (GQA) - 16 heads for Q, 8 heads for KV
• Context Length: 32,768 tokens (40,960 in config)
• Vocabulary Size: 151,936 tokens
• Special Features:
  • Thinking/non-thinking mode with <think>...</think> tags
  • Multilingual support (100+ languages)
  • Tool calling capabilities
  • Enhanced reasoning

Usage Instructions

1. Using llama.cpp CLI

# Basic text generation
/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf -p "Hello, I am" -n 512 --temp 0.7

# Interactive chat mode
/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf --interactive-first --reverse-prompt "User:"

# With GPU offloading (if available)
/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf -ngl 28 -p "Explain quantum computing"

2. Using Ollama

Create a Modelfile:

FROM ./qwen3-0.6b-q5_k_m.gguf

PARAMETER temperature 0.7
PARAMETER top_k 40
PARAMETER top_p 0.9

TEMPLATE """<|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

Import and run:

# Create the model
ollama create qwen3-0.6b -f Modelfile

# Run the model
ollama run qwen3-0.6b "What is machine learning?"

3. Using Python (llama-cpp-python)

from llama_cpp import Llama

# Initialize model
llm = Llama(
    model_path="qwen3-0.6b-q5_k_m.gguf",
    n_ctx=2048,
    n_threads=4,
    n_gpu_layers=0,
    verbose=False
)

# Generate text
output = llm(
    "Write a short poem about AI:",
    max_tokens=256,
    temperature=0.7,
    stop=["<|im_end|>"]
)

print(output['choices'][0]['text'])

Performance Characteristics

Inference Speed (approximate, CPU-dependent)

Model	Tokens/sec (2-core)	Tokens/sec (8-core)	Memory Usage
Q4_K_M	~15-20	~40-60	~800 MB
Q5_K_M	~12-18	~35-50	~900 MB
Q8_0	~10-15	~30-40	~1.2 GB
FP16	~8-12	~25-35	~1.8 GB

Note: Actual performance depends on CPU architecture, cache size, and prompt complexity.

Recommended CPU Configurations

• Minimum: 2 cores, 2GB RAM - Use Q4_K_M
• Recommended: 4 cores, 4GB RAM - Use Q5_K_M
• Optimal: 8+ cores, 8GB RAM - Use Q8_0 or FP16

Context Length Management

The model supports up to 32,768 tokens but uses less memory with smaller contexts:

# Short context (faster, less memory)
llama-cli -m qwen3-0.6b-q5_k_m.gguf --ctx-size 2048

# Long context (slower, more memory)
llama-cli -m qwen3-0.6b-q5_k_m.gguf --ctx-size 32768

Multilingual Support

Qwen3-0.6B supports 100+ languages including English, Chinese, Spanish, French, German, Japanese, Korean, and many more.

Troubleshooting

Model fails to load

• Solution: Use smaller quantization (Q4_K_M) or reduce context size

Slow generation

• Solution: Increase thread count, use Q4_K_M, or reduce batch size

Poor quality outputs

• Solution: Use higher quantization (Q8_0 or FP16), adjust temperature

Conversion Details

• Source: Qwen/Qwen3-0.6B from Hugging Face Hub
• Conversion Tool: llama.cpp convert_hf_to_gguf.py
• Base Format: FP16 (converted from BF16)
• Quantization Tool: llama-quantize
• Validated: All models tested for loading and inference

License

This model follows the Apache 2.0 license from the original Qwen3-0.6B model.

Original Model: https://huggingface.co/Qwen/Qwen3-0.6B

Support & Resources

• llama.cpp Documentation: https://github.com/ggerganov/llama.cpp
• Ollama Documentation: https://ollama.ai/
• Qwen3 Model Card: https://huggingface.co/Qwen/Qwen3-0.6B

Version Information

• Conversion Date: 2024-12-17
• llama.cpp Version: Build 7451 (669696e00)
• GGUF Version: V3 (latest)

---

Note: Performance metrics are approximate and will vary based on hardware. Test different quantization levels to find the optimal balance for your use case.