qwen3-0.6b-gguf/USAGE_GUIDE.md

# Qwen3-0.6B GGUF Models

This directory contains Qwen3-0.6B language model converted to GGUF format with multiple quantization levels optimized for CPU inference.

## Available Model Variants

| Variant | File Size | Use Case | Quality | Speed |
|---------|-----------|----------|---------|-------|
| **FP16** | 1.4 GB | Maximum accuracy, baseline reference | Highest | Slowest |
| **Q8_0** | 767 MB | High-quality CPU inference | Very High | Medium |
| **Q5_K_M** | 526 MB | Balanced quality and performance | High | Fast |
| **Q4_K_M** | 462 MB | Edge devices, fastest inference | Good | Fastest |

### Quantization Recommendations

- **Q4_K_M**: Best for edge devices, mobile, or when speed is critical. Minimal quality loss for most tasks.
- **Q5_K_M**: Recommended for production use. Excellent balance of quality and resource efficiency.
- **Q8_0**: Use when quality is paramount and you have sufficient memory. Close to FP16 performance.
- **FP16**: Reference model for validation and quality comparison. Use for benchmarking.

## Model Specifications

- **Architecture**: Qwen3 Causal Language Model
- **Parameters**: 0.6B (Non-embedding: 0.44B)
- **Layers**: 28
- **Attention**: Grouped Query Attention (GQA) - 16 heads for Q, 8 heads for KV
- **Context Length**: 32,768 tokens (40,960 in config)
- **Vocabulary Size**: 151,936 tokens
- **Special Features**:
  - Thinking/non-thinking mode with `<think>...</think>` tags
  - Multilingual support (100+ languages)
  - Tool calling capabilities
  - Enhanced reasoning

## Usage Instructions

### 1. Using llama.cpp CLI

```bash
# Basic text generation
/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf -p "Hello, I am" -n 512 --temp 0.7

# Interactive chat mode
/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf --interactive-first --reverse-prompt "User:"

# With GPU offloading (if available)
/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf -ngl 28 -p "Explain quantum computing"
```

### 2. Using Ollama

Create a `Modelfile`:

```dockerfile
FROM ./qwen3-0.6b-q5_k_m.gguf

PARAMETER temperature 0.7
PARAMETER top_k 40
PARAMETER top_p 0.9

TEMPLATE """<|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
```

Import and run:

```bash
# Create the model
ollama create qwen3-0.6b -f Modelfile

# Run the model
ollama run qwen3-0.6b "What is machine learning?"
```

### 3. Using Python (llama-cpp-python)

```python
from llama_cpp import Llama

# Initialize model
llm = Llama(
    model_path="qwen3-0.6b-q5_k_m.gguf",
    n_ctx=2048,
    n_threads=4,
    n_gpu_layers=0,
    verbose=False
)

# Generate text
output = llm(
    "Write a short poem about AI:",
    max_tokens=256,
    temperature=0.7,
    stop=["<|im_end|>"]
)

print(output['choices'][0]['text'])
```

## Performance Characteristics

### Inference Speed (approximate, CPU-dependent)

| Model | Tokens/sec (2-core) | Tokens/sec (8-core) | Memory Usage |
|-------|---------------------|---------------------|--------------|
| Q4_K_M | ~15-20 | ~40-60 | ~800 MB |
| Q5_K_M | ~12-18 | ~35-50 | ~900 MB |
| Q8_0 | ~10-15 | ~30-40 | ~1.2 GB |
| FP16 | ~8-12 | ~25-35 | ~1.8 GB |

*Note: Actual performance depends on CPU architecture, cache size, and prompt complexity.*

### Recommended CPU Configurations

- **Minimum**: 2 cores, 2GB RAM - Use Q4_K_M
- **Recommended**: 4 cores, 4GB RAM - Use Q5_K_M
- **Optimal**: 8+ cores, 8GB RAM - Use Q8_0 or FP16

## Context Length Management

The model supports up to 32,768 tokens but uses less memory with smaller contexts:

```bash
# Short context (faster, less memory)
llama-cli -m qwen3-0.6b-q5_k_m.gguf --ctx-size 2048

# Long context (slower, more memory)
llama-cli -m qwen3-0.6b-q5_k_m.gguf --ctx-size 32768
```

## Multilingual Support

Qwen3-0.6B supports 100+ languages including English, Chinese, Spanish, French, German, Japanese, Korean, and many more.

## Troubleshooting

### Model fails to load
- **Solution**: Use smaller quantization (Q4_K_M) or reduce context size

### Slow generation
- **Solution**: Increase thread count, use Q4_K_M, or reduce batch size

### Poor quality outputs
- **Solution**: Use higher quantization (Q8_0 or FP16), adjust temperature

## Conversion Details

- **Source**: Qwen/Qwen3-0.6B from Hugging Face Hub
- **Conversion Tool**: llama.cpp convert_hf_to_gguf.py
- **Base Format**: FP16 (converted from BF16)
- **Quantization Tool**: llama-quantize
- **Validated**: All models tested for loading and inference

## License

This model follows the Apache 2.0 license from the original Qwen3-0.6B model.

**Original Model**: https://huggingface.co/Qwen/Qwen3-0.6B

## Support & Resources

- **llama.cpp Documentation**: https://github.com/ggerganov/llama.cpp
- **Ollama Documentation**: https://ollama.ai/
- **Qwen3 Model Card**: https://huggingface.co/Qwen/Qwen3-0.6B

## Version Information

- **Conversion Date**: 2024-12-17
- **llama.cpp Version**: Build 7451 (669696e00)
- **GGUF Version**: V3 (latest)

---

**Note**: Performance metrics are approximate and will vary based on hardware. Test different quantization levels to find the optimal balance for your use case.