---
license: apache-2.0
base_model:
  - Qwen/Qwen2.5-0.5B
base_model_relation: quantized
tags:
  - quantization
  - float16
  - half-precision
  - pytorch
  - edge-deployment
  - qwen2
language:
  - en
pipeline_tag: text-generation
---

# Two_and_a_half_Qwen2.5-MiniFP16

## Overview
This is a **float16 (half precision) quantized** version of [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B).
All model weights are converted from float32 to float16, reducing model size by ~50% while
maintaining near-identical text generation quality.

## Key Features
- **Half the size**: 942.4 MB (down from 1884.7 MB)
- **No GPU required**: Runs on CPU and Apple Silicon Macs
- **Near-lossless**: Float16 preserves most of the original precision
- **Zero training**: Pure post-training quantization
- **HuggingFace native**: Standard safetensors format, load with AutoModelForCausalLM

## Quantization Details
- **Method**: PyTorch `.half()` conversion (float32 -> float16)
- **Target**: All model parameters (weights, biases, embeddings)
- **Original dtype**: torch.float32 (32-bit, 4 bytes per weight)
- **Quantized dtype**: torch.float16 (16-bit, 2 bytes per weight)
- **Compression ratio**: ~2x

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("Ringkvist/Two_and_a_half_Qwen2.5-MiniFP16")
model = AutoModelForCausalLM.from_pretrained(
    "Ringkvist/Two_and_a_half_Qwen2.5-MiniFP16",
    torch_dtype=torch.float16,
)

inputs = tokenizer("The future of AI is", return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Limitations
- Slight numerical precision loss vs float32 (negligible for inference)
- Some operations may need float32 upcasting on certain hardware
- Not as aggressive as int8/int4 quantization but much simpler and more portable

## Base Model
- **Model**: [Qwen/Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)
- **Parameters**: ~494M
- **Architecture**: Qwen2 (decoder-only transformer)