---
license: mit
library_name: transformers
tags:
- zomi
- qwen
- qlora
- low-resource-language
- text-generation
pipeline_tag: text-generation
base_model: Qwen/Qwen2.5-3B
language:
- zom
- en
---

# Zomi QLoRA v1 — Zomi Language Model

A **Qwen 2.5 3B** fine-tuned on **Zomi (Zou)** language text using QLoRA (rank 128).

This model can generate and understand Zomi text. It is the first publicly available language model for the Zomi language (spoken by ~500,000 people across Myanmar, India, and Bangladesh).

## What This Model Can Do

- Generate coherent Zomi text
- Complete Zomi sentences and passages
- Understand Zomi grammar, vocabulary, and structure
- Serve as a foundation for further fine-tuning

## What This Model Cannot Do (Yet)

- Follow instructions or commands (Phase 2 — instruction fine-tuning — is in progress)
- Translate between Zomi and English on demand
- Hold conversations

This is a **continued pretraining / language adaptation** model. It speaks Zomi but does not yet follow directions. Phase 2 (instruction tuning) will add command-following ability.

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "paumkim/zomi-qlora-v1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype="auto",
)

# Generate Zomi text
prompt = "Pasian in hong it"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Fine-Tune It Yourself

This model is a QLoRA adapter (rank 128) on top of Qwen 2.5 3B. You can load it and fine-tune further:

```python
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B", device_map="auto")
model = PeftModel.from_pretrained(base, model_name)
# Continue training with your own Zomi data
```

## Training Details

| Parameter | Value |
|-----------|-------|
| Base model | Qwen 2.5 3B |
| Method | QLoRA (rank 128, alpha 256) |
| Trainable params | 239M (7.2% of base) |
| Data | ~54,000 chunks of Zomi text from public sources |
| Hardware | NVIDIA A100 80GB |
| GPU hours | ~20 hours |

## Dataset

The model was trained on a corpus of Zomi text collected from public sources including:
- Zomi literature and publications
- Community-contributed content
- Public domain religious texts in Zomi

All data was collected with respect for community ownership of the Zomi language.

## Why This Model Exists

Most AI models do not support Zomi. The Zomi language has ~500,000 speakers but zero commercial AI representation. This model is an attempt to change that — built by a Zomi speaker, for Zomi speakers.

## Phase 2 (Coming)

Instruction fine-tuning is in progress. The Phase 2 model will follow commands in Zomi, enabling translation,问答, content generation, and more.

## License

MIT — free to use, modify, and distribute. We believe the Zomi language should belong to its community, not to any company.