license, library_name, tags, pipeline_tag, base_model, language

license	library_name	tags	pipeline_tag	base_model	language
mit	transformers	zomi qwen qlora low-resource-language text-generation	text-generation	Qwen/Qwen2.5-3B	zom en

Zomi QLoRA v1 — Zomi Language Model

A Qwen 2.5 3B fine-tuned on Zomi (Zou) language text using QLoRA (rank 128).

This model can generate and understand Zomi text. It is the first publicly available language model for the Zomi language (spoken by ~500,000 people across Myanmar, India, and Bangladesh).

What This Model Can Do

• Generate coherent Zomi text
• Complete Zomi sentences and passages
• Understand Zomi grammar, vocabulary, and structure
• Serve as a foundation for further fine-tuning

What This Model Cannot Do (Yet)

• Follow instructions or commands (Phase 2 — instruction fine-tuning — is in progress)
• Translate between Zomi and English on demand
• Hold conversations

This is a continued pretraining / language adaptation model. It speaks Zomi but does not yet follow directions. Phase 2 (instruction tuning) will add command-following ability.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "paumkim/zomi-qlora-v1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype="auto",
)

# Generate Zomi text
prompt = "Pasian in hong it"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Fine-Tune It Yourself

This model is a QLoRA adapter (rank 128) on top of Qwen 2.5 3B. You can load it and fine-tune further:

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B", device_map="auto")
model = PeftModel.from_pretrained(base, model_name)
# Continue training with your own Zomi data

Training Details

Parameter	Value
Base model	Qwen 2.5 3B
Method	QLoRA (rank 128, alpha 256)
Trainable params	239M (7.2% of base)
Data	~54,000 chunks of Zomi text from public sources
Hardware	NVIDIA A100 80GB
GPU hours	~20 hours

Dataset

The model was trained on a corpus of Zomi text collected from public sources including:

• Zomi literature and publications
• Community-contributed content
• Public domain religious texts in Zomi

All data was collected with respect for community ownership of the Zomi language.

Why This Model Exists

Most AI models do not support Zomi. The Zomi language has ~500,000 speakers but zero commercial AI representation. This model is an attempt to change that — built by a Zomi speaker, for Zomi speakers.

Phase 2 (Coming)

Instruction fine-tuning is in progress. The Phase 2 model will follow commands in Zomi, enabling translation,问答, content generation, and more.

License

MIT — free to use, modify, and distribute. We believe the Zomi language should belong to its community, not to any company.