150 lines
3.8 KiB
Markdown
150 lines
3.8 KiB
Markdown
---
|
||
base_model: TSjB/QM-4B-embeddings-only
|
||
library_name: transformers
|
||
model_name: QM-4B
|
||
tags:
|
||
- qarachay-malqar
|
||
- caucasian-languages
|
||
- turkic-languages
|
||
- karachay-balkar
|
||
- multilingual
|
||
- trl
|
||
- sft
|
||
- unsloth
|
||
language:
|
||
- krc
|
||
- ru
|
||
- en
|
||
license: cc-by-nc-sa-4.0
|
||
---
|
||
|
||
# QM-4B: with Qarachay-Malqar Language
|
||
|
||
A model based on Qwen3-4B-Instruct-2507, fine-tuned to support the Qarachay-Malqar language.
|
||
|
||
## Description
|
||
|
||
QM-4B is a language model built on Qwen3-4B-Instruct-2507 with an extended tokenizer and fine-tuning for Qarachay-Malqar language support (къарачай-малкъар тил).
|
||
|
||
### Training Stages:
|
||
|
||
1. **Tokenizer expansion** — added tokens for Qarachay-Malqar: replacement from 150k to 130k tokens (tokenizer trained in Qarachay-Malqar (76.5%), English (11.5%), Russian (11.5%) and Circassian (5%)) (the number of symbols/tokens has been increased in Qarachay-Malqar compared to the original tokenizer: 1.78 -> 5.38)
|
||
2. **Embeddings-only Training** — training only embedding layers (3 epochs, LR=2e-4)
|
||
3. **Full Fine-Tune** — full fine-tuning of all model layers (1 epoch, LR=5e-6)
|
||
|
||
## Training Metrics
|
||
|
||
| Stage | Train Loss | Eval Loss | Parameters |
|
||
|-------|------------|-----------|------------|
|
||
| Embeddings-only | 4.27 | 4.49 | 8.4% (332M) |
|
||
| Full FT (1 epoch) | 4.16 | 4.36 | 100% (3.97B) |
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"TSjB/QM-4B",
|
||
dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
trust_remote_code=True
|
||
)
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(
|
||
"TSjB/QM-4B",
|
||
trust_remote_code=True
|
||
)
|
||
|
||
# With chat template
|
||
messages = [
|
||
{"role": "system", "content": "Сен къарачай-малкъар тилде болушлукъчуса. Соруўлагъа къысха, тюз эм ачыкъ джуўабла бер. Орусча неда ингилизче сорсала — ол тилде джуўаб бер."},
|
||
{"role": "user", "content": "Не зат билесе Къарачай юсюнден?"}
|
||
]
|
||
text = tokenizer.apply_chat_template(
|
||
messages,
|
||
tokenize=False,
|
||
add_generation_prompt=True,
|
||
enable_thinking=False
|
||
)
|
||
|
||
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
||
|
||
if 'token_type_ids' in inputs:
|
||
inputs.pop('token_type_ids')
|
||
|
||
outputs = model.generate(
|
||
**inputs,
|
||
max_new_tokens=100,
|
||
temperature=0.7,
|
||
top_p=0.9,
|
||
do_sample=True,
|
||
repetition_penalty=1.2,
|
||
no_repeat_ngram_size=4,
|
||
pad_token_id=tokenizer.pad_token_id,
|
||
eos_token_id=tokenizer.eos_token_id,
|
||
)
|
||
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
```
|
||
|
||
## Recommended Generation Parameters
|
||
|
||
```python
|
||
generation_config = {
|
||
"max_new_tokens": 200,
|
||
"temperature": 0.7,
|
||
"top_p": 0.9,
|
||
"do_sample": True,
|
||
"repetition_penalty": 1.2, # important to avoid repetitions
|
||
"no_repeat_ngram_size": 3, # optional
|
||
}
|
||
```
|
||
|
||
## Supported Languages
|
||
|
||
- Qarachay-Malqar (къарачай-малкъар тил)
|
||
- Russian
|
||
- English
|
||
- Other languages from the base Qwen3 model
|
||
|
||
## Limitations
|
||
|
||
- The model was fine-tuned on text data (continued pretraining), not on dialogues
|
||
- May switch between languages within a single response
|
||
- Additional instruction tuning is recommended for better instruction following
|
||
|
||
## Training Data
|
||
|
||
The model was trained on a multilingual text corpus including:
|
||
- Qarachay-Malqar texts
|
||
- Russian texts
|
||
- English texts
|
||
|
||
## License
|
||
|
||
cc-by-nc-sa-4.0
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{qm4b2026,
|
||
title={QM-4B: Qarachay-Malqar language support},
|
||
author={TSjB},
|
||
year={2026},
|
||
publisher={HuggingFace},
|
||
url={https://huggingface.co/TSjB/QM-4B}
|
||
}
|
||
```
|
||
|
||
## Framework Versions
|
||
|
||
- TRL: 0.24.0
|
||
- Transformers: 4.57.3
|
||
- Pytorch: 2.9.0
|
||
- Unsloth: optimized training
|
||
|
||
|
||
## Authors
|
||
[Bogdan Tewunalany](https://t.me/bogdan_tewunalany), [Ali Berberov](https://t.me/ali_berberov) |