90 lines
4.0 KiB
Markdown
90 lines
4.0 KiB
Markdown
|
---
|
|||
|
|
base_model:
|
|||
|
|
- LVSTCK/domestic-yak-8B
|
|||
|
|
datasets:
|
|||
|
|
- LVSTCK/sft-mk
|
|||
|
|
- LVSTCK/Open-Platypus-MK
|
|||
|
|
- LVSTCK/ultrachat-sft-mk
|
|||
|
|
language:
|
|||
|
|
- mk
|
|||
|
|
license: llama3.1
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- mkd
|
|||
|
|
- mk
|
|||
|
|
- macedonia
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 🐂 domestic-yak, a Macedonian LM (instruct version)
|
|||
|
|
|
|||
|
|
This repository contains the model of the paper [Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language](https://huggingface.co/papers/2506.09560).
|
|||
|
|
|
|||
|
|
Code: https://github.com/LVSTCK
|
|||
|
|
|
|||
|
|
## Model Summary
|
|||
|
|
This is the **instruct-tuned version** of [domestic-yak-8B](https://huggingface.co/LVSTCK/domestic-yak-8B). It has been fine-tuned specifically to improve instruction-following capabilities in Macedonian. It was fine-tuned on the [sft-mk dataset](https://huggingface.co/datasets/LVSTCK/sft-mk) for three epochs. Building on the foundation of `domestic-yak-8B`, this version is optimized for generating coherent, task-specific responses to user queries, making it ideal for chatbots, virtual assistants, and other interactive applications.
|
|||
|
|
|
|||
|
|
### 📊 Results
|
|||
|
|
The table below compares the performance of our model, domestic-yak-8B-instruct with 4 other models. As we can see our model is on par with Llama 70B, and even beats it on three of the benchmarks. It is also worth noting that this model is currently the best in the 8B parameter range.
|
|||
|
|
|
|||
|
|
The results were obtained using the [macedonian-llm-eval](https://github.com/LVSTCK/macedonian-llm-eval) benchmark.
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
wn.png)
|
|||
|
|
|
|||
|
|
## 🔑 Key Details
|
|||
|
|
- **Language:** Macedonian (`mk`)
|
|||
|
|
- **Base Model:** [domestic-yak-8B](https://huggingface.co/LVSTCK/domestic-yak-8B)
|
|||
|
|
- **Dataset:** ~100k samples across multiple categories (Question answering (QA), chat-like conversations, reasoning, essays, and code) consolidated from translating publicly available datasets and custom synthetic data. Dataset can be found [here](https://huggingface.co/datasets/LVSTCK/sft-mk).
|
|||
|
|
- **Fine-tuning Objective:** Supervised fine-tuning (SFT) on Macedonian-specific instruction-following data
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
Pipeline automatically uses apply_chat_template which formats the input appropriately. The model was trained using the default Llama 3.1 [format](https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/).
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import transformers
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
model_id = "LVSTCK/domestic-yak-8B-instruct"
|
|||
|
|
|
|||
|
|
pipeline = transformers.pipeline(
|
|||
|
|
"text-generation",
|
|||
|
|
model=model_id,
|
|||
|
|
model_kwargs={"torch_dtype": torch.bfloat16},
|
|||
|
|
device_map="auto",
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": "Ти си виртуелен асистент кој помага на корисници на македонски јазик. Одговарај на прашања на јасен, разбирлив и професионален начин. Користи правилна граматика и обиди се одговорите да бидат што е можно покорисни и релевантни."},
|
|||
|
|
{"role": "user", "content": "Кој е највисок врв во Македонија?"},
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
outputs = pipeline(
|
|||
|
|
messages,
|
|||
|
|
max_new_tokens=256, # You can increase this
|
|||
|
|
temperature=0.1,
|
|||
|
|
)
|
|||
|
|
print(outputs[0]["generated_text"][-1])
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 📬 Contact
|
|||
|
|
|
|||
|
|
For inquiries, feedback, or contributions, please feel free to reach out to the core team:
|
|||
|
|
|
|||
|
|
- [Stefan Krsteski](https://www.linkedin.com/in/stefan-krsteski-136abb235/) [📧](mailto:stefan.krsteski@gmail.com)
|
|||
|
|
- [Matea Tashkovska](https://www.linkedin.com/in/matea-tashkovska-774603198/) [📧](mailto:matea_tas@yahoo.com)
|
|||
|
|
- [Borjan Sazdov](https://www.linkedin.com/in/borjan-sazdov-4b2187211/) [📧](mailto:borjansazdov@yahoo.com)
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Citation
|
|||
|
|
```
|
|||
|
|
@article{krsteski2025towards,
|
|||
|
|
title={Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language},
|
|||
|
|
author={Krsteski, Stefan and Tashkovska, Matea and Sazdov, Borjan and Gjoreski, Hristijan and Gerazov, Branislav},
|
|||
|
|
journal={arXiv preprint arXiv:2506.09560},
|
|||
|
|
year={2025}
|
|||
|
|
}
|
|||
|
|
```
|