102 lines
3.8 KiB
Markdown
102 lines
3.8 KiB
Markdown
---
|
||
language:
|
||
- tr
|
||
- en
|
||
license: apache-2.0
|
||
tags:
|
||
- instruct
|
||
- sft
|
||
- chatml
|
||
- Turkish
|
||
- llama
|
||
- causal-lm
|
||
- slm
|
||
- experimental
|
||
- tr-llm
|
||
- Ahıska
|
||
- AhiskaTurks
|
||
- MeskhetianTurks
|
||
- AhıskaTürkleri
|
||
datasets:
|
||
- AhiskaAI/AhiskaAI-Instruct-v0.2-Sample-Dataset
|
||
pipeline_tag: text-generation
|
||
---
|
||
|
||
# AhıskaAI-110M-IT-Experimental-v0.1
|
||
|
||
**AhıskaAI-110M-IT-Experimental-v0.1** is an instruction-tuned Small Language Model (~110M parameters) fine-tuned from [AhıskaAI-110M-Experimental-v0.1](https://huggingface.co/AhıskaAI/AhıskaAI-110M-Experimental-v0.1). It is designed to evaluate early instruction-following capabilities on top of synthetic pre-trained representations, using multi-turn Turkish conversations with the **ChatML** template.
|
||
|
||
## Model Highlights
|
||
|
||
- **Instruction Alignment:** Supervised Fine-Tuned (SFT) using custom-cleaned multi-turn instruction datasets formatted in ChatML.
|
||
- **Loss Masking Strategy:** Trained using custom prompt masking (`labels = -100` for user/system tokens), ensuring loss is calculated **only** on assistant responses for concise and non-hallucinating outputs.
|
||
- **System Prompt Support:** Native support for fixed system prompts prioritizing polite, short, and accurate Turkish answers.
|
||
- **Hardware-Efficient Fine-Tuning:** Trained on consumer-grade hardware (NVIDIA RTX 4050 6GB GPU) using Liger Kernel acceleration (`apply_liger_kernel_to_llama`) and `adamw_torch_fused`.
|
||
|
||
## Model Details
|
||
|
||
- **Base Model:** `AhıskaAI/AhıskaAI-110M-Experimental-v0.1`
|
||
- **Architecture:** `LlamaForCausalLM` (MHA)
|
||
- **Parameters:** ~110M
|
||
- **Fine-Tuning Method:** Full Parameter SFT (Supervised Fine-Tuning)
|
||
- **Context Length:** 1024 tokens
|
||
- **Template:** ChatML (`<|im_start|>` and `<|im_end|>`)
|
||
- **Precision:** Bf16
|
||
|
||
## Supported System Prompts
|
||
|
||
The model has been optimized around two primary system personas:
|
||
|
||
1. `"Sen kibar, sorulan soruları tam cümlelerle yanıtlayan Türkçe bir asistansın."`
|
||
2. `"Sen AhıskaAI adında Türkçe bir yapay zeka asistansın.\nGörevlerin:\n1. Sorulara doğrudan, net ve kısa cümlelerle cevap ver.\n2. Bilmediğin veya emin olmadığın konularda uydurma yapma, bilmiyorum de.\n3. Kullanıcının verdiği metin veya listeleri istenen formata sadık kalarak düzenle."`
|
||
|
||
## Usage (with Transformers)
|
||
|
||
```python
|
||
import torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model_id = "AhıskaAI/AhıskaAI-110M-IT-Experimental-v0.1"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_id,
|
||
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
|
||
device_map="auto"
|
||
)
|
||
|
||
# ChatML Formatting
|
||
system_prompt = "Sen kibar, sorulan soruları tam cümlelerle yanıtlayan Türkçe bir asistansın."
|
||
messages = [
|
||
{"role": "system", "content": system_prompt},
|
||
{"role": "user", "content": "11 sayısından 2 çıkarırsak kaç kalır? Açıkla."}
|
||
]
|
||
|
||
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
|
||
outputs = model.generate(
|
||
**inputs,
|
||
max_new_tokens=150,
|
||
temperature=0.3,
|
||
top_p=0.9,
|
||
do_sample=True
|
||
)
|
||
|
||
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
|
||
print(response)
|
||
```
|
||
|
||
### Training Parameters & Hardware
|
||
**Epochs:** 1
|
||
**Learning Rate:** 5e-5 (Cosine Schedule)
|
||
**Batch Size:** 32 (effective)
|
||
**Optimizer:** AdamW Fused (adamw_torch_fused)
|
||
**Acceleration:** Liger Kernel (liger-kernel) & SDPA
|
||
**Hardware:** NVIDIA RTX 4050 Laptop GPU (6GB VRAM)
|
||
|
||
### Related Resources
|
||
**Base Model:** AhıskaAI-110M-Experimental-v0.1
|
||
|
||
### About AhıskaAI
|
||
**AhıskaAI** is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem. |