AhiskaAI-110M-Experimental-v0.1-Instruct/README.md

---
language:
- tr
- en
license: apache-2.0
tags:
- instruct
- sft
- chatml
- Turkish
- llama
- causal-lm
- slm
- experimental
- tr-llm
- Ahıska
- AhiskaTurks
- MeskhetianTurks
- AhıskaTürkleri
datasets:
- AhiskaAI/AhiskaAI-Instruct-v0.2-Sample-Dataset
pipeline_tag: text-generation
---

# AhıskaAI-110M-IT-Experimental-v0.1

**AhıskaAI-110M-IT-Experimental-v0.1** is an instruction-tuned Small Language Model (~110M parameters) fine-tuned from [AhıskaAI-110M-Experimental-v0.1](https://huggingface.co/AhıskaAI/AhıskaAI-110M-Experimental-v0.1). It is designed to evaluate early instruction-following capabilities on top of synthetic pre-trained representations, using multi-turn Turkish conversations with the **ChatML** template.

## Model Highlights

- **Instruction Alignment:** Supervised Fine-Tuned (SFT) using custom-cleaned multi-turn instruction datasets formatted in ChatML.
- **Loss Masking Strategy:** Trained using custom prompt masking (`labels = -100` for user/system tokens), ensuring loss is calculated **only** on assistant responses for concise and non-hallucinating outputs.
- **System Prompt Support:** Native support for fixed system prompts prioritizing polite, short, and accurate Turkish answers.
- **Hardware-Efficient Fine-Tuning:** Trained on consumer-grade hardware (NVIDIA RTX 4050 6GB GPU) using Liger Kernel acceleration (`apply_liger_kernel_to_llama`) and `adamw_torch_fused`.

## Model Details

- **Base Model:** `AhıskaAI/AhıskaAI-110M-Experimental-v0.1`
- **Architecture:** `LlamaForCausalLM` (MHA)
- **Parameters:** ~110M
- **Fine-Tuning Method:** Full Parameter SFT (Supervised Fine-Tuning)
- **Context Length:** 1024 tokens
- **Template:** ChatML (`<|im_start|>` and `<|im_end|>`)
- **Precision:** Bf16

## Supported System Prompts

The model has been optimized around two primary system personas:

1. `"Sen kibar, sorulan soruları tam cümlelerle yanıtlayan Türkçe bir asistansın."`
2. `"Sen AhıskaAI adında Türkçe bir yapay zeka asistansın.\nGörevlerin:\n1. Sorulara doğrudan, net ve kısa cümlelerle cevap ver.\n2. Bilmediğin veya emin olmadığın konularda uydurma yapma, bilmiyorum de.\n3. Kullanıcının verdiği metin veya listeleri istenen formata sadık kalarak düzenle."`

## Usage (with Transformers)

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AhıskaAI/AhıskaAI-110M-IT-Experimental-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)

# ChatML Formatting
system_prompt = "Sen kibar, sorulan soruları tam cümlelerle yanıtlayan Türkçe bir asistansın."
messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "11 sayısından 2 çıkarırsak kaç kalır? Açıkla."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=150,
    temperature=0.3,
    top_p=0.9,
    do_sample=True
)

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
```

### Training Parameters & Hardware
**Epochs:** 1
**Learning Rate:** 5e-5 (Cosine Schedule)
**Batch Size:** 32 (effective)
**Optimizer:** AdamW Fused (adamw_torch_fused)
**Acceleration:** Liger Kernel (liger-kernel) & SDPA
**Hardware:** NVIDIA RTX 4050 Laptop GPU (6GB VRAM)

### Related Resources
**Base Model:** AhıskaAI-110M-Experimental-v0.1

### About AhıskaAI
**AhıskaAI** is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.