133 lines
3.2 KiB
Markdown
133 lines
3.2 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- tr
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- instruct
|
||
- chat
|
||
- conversational
|
||
- fine-tuned
|
||
- slm
|
||
- llama
|
||
- tr-llm
|
||
- Ahiska
|
||
library_name: transformers
|
||
---
|
||
|
||
|
||
# AhiskaAI-308m-IT-v0.2
|
||
|
||
AhiskaAI-308m-IT-v0.2 is the instruction-tuned version of our 308M parameter Small Language Model. Fine-tuned on 16,000+ curated Turkish instruction-response pairs, it is designed to provide stronger conversational ability and improved instruction following while remaining efficient enough to run on consumer hardware.
|
||
|
||
**Base Model:** AhiskaAI-308m-Base-v0.2
|
||
|
||
---
|
||
|
||
## Model Details
|
||
|
||
- **Architecture:** Llama-based architecture.
|
||
- **Fine-tuning:** Supervised Fine-Tuning (SFT).
|
||
- **Format:** ChatML.
|
||
- **Parameters:** 308M.
|
||
- **Context Window:** 1024 tokens.
|
||
- **Tokenizer:** Custom BPE Tokenizer (Vocabulary Size: 32,000).
|
||
- **Training Framework:** PyTorch & Transformers.
|
||
- **Hardware:** NVIDIA RTX 4050 6GB Laptop GPU.
|
||
|
||
---
|
||
|
||
## Fine-tuning Dataset
|
||
|
||
The model was fine-tuned using more than **16,000** carefully curated Turkish instruction-response pairs.
|
||
|
||
The dataset includes tasks such as:
|
||
|
||
- Question answering
|
||
- General conversation
|
||
- Summarization
|
||
- Text generation
|
||
- Instruction following
|
||
- Basic reasoning
|
||
|
||
---
|
||
|
||
## Design Goal
|
||
|
||
The 308M-IT model serves as the flagship conversational model of the AhiskaAI v0.2 family.
|
||
|
||
Its primary objectives are:
|
||
|
||
- Improved Turkish instruction following.
|
||
- Better contextual understanding.
|
||
- More natural conversational responses.
|
||
- A strong research foundation for future preference alignment methods such as DPO.
|
||
|
||
---
|
||
|
||
## Training Logs
|
||
|
||
|
||
|
||
*The graph above demonstrates the supervised fine-tuning convergence of AhiskaAI-308m-IT-v0.2.*
|
||
|
||
---
|
||
|
||
## Usage (ChatML Format)
|
||
|
||
### Recommended System Prompt
|
||
|
||
```
|
||
Sen kibar, sorulan soruları tam cümlelerle yanıtlayan Türkçe bir asistansın.
|
||
```
|
||
|
||
### Example Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model = AutoModelForCausalLM.from_pretrained("AhiskaAI/AhiskaAI-308m-IT-v0.2")
|
||
tokenizer = AutoTokenizer.from_pretrained("AhiskaAI/AhiskaAI-308m-IT-v0.2")
|
||
|
||
SYSTEM_PROMPT = "Sen kibar, sorulan soruları tam cümlelerle yanıtlayan Türkçe bir asistansın."
|
||
|
||
user_query = "Ahıska Türkleri hakkında bilgi verir misin?"
|
||
|
||
prompt = (
|
||
f"<|im_start|>system\n{SYSTEM_PROMPT}<|im_end|>\n"
|
||
f"<|im_start|>user\n{user_query}<|im_end|>\n"
|
||
f"<|im_start|>assistant\n"
|
||
)
|
||
|
||
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
||
outputs = model.generate(**inputs, max_new_tokens=200)
|
||
|
||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
```
|
||
|
||
---
|
||
|
||
## Known Limitations
|
||
|
||
- Optimized primarily for Turkish.
|
||
- Context window is limited to 1024 tokens.
|
||
- Factual accuracy is still limited by model size and pretraining data.
|
||
- May generate incorrect or incomplete responses on complex reasoning tasks.
|
||
|
||
---
|
||
|
||
## Future Plans
|
||
|
||
- Preference alignment using DPO.
|
||
- Larger, higher-quality Turkish datasets.
|
||
- Expanded evaluation benchmarks.
|
||
- Future AhiskaAI v0.3 model family.
|
||
|
||
---
|
||
|
||
## About AhiskaAI
|
||
|
||
AhiskaAI is an independent open-source initiative dedicated to developing efficient Turkish Small Language Models trained completely from scratch.
|
||
|
||
Follow us on Hugging Face for updates and future releases. |