82 lines
2.8 KiB
Markdown
82 lines
2.8 KiB
Markdown
|
---
|
|||
|
|
language:
|
|||
|
|
- tr
|
|||
|
|
license: apache-2.0
|
|||
|
|
tags:
|
|||
|
|
- llama
|
|||
|
|
- causal-lm
|
|||
|
|
- slm
|
|||
|
|
- experimental
|
|||
|
|
- turkish
|
|||
|
|
- tr-llm
|
|||
|
|
- AhiskaAI
|
|||
|
|
datasets:
|
|||
|
|
- BILGEM-AI/BILGE-Synthetic-Math
|
|||
|
|
- BILGEM-AI/BILGE-Synthetic-Web
|
|||
|
|
- BILGEM-AI/BILGE-Synthetic-Stories
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# AhıskaAI-110M-Experimental-v0.1
|
|||
|
|
|
|||
|
|
**AhıskaAI-110M-Experimental-v0.1** is a 110M parameter Small Language Model (SLM) trained from scratch as part of the experimental release series by AhıskaAI. This series focuses on evaluating early-stage convergence and reasoning behavior on synthetic Turkish datasets (covering math, web content, and narrative stories).
|
|||
|
|
|
|||
|
|
## Experimental Setup & Scope
|
|||
|
|
|
|||
|
|
- **Objective:** Test synthetic data density (Math + Web + Stories) and sequence modeling performance on a compact 110M Llama architecture.
|
|||
|
|
- **Training Epochs:** Pre-trained for **0.3 epochs** across a ~3.5 GB synthetic Turkish corpus.
|
|||
|
|
- **Context Length:** 1024 tokens.
|
|||
|
|
|
|||
|
|
## Model Details
|
|||
|
|
|
|||
|
|
- **Architecture:** `LlamaForCausalLM` (Standard MHA)
|
|||
|
|
- **Parameters:** ~110M
|
|||
|
|
- **Hidden Size:** 768
|
|||
|
|
- **Intermediate Size:** 2048
|
|||
|
|
- **Attention Heads:** 12 (Query) / 12 (Key/Value)
|
|||
|
|
- **Layers:** 12
|
|||
|
|
- **Context Length:** 1024 tokens
|
|||
|
|
- **Vocabulary Size:** 32,000
|
|||
|
|
- **Precision:** `float32` / `float16`
|
|||
|
|
|
|||
|
|
## Pre-training Dataset Mixture
|
|||
|
|
|
|||
|
|
The model was pre-trained on a synthetic Turkish corpus (~3.5 GB total):
|
|||
|
|
|
|||
|
|
| Data Source | Approx. Size | Primary Focus | Link |
|
|||
|
|
| :--- | :--- | :--- | :--- |
|
|||
|
|
| **BILGEM AI Synthetic Web** | ~2.0 GB | General Web, Knowledge & Grammar | [BILGEM-AI/BILGE-Synthetic-Web](https://huggingface.co/datasets/BILGEM-AI/BILGE-Synthetic-Web) |
|
|||
|
|
| **BILGEM AI Synthetic Math** | ~750 MB | Mathematical Reasoning & Logic | [BILGEM-AI/BILGE-Synthetic-Math](https://huggingface.co/datasets/BILGEM-AI/BILGE-Synthetic-Math) |
|
|||
|
|
| **BILGEM AI Synthetic Stories** | ~750 MB | Narrative Flow & Text Generation | [BILGEM-AI/BILGE-Synthetic-Stories](https://huggingface.co/datasets/BILGEM-AI/BILGE-Synthetic-Stories) |
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import torch
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
model_id = "AhıskaAI/AhıskaAI-110M-Experimental-v0.1"
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
model_id,
|
|||
|
|
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
|
|||
|
|
device_map="auto"
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
prompt = "Bir zamanlar uzak bir ülkede"
|
|||
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|||
|
|
|
|||
|
|
outputs = model.generate(
|
|||
|
|
**inputs,
|
|||
|
|
max_new_tokens=100,
|
|||
|
|
temperature=0.7,
|
|||
|
|
top_p=0.9,
|
|||
|
|
do_sample=True
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### About AhıskaAI
|
|||
|
|
**AhıskaAI** is an independent initiative dedicated to developing efficient, high-performance Small Language Models (SLMs) tailored for the Turkish language ecosystem.
|