初始化项目，由ModelHub XC社区提供模型

Model: AhiskaAI/AhiskaAI-25m-Base-v0.1 Source: Original Platform
2026-08-15 14:07:17 +08:00
commit 1f4011b880
8 changed files with 79419 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,84 @@
+---
+license: apache-2.0
+language:
+- tr
+pipeline_tag: text-generation
+tags:
+- gpt2
+- slm
+- base-model
+- causal-lm
+- pre-trained
+- tr-llm
+- Ahıska
+- AhiskaTurks
+- MeskhetianTurks
+- AhıskaTürkleri
+datasets:
+- wikipedia
+library_name: transformers
+metrics:
+- perplexity
+---
+
+# AhiskaAI 25M Base v0.1 (Pre-trained From Scratch)
+
+**AhiskaAI 25M Base v0.1** is an ultra-lightweight, foundational causal language model trained entirely **from scratch** for the Turkish language ecosystem. 
+
+With only **~25 Million active parameters**, this model was initialized from absolute zero to explore the lower limits of grammatical compression, syntax mapping, and localized semantic density within highly constrained computational environments. It serves as the raw, unaligned backbone for the AhıskaAI research pipeline.
+
+---
+
+## 🧠 Training Profile & Dataset
+
+Despite its miniature footprint, the model captures core Turkish morphology and factual token paths due to a highly strategic, clean data mixture of a **5.3 GB** base corpus:
+
+* **CulturaX (Turkish Split):** Utilized for deep web-scale text distributions, teaching the network core token connectivity, basic sentence boundaries, and general Turkish vocabulary.
+* **Custom Filtered Wikipedia (75 MB):** A highly curated, hand-filtered subset of Turkish Wikipedia explicitly processed by AhıskaAI to target rich historical timelines, cultural identity milestones, and factual knowledge. This dense sub-matrix is the primary driver behind the model's factual recall capabilities despite its size.
+
+### 💻 Hardware & Infrastructure
+* **Hardware:** NVIDIA GeForce RTX 4050 Laptop GPU (6GB VRAM)
+* **Training Depth:** Trained under strict local VRAM constraints with an indie "Build in Public" ethos.
+
+---
+
+## 🛠️ Quickstart & Inference
+
+You can easily load and run text generation using the Hugging Face `transformers` library. 
+
+```python
+from transformers import GPT2LMHeadModel, AutoTokenizer
+import torch
+
+model_name = "AhiskaAI/AhiskaAI-25m-Base-v0.1"
+
+# Load model and custom Turkish tokenizer
+model = GPT2LMHeadModel.from_pretrained(model_name)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+# Configure pad token for GPT-2 architecture
+tokenizer.pad_token = tokenizer.eos_token
+
+# Sample generation
+prompt = "Ahıska Türkleri,"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs, 
+        max_length=100, 
+        do_sample=True, 
+        top_k=50, 
+        top_p=0.95,
+        temperature=0.7,
+        no_repeat_ngram_size=2
+    )
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+⚠️ Model Limitations & Intended Use
+Pure Base Model: This is a raw text completer and has not been aligned using Instruction Fine-Tuning (SFT) or RLHF. It will not act as a conversational chatbot out of the box and might loop sequences if not sampled correctly.
+
+Intended Use: This model is highly receptive to immediate downstream SFT fine-tuning, vocabulary adaptations, or continuation of pre-training checkpoints.
+
+For the aligned chat version, please check out: AhiskaAI-25m-Chat-v0.1-Experimental
--- a/config.json
+++ b/config.json
@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 3,
+  "dtype": "float32",
+  "eos_token_id": 4,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 512,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "max_position_embeddings": 1024,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 10,
+  "num_key_value_heads": 8,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.5.0",
+  "use_cache": false,
+  "vocab_size": 16000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 3,
+  "eos_token_id": 4,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.5.0",
+  "use_cache": true
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25fe9b77683bde57710ef8d1b70cf85528edfbc97a2d2559fd5c165b1045850b
+size 201904144
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,8 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|im_start|>",
+  "eos_token": "<|im_end|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend"
+}
--- a/trainer_state.json
+++ b/trainer_state.json
@@ -0,0 +1,181 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 1076,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.04650622020695268,
+      "grad_norm": 0.14487601816654205,
+      "learning_rate": 4.9000000000000005e-05,
+      "loss": 9.355223999023437,
+      "step": 50
+    },
+    {
+      "epoch": 0.09301244041390536,
+      "grad_norm": 0.07256779819726944,
+      "learning_rate": 9.900000000000001e-05,
+      "loss": 8.395075073242188,
+      "step": 100
+    },
+    {
+      "epoch": 0.13951866062085805,
+      "grad_norm": 0.08071431517601013,
+      "learning_rate": 0.000149,
+      "loss": 7.945545043945312,
+      "step": 150
+    },
+    {
+      "epoch": 0.1860248808278107,
+      "grad_norm": 0.11445911228656769,
+      "learning_rate": 0.000199,
+      "loss": 7.699019775390625,
+      "step": 200
+    },
+    {
+      "epoch": 0.2325311010347634,
+      "grad_norm": 0.10978443920612335,
+      "learning_rate": 0.000249,
+      "loss": 7.29552734375,
+      "step": 250
+    },
+    {
+      "epoch": 0.2790373212417161,
+      "grad_norm": 0.11569751799106598,
+      "learning_rate": 0.000299,
+      "loss": 6.84951416015625,
+      "step": 300
+    },
+    {
+      "epoch": 0.32554354144866876,
+      "grad_norm": 0.11708350479602814,
+      "learning_rate": 0.00034899999999999997,
+      "loss": 6.479200439453125,
+      "step": 350
+    },
+    {
+      "epoch": 0.3720497616556214,
+      "grad_norm": 0.10155152529478073,
+      "learning_rate": 0.00039900000000000005,
+      "loss": 6.209773559570312,
+      "step": 400
+    },
+    {
+      "epoch": 0.41855598186257414,
+      "grad_norm": 0.11156516522169113,
+      "learning_rate": 0.000449,
+      "loss": 5.98971923828125,
+      "step": 450
+    },
+    {
+      "epoch": 0.4650622020695268,
+      "grad_norm": 0.10470128804445267,
+      "learning_rate": 0.000499,
+      "loss": 5.820839233398438,
+      "step": 500
+    },
+    {
+      "epoch": 0.5115684222764795,
+      "grad_norm": 0.0922137200832367,
+      "learning_rate": 0.0004911249568335938,
+      "loss": 5.658131103515625,
+      "step": 550
+    },
+    {
+      "epoch": 0.5580746424834322,
+      "grad_norm": 0.09017608314752579,
+      "learning_rate": 0.00046443215250006805,
+      "loss": 5.505938720703125,
+      "step": 600
+    },
+    {
+      "epoch": 0.6045808626903848,
+      "grad_norm": 0.10910492390394211,
+      "learning_rate": 0.00042189072640434186,
+      "loss": 5.364219970703125,
+      "step": 650
+    },
+    {
+      "epoch": 0.6510870828973375,
+      "grad_norm": 0.0955413430929184,
+      "learning_rate": 0.000366644903717664,
+      "loss": 5.2475146484375,
+      "step": 700
+    },
+    {
+      "epoch": 0.6975933031042902,
+      "grad_norm": 0.07590549439191818,
+      "learning_rate": 0.0003027778880897413,
+      "loss": 5.147374267578125,
+      "step": 750
+    },
+    {
+      "epoch": 0.7440995233112428,
+      "grad_norm": 0.07280214875936508,
+      "learning_rate": 0.00023501007318339978,
+      "loss": 5.035035705566406,
+      "step": 800
+    },
+    {
+      "epoch": 0.7906057435181956,
+      "grad_norm": 0.07863806188106537,
+      "learning_rate": 0.0001683501596695945,
+      "loss": 4.933069152832031,
+      "step": 850
+    },
+    {
+      "epoch": 0.8371119637251483,
+      "grad_norm": 0.08269604295492172,
+      "learning_rate": 0.00010772496353027538,
+      "loss": 4.838182983398437,
+      "step": 900
+    },
+    {
+      "epoch": 0.8836181839321009,
+      "grad_norm": 0.0789259672164917,
+      "learning_rate": 5.761527643622996e-05,
+      "loss": 4.838483276367188,
+      "step": 950
+    },
+    {
+      "epoch": 0.9301244041390536,
+      "grad_norm": 0.07977437973022461,
+      "learning_rate": 2.172469165981239e-05,
+      "loss": 4.8021307373046875,
+      "step": 1000
+    },
+    {
+      "epoch": 0.9766306243460062,
+      "grad_norm": 0.09115710109472275,
+      "learning_rate": 2.7058725088047465e-06,
+      "loss": 4.787516784667969,
+      "step": 1050
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 1076,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 107,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.11175261869312e+16,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}