130 lines
3.7 KiB
Markdown
130 lines
3.7 KiB
Markdown
---
|
||
language:
|
||
- ru
|
||
- en
|
||
license: other
|
||
base_model: NickupAI/LFM2.5-1.2B-Saiga-It-v3
|
||
tags:
|
||
- russian
|
||
- saiga
|
||
- continued-pretraining
|
||
- sft
|
||
- lfm
|
||
- liquid-ai
|
||
datasets:
|
||
- IlyaGusev/saiga_scored
|
||
- d0rj/alpaca-cleaned-ru
|
||
- IlyaGusev/ru_sharegpt_cleaned
|
||
- IlyaGusev/ru_turbo_saiga
|
||
- lksy/ru_instruct_gpt4
|
||
- wikimedia/wikipedia
|
||
- uonlp/CulturaX
|
||
- allenai/c4
|
||
- IlyaGusev/habr
|
||
---
|
||
|
||
# LFM2.5-1.2B-Saiga-It-v3
|
||
|
||
Continued attempt to make a Russian-language Saiga out of [LFM2.5 1.2B](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct) by Liquid AI. This is v3 — a significant upgrade over [v2](https://huggingface.co/NickupAI/LFM2.5-1.2B-Saiga-It-v2) with much more CPT data.
|
||
|
||
Inspired by the amazing work of [Ilya Gusev](https://huggingface.co/IlyaGusev) and the [Saiga project](https://huggingface.co/IlyaGusev/saiga_llama3_8b).
|
||
|
||
## What changed vs v2?
|
||
|
||
**CPT dataset grew from ~1.25M to ~2.69M examples:**
|
||
- Full Russian Wikipedia (~1.9M articles, up from 350k)
|
||
- Added Habr (~295k technical articles in Russian)
|
||
- Replaced c4 ru with CulturaX ru (200k)
|
||
- Kept English c4 (300k) for retention
|
||
|
||
**SFT dataset expanded:**
|
||
- Added `IlyaGusev/ru_turbo_saiga`
|
||
- Added `lksy/ru_instruct_gpt4`
|
||
|
||
## Training
|
||
|
||
**Stage 1 — Continued Pre-Training (CPT):**
|
||
- `wikimedia/wikipedia` (Russian, ~1.9M articles)
|
||
- `IlyaGusev/habr` (Russian, ~295k technical articles)
|
||
- `uonlp/CulturaX` (Russian, 200k)
|
||
- `allenai/c4` (English, 300k for retention)
|
||
- 10,000 steps, lr=3e-5, loss 2.26 → 2.13
|
||
|
||
**Stage 2 — Supervised Fine-Tuning (SFT):**
|
||
- `IlyaGusev/saiga_scored` (opus_score ≥ 8, ~27k)
|
||
- `d0rj/alpaca-cleaned-ru` (15k)
|
||
- `IlyaGusev/ru_sharegpt_cleaned`
|
||
- `IlyaGusev/ru_turbo_saiga`
|
||
- `lksy/ru_instruct_gpt4`
|
||
- 2 epochs, lr=5e-6, final loss ~1.21
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"NickupAI/LFM2.5-1.2B-Saiga-It-v3",
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
trust_remote_code=True,
|
||
)
|
||
tokenizer = AutoTokenizer.from_pretrained(
|
||
"NickupAI/LFM2.5-1.2B-Saiga-It-v3",
|
||
trust_remote_code=True,
|
||
)
|
||
|
||
messages = [{"role": "user", "content": "Привет! Как дела?"}]
|
||
text = tokenizer.apply_chat_template(
|
||
messages, tokenize=False, add_generation_prompt=True
|
||
)
|
||
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
||
outputs = model.generate(
|
||
**inputs,
|
||
max_new_tokens=200,
|
||
temperature=0.1,
|
||
top_k=50,
|
||
top_p=0.1,
|
||
repetition_penalty=1.05,
|
||
do_sample=True,
|
||
)
|
||
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
||
```
|
||
|
||
## Honest Warning ⚠️
|
||
|
||
Still hallucinates. Training a model of this size properly requires significantly more compute
|
||
and time than was available for this project — the CPT data barely scratches the surface of
|
||
what would be needed for reliable factual knowledge. The model knows the *format* of answers
|
||
but sometimes invents the content.
|
||
|
||
**Recommended for:**
|
||
- Creative nonsense generation, lol
|
||
- Experiments and research
|
||
- Russian language generation tasks
|
||
|
||
**Not recommended for:**
|
||
- Factual questions
|
||
- Medicine, law or any serious topics
|
||
- Astronomy (the dwarf planet situation has not improved)
|
||
|
||
> — Назови все карликовые планеты Солнечной системы
|
||
>
|
||
> 1. Марс, 2. Венус, 3. Земля, 4. Юпитер... 8. Эритрея
|
||
|
||
v2 had Kvass and Gamma-Tit. v3 replaced them with Eritrea. Progress.
|
||
|
||
## GGUF versions
|
||
|
||
Quantized versions available at [NickupAI/LFM2.5-1.2B-Saiga-It-v3-GGUF](https://huggingface.co/NickupAI/LFM2.5-1.2B-Saiga-It-v3-GGUF).
|
||
|
||
## What's next
|
||
|
||
LoRA experiments on larger base models (7B+). The 1.2B size has fundamental limitations for factual knowledge — time to scale up.
|
||
|
||
|
||
## GGUF Versions
|
||
|
||
Quantized versions of the model.
|