85 lines
3.0 KiB
Markdown
85 lines
3.0 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- ro
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- llama
|
|
- romanian
|
|
- synthetic-data
|
|
- distillation
|
|
- tinyfabulist
|
|
- fables
|
|
base_model: klusai/tf3-50m-base
|
|
datasets:
|
|
- klusai/ds-tf2-en-ro-15k
|
|
---
|
|
|
|
# TF3 Student: Distilled Romanian Language Model
|
|
|
|
A compact **22.9M-parameter** Romanian language model distilled from the [TF3-50M teacher](https://huggingface.co/klusai/tf3-50m-base) using logit-based knowledge distillation. Part of the [TinyFabulist](https://arxiv.org/abs/2601.10410) research project.
|
|
|
|
## Model Details
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Parameters | 22.9M (26.45M with untied embeddings) |
|
|
| Architecture | LLaMA-style decoder-only Transformer |
|
|
| Hidden size | 384 |
|
|
| Attention heads | 6 (head dim 64) |
|
|
| Layers | 6 |
|
|
| MLP intermediate | 1,024 |
|
|
| Vocab size | 32,000 (Unigram, Romanian-specific) |
|
|
| Context length | 2,048 tokens |
|
|
| Tied embeddings | Yes |
|
|
| Training | Knowledge distillation from klusai/tf3-50m-base |
|
|
|
|
## Training
|
|
|
|
- **Method**: Logit-based knowledge distillation (KL + CE loss, alpha=0.009)
|
|
- **Teacher**: [klusai/tf3-50m-base](https://huggingface.co/klusai/tf3-50m-base) (51.65M params, frozen)
|
|
- **Data**: [klusai/ds-tf2-en-ro-15k](https://huggingface.co/datasets/klusai/ds-tf2-en-ro-15k) (15k Romanian fables)
|
|
- **Temperature**: T=1.0
|
|
- **Epochs**: 3
|
|
- **Learning rate**: 3e-4 (cosine schedule, 50-step warmup)
|
|
- **Hardware**: Apple M3 Ultra (96GB unified memory)
|
|
|
|
## Intended Use
|
|
|
|
This model is a research artifact demonstrating knowledge distillation for compact Romanian language models trained on synthetic moral microfiction. It is designed for:
|
|
|
|
- Research on compact language model compression
|
|
- Romanian text generation in the fable/moral story domain
|
|
- Downstream fine-tuning for Romanian NLP tasks
|
|
|
|
**Not intended for**: Production text generation, factual question answering, or safety-critical applications.
|
|
|
|
## Limitations
|
|
|
|
- Domain-restricted to moral microfiction (fables)
|
|
- Trained exclusively on synthetic data
|
|
- May exhibit repetitive patterns and simplified phrasing compared to the teacher
|
|
- Gender agreement errors may occur in generated text
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@article{nadas2026tf3,
|
|
title={TF3-RO-50M: Training Compact Romanian Language Models from Scratch on Synthetic Moral Microfiction},
|
|
author={Nada\c{s}, Mihai Dan and Dio\c{s}an, Laura and Tomescu, Andreea and Pi\c{s}coran, Andrei},
|
|
journal={arXiv preprint arXiv:2601.10410},
|
|
year={2026}
|
|
}
|
|
```
|
|
|
|
## Related Models and Datasets
|
|
|
|
| Artifact | Description |
|
|
|----------|-------------|
|
|
| [klusai/tf3-50m-base](https://huggingface.co/klusai/tf3-50m-base) | Teacher model (51.65M) |
|
|
| [klusai/tf3-50m-sft](https://huggingface.co/klusai/tf3-50m-sft) | SFT-tuned teacher |
|
|
| [klusai/tf3-bert](https://huggingface.co/klusai/tf3-bert) | NER model for entity coherence evaluation |
|
|
| [klusai/ds-tf2-en-ro-3m](https://huggingface.co/datasets/klusai/ds-tf2-en-ro-3m) | 3M bilingual fable corpus |
|
|
| [klusai/ds-tf2-en-ro-15k](https://huggingface.co/datasets/klusai/ds-tf2-en-ro-15k) | 15k curated subset for distillation/SFT |
|