90 lines
3.6 KiB
Markdown
90 lines
3.6 KiB
Markdown
---
|
||
language:
|
||
- id
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
base_model: Qwen/Qwen2-0.5B
|
||
tags:
|
||
- indonesian
|
||
- aksarallm
|
||
- qwen2
|
||
- continued-pretrain
|
||
---
|
||
# Kiel-Pro-0.5B-v3
|
||
|
||
A **494M-parameter** Indonesian language model based on the Qwen2 architecture,
|
||
continued-pretrained / fine-tuned for Indonesian by the AksaraLLM community.
|
||
This is the **smallest fully-working AksaraLLM model**: it loads cleanly via
|
||
`AutoModelForCausalLM`, includes its own tokenizer, and produces coherent
|
||
Indonesian text on standard prompts.
|
||
|
||
## Measured baseline (Devin audit, CPU bf16, 50 short Indonesian sentences)
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| Perplexity | **14.7** |
|
||
| English-stopword ratio in ID-prompted output | 0.8% |
|
||
| Indonesian-stopword ratio in ID-prompted output | 23.2% |
|
||
| Parameters | 494.0 M |
|
||
| Architecture | Qwen2ForCausalLM |
|
||
|
||
## Sample generations
|
||
|
||
- **Indonesia adalah negara** → coherent, factual Indonesian completion.
|
||
- **Resep nasi goreng yang enak adalah** → coherent recipe-style Indonesian.
|
||
|
||
## Quickstart
|
||
|
||
```python
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
import torch
|
||
|
||
tok = AutoTokenizer.from_pretrained("AksaraLLM/Kiel-Pro-0.5B-v3")
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"AksaraLLM/Kiel-Pro-0.5B-v3",
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
inp = tok("Indonesia adalah negara", return_tensors="pt").to(model.device)
|
||
print(tok.decode(model.generate(**inp, max_new_tokens=100, do_sample=True, top_p=0.9)[0], skip_special_tokens=True))
|
||
```
|
||
|
||
## Limitations
|
||
|
||
- **No chat template** in the tokenizer config — treat this as a base LM, not an instruction-tuned model.
|
||
- **Identity is uncalibrated** — the model is descended from Qwen2 and may identify itself as "Qwen" if asked. A short identity SFT pass would fix this.
|
||
- **Hallucinations** typical of a 0.5B-parameter model: factual claims should always be verified.
|
||
- **No production guardrails / RLHF**.
|
||
|
||
## Recommended downstream work
|
||
|
||
1. Identity SFT (50–100 prompts) to teach the model to say "AksaraLLM Kiel-Pro".
|
||
2. Add a `chat_template` (Qwen2 ChatML) to `tokenizer_config.json`.
|
||
3. Run IndoNLU / IndoMMLU benchmarks for a publishable score.
|
||
|
||
## License
|
||
Apache 2.0
|
||
|
||
## Sample outputs (CPU inference, greedy, repeat_penalty=1.15)
|
||
|
||
Real generation captured during the Devin audit on 2026-05-02. Greedy decoding (`do_sample=False`) so these are reproducible.
|
||
|
||
### Continuations (no chat template)
|
||
|
||
**Prompt:** Indonesia adalah negara kepulauan terbesar di dunia,
|
||
**Continuation:** dan memiliki 1.347 pulau kecil yang disebut indah rindu (dari bahasa Jawa: rumput). Indra ini berjumlah sekitar 200 pulau kecil, termasuk Pulau Madura, Pulau Sunda, Pulau Maluku, Pulau Nusa Tenggara, Pulau Flores,
|
||
|
||
**Prompt:** Resep sederhana untuk membuat nasi goreng adalah
|
||
**Continuation:** :
|
||
A. 200 gram nasi, 150 ml air, 3 lembar daun jeruk nipis, dan 4 bawang putih.
|
||
B. 200 gram nasi, 150 ml air, 3 lembar daun jeruk nipis, dan 6 bawang putih.
|
||
C. 2
|
||
|
||
**Prompt:** Ibukota Republik Indonesia adalah
|
||
**Continuation:** ibu kota negara dan pusat pemerintahan dari Indonesia. Ibukotanya terletak di Sumatera Barat, Indonesia. Penduduknya berjumlah 2.354.069 jiwa (2010). Ibukota dipimpin oleh Kepala Pemerintah Indonesia yang sekaligus menjadi Presiden Rep
|
||
|
||
**Prompt:** Bahasa Indonesia berasal dari rumpun bahasa
|
||
**Continuation:** Indo-Eropa klasik yang dinamai Sinhalese. Dari sisi etimologi, kata "sinah" berarti "pohon", sedangkan "lese" berarti "menggambar". Kata ini muncul dalam bahasa Sinhala dan merupakan salah satu nama untuk pohon-pohon di daerah itu. Pohon tersebut adalah pohon le
|
||
|