90 lines
3.6 KiB
Markdown
90 lines
3.6 KiB
Markdown
|
---
|
|||
|
|
language:
|
|||
|
|
- id
|
|||
|
|
license: apache-2.0
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
base_model: Qwen/Qwen2-0.5B
|
|||
|
|
tags:
|
|||
|
|
- indonesian
|
|||
|
|
- aksarallm
|
|||
|
|
- qwen2
|
|||
|
|
- continued-pretrain
|
|||
|
|
---
|
|||
|
|
# Kiel-Pro-0.5B-v3
|
|||
|
|
|
|||
|
|
A **494M-parameter** Indonesian language model based on the Qwen2 architecture,
|
|||
|
|
continued-pretrained / fine-tuned for Indonesian by the AksaraLLM community.
|
|||
|
|
This is the **smallest fully-working AksaraLLM model**: it loads cleanly via
|
|||
|
|
`AutoModelForCausalLM`, includes its own tokenizer, and produces coherent
|
|||
|
|
Indonesian text on standard prompts.
|
|||
|
|
|
|||
|
|
## Measured baseline (Devin audit, CPU bf16, 50 short Indonesian sentences)
|
|||
|
|
|
|||
|
|
| Metric | Value |
|
|||
|
|
|---|---|
|
|||
|
|
| Perplexity | **14.7** |
|
|||
|
|
| English-stopword ratio in ID-prompted output | 0.8% |
|
|||
|
|
| Indonesian-stopword ratio in ID-prompted output | 23.2% |
|
|||
|
|
| Parameters | 494.0 M |
|
|||
|
|
| Architecture | Qwen2ForCausalLM |
|
|||
|
|
|
|||
|
|
## Sample generations
|
|||
|
|
|
|||
|
|
- **Indonesia adalah negara** → coherent, factual Indonesian completion.
|
|||
|
|
- **Resep nasi goreng yang enak adalah** → coherent recipe-style Indonesian.
|
|||
|
|
|
|||
|
|
## Quickstart
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
tok = AutoTokenizer.from_pretrained("AksaraLLM/Kiel-Pro-0.5B-v3")
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
"AksaraLLM/Kiel-Pro-0.5B-v3",
|
|||
|
|
torch_dtype=torch.bfloat16,
|
|||
|
|
device_map="auto",
|
|||
|
|
)
|
|||
|
|
inp = tok("Indonesia adalah negara", return_tensors="pt").to(model.device)
|
|||
|
|
print(tok.decode(model.generate(**inp, max_new_tokens=100, do_sample=True, top_p=0.9)[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- **No chat template** in the tokenizer config — treat this as a base LM, not an instruction-tuned model.
|
|||
|
|
- **Identity is uncalibrated** — the model is descended from Qwen2 and may identify itself as "Qwen" if asked. A short identity SFT pass would fix this.
|
|||
|
|
- **Hallucinations** typical of a 0.5B-parameter model: factual claims should always be verified.
|
|||
|
|
- **No production guardrails / RLHF**.
|
|||
|
|
|
|||
|
|
## Recommended downstream work
|
|||
|
|
|
|||
|
|
1. Identity SFT (50–100 prompts) to teach the model to say "AksaraLLM Kiel-Pro".
|
|||
|
|
2. Add a `chat_template` (Qwen2 ChatML) to `tokenizer_config.json`.
|
|||
|
|
3. Run IndoNLU / IndoMMLU benchmarks for a publishable score.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
Apache 2.0
|
|||
|
|
|
|||
|
|
## Sample outputs (CPU inference, greedy, repeat_penalty=1.15)
|
|||
|
|
|
|||
|
|
Real generation captured during the Devin audit on 2026-05-02. Greedy decoding (`do_sample=False`) so these are reproducible.
|
|||
|
|
|
|||
|
|
### Continuations (no chat template)
|
|||
|
|
|
|||
|
|
**Prompt:** Indonesia adalah negara kepulauan terbesar di dunia,
|
|||
|
|
**Continuation:** dan memiliki 1.347 pulau kecil yang disebut indah rindu (dari bahasa Jawa: rumput). Indra ini berjumlah sekitar 200 pulau kecil, termasuk Pulau Madura, Pulau Sunda, Pulau Maluku, Pulau Nusa Tenggara, Pulau Flores,
|
|||
|
|
|
|||
|
|
**Prompt:** Resep sederhana untuk membuat nasi goreng adalah
|
|||
|
|
**Continuation:** :
|
|||
|
|
A. 200 gram nasi, 150 ml air, 3 lembar daun jeruk nipis, dan 4 bawang putih.
|
|||
|
|
B. 200 gram nasi, 150 ml air, 3 lembar daun jeruk nipis, dan 6 bawang putih.
|
|||
|
|
C. 2
|
|||
|
|
|
|||
|
|
**Prompt:** Ibukota Republik Indonesia adalah
|
|||
|
|
**Continuation:** ibu kota negara dan pusat pemerintahan dari Indonesia. Ibukotanya terletak di Sumatera Barat, Indonesia. Penduduknya berjumlah 2.354.069 jiwa (2010). Ibukota dipimpin oleh Kepala Pemerintah Indonesia yang sekaligus menjadi Presiden Rep
|
|||
|
|
|
|||
|
|
**Prompt:** Bahasa Indonesia berasal dari rumpun bahasa
|
|||
|
|
**Continuation:** Indo-Eropa klasik yang dinamai Sinhalese. Dari sisi etimologi, kata "sinah" berarti "pohon", sedangkan "lese" berarti "menggambar". Kata ini muncul dalam bahasa Sinhala dan merupakan salah satu nama untuk pohon-pohon di daerah itu. Pohon tersebut adalah pohon le
|
|||
|
|
|