初始化项目，由ModelHub XC社区提供模型
Model: wongwian-org/wongwian-micro-instruct Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
ProfilePic.png filter=lfs diff=lfs merge=lfs -text
|
||||
3
ProfilePic.png
Normal file
3
ProfilePic.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:e03f833e05faf1592d923061d9e5d28eda28f50e778c3807831c79865bfe7365
|
||||
size 1537161
|
||||
242
README.md
Normal file
242
README.md
Normal file
@@ -0,0 +1,242 @@
|
||||
---
|
||||
license: mit
|
||||
language:
|
||||
- th
|
||||
- en
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
tags:
|
||||
- thai
|
||||
- instruct
|
||||
- causal-lm
|
||||
- llama-architecture
|
||||
- sentencepiece
|
||||
- wongwian
|
||||
base_model: wongwian-org/wongwian-micro-instruct
|
||||
model_type: llama
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<img src="./ProfilePic.png" width="20%"/>
|
||||
</p>
|
||||
|
||||
|
||||
# Wongwian Micro Instruct — 272M
|
||||
|
||||
**The first model in the Wongwian series.** A compact 272M-parameter Thai-centric language model trained entirely from scratch on 40B tokens of Thai-dominant data and then instruction-fine-tuned. No base weights from existing open models were used at any stage.
|
||||
|
||||
> _"Cultural & Localization AI for the world"_ — Wongwian
|
||||
|
||||
---
|
||||
# Wongwian Micro Instruct — 272M (ภาษาไทย)
|
||||
|
||||
**โมเดลตัวแรกในซีรีส์ Wongwian** เป็นโมเดลภาษาขนาดเล็ก 272 ล้านพารามิเตอร์ที่เน้นภาษาไทยเป็นหลัก **ฝึกขึ้นมาจากศูนย์ (train from scratch) บนข้อมูล 40 พันล้าน token** แล้วผ่านการ instruction fine-tuning โดยไม่ได้นำ open weights ของโมเดลอื่นมาต่อยอดแต่อย่างใด
|
||||
|
||||
> _"Cultural & Localization AI for the world"_ — Wongwian
|
||||
|
||||
---
|
||||
|
||||
## ภาพรวมโมเดล
|
||||
|
||||
| คุณสมบัติ | ค่า |
|
||||
|---|---|
|
||||
| **ตระกูล** | Wongwian |
|
||||
| **ซีรีส์** | Micro |
|
||||
| **เวอร์ชัน** | Instruct v1 (step 450) |
|
||||
| **สถาปัตยกรรม** | LlamaForCausalLM |
|
||||
| **จำนวนพารามิเตอร์** | ~272 ล้าน |
|
||||
| **Context length** | 2,048 tokens |
|
||||
| **Precision** | bfloat16 |
|
||||
| **Token ที่ใช้ pre-train** | ~40 พันล้าน |
|
||||
| **ภาษาหลัก** | ไทย 🇹🇭 |
|
||||
| **ภาษารอง** | อังกฤษ 🇬🇧 |
|
||||
| **License** | MIT |
|
||||
|
||||
---
|
||||
|
||||
## เกี่ยวกับ Wongwian
|
||||
|
||||
Wongwian คือโครงการวิจัย AI ภาษาไทยที่มุ่งสร้าง **โมเดลภาษา AI ที่มีประสิทธิภาพสูงและเข้าใจบริบทเชิงวัฒนธรรม** สำหรับภาษาไทยและภาษาอื่น ๆ ที่ขาดแคลนทรัพยากร โครงการนี้แสดงให้เห็นว่าโมเดล AI ที่มีคุณภาพสูงไม่จำเป็นต้องมีพารามิเตอร์มหาศาล โมเดลขนาดเล็กที่ออกแบบอย่างพิถีพิถันและใช้ข้อมูลที่เหมาะสม สามารถตอบโจทย์การใช้งานจริงได้อย่างมีประสิทธิผล
|
||||
|
||||
**วิสัยทัศน์:** _Cultural & Localization AI for the world_ — สร้าง AI ที่เข้าใจความละเอียดอ่อนทางภาษา วัฒนธรรม และบริบทของแต่ละชุมชนอย่างลึกซึ้ง
|
||||
|
||||
**จุดแข็งของโมเดลนี้:**
|
||||
|
||||
- **ฝึกจากศูนย์ (from scratch)** บนข้อมูลภาษาไทยเป็นหลัก ไม่ได้ต่อยอดจากโมเดลสาธารณะใดทั้งสิ้น
|
||||
- **SentencePiece Unigram tokenizer** (32K vocab) ออกแบบมาเฉพาะสำหรับโครงสร้างภาษาไทยและข้อความผสมไทย-อังกฤษ
|
||||
- สร้างบนเฟรมเวิร์ค [OLMo-core](https://github.com/allenai/OLMo-core) แบบ open-source
|
||||
- ผ่านการ instruction fine-tuning (SFT) ด้วยข้อมูลบทสนทนาภาษาไทย
|
||||
- รูปแบบ prompt สะท้อนการโต้ตอบผู้ช่วยภาษาไทยอย่างเป็นธรรมชาติ
|
||||
|
||||
---
|
||||
|
||||
## Model Overview
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| **Model family** | Wongwian |
|
||||
| **Series** | Micro |
|
||||
| **Version** | Instruct v1 (step 450) |
|
||||
| **Architecture** | LlamaForCausalLM |
|
||||
| **Parameters** | ~272 M |
|
||||
| **Context length** | 2 048 tokens |
|
||||
| **Precision** | bfloat16 |
|
||||
| **Pre-train tokens** | ~40 B |
|
||||
| **Primary language** | Thai 🇹🇭 |
|
||||
| **Secondary language** | English 🇬🇧 |
|
||||
| **License** | MIT |
|
||||
|
||||
---
|
||||
|
||||
## About Wongwian
|
||||
|
||||
Wongwian is a Thai AI research initiative focused on building **efficient, culturally-grounded language models** for Thai and other under-resourced languages. The project demonstrates that high-quality language AI does not require massive parameter counts — a carefully designed small model, trained on the right data, can serve real-world use cases effectively.
|
||||
|
||||
**Vision:** _Cultural & Localization AI for the world_ — delivering language models that deeply understand the cultural, linguistic, and contextual nuances of each community they serve.
|
||||
|
||||
**What makes this model different:**
|
||||
|
||||
- Trained **from scratch** on Thai-dominant data — not a fine-tune of any existing public model
|
||||
- Custom **SentencePiece Unigram** tokenizer (32K vocab) designed for Thai morphology and mixed Thai-English text
|
||||
- Built on the open [OLMo-core](https://github.com/allenai/OLMo-core) training framework
|
||||
- Full instruction fine-tuning (SFT) with Thai conversation data
|
||||
- Chat prompt format mirrors natural Thai assistant interaction
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### pip install
|
||||
|
||||
```bash
|
||||
pip install transformers sentencepiece
|
||||
```
|
||||
|
||||
### Chat generation (recommended)
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
import torch
|
||||
|
||||
model_id = "wongwian-org/wongwian-micro-instruct"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
model.eval()
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": "สวัสดี อธิบาย AI ให้ฟังหน่อย"},
|
||||
]
|
||||
|
||||
input_ids = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_tensors="pt",
|
||||
)
|
||||
with torch.no_grad():
|
||||
output = model.generate(
|
||||
input_ids.to(model.device),
|
||||
max_new_tokens=200,
|
||||
do_sample=True,
|
||||
temperature=0.7,
|
||||
top_p=0.9,
|
||||
repetition_penalty=1.3,
|
||||
no_repeat_ngram_size=4,
|
||||
eos_token_id=tokenizer.eos_token_id,
|
||||
pad_token_id=tokenizer.pad_token_id,
|
||||
)
|
||||
new_tokens = output[0, input_ids.shape[1]:]
|
||||
print(tokenizer.decode(new_tokens, skip_special_tokens=True))
|
||||
```
|
||||
|
||||
### Multi-turn conversation
|
||||
|
||||
```python
|
||||
history = []
|
||||
|
||||
def chat(user_message: str, system_prompt: str | None = None) -> str:
|
||||
messages = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.extend(history)
|
||||
messages.append({"role": "user", "content": user_message})
|
||||
|
||||
input_ids = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
tokenize=True,
|
||||
add_generation_prompt=True,
|
||||
return_tensors="pt",
|
||||
)
|
||||
with torch.no_grad():
|
||||
output = model.generate(
|
||||
input_ids.to(model.device),
|
||||
max_new_tokens=200,
|
||||
do_sample=True,
|
||||
temperature=0.7,
|
||||
top_p=0.9,
|
||||
repetition_penalty=1.3,
|
||||
no_repeat_ngram_size=4,
|
||||
eos_token_id=tokenizer.eos_token_id,
|
||||
pad_token_id=tokenizer.pad_token_id,
|
||||
)
|
||||
reply = tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True).strip()
|
||||
history.append({"role": "user", "content": user_message})
|
||||
history.append({"role": "assistant", "content": reply})
|
||||
return reply
|
||||
|
||||
print(chat("สวัสดี คุณคือใคร?"))
|
||||
print(chat("แล้วคุณทำอะไรได้บ้าง?"))
|
||||
```
|
||||
|
||||
### Chat template format
|
||||
|
||||
The model uses the following plain-text prompt format (applied automatically by `apply_chat_template`):
|
||||
|
||||
```
|
||||
<s>system: <system_prompt>
|
||||
ผู้ใช้: <user_turn>
|
||||
ผู้ช่วย: <assistant_turn></s>
|
||||
ผู้ใช้: <user_turn>
|
||||
ผู้ช่วย:
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prompt Engineering Tips
|
||||
|
||||
| Tip | Detail |
|
||||
|---|---|
|
||||
| **Temperature** | 0.6 – 0.8 for natural Thai conversation |
|
||||
| **Repetition penalty** | 1.2 – 1.4 — recommended; prevents looping |
|
||||
| **no_repeat_ngram_size** | 4 — prevents phrase repetition |
|
||||
| **max_new_tokens** | 150 – 300 for chat; 512 for long-form |
|
||||
| **System prompt** | Optional but improves role adherence |
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Size:** At 272M parameters, this model is suited for assistive tasks and demonstrations, not advanced reasoning or complex long-form analysis.
|
||||
- **Hallucination:** Like all language models, the model may produce inaccurate information. Always verify critical outputs.
|
||||
- **Context length:** Maximum 2 048 tokens per call; performance may degrade near the limit.
|
||||
- **Bias:** The model may reflect biases present in the training corpus.
|
||||
|
||||
---
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{wongwian2026micro,
|
||||
title = {Wongwian Micro Instruct: A Thai-Centric Small Language Model Trained from Scratch},
|
||||
author = {Wongwian AI Research},
|
||||
year = {2026},
|
||||
url = {https://huggingface.co/wongwian-org/wongwian-micro-instruct}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
22
config.json
Normal file
22
config.json
Normal file
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"model_type": "llama",
|
||||
"hidden_size": 1024,
|
||||
"intermediate_size": 2816,
|
||||
"num_attention_heads": 8,
|
||||
"num_hidden_layers": 16,
|
||||
"num_key_value_heads": 8,
|
||||
"vocab_size": 32000,
|
||||
"hidden_act": "silu",
|
||||
"max_position_embeddings": 2048,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_theta": 10000,
|
||||
"tie_word_embeddings": false,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"pad_token_id": 3,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.40.0"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:0d95599589e877ae477c07b8a0ba1e2fd97fa6eb3411331fde0d02e19ad15c28
|
||||
size 542198056
|
||||
6
special_tokens_map.json
Normal file
6
special_tokens_map.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"bos_token": "<s>",
|
||||
"eos_token": "</s>",
|
||||
"unk_token": "<unk>",
|
||||
"pad_token": "<pad>"
|
||||
}
|
||||
128126
tokenizer.json
Normal file
128126
tokenizer.json
Normal file
File diff suppressed because one or more lines are too long
3
tokenizer.model
Normal file
3
tokenizer.model
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:5b376fbc6e5c8e2dca8e85fd1844a1da5e4a9a94e0ed8708a2f0e01768f883ca
|
||||
size 987961
|
||||
13
tokenizer_config.json
Normal file
13
tokenizer_config.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"model_max_length": 2048,
|
||||
"pad_token": "<pad>",
|
||||
"tokenizer_class": "TokenizersBackend",
|
||||
"unk_token": "<unk>",
|
||||
"add_bos_token": true,
|
||||
"add_eos_token": false,
|
||||
"chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}system: {{ message['content'] }}\n{% elif message['role'] == 'user' %}User: {{ message['content'] }}\n{% elif message['role'] == 'assistant' %}Assistant: {{ message['content'] }}\n{{ eos_token }}{% endif %}{% endfor %}{% if add_generation_prompt %}Assistant:{% endif %}"
|
||||
}
|
||||
Reference in New Issue
Block a user