214 lines
6.8 KiB
Markdown
214 lines
6.8 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
---
|
||
|
||
# ThaiLLM-8B info
|
||
|
||
This model is a continued pre-training from [Qwen3-8-Base](https://huggingface.co/Qwen/Qwen3-8B-Base), which underwent training on a diverse corpus of approximately 63 billion tokens.
|
||
|
||
**Important Note**: This is a base model that requires instruction fine-tuning to align with specific user requirements and use cases.
|
||
|
||
**For example**, the following models have been instruction fine-tuned based on ThaiLLM-8B:
|
||
|
||
- Typhoon by SCB10X: https://huggingface.co/typhoon-ai/typhoon-s-thaillm-8b-instruct-research-preview
|
||
|
||
- THaLLE by KBTG: https://huggingface.co/KBTG-Labs/THaLLE-0.2-ThaiLLM-8B-fa
|
||
|
||
- OpenThaiGPT by AIEAT: https://huggingface.co/openthaigpt/openthaigpt-thaillm-8b-instruct-v0.7.2-research-preview/
|
||
|
||
- Pathumma by NECTEC: https://huggingface.co/nectec/Pathumma-ThaiLLM-qwen3-8b-it-2.0.0
|
||
|
||
## Data
|
||
|
||
The training corpus consists of the following datasets:
|
||
|
||
| Dataset | Tokens |
|
||
|---------|--------|
|
||
| Fineweb2-ENG | 24,000,000,000 |
|
||
| Fineweb2-TH | 31,525,674,209 |
|
||
| CuratedData | 8,054,246,789 |
|
||
|
||
### CuratedData Breakdown
|
||
|
||
| Category | Token Count |
|
||
|----------|-------------|
|
||
| Business & Finance | 736,071,807 |
|
||
| News | 1,700,662,378 |
|
||
| Education | 576,489,778 |
|
||
| Social | 211,000,000 |
|
||
| Government | 40,492,117 |
|
||
| Medical | 42,987,587 |
|
||
| Conversation | 80,919,390 |
|
||
| Code | 620,218 |
|
||
| Research Articles | 4,185,649,758 |
|
||
| Law | 467,994,847 |
|
||
| Travel | 6,948,290 |
|
||
| Others | 4,410,619 |
|
||
|
||
*Token counts calculated using Qwen3 Tokenizer
|
||
|
||
## Requirements
|
||
|
||
The code of Qwen3 has been integrated into the latest Hugging Face `transformers` library. We strongly recommend using the latest version of `transformers`.
|
||
|
||
With `transformers<4.51.0`, you will encounter the following error:
|
||
```
|
||
KeyError: 'qwen3'
|
||
```
|
||
|
||
## Usage Training
|
||
|
||
**Important**: This is a base model and requires instruction fine-tuning before use to ensure optimal performance for your specific tasks and requirements.
|
||
|
||
### Recommended Training Setup
|
||
|
||
We recommend using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory.git) for instruction fine-tuning. This framework provides an easy-to-use interface for training language models with various optimization techniques.
|
||
|
||
#### Quick Start with LLaMA-Factory
|
||
|
||
```bash
|
||
# Clone the repository
|
||
git clone https://github.com/hiyouga/LLaMA-Factory.git
|
||
cd LLaMA-Factory
|
||
|
||
# Install dependencies
|
||
pip install -e .
|
||
|
||
# Example training command for LoRA
|
||
llamafactory-cli train \
|
||
--model_name_or_path ThaiLLM/ThaiLLM-8B \
|
||
--stage sft \
|
||
--do_train \
|
||
--finetuning_type lora \
|
||
--dataset your_dataset \
|
||
--template qwen3 \
|
||
--cutoff_len 8192 \
|
||
--learning_rate 5e-05 \
|
||
--num_train_epochs 3.0 \
|
||
--per_device_train_batch_size 2 \
|
||
--gradient_accumulation_steps 8 \
|
||
--lr_scheduler_type cosine \
|
||
--max_grad_norm 1.0 \
|
||
--logging_steps 5 \
|
||
--save_steps 100 \
|
||
--warmup_steps 0 \
|
||
--output_dir saves/ThaiLLM-8B-lora \
|
||
--bf16
|
||
```
|
||
|
||
## Usage Inference
|
||
|
||
Below are code snippets to get quickly started with running the model. First, install the necessary libraries.
|
||
|
||
```bash
|
||
pip install -U transformers torch accelerate
|
||
```
|
||
|
||
```python
|
||
from transformers import AutoTokenizer, AutoModelForCausalLM,
|
||
import torch
|
||
|
||
model_id = "ThaiLLM/ThaiLLM-8B"
|
||
|
||
# Load model and tokenizer
|
||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_id,
|
||
device_map="auto",
|
||
torch_dtype=torch.bfloat16
|
||
)
|
||
|
||
# Example prompt
|
||
prompt = "น้ำบริสุทธิ์มีค่า pH เท่าใด"
|
||
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
||
# Generate response
|
||
with torch.inference_mode():
|
||
generate_ids = model.generate(
|
||
inputs.input_ids,
|
||
max_new_tokens=500,
|
||
repetition_penalty=1.2,
|
||
num_beams=1,
|
||
do_sample=True,
|
||
top_k=40,
|
||
top_p=0.75,
|
||
temperature=0.4,
|
||
pad_token_id=tokenizer.eos_token_id,
|
||
)
|
||
|
||
response = tokenizer.batch_decode(
|
||
generate_ids,
|
||
skip_special_tokens=True,
|
||
clean_up_tokenization_spaces=True
|
||
)[0]
|
||
|
||
print(response)
|
||
```
|
||
|
||
## Benchmarks
|
||
|
||
We evaluated **ThaiLLM-8B** against **Qwen3-8B-Base** using multiple-choice question datasets in both Thai and English.
|
||
Each benchmark measures the probability of selecting the correct choice based on the model’s next-token prediction.
|
||
|
||
### 1. Natural Language Understanding (NLU)
|
||
|
||
| Task | Qwen3-8B-Base | ThaiLLM-8B | Δ |
|
||
|------|--------------:|-----------:|---:|
|
||
| **[MMLU](https://huggingface.co/datasets/cais/mmlu) (ENG, 5-shot)** | **0.7691** | 0.7565 | -0.0126 |
|
||
| **[MMLU (TH)](https://huggingface.co/datasets/SeaLLMs/SeaExam/)** | 0.6259 | **0.6459** | +0.0200 |
|
||
| **[ThaiExam](https://huggingface.co/datasets/scb10x/thai_exam) Avg.** (ONET, IC, TGAT, TPAT-1, A-Level) | 0.31396 | **0.48292** | +0.16896 |
|
||
| ├── ONET | 0.4074 | **0.5864** | +0.1790 |
|
||
| ├── IC | 0.5157 | **0.7052** | +0.1895 |
|
||
| ├── TGAT | 0.3384 | **0.6307** | +0.2923 |
|
||
| ├── TPAT-1 | 0.1379 | **0.3965** | +0.2586 |
|
||
| └── A-Level | 0.1653 | **0.5275** | +0.3622 |
|
||
| [M3Exam](https://github.com/DAMO-NLP-SG/M3Exam) | 0.5802 | **0.6369** | +0.0567 |
|
||
| [M6Exam](https://huggingface.co/datasets/openthaigpt/thai-onet-m6-exam) Avg. | 0.54844 | **0.55792** | +0.00948 |
|
||
| ├── Thai | **0.4833** | 0.5023 | +0.0190 |
|
||
| ├── Math | 0.4090 | **0.2727** | -0.1363 |
|
||
| ├── Social | 0.5844 | **0.7088** | +0.1244 |
|
||
| ├── Science | 0.4603 | **0.5238** | +0.0635 |
|
||
| └── English | 0.7552 | **0.7864** | +0.0312 |
|
||
| [XNLI-Thai](https://huggingface.co/datasets/facebook/xnli/viewer/th) | 0.7529 | **0.6667** | -0.0862 |
|
||
| [XCOPA-Thai](https://github.com/cambridgeltl/xcopa/blob/master/data/th/test.th.jsonl) | 0.8220 | **0.8340** | +0.0120 |
|
||
| [Belebele-Thai](https://huggingface.co/datasets/facebook/belebele/viewer/tha_Thai) | 0.3880 | **0.8447** | +0.4567 |
|
||
|
||
---
|
||
|
||
### 2. Average Performance
|
||
|
||
| Model | Average Score |
|
||
|-------|--------------:|
|
||
| Qwen3-8B-Base | 0.5987 |
|
||
| ThaiLLM-8B | **0.6891** |
|
||
|
||
> **Highlights**:
|
||
> - **ThaiLLM-8B** shows large improvements in **ThaiExam**, **Belebele-Thai**, and **MMLU-TH**.
|
||
> - Gains are especially strong in **A-Level** (+0.36) and **TGAT** (+0.29).
|
||
> - Some slight regressions are seen in **MMLU-ENG** and **Math** in M6Exam.
|
||
|
||
|
||
## Limitations
|
||
|
||
- This is a base model and requires instruction fine-tuning for optimal performance
|
||
- Performance on specialized domains may require domain-specific fine-tuning
|
||
- As with all language models, outputs should be verified for accuracy in critical applications
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{qwen3technicalreport,
|
||
title={Qwen3 Technical Report},
|
||
author={Qwen Team},
|
||
year={2025},
|
||
eprint={2505.09388},
|
||
archivePrefix={arXiv},
|
||
primaryClass={cs.CL},
|
||
url={https://arxiv.org/abs/2505.09388},
|
||
}
|
||
```
|
||
|
||
## Dataset Contributor
|
||
|
||
 |