167 lines
8.5 KiB
Markdown
167 lines
8.5 KiB
Markdown
|
---
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
- th
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: Qwen/Qwen3-4B
|
|||
|
|
tags:
|
|||
|
|
- translation
|
|||
|
|
- thai
|
|||
|
|
- english
|
|||
|
|
- machine-translation
|
|||
|
|
- qwen3
|
|||
|
|
- distillation
|
|||
|
|
datasets:
|
|||
|
|
- KordAI/Translation-Pairs-8K
|
|||
|
|
pipeline_tag: translation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Kord Translate ENTH V2 — 4B
|
|||
|
|
|
|||
|
|
Bidirectional Thai ⇄ English translation model, fine-tuned from **Qwen3-4B** via **rationale-free distillation** from a large reasoning teacher (DeepSeek-V4-Flash).
|
|||
|
|
|
|||
|
|
This model is part of the Kord Translate ENTH V2 family, accompanying the paper *"Teaching the Student to Skip the Homework: Rationale-Free Distillation for Thai-English Translation"* (KordAI, 2026). Other models in the family: [1.7B](https://huggingface.co/KordAI/Kord-Translate-ENTH-V2-1.7B), [8B](https://huggingface.co/KordAI/Kord-Translate-ENTH-V2-8B), [mBART50](https://huggingface.co/KordAI/Kord-Translate-MBART50-ENTH-V2).
|
|||
|
|
|
|||
|
|
## Model Description
|
|||
|
|
|
|||
|
|
- **Base model:** Qwen3-4B
|
|||
|
|
- **Adaptation:** LoRA (rank 8, alpha 16, dropout 0.02) applied to all attention and MLP projection matrices, trained in 4-bit precision with gradient checkpointing (Unsloth)
|
|||
|
|
- **Training data:** [`KordAI/Translation-Pairs-8K`](https://huggingface.co/datasets/KordAI/Translation-Pairs-8K) — ~8,000 bidirectional Thai/English pairs generated by prompting DeepSeek-V4-Flash through an explicit 4-stage reasoning procedure (literal meaning → genre/formality → vocabulary/honorifics → natural rewrite), keeping **only the final translation** and discarding the reasoning trace
|
|||
|
|
- **Loss masking:** assistant-only, so gradients only flow through the translation output, not the system prompt or source text
|
|||
|
|
- **Epochs:** 3, LoRA learning rate 2e-4 (cosine decay, 5 warmup steps), paged AdamW 8-bit optimizer
|
|||
|
|
- **Compute:** 1× NVIDIA Tesla T4 (16GB), per-device batch 4, gradient accumulation 32
|
|||
|
|
|
|||
|
|
## Results (FLORES devtest, 1,012 samples/direction)
|
|||
|
|
|
|||
|
|
| Direction | Model | BLEU | chrF | chrF++ | BERTScore-P | BERTScore-F1 | COMET |
|
|||
|
|
|---|---|---|---|---|---|---|---|
|
|||
|
|
| en→th | Qwen3-4B (base) | 8.87 | 48.19 | 39.87 | 0.82 | 0.82 | 0.86 |
|
|||
|
|
| en→th | **Kord Translate 4B** | **9.46** | **49.01** | **40.57** | 0.82 | 0.82 | 0.86 |
|
|||
|
|
| th→en | Qwen3-4B (base) | 26.94 | 56.79 | 54.39 | 0.95 | 0.95 | 0.87 |
|
|||
|
|
| th→en | **Kord Translate 4B** | 25.74 | 55.79 | 53.34 | 0.95 | 0.95 | 0.87 |
|
|||
|
|
|
|||
|
|
Rationale-free distillation produces a small but clear BLEU/chrF gain on en→th (+0.59 BLEU), while th→en is essentially flat to slightly down relative to the untuned Qwen3-4B base — consistent with the paper's finding that gains shrink as base model competence increases. See the paper for comparison against other scales and specialized Thai-English systems.
|
|||
|
|
|
|||
|
|
## Inference
|
|||
|
|
|
|||
|
|
This is a chat/instruction-tuned model. Prompt with a system message asking for translation and a user message containing the source text.
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import torch
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
MODEL_ID = "KordAI/Kord-Translate-ENTH-V2-4B"
|
|||
|
|
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
MODEL_ID,
|
|||
|
|
torch_dtype=torch.bfloat16,
|
|||
|
|
device_map="auto",
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
def translate(text: str, direction: str = "en2th") -> str:
|
|||
|
|
"""direction: 'en2th' or 'th2en'"""
|
|||
|
|
src_lang, tgt_lang = ("English", "Thai") if direction == "en2th" else ("Thai", "English")
|
|||
|
|
messages = [
|
|||
|
|
{
|
|||
|
|
"role": "system",
|
|||
|
|
"content": (
|
|||
|
|
f"You are a professional {src_lang}-{tgt_lang} translator. "
|
|||
|
|
f"Translate the user's text from {src_lang} to {tgt_lang}. "
|
|||
|
|
"Output only the translation, with no explanation, notes, or extra text."
|
|||
|
|
),
|
|||
|
|
},
|
|||
|
|
{"role": "user", "content": text},
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
prompt = tokenizer.apply_chat_template(
|
|||
|
|
messages, tokenize=False, add_generation_prompt=True
|
|||
|
|
)
|
|||
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
|||
|
|
|
|||
|
|
with torch.no_grad():
|
|||
|
|
output_ids = model.generate(
|
|||
|
|
**inputs,
|
|||
|
|
max_new_tokens=256,
|
|||
|
|
do_sample=False,
|
|||
|
|
temperature=None,
|
|||
|
|
top_p=None,
|
|||
|
|
top_k=None,
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
generated = output_ids[0][inputs["input_ids"].shape[-1]:]
|
|||
|
|
return tokenizer.decode(generated, skip_special_tokens=True).strip()
|
|||
|
|
|
|||
|
|
|
|||
|
|
print(translate("How is the weather today in Bangkok?", direction="en2th"))
|
|||
|
|
print(translate("วันนี้อากาศที่กรุงเทพเป็นอย่างไรบ้าง", direction="th2en"))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Using Unsloth (faster 4-bit inference, matches training setup):**
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from unsloth import FastLanguageModel
|
|||
|
|
|
|||
|
|
model, tokenizer = FastLanguageModel.from_pretrained(
|
|||
|
|
model_name="KordAI/Kord-Translate-ENTH-V2-4B",
|
|||
|
|
max_seq_length=2048,
|
|||
|
|
load_in_4bit=True,
|
|||
|
|
)
|
|||
|
|
FastLanguageModel.for_inference(model) # enable native 2x faster inference
|
|||
|
|
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": "You are a professional English-Thai translator. Translate the user's text from English to Thai. Output only the translation, with no explanation, notes, or extra text."},
|
|||
|
|
{"role": "user", "content": "How is the weather today in Bangkok?"},
|
|||
|
|
]
|
|||
|
|
inputs = tokenizer.apply_chat_template(
|
|||
|
|
messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
|
|||
|
|
).to("cuda")
|
|||
|
|
|
|||
|
|
output_ids = model.generate(input_ids=inputs, max_new_tokens=256, do_sample=False)
|
|||
|
|
print(tokenizer.decode(output_ids[0][inputs.shape[-1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Sample Translations
|
|||
|
|
|
|||
|
|
### English → Thai
|
|||
|
|
|
|||
|
|
| Source | Translation |
|
|||
|
|
|--------|-------------|
|
|||
|
|
| *Ring also settled a lawsuit with competing security company, the ADT Corporation.* | ริงยังได้ตัดสินพิจารณาคดีกับบริษัทความปลอดภัยที่แข่งขันกันอย่าง ADT Corporation |
|
|||
|
|
| *USA Gymnastics and the USOC have the same goal — making the sport of gymnastics, and others, as safe as possible for athletes to follow their dreams in a safe, positive and empowered environment.* | USA Gymnastics และ USOC มีเป้าหมายเดียวกัน คือ การทำให้กีฬากรีฑศาสตร์และกีฬาอื่นๆ เป็นกีฬาที่ปลอดภัยที่สุดเท่าที่จะทำได้สำหรับนักกีฬาที่จะได้ไปตามฝันของพวกเขาในสภาพแวดล้อมที่ปลอดภัย บวก สร้างสรรค์ และมีอำนาจ |
|
|||
|
|
|
|||
|
|
### Thai → English
|
|||
|
|
|
|||
|
|
| Source | Translation |
|
|||
|
|
|--------|-------------|
|
|||
|
|
| *แกงอาจมีทั้งชนิด "แห้ง" หรือ "น้ำ" ขึ้นอยู่กับปริมาณของเหลว* | There are both dry and wet types of curry, depending on the amount of liquid used. |
|
|||
|
|
| *เนื่องจากมีหมู่เกาะให้เลือกถึง 17,000 เกาะ คำว่าอาหารอินโดนีเซียจึงเป็นคำเรียกกว้าง ๆ ที่ครอบคลุมถึงอาหารประจำภูมิภาคทั่วประเทศ* | Because there are 17,000 islands to choose from, the term \"Indonesian food\" is a broad term that encompasses the cuisine of all regions in the country. |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- Trained on a small (~8K pair), single-teacher distillation set; may not generalize to document-level or highly colloquial Thai.
|
|||
|
|
- Evaluated only on FLORES devtest (sentence-level general-domain text).
|
|||
|
|
- Gains over the untuned Qwen3-4B base are modest and direction-dependent (positive on en→th, roughly flat on th→en).
|
|||
|
|
|
|||
|
|
## Citation
|
|||
|
|
|
|||
|
|
```bibtex
|
|||
|
|
@article{kordai2026rationalefree,
|
|||
|
|
title = {Teaching the Student to Skip the Homework: Rationale-Free Distillation for Thai-English Translation},
|
|||
|
|
author = {Jangjit, Naphon and Komsang, Jeerawat and Boran, Kord C.},
|
|||
|
|
year = {2026},
|
|||
|
|
organization = {KordAI}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Acknowledgements
|
|||
|
|
|
|||
|
|
Built on [Qwen3](https://arxiv.org/abs/2505.09388), with teacher supervision from [DeepSeek-V4](https://arxiv.org/html/2606.19348v1). LoRA fine-tuning follows [Hu et al., 2021](https://arxiv.org/abs/2106.09685) and the 4-bit recipe popularized by [QLoRA](https://arxiv.org/abs/2305.14314).
|
|||
|
|
|
|||
|
|
This qwen3 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
|
|||
|
|
|
|||
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|