154 lines
5.5 KiB
Markdown
154 lines
5.5 KiB
Markdown
---
|
|
language:
|
|
- ms
|
|
- en
|
|
- zh
|
|
- ta
|
|
---
|
|
|
|
# Malaysian Qwen 2.5 7B Instruct
|
|
|
|
Continue finetuning https://huggingface.co/Qwen/Qwen2.5-7B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.
|
|
|
|
We provide 2 different revisions,
|
|
|
|
1. Rank 128, Alpha 256, [83a0e145c726385502898ab7e016982eae1b684d](https://huggingface.co/mesolitica/Malaysian-Qwen2.5-7B-Instruct/commit/83a0e145c726385502898ab7e016982eae1b684d)
|
|
2. Rank 256, Alpha 512, [5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1](https://huggingface.co/mesolitica/Malaysian-Qwen2.5-7B-Instruct/commit/5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1)
|
|
|
|
## Improvement
|
|
|
|
1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
|
|
2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
|
|
3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.
|
|
|
|
## Training session
|
|
|
|
Finetune on [mesolitica/Malaysian-SFT](https://huggingface.co/datasets/mesolitica/Malaysian-SFT) to make the model understand Malaysian context.
|
|
|
|
## How we train
|
|
|
|
1. LoRA on `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"]`.
|
|
2. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
|
|
3. Chunk CCE loss for LoRA.
|
|
|
|
### Revision 83a0e145c726385502898ab7e016982eae1b684d
|
|
|
|
1. Rank 128, Alpha 256.
|
|
2. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-qwen2.5-7b-malaysian-8k
|
|
|
|
Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5
|
|
|
|
### Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1
|
|
|
|
1. Rank 256, Alpha 512.
|
|
2. WanDB at https://wandb.ai/huseinzol05/lora-embedding-256-qwen2.5-7b-malaysian-8k
|
|
|
|
Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5
|
|
|
|
## Benchmark
|
|
|
|
### MalayMMLU
|
|
|
|
#### Probability next tokens
|
|
|
|
Based on 0-shot official MalayMMLU First token accuracy,
|
|
|
|
Revision 83a0e145c726385502898ab7e016982eae1b684d,
|
|
|
|
```
|
|
Model Accuracy shot by_letter category
|
|
0 Malaysian-Qwen2.5-7B-Instruct 72.042571 0shot True STEM
|
|
1 Malaysian-Qwen2.5-7B-Instruct 70.690204 0shot True Language
|
|
2 Malaysian-Qwen2.5-7B-Instruct 66.536571 0shot True Social science
|
|
3 Malaysian-Qwen2.5-7B-Instruct 67.306308 0shot True Others
|
|
4 Malaysian-Qwen2.5-7B-Instruct 71.808874 0shot True Humanities
|
|
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
|
|
Model : Malaysian-Qwen2.5-7B-Instruct
|
|
Metric : first
|
|
Shot : 0shot
|
|
average accuracy 69.26031470697559
|
|
accuracy for STEM 72.04257060990585
|
|
accuracy for Language 70.69020356234097
|
|
accuracy for Social science 66.53657126337092
|
|
accuracy for Others 67.30630846725833
|
|
accuracy for Humanities 71.80887372013652
|
|
```
|
|
|
|
Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1,
|
|
|
|
```
|
|
```
|
|
|
|
While the original model,
|
|
|
|
```
|
|
Model Accuracy shot by_letter category
|
|
0 Qwen2.5-7B-Instruct 70.609906 0shot True STEM
|
|
1 Qwen2.5-7B-Instruct 68.034351 0shot True Language
|
|
2 Qwen2.5-7B-Instruct 63.486557 0shot True Social science
|
|
3 Qwen2.5-7B-Instruct 64.164068 0shot True Others
|
|
4 Qwen2.5-7B-Instruct 69.101251 0shot True Humanities
|
|
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
|
|
Model : Qwen2.5-7B-Instruct
|
|
Metric : first
|
|
Shot : 0shot
|
|
average accuracy 66.52211621856027
|
|
accuracy for STEM 70.60990585345887
|
|
accuracy for Language 68.03435114503816
|
|
accuracy for Social science 63.486556808326114
|
|
accuracy for Others 64.16406812185176
|
|
accuracy for Humanities 69.10125142207053
|
|
```
|
|
|
|
#### First token match using vLLM
|
|
|
|
Based on 0-shot exact first token match using vLLM Guided Decoding,
|
|
|
|
Revision 83a0e145c726385502898ab7e016982eae1b684d,
|
|
|
|
```
|
|
Model Accuracy shot category
|
|
0 Malaysian-Qwen2.5-7B-Instruct 70.159640 0 STEM
|
|
1 Malaysian-Qwen2.5-7B-Instruct 66.682570 0 Language
|
|
2 Malaysian-Qwen2.5-7B-Instruct 62.893900 0 Social science
|
|
3 Malaysian-Qwen2.5-7B-Instruct 64.379947 0 Others
|
|
4 Malaysian-Qwen2.5-7B-Instruct 66.780432 0 Humanities
|
|
Model : Malaysian-Qwen2.5-7B-Instruct
|
|
Metric : full
|
|
Shot : 0
|
|
average accuracy 65.57221327386115
|
|
accuracy for STEM 70.15963978714696
|
|
accuracy for Language 66.68256997455471
|
|
accuracy for Social science 62.89389997108991
|
|
accuracy for Others 64.37994722955145
|
|
accuracy for Humanities 66.78043230944255
|
|
```
|
|
|
|
Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1,
|
|
|
|
```
|
|
```
|
|
|
|
While the original model,
|
|
|
|
```
|
|
Model Accuracy shot category
|
|
0 Qwen2.5-7B-Instruct 70.978305 0 STEM
|
|
1 Qwen2.5-7B-Instruct 68.177481 0 Language
|
|
2 Qwen2.5-7B-Instruct 64.238219 0 Social science
|
|
3 Qwen2.5-7B-Instruct 64.643799 0 Others
|
|
4 Qwen2.5-7B-Instruct 70.443686 0 Humanities
|
|
Model : Qwen2.5-7B-Instruct
|
|
Metric : full
|
|
Shot : 0
|
|
average accuracy 67.13748812621319
|
|
accuracy for STEM 70.97830536225952
|
|
accuracy for Language 68.17748091603053
|
|
accuracy for Social science 64.23821913847932
|
|
accuracy for Others 64.64379947229551
|
|
accuracy for Humanities 70.44368600682593
|
|
```
|
|
|
|
## Acknowledgement
|
|
|
|
Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node! |