125 lines
4.7 KiB
Markdown
125 lines
4.7 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- ms
|
||
|
|
- en
|
||
|
|
- zh
|
||
|
|
- ta
|
||
|
|
---
|
||
|
|
|
||
|
|
# Malaysian Llama-3.1-8B-Instruct
|
||
|
|
|
||
|
|
Continue finetuning https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.
|
||
|
|
|
||
|
|
## Improvement
|
||
|
|
|
||
|
|
1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
|
||
|
|
2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
|
||
|
|
3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.
|
||
|
|
|
||
|
|
## Training session
|
||
|
|
|
||
|
|
Finetune on [mesolitica/Malaysian-SFT](https://huggingface.co/datasets/mesolitica/Malaysian-SFT) to make the model understand Malaysian context.
|
||
|
|
|
||
|
|
## How we train
|
||
|
|
|
||
|
|
1. LoRA on `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"]`.
|
||
|
|
2. 128 Rank with alpha 256, or alpha of 2.0
|
||
|
|
3. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
|
||
|
|
4. Chunk CCE loss for LoRA.
|
||
|
|
5. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-llama3.1-8b-malaysian-8k?nw=nwuserhuseinzol05
|
||
|
|
|
||
|
|
Source code at https://github.com/mesolitica/malaya/tree/master/session/llama3
|
||
|
|
|
||
|
|
## Benchmark
|
||
|
|
|
||
|
|
### MalayMMLU
|
||
|
|
|
||
|
|
#### Probability next tokens
|
||
|
|
|
||
|
|
Based on 0-shot official MalayMMLU First token accuracy,
|
||
|
|
|
||
|
|
```
|
||
|
|
Model Accuracy shot by_letter category
|
||
|
|
0 Malaysian-Llama-3.1-8B-Instruct 61.522718 0shot True STEM
|
||
|
|
1 Malaysian-Llama-3.1-8B-Instruct 61.784351 0shot True Language
|
||
|
|
2 Malaysian-Llama-3.1-8B-Instruct 60.610003 0shot True Social science
|
||
|
|
3 Malaysian-Llama-3.1-8B-Instruct 60.254258 0shot True Others
|
||
|
|
4 Malaysian-Llama-3.1-8B-Instruct 62.434585 0shot True Humanities
|
||
|
|
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
|
||
|
|
Model : Malaysian-Llama-3.1-8B-Instruct
|
||
|
|
Metric : first
|
||
|
|
Shot : 0shot
|
||
|
|
average accuracy 61.276999958699875
|
||
|
|
accuracy for STEM 61.522717969709376
|
||
|
|
accuracy for Language 61.784351145038165
|
||
|
|
accuracy for Social science 60.61000289100896
|
||
|
|
accuracy for Others 60.254257615735185
|
||
|
|
accuracy for Humanities 62.43458475540387
|
||
|
|
```
|
||
|
|
|
||
|
|
While the original model,
|
||
|
|
|
||
|
|
```
|
||
|
|
Model Accuracy shot by_letter category
|
||
|
|
0 Llama-3.1-8B-Instruct 64.019648 0shot True STEM
|
||
|
|
1 Llama-3.1-8B-Instruct 65.505725 0shot True Language
|
||
|
|
2 Llama-3.1-8B-Instruct 62.604799 0shot True Social science
|
||
|
|
3 Llama-3.1-8B-Instruct 62.197170 0shot True Others
|
||
|
|
4 Llama-3.1-8B-Instruct 67.167235 0shot True Humanities
|
||
|
|
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
|
||
|
|
Model : Llama-3.1-8B-Instruct
|
||
|
|
Metric : first
|
||
|
|
Shot : 0shot
|
||
|
|
average accuracy 64.25886920249452
|
||
|
|
accuracy for STEM 64.0196479738027
|
||
|
|
accuracy for Language 65.5057251908397
|
||
|
|
accuracy for Social science 62.60479907487713
|
||
|
|
accuracy for Others 62.197169585032384
|
||
|
|
accuracy for Humanities 67.16723549488054
|
||
|
|
```
|
||
|
|
|
||
|
|
#### First token match using vLLM
|
||
|
|
|
||
|
|
Based on 0-shot exact first token match using vLLM Guided Decoding,
|
||
|
|
|
||
|
|
```
|
||
|
|
Model Accuracy shot category
|
||
|
|
0 Malaysian-Llama-3.1-8B-Instruct 58.616455 0 STEM
|
||
|
|
1 Malaysian-Llama-3.1-8B-Instruct 60.178117 0 Language
|
||
|
|
2 Malaysian-Llama-3.1-8B-Instruct 57.213067 0 Social science
|
||
|
|
3 Malaysian-Llama-3.1-8B-Instruct 56.896138 0 Others
|
||
|
|
4 Malaysian-Llama-3.1-8B-Instruct 59.704209 0 Humanities
|
||
|
|
Model : Malaysian-Llama-3.1-8B-Instruct
|
||
|
|
Metric : full
|
||
|
|
Shot : 0
|
||
|
|
average accuracy 58.5222814190724
|
||
|
|
accuracy for STEM 58.616455178059766
|
||
|
|
accuracy for Language 60.17811704834606
|
||
|
|
accuracy for Social science 57.213067360508816
|
||
|
|
accuracy for Others 56.89613816262893
|
||
|
|
accuracy for Humanities 59.70420932878271
|
||
|
|
```
|
||
|
|
|
||
|
|
While the original model,
|
||
|
|
|
||
|
|
```
|
||
|
|
Model Accuracy shot category
|
||
|
|
0 Llama-3.1-8B-Instruct 58.739255 0 STEM
|
||
|
|
1 Llama-3.1-8B-Instruct 61.577608 0 Language
|
||
|
|
2 Llama-3.1-8B-Instruct 57.487713 0 Social science
|
||
|
|
3 Llama-3.1-8B-Instruct 56.872152 0 Others
|
||
|
|
4 Llama-3.1-8B-Instruct 63.890785 0 Humanities
|
||
|
|
Model : Llama-3.1-8B-Instruct
|
||
|
|
Metric : full
|
||
|
|
Shot : 0
|
||
|
|
average accuracy 59.73237517036303
|
||
|
|
accuracy for STEM 58.73925501432665
|
||
|
|
accuracy for Language 61.57760814249363
|
||
|
|
accuracy for Social science 57.487713211910965
|
||
|
|
accuracy for Others 56.872151595106736
|
||
|
|
accuracy for Humanities 63.89078498293516
|
||
|
|
```
|
||
|
|
|
||
|
|
## Acknowledgement
|
||
|
|
|
||
|
|
Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node!
|