初始化项目，由ModelHub XC社区提供模型

Model: mesolitica/Malaysian-Llama-3.1-8B-Instruct Source: Original Platform
2026-08-11 15:03:18 +08:00
commit a580c21353
418 changed files with 702635 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,125 @@
+---
+language:
+- ms
+- en
+- zh
+- ta
+---
+
+# Malaysian Llama-3.1-8B-Instruct
+
+Continue finetuning https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.
+
+## Improvement
+
+1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
+2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
+3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.
+
+## Training session
+
+Finetune on [mesolitica/Malaysian-SFT](https://huggingface.co/datasets/mesolitica/Malaysian-SFT) to make the model understand Malaysian context.
+  
+## How we train
+
+1. LoRA on `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"]`.
+2. 128 Rank with alpha 256, or alpha of 2.0
+3. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
+4. Chunk CCE loss for LoRA.
+5. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-llama3.1-8b-malaysian-8k?nw=nwuserhuseinzol05
+
+Source code at https://github.com/mesolitica/malaya/tree/master/session/llama3
+
+## Benchmark
+
+### MalayMMLU
+
+#### Probability next tokens
+
+Based on 0-shot official MalayMMLU First token accuracy,
+
+```
+                             Model   Accuracy   shot by_letter        category
+0  Malaysian-Llama-3.1-8B-Instruct  61.522718  0shot      True            STEM
+1  Malaysian-Llama-3.1-8B-Instruct  61.784351  0shot      True        Language
+2  Malaysian-Llama-3.1-8B-Instruct  60.610003  0shot      True  Social science
+3  Malaysian-Llama-3.1-8B-Instruct  60.254258  0shot      True          Others
+4  Malaysian-Llama-3.1-8B-Instruct  62.434585  0shot      True      Humanities
+{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
+Model : Malaysian-Llama-3.1-8B-Instruct
+Metric : first
+Shot : 0shot
+average accuracy 61.276999958699875
+accuracy for STEM 61.522717969709376
+accuracy for Language 61.784351145038165
+accuracy for Social science 60.61000289100896
+accuracy for Others 60.254257615735185
+accuracy for Humanities 62.43458475540387
+```
+
+While the original model,
+
+```
+                   Model   Accuracy   shot by_letter        category
+0  Llama-3.1-8B-Instruct  64.019648  0shot      True            STEM
+1  Llama-3.1-8B-Instruct  65.505725  0shot      True        Language
+2  Llama-3.1-8B-Instruct  62.604799  0shot      True  Social science
+3  Llama-3.1-8B-Instruct  62.197170  0shot      True          Others
+4  Llama-3.1-8B-Instruct  67.167235  0shot      True      Humanities
+{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
+Model : Llama-3.1-8B-Instruct
+Metric : first
+Shot : 0shot
+average accuracy 64.25886920249452
+accuracy for STEM 64.0196479738027
+accuracy for Language 65.5057251908397
+accuracy for Social science 62.60479907487713
+accuracy for Others 62.197169585032384
+accuracy for Humanities 67.16723549488054
+```
+
+#### First token match using vLLM
+
+Based on 0-shot exact first token match using vLLM Guided Decoding,
+
+```
+                             Model   Accuracy  shot        category
+0  Malaysian-Llama-3.1-8B-Instruct  58.616455     0            STEM
+1  Malaysian-Llama-3.1-8B-Instruct  60.178117     0        Language
+2  Malaysian-Llama-3.1-8B-Instruct  57.213067     0  Social science
+3  Malaysian-Llama-3.1-8B-Instruct  56.896138     0          Others
+4  Malaysian-Llama-3.1-8B-Instruct  59.704209     0      Humanities
+Model : Malaysian-Llama-3.1-8B-Instruct
+Metric : full
+Shot : 0
+average accuracy 58.5222814190724
+accuracy for STEM 58.616455178059766
+accuracy for Language 60.17811704834606
+accuracy for Social science 57.213067360508816
+accuracy for Others 56.89613816262893
+accuracy for Humanities 59.70420932878271
+```
+
+While the original model,
+
+```
+                   Model   Accuracy  shot        category
+0  Llama-3.1-8B-Instruct  58.739255     0            STEM
+1  Llama-3.1-8B-Instruct  61.577608     0        Language
+2  Llama-3.1-8B-Instruct  57.487713     0  Social science
+3  Llama-3.1-8B-Instruct  56.872152     0          Others
+4  Llama-3.1-8B-Instruct  63.890785     0      Humanities
+Model : Llama-3.1-8B-Instruct
+Metric : full
+Shot : 0
+average accuracy 59.73237517036303
+accuracy for STEM 58.73925501432665
+accuracy for Language 61.57760814249363
+accuracy for Social science 57.487713211910965
+accuracy for Others 56.872151595106736
+accuracy for Humanities 63.89078498293516
+```
+
+## Acknowledgement
+
+Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node!