--- language: - ms - en - zh - ta --- # Malaysian Qwen 2.5 7B Instruct Continue finetuning https://huggingface.co/Qwen/Qwen2.5-7B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset. We provide 2 different revisions, 1. Rank 128, Alpha 256, [83a0e145c726385502898ab7e016982eae1b684d](https://huggingface.co/mesolitica/Malaysian-Qwen2.5-7B-Instruct/commit/83a0e145c726385502898ab7e016982eae1b684d) 2. Rank 256, Alpha 512, [5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1](https://huggingface.co/mesolitica/Malaysian-Qwen2.5-7B-Instruct/commit/5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1) ## Improvement 1. Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu. 2. Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu. 3. Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages. ## Training session Finetune on [mesolitica/Malaysian-SFT](https://huggingface.co/datasets/mesolitica/Malaysian-SFT) to make the model understand Malaysian context. ## How we train 1. LoRA on `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"]`. 2. Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids. 3. Chunk CCE loss for LoRA. ### Revision 83a0e145c726385502898ab7e016982eae1b684d 1. Rank 128, Alpha 256. 2. WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-qwen2.5-7b-malaysian-8k Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5 ### Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1 1. Rank 256, Alpha 512. 2. WanDB at https://wandb.ai/huseinzol05/lora-embedding-256-qwen2.5-7b-malaysian-8k Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5 ## Benchmark ### MalayMMLU #### Probability next tokens Based on 0-shot official MalayMMLU First token accuracy, Revision 83a0e145c726385502898ab7e016982eae1b684d, ``` Model Accuracy shot by_letter category 0 Malaysian-Qwen2.5-7B-Instruct 72.042571 0shot True STEM 1 Malaysian-Qwen2.5-7B-Instruct 70.690204 0shot True Language 2 Malaysian-Qwen2.5-7B-Instruct 66.536571 0shot True Social science 3 Malaysian-Qwen2.5-7B-Instruct 67.306308 0shot True Others 4 Malaysian-Qwen2.5-7B-Instruct 71.808874 0shot True Humanities {'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443} Model : Malaysian-Qwen2.5-7B-Instruct Metric : first Shot : 0shot average accuracy 69.26031470697559 accuracy for STEM 72.04257060990585 accuracy for Language 70.69020356234097 accuracy for Social science 66.53657126337092 accuracy for Others 67.30630846725833 accuracy for Humanities 71.80887372013652 ``` Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1, ``` ``` While the original model, ``` Model Accuracy shot by_letter category 0 Qwen2.5-7B-Instruct 70.609906 0shot True STEM 1 Qwen2.5-7B-Instruct 68.034351 0shot True Language 2 Qwen2.5-7B-Instruct 63.486557 0shot True Social science 3 Qwen2.5-7B-Instruct 64.164068 0shot True Others 4 Qwen2.5-7B-Instruct 69.101251 0shot True Humanities {'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443} Model : Qwen2.5-7B-Instruct Metric : first Shot : 0shot average accuracy 66.52211621856027 accuracy for STEM 70.60990585345887 accuracy for Language 68.03435114503816 accuracy for Social science 63.486556808326114 accuracy for Others 64.16406812185176 accuracy for Humanities 69.10125142207053 ``` #### First token match using vLLM Based on 0-shot exact first token match using vLLM Guided Decoding, Revision 83a0e145c726385502898ab7e016982eae1b684d, ``` Model Accuracy shot category 0 Malaysian-Qwen2.5-7B-Instruct 70.159640 0 STEM 1 Malaysian-Qwen2.5-7B-Instruct 66.682570 0 Language 2 Malaysian-Qwen2.5-7B-Instruct 62.893900 0 Social science 3 Malaysian-Qwen2.5-7B-Instruct 64.379947 0 Others 4 Malaysian-Qwen2.5-7B-Instruct 66.780432 0 Humanities Model : Malaysian-Qwen2.5-7B-Instruct Metric : full Shot : 0 average accuracy 65.57221327386115 accuracy for STEM 70.15963978714696 accuracy for Language 66.68256997455471 accuracy for Social science 62.89389997108991 accuracy for Others 64.37994722955145 accuracy for Humanities 66.78043230944255 ``` Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1, ``` ``` While the original model, ``` Model Accuracy shot category 0 Qwen2.5-7B-Instruct 70.978305 0 STEM 1 Qwen2.5-7B-Instruct 68.177481 0 Language 2 Qwen2.5-7B-Instruct 64.238219 0 Social science 3 Qwen2.5-7B-Instruct 64.643799 0 Others 4 Qwen2.5-7B-Instruct 70.443686 0 Humanities Model : Qwen2.5-7B-Instruct Metric : full Shot : 0 average accuracy 67.13748812621319 accuracy for STEM 70.97830536225952 accuracy for Language 68.17748091603053 accuracy for Social science 64.23821913847932 accuracy for Others 64.64379947229551 accuracy for Humanities 70.44368600682593 ``` ## Acknowledgement Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node!