105 lines
5.0 KiB
Markdown
105 lines
5.0 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- sw
|
|
- bn
|
|
- te
|
|
- th
|
|
- ja
|
|
- zh
|
|
- ru
|
|
- es
|
|
- fr
|
|
- de
|
|
- en
|
|
tags:
|
|
- text-generation
|
|
datasets:
|
|
- laihuiyuan/mCoT-MATH
|
|
base_model:
|
|
- mistralai/Mistral-7B-v0.1
|
|
---
|
|
# mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models
|
|
|
|
Paper: https://arxiv.org/abs/2406.02301
|
|
|
|
Code: https://github.com/laihuiyuan/mCoT
|
|
|
|
Dataset: https://huggingface.co/datasets/laihuiyuan/mCoT-MATH
|
|
|
|
### Introduction
|
|
We introduce mCoT, a 7B parameter model for multilingual math reasoning that achieves impressive multilingual reasoning consistency across multiple languages.
|
|
Based on [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), mCoT is trained on [mCoT-MATH](https://huggingface.co/datasets/laihuiyuan/mCoT-MATH), the first large-scale multilingual math CoT reasoning dataset containing around 6.3 million samples for 11 diverse languages.
|
|
|
|
|
|
### Results on [MGSM](https://arxiv.org/abs/2210.03057v1)
|
|
| Language | SW | BN | TE | TH | JA | ZH | RU | ES | FR | DE | EN |
|
|
|:-----------------------|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|
|
|
| GPT-3 few-shot | 11.2 | 6.4 | 0.4 | 0.8 | 26.0 | 40.0 | 28.4 | 40.4 | 37.6 | 36.0 | 53.6 |
|
|
| GPT-3.5-En 2-shot | 40.0 | 7.6 | - | 15.6 | 46.8 | 52.8 | 50.4 | 61.2 | 59.2 | 62.0 | 67.2 |
|
|
| GPT4-En 2-shot | 64.4 | 17.6 | - | 40.4 | 71.6 | 70.0 | 64.0 | 71.2 | 72.0 | 73.6 | 80.0 |
|
|
| PaLM-540B few-shot | 35.2 | 46.0 | 45.6 | 52.8 | 40.0 | 46.8 | 48.4 | 56.8 | 46.4 | 49.2 | 62.4 |
|
|
| WizardMath-7B | 3.4 | 2.0 | - | 4.0 | 24.0 | 22.4 | 30.8 | 34.8 | 30.4 | 30.4 | 47.6 |
|
|
| MathOctopus-7B | 38.4 | 33.2 | - | 36.4 | 35.6 | 45.2 | 48.4 | 45.2 | 38.0 | 43.6 | 54.8 |
|
|
| MathOctopus-Mistral-7B | 51.6 | 44.0 | - | 48.8 | 48.0 | 51.6 | 49.6 | 53.2 | 47.2 | 50.0 | 58.4 |
|
|
| xCoT-7B | 48.4 | 40.4 | 42.8 | 49.2 | 50.0 | 50.0 | 50.0 | 48.8 | 49.6 | 47.2 | 48.4 |
|
|
| WizardMath-13B | 5.6 | 6.4 | - | 5.6 | 22.0 | 28.0 | 34.4 | 45.6 | 42.0 | 40.4 | 52.8 |
|
|
| MathOctopus-13B | 46.0 | 42.0 | - | 46.0 | 39.6 | 51.2 | 47.6 | 53.2 | 49.6 | 49.2 | 51.6 |
|
|
| xCoT-13B | 51.6 | 50.0 | 47.2 | 50.0 | 49.6 | 54.0 | 56.8 | 54.8 | 46.4 | 52.4 | 54.4 |
|
|
| mCoT-7B | 67.2 | 65.6 | 62.4 | 67.6 | 65.2 | 64.8 | 66.8 | 68.4 | 63.8 | 61.2 | 71.6 |
|
|
|
|
|
|
### Results on [MSVAMP](https://arxiv.org/abs/2310.20246)
|
|
| Language | SW | BN | TH | JA | ZH | RU | ES | FR | DE | EN | AVG |
|
|
|:-----------------------|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|:-----|
|
|
| GPT-3.5-En zero-shot | 63.2 | 3.1 | 24.4 | 63.3 | 72.4 | 62.3 | 69.5 | 71.9 | 66.7 | 76.1 | 57.3 |
|
|
| GPT-3.5-En 2-shot | 68.4 | 14.4 | 46.0 | 74.0 | 78.4 | 70.9 | 74.6 | 78.2 | 73.9 | 81.2 | 66.0 |
|
|
| GPT4-En 2-shot | 75.7 | 31.2 | 68.1 | 74.8 | 78.9 | 77.9 | 81.5 | 83.9 | 78.1 | 80.1 | 73.0 |
|
|
| PaLM-540B few-shot | 35.2 | 46.0 | 45.6 | 52.8 | 40.0 | 46.8 | 48.4 | 56.8 | 46.4 | 49.2 | 62.4 |
|
|
| WizardMath-7B | 10.3 | 16.1 | 6.3 | 26.7 | 26.8 | 33.7 | 42.9 | 39.9 | 39.6 | 45.1 | 27.0 |
|
|
| MathOctopus-7B | 42.3 | 32.8 | 40.5 | 43.2 | 43.2 | 42.1 | 44.5 | 45.3 | 43.1 | 46.8 | 42.4 |
|
|
| MathOctopus-Mistral-7B | 41.2 | 36.7 | 40.2 | 41.5 | 43.1 | 44.0 | 47.0 | 49.0 | 46.4 | 49.7 | 43.9 |
|
|
| WizardMath-13B | 12.5 | 13.7 | 16.3 | 29.5 | 37.0 | 43.8 | 50.4 | 49.4 | 48.7 | 56.3 | 35.8 |
|
|
| MathOctopus-13B | 43.4 | 34.2 | 39.5 | 43.1 | 46.4 | 48.2 | 48.2 | 49.9 | 47.7 | 44.6 | 44.5 |
|
|
| mCoT-7B | 55.0 | 53.7 | 56.4 | 58.8 | 58.2 | 58.1 | 58.9 | 58.8 | 61.1 | 58.3 | 57.7 |
|
|
|
|
### Prompt Template
|
|
```bash
|
|
# Template
|
|
template = "Question: \n{question} \nAnswer: \n{language}\n"
|
|
|
|
# Language prompt
|
|
bn = "আসুন ধাপে ধাপে চিন্তা করি।"
|
|
de = "Denken wir Schritt für Schritt."
|
|
en = "Let's think step by step."
|
|
es = "Pensemos paso a paso."
|
|
fr = "Réfléchissons étape par étape."
|
|
ja = "段階的に考えてみましょう。"
|
|
ru = "Давайте думать поэтапно."
|
|
sw = "Hebu fikiria hatua kwa hatua."
|
|
te = "అంచెలంచెలుగా ఆలోచిద్దాం."
|
|
th = "ลองคิดทีละขั้นตอน"
|
|
zh = "让我们一步步思考。"
|
|
|
|
# Math question
|
|
math_en = "A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?"
|
|
|
|
# An example for the English question
|
|
prompt = template.format(question=math_en, language=en)
|
|
```
|
|
|
|
### Citation
|
|
If you use any content from this repository, please cite our paper:
|
|
```
|
|
@inproceedings{lai-etal-2024-mcot,
|
|
title = "mCoT: Multilingual Instruction Tuning for Reasoning Consistency
|
|
in Language Models",
|
|
author = "Lai, Huiyuan and Nissim, Malvina",
|
|
booktitle = "Proceedings of the 62nd Annual Meeting of the Association
|
|
for Computational Linguistics,
|
|
month = aug,
|
|
address = "Bangkok, Thailand",
|
|
year = "2024",
|
|
publisher = "Association for Computational Linguistics"
|
|
}
|
|
``` |