llama-lang-adapt/pretrain-wura
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: llama-lang-adapt/pretrain-wura
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-05-17 14:22:53 +08:00
commit d7dc2b1708
11 changed files with 446 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
README.md Normal file
View File

@@ -0,0 +1,13 @@
---
datasets:
- llama-lang-adapt/wura
---
We continual pre-train **meta-llama/Llama-2-7b-hf** on monolingual WURA corpus for **20 languages**. All languages are uniformly sampled.
## Important Parameters
- num_gpus: 8
- max_steps: 8000 # see [here](https://github.com/AfricanLlama/ALMA?tab=readme-ov-file#when-should-i-stop-fine-tuning-at-stage-1)
- gradient_accumulation_steps: 16
- per_device_batch_size: 2
- learning_rate: 2e-5