28 lines
733 B
Markdown
28 lines
733 B
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- et
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
base_model:
|
||
|
|
- meta-llama/Llama-2-7b-hf
|
||
|
|
license: llama2
|
||
|
|
---
|
||
|
|
# LLammas-base 🐑
|
||
|
|
|
||
|
|
Llama-2-7B with continued pre-training of 5B tokens of CulturaX (75% Estonian, 25% English documents).
|
||
|
|
|
||
|
|
This model is also instruction-tuned resulting in [Llammas](https://huggingface.co/tartuNLP/Llammas).
|
||
|
|
|
||
|
|
More details in our [paper](https://arxiv.org/abs/2404.04042).
|
||
|
|
|
||
|
|
### Citation
|
||
|
|
```
|
||
|
|
@misc{kuulmets2024teaching,
|
||
|
|
title={Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer},
|
||
|
|
author={Hele-Andra Kuulmets and Taido Purason and Agnes Luhtaru and Mark Fishel},
|
||
|
|
year={2024},
|
||
|
|
eprint={2404.04042},
|
||
|
|
archivePrefix={arXiv},
|
||
|
|
primaryClass={cs.CL}
|
||
|
|
}
|
||
|
|
```
|