78 lines
2.3 KiB
Markdown
78 lines
2.3 KiB
Markdown
---
|
||
license: llama3.1
|
||
language:
|
||
- hu
|
||
- en
|
||
tags:
|
||
- puli
|
||
---
|
||
|
||
# PULI-LlumiX-Llama-3.1 8B base (8.03B billion parameter)
|
||
|
||
|
||
- Trained with LLaMA-Factory [github](https://github.com/hiyouga/LLaMA-Factory)
|
||
- The [Llama 3.1 8B Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) model were continual pretrained on Hungarian dataset
|
||
|
||
## Dataset for continued pretraining
|
||
|
||
- Hungarian (8.7 billion words): documents (763K) that exceed 5000 words in length + Hungarian Wikipedia and news
|
||
- English: Long Context QA (1 billion words), BookSum (42 million words)
|
||
|
||
## Limitations
|
||
|
||
- max_seq_length = 16 384
|
||
- bfloat16
|
||
|
||
## Usage with pipeline
|
||
|
||
```python
|
||
from transformers import pipeline, LlamaForCausalLM, AutoTokenizer
|
||
|
||
model = LlamaForCausalLM.from_pretrained("NYTK/PULI-LlumiX-Llama-3.1")
|
||
tokenizer = AutoTokenizer.from_pretrained("NYTK/PULI-LlumiX-Llama-3.1")
|
||
prompt = "Elmesélek egy történetet a nyelvtechnológiáról."
|
||
generator = pipeline(task="text-generation", model=model, tokenizer=tokenizer, device=0)
|
||
|
||
print(generator(prompt, max_new_tokens=30)[0]["generated_text"])
|
||
```
|
||
|
||
Since the model was continuously pre-trained from Llama 3.1 8B **_Instruct_**, it can be used as a chat model.
|
||
|
||
```python
|
||
import torch
|
||
from transformers import pipeline
|
||
|
||
model_id = "NYTK/PULI-LlumiX-Llama-3.1"
|
||
pipe = pipeline(
|
||
"text-generation",
|
||
model=model_id,
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
messages = [
|
||
{"role": "system", "content": "You are a helpful assistant"},
|
||
{"role": "user", "content": "Mit gondolsz a nyelvtechnológiáról?"},
|
||
]
|
||
outputs = pipe(
|
||
messages,
|
||
max_new_tokens=256,
|
||
)
|
||
print(outputs[0]["generated_text"][-1])
|
||
```
|
||
|
||
|
||
## Citation
|
||
If you use this model, please cite the following paper:
|
||
|
||
```
|
||
@article{chatpuli,
|
||
title={ChatPULI: Enhancement to the first Hungarian conversational model},
|
||
author={Yang, Zijian Győző and Bánfi, Ágnes and Dodé, Réka and Ferenczi, Gergő and Földesi, Flóra and Hatvani, Péter and Héja, Enikő and Lengyel, Mariann and Madarász, Gábor and Osváth, Mátyás and Sárossy, Bence and Varga, Kristóf and Váradi, Tamás and Prószéky, Gábor and Ligeti-Nagy, Noémi},
|
||
journal={Annales Mathematicae et Informaticae},
|
||
doi = {https://doi.org/10.33039/ami.2025.10.010},
|
||
url = { https://ami.uni-eszterhazy.hu},
|
||
year={2025},
|
||
volume={61},
|
||
pages={261–-274}
|
||
}
|
||
``` |