92 lines
3.8 KiB
Markdown
92 lines
3.8 KiB
Markdown
|
---
|
|||
|
|
base_model: mistralai/Mistral-7B-v0.1
|
|||
|
|
tags:
|
|||
|
|
- mistral
|
|||
|
|
- instruct
|
|||
|
|
- bggpt
|
|||
|
|
- insait
|
|||
|
|
language:
|
|||
|
|
- bg
|
|||
|
|
- en
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
license: apache-2.0
|
|||
|
|
---
|
|||
|
|
# INSAIT-Institute/BgGPT-7B-Instruct-v0.2
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
Meet BgGPT-7B, a Bulgarian language model trained from mistralai/Mistral-7B-v0.1. BgGPT is distributed under Apache 2.0 license.
|
|||
|
|
|
|||
|
|
This model was created by [`INSAIT Institute`](https://insait.ai/), part of Sofia University, in Sofia, Bulgaria.
|
|||
|
|
|
|||
|
|
This is an improved version of the model - v0.2.
|
|||
|
|
|
|||
|
|
## Model description
|
|||
|
|
|
|||
|
|
The model is continously pretrained to gain its Bulgarian language and culture capabilities using multiple datasets, including Bulgarian web crawl data, a range of specialized Bulgarian datasets sourced by INSAIT Institute, and machine translations of popular English datasets.
|
|||
|
|
This Bulgarian data was augmented with English datasets to retain English and logical reasoning skills.
|
|||
|
|
|
|||
|
|
The model's tokenizer has been extended to allow for a more efficient encoding of Bulgarian words written in Cyrillic.
|
|||
|
|
This not only increases throughput of Cyrillic text but also performance.
|
|||
|
|
|
|||
|
|
## Instruction format
|
|||
|
|
|
|||
|
|
In order to leverage instruction fine-tuning, your prompt should be surrounded by `[INST]` and `[/INST]` tokens.
|
|||
|
|
The very first instruction should begin with a begin of sequence token `<s>`. Following instructions should not.
|
|||
|
|
The assistant generation will be ended by the end-of-sequence token.
|
|||
|
|
|
|||
|
|
E.g.
|
|||
|
|
```
|
|||
|
|
text = "<s>[INST] Кога е основан Софийският университет? [/INST]"
|
|||
|
|
"Софийският университет „Св. Климент Охридски“ е създаден на 1 октомври 1888 г.</s> "
|
|||
|
|
"[INST] Кой го е основал? [/INST]"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
This format is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating) via the `apply_chat_template()` method:
|
|||
|
|
|
|||
|
|
## Benchmarks
|
|||
|
|
|
|||
|
|
The model comes with a set of Benchmarks that are translations of the corresponding English-benchmarks. These are provided at [`https://github.com/insait-institute/lm-evaluation-harness-bg`](https://github.com/insait-institute/lm-evaluation-harness-bg)
|
|||
|
|
|
|||
|
|
As this is an improved version over version 0.1 of the same model and we include benchmark comparisons.
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Summary
|
|||
|
|
- **Finetuned from:** [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
|
|||
|
|
- **Model type:** Causal decoder-only transformer language model
|
|||
|
|
- **Language:** Bulgarian and English
|
|||
|
|
- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)
|
|||
|
|
- **Contact:** [bggpt@insait.ai](mailto:bggpt@insait.ai)
|
|||
|
|
|
|||
|
|
## Use in 🤗Transformers
|
|||
|
|
First install direct dependencies:
|
|||
|
|
```
|
|||
|
|
pip install transformers torch accelerate
|
|||
|
|
```
|
|||
|
|
If you want faster inference using flash-attention2, you need to install these dependencies:
|
|||
|
|
```bash
|
|||
|
|
pip install packaging ninja
|
|||
|
|
pip install flash-attn
|
|||
|
|
```
|
|||
|
|
Then load the model in transformers:
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
model="INSAIT-Institute/BgGPT-7B-Instruct-v0.2",
|
|||
|
|
device_map="auto",
|
|||
|
|
torch_dtype=torch.bfloat16,
|
|||
|
|
use_flash_attn_2=True # optional
|
|||
|
|
)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Use with GGML / llama.cpp
|
|||
|
|
|
|||
|
|
The model in GGUF format [INSAIT-Institute/BgGPT-7B-Instruct-v0.2-GGUF](https://huggingface.co/INSAIT-Institute/BgGPT-7B-Instruct-v0.2-GGUF)
|