100 lines
3.2 KiB
Markdown
100 lines
3.2 KiB
Markdown
---
|
|
language:
|
|
- sv
|
|
license: apache-2.0
|
|
library_name: transformers
|
|
tags:
|
|
- unsloth
|
|
datasets:
|
|
- neph1/bellman-7b-finetune
|
|
- neph1/codefeedback-swedish
|
|
base_model:
|
|
- mistralai/Mistral-Nemo-Instruct-2407
|
|
---
|
|
|
|
# Model Card for Bellman
|
|
|
|
This version of bellman is finetuned from Mistral-Nemo-Instruct-2407.
|
|
It's a rank 128 qlora trained for about 1 epoch.
|
|
It's finetuned for prompt question answering, based on a dataset created from
|
|
Swedish wikipedia, with a lot of Sweden-centric questions.
|
|
New from previous versions is questions from a translated code-feedback dataset, as well as a number of stories.
|
|
|
|
Consider this a work in progress as I adjust the training for this new model size. Will provide a few updates to the model.
|
|
|
|
Quality seems to be dependent on "initial seed". Sometimes it's just amazing, and sometimes quite bad. If you're having trouble getting good results, try lowering temperature.
|
|
|
|
|
|
For GGUFs, please look to: <br> https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-bellman-12b-GGUF and <br>
|
|
https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-bellman-12b-i1-GGUF
|
|
|
|
|
|
|
|
[](https://ko-fi.com/T6T3S8VXY)
|
|
|
|
## Model Details
|
|
|
|
Rank: 128
|
|
|
|
Trained with Unsloth on a 3090
|
|
|
|
Differences from base model:
|
|
|
|
The base model is pretty good at Swedish already, but my 'vibe check' says this finetune is slightly improved. There are less weird wordings.
|
|
|
|
Bellman is trained on fairly short answers and tends to be less verbose.
|
|
|
|
Output example (Mistral-Nemo-Instruct-bellman-12b.i1-Q4_K_M.gguf):
|
|
|
|
User: "Hej!"
|
|
|
|
Bellman: "Hej! Vad kan jag hjälpa dig med?"
|
|
|
|
User: "Berätta lite om Sverige"
|
|
|
|
Bellman: "Sverige är ett land i norra Europa med en rik historia. Det består av 21 län och har en yta på drygt 450,000 kvadratkilometer. Stockholm är Sveriges huvudstad."
|
|
|
|
User: "När grundades det?"
|
|
|
|
Bellman: "Det är inte exakt känt när Sverige grundades. Det var först under 1300-talet att Sverige började ge sig uttryck i internationella affärer."
|
|
|
|
User: "Vad hände 1523?"
|
|
|
|
Bellman: "År 1523 blev Gustav Vasa kung över Sverige och inledde reformationen i landet. Han var också den förste monarken att bära titeln "Sveriges Konung"."
|
|
|
|
User: "Hur länge levde han?"
|
|
|
|
Bellman: "Gustav Vasa levde från år 1496 till 1560."
|
|
|
|
### Training Parameters
|
|
per_device_train_batch_size = 2,<br>
|
|
gradient_accumulation_steps = 64,<br>
|
|
num_train_epochs=3,<br>
|
|
warmup_steps = 5,<br>
|
|
learning_rate = 1e-4,<br>
|
|
logging_steps = 15,<br>
|
|
optim = "adamw_8bit",<br>
|
|
weight_decay = 0.01,<br>
|
|
lr_scheduler_type = "linear",<br>
|
|
seed = 3407,<br>
|
|
per_device_eval_batch_size = 2,<br>
|
|
evaluation_strategy="steps",<br>
|
|
eval_accumulation_steps = 64,<br>
|
|
eval_steps = 15,<br>
|
|
eval_delay = 0,<br>
|
|
save_strategy="steps",<br>
|
|
save_steps=50,<br>
|
|
|
|
### Model Description
|
|
|
|
|
|
- **Developed by:** Me
|
|
- **Funded by:** Me
|
|
- **Model type:** Instruct
|
|
- **Language(s) (NLP):** Swedish
|
|
- **License:** Apache 2 License
|
|
- **Finetuned from model:** Mistral-Nemo-Instruct-2407
|
|
|
|
## Model Card Contact
|
|
|
|
rickard@mindemia.com |