168 lines
5.1 KiB
Markdown
168 lines
5.1 KiB
Markdown
---
|
|
library_name: transformers
|
|
license: apache-2.0
|
|
language:
|
|
- sk
|
|
- en
|
|
base_model:
|
|
- slovak-nlp/mistral-sk-7b
|
|
datasets:
|
|
- saillab/alpaca-slovak-cleaned
|
|
tags:
|
|
- slovak
|
|
- instruction-tuned
|
|
- mistral
|
|
- lora
|
|
- merged-lora
|
|
- text-generation
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# Mistral-sk-7B Alpaca Slovak IT
|
|
|
|
Mistral-sk-7B Alpaca Slovak IT is an instruction-tuned Slovak assistant model
|
|
derived from `slovak-nlp/mistral-sk-7b`. It was trained with a LoRA supervised
|
|
fine-tuning recipe on Slovak Alpaca-style instruction data and then merged back
|
|
into the base model, so the released artifact can be loaded directly with
|
|
`transformers`.
|
|
|
|
This is the compact 7B release candidate. It is intended for Slovak assistant
|
|
experiments where lower memory use is more important than using the strongest
|
|
available model in this release set.
|
|
|
|
## Intended Use
|
|
|
|
This model is intended for Slovak instruction following, Slovak question
|
|
answering, drafting, rewriting, summarization-style prompts, and general
|
|
assistant workflows where Slovak is the primary language.
|
|
|
|
It can also respond to English prompts and translation-style requests, but
|
|
language control is weaker than in larger models. Use additional
|
|
application-level checks for strict formatting, policy compliance, or
|
|
high-reliability translation.
|
|
|
|
Do not use this model as the sole source for medical, legal, financial, safety,
|
|
or other high-stakes decisions. It has not been safety aligned, red-teamed, or
|
|
moderated for production deployment.
|
|
|
|
## Model Details
|
|
|
|
| Field | Value |
|
|
| --- | --- |
|
|
| Base model | `slovak-nlp/mistral-sk-7b` |
|
|
| Base revision | `089497ae0d72018e9591895c54f774bf0b4a83a4` |
|
|
| Release repository | [`mrshu/mistral-sk-7b-alpaca-slovak-it`](https://huggingface.co/mrshu/mistral-sk-7b-alpaca-slovak-it) |
|
|
| Architecture | Mistral causal language model |
|
|
| Tuned artifact | Merged full-weight checkpoint |
|
|
| Adapter used during training | LoRA |
|
|
| Weight dtype | bfloat16 |
|
|
| License | Apache-2.0 |
|
|
|
|
## Training Data
|
|
|
|
The model was instruction-tuned on
|
|
[`saillab/alpaca-slovak-cleaned`](https://huggingface.co/datasets/saillab/alpaca-slovak-cleaned),
|
|
a Slovak Alpaca-style instruction dataset.
|
|
|
|
Dataset preparation converted each example into a chat-style conversation with
|
|
system, user, and assistant messages. Empty instruction/output examples were
|
|
excluded, and duplicate instruction/input/output triples were removed across
|
|
the prepared splits.
|
|
|
|
| Split | Rows |
|
|
| --- | ---: |
|
|
| Train | 41,601 |
|
|
| Held-out | 10,401 |
|
|
|
|
The data preparation used dataset revision
|
|
`058172466eb1d6a28b161f29c74350911d154161`.
|
|
|
|
Each training conversation used this system prompt:
|
|
|
|
```text
|
|
Si užitočný asistent. Riaď sa jazykom a požadovaným formátom používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky.
|
|
```
|
|
|
|
## Training Recipe
|
|
|
|
The model was trained with supervised fine-tuning using PEFT LoRA. Only the
|
|
assistant turns were included in the training loss.
|
|
|
|
| Setting | Value |
|
|
| --- | --- |
|
|
| LoRA rank / alpha / dropout | `32` / `64` / `0.05` |
|
|
| Target modules | linear projection modules |
|
|
| Sequence length | `4096` |
|
|
| Sample packing | enabled |
|
|
| Epochs | `1` |
|
|
| Effective batch size | `16` |
|
|
| Micro batch size | `2` |
|
|
| Gradient accumulation | `8` |
|
|
| Learning rate | `1e-4` |
|
|
| Scheduler | cosine |
|
|
| Warmup ratio | `0.06` |
|
|
| Optimizer | fused AdamW |
|
|
| Precision | bfloat16 |
|
|
| Gradient checkpointing | enabled |
|
|
| Seed | `42` |
|
|
|
|
After training, the LoRA adapter was merged into the base model with PEFT and
|
|
saved as a standalone safetensors checkpoint with a Mistral chat template in
|
|
the tokenizer.
|
|
|
|
## Usage
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "mrshu/mistral-sk-7b-alpaca-slovak-it"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
trust_remote_code=True,
|
|
)
|
|
|
|
messages = [
|
|
{
|
|
"role": "system",
|
|
"content": (
|
|
"Si užitočný asistent. Riaď sa jazykom a požadovaným formátom "
|
|
"používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky."
|
|
),
|
|
},
|
|
{"role": "user", "content": "Stručne vysvetli, čo je LoRA."},
|
|
]
|
|
|
|
inputs = tokenizer.apply_chat_template(
|
|
messages,
|
|
add_generation_prompt=True,
|
|
return_tensors="pt",
|
|
).to(model.device)
|
|
|
|
outputs = model.generate(
|
|
inputs,
|
|
max_new_tokens=512,
|
|
do_sample=True,
|
|
temperature=0.7,
|
|
top_p=0.95,
|
|
)
|
|
|
|
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
|
|
```
|
|
|
|
## Limitations
|
|
|
|
- The supervised fine-tuning data is translated instruction data, so the model
|
|
may inherit translation artifacts, unnatural phrasing, or source-dataset
|
|
biases.
|
|
- The model is strongly biased toward Slovak responses. Explicit English-only
|
|
or strict-format requests may not be followed reliably.
|
|
- Strict JSON, exact labels, citations, and other constrained formats should be
|
|
validated outside the model.
|
|
- The model may hallucinate facts, produce unsafe content, or follow malicious
|
|
instructions without additional safeguards.
|