---
library_name: transformers
license: apache-2.0
language:
- sk
- en
base_model:
- slovak-nlp/mistral-sk-7b
datasets:
- saillab/alpaca-slovak-cleaned
tags:
- slovak
- instruction-tuned
- mistral
- lora
- merged-lora
- text-generation
pipeline_tag: text-generation
---

# Mistral-sk-7B Alpaca Slovak IT

Mistral-sk-7B Alpaca Slovak IT is an instruction-tuned Slovak assistant model
derived from `slovak-nlp/mistral-sk-7b`. It was trained with a LoRA supervised
fine-tuning recipe on Slovak Alpaca-style instruction data and then merged back
into the base model, so the released artifact can be loaded directly with
`transformers`.

This is the compact 7B release candidate. It is intended for Slovak assistant
experiments where lower memory use is more important than using the strongest
available model in this release set.

## Intended Use

This model is intended for Slovak instruction following, Slovak question
answering, drafting, rewriting, summarization-style prompts, and general
assistant workflows where Slovak is the primary language.

It can also respond to English prompts and translation-style requests, but
language control is weaker than in larger models. Use additional
application-level checks for strict formatting, policy compliance, or
high-reliability translation.

Do not use this model as the sole source for medical, legal, financial, safety,
or other high-stakes decisions. It has not been safety aligned, red-teamed, or
moderated for production deployment.

## Model Details

| Field | Value |
| --- | --- |
| Base model | `slovak-nlp/mistral-sk-7b` |
| Base revision | `089497ae0d72018e9591895c54f774bf0b4a83a4` |
| Release repository | [`mrshu/mistral-sk-7b-alpaca-slovak-it`](https://huggingface.co/mrshu/mistral-sk-7b-alpaca-slovak-it) |
| Architecture | Mistral causal language model |
| Tuned artifact | Merged full-weight checkpoint |
| Adapter used during training | LoRA |
| Weight dtype | bfloat16 |
| License | Apache-2.0 |

## Training Data

The model was instruction-tuned on
[`saillab/alpaca-slovak-cleaned`](https://huggingface.co/datasets/saillab/alpaca-slovak-cleaned),
a Slovak Alpaca-style instruction dataset.

Dataset preparation converted each example into a chat-style conversation with
system, user, and assistant messages. Empty instruction/output examples were
excluded, and duplicate instruction/input/output triples were removed across
the prepared splits.

| Split | Rows |
| --- | ---: |
| Train | 41,601 |
| Held-out | 10,401 |

The data preparation used dataset revision
`058172466eb1d6a28b161f29c74350911d154161`.

Each training conversation used this system prompt:

```text
Si užitočný asistent. Riaď sa jazykom a požadovaným formátom používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky.
```

## Training Recipe

The model was trained with supervised fine-tuning using PEFT LoRA. Only the
assistant turns were included in the training loss.

| Setting | Value |
| --- | --- |
| LoRA rank / alpha / dropout | `32` / `64` / `0.05` |
| Target modules | linear projection modules |
| Sequence length | `4096` |
| Sample packing | enabled |
| Epochs | `1` |
| Effective batch size | `16` |
| Micro batch size | `2` |
| Gradient accumulation | `8` |
| Learning rate | `1e-4` |
| Scheduler | cosine |
| Warmup ratio | `0.06` |
| Optimizer | fused AdamW |
| Precision | bfloat16 |
| Gradient checkpointing | enabled |
| Seed | `42` |

After training, the LoRA adapter was merged into the base model with PEFT and
saved as a standalone safetensors checkpoint with a Mistral chat template in
the tokenizer.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mrshu/mistral-sk-7b-alpaca-slovak-it"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "system",
        "content": (
            "Si užitočný asistent. Riaď sa jazykom a požadovaným formátom "
            "používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky."
        ),
    },
    {"role": "user", "content": "Stručne vysvetli, čo je LoRA."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
)

print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
```

## Limitations

- The supervised fine-tuning data is translated instruction data, so the model
  may inherit translation artifacts, unnatural phrasing, or source-dataset
  biases.
- The model is strongly biased toward Slovak responses. Explicit English-only
  or strict-format requests may not be followed reliably.
- Strict JSON, exact labels, citations, and other constrained formats should be
  validated outside the model.
- The model may hallucinate facts, produce unsafe content, or follow malicious
  instructions without additional safeguards.