mistral-sk-7b-alpaca-slovak-it/README.md

library_name, license, language, base_model, datasets, tags, pipeline_tag

library_name	license	language	base_model	datasets	tags	pipeline_tag
transformers	apache-2.0	sk en	slovak-nlp/mistral-sk-7b	saillab/alpaca-slovak-cleaned	slovak instruction-tuned mistral lora merged-lora text-generation	text-generation

Mistral-sk-7B Alpaca Slovak IT

Mistral-sk-7B Alpaca Slovak IT is an instruction-tuned Slovak assistant model derived from slovak-nlp/mistral-sk-7b. It was trained with a LoRA supervised fine-tuning recipe on Slovak Alpaca-style instruction data and then merged back into the base model, so the released artifact can be loaded directly with transformers.

This is the compact 7B release candidate. It is intended for Slovak assistant experiments where lower memory use is more important than using the strongest available model in this release set.

Intended Use

This model is intended for Slovak instruction following, Slovak question answering, drafting, rewriting, summarization-style prompts, and general assistant workflows where Slovak is the primary language.

It can also respond to English prompts and translation-style requests, but language control is weaker than in larger models. Use additional application-level checks for strict formatting, policy compliance, or high-reliability translation.

Do not use this model as the sole source for medical, legal, financial, safety, or other high-stakes decisions. It has not been safety aligned, red-teamed, or moderated for production deployment.

Model Details

Field	Value
Base model	slovak-nlp/mistral-sk-7b
Base revision	089497ae0d72018e9591895c54f774bf0b4a83a4
Release repository	mrshu/mistral-sk-7b-alpaca-slovak-it
Architecture	Mistral causal language model
Tuned artifact	Merged full-weight checkpoint
Adapter used during training	LoRA
Weight dtype	bfloat16
License	Apache-2.0

Training Data

The model was instruction-tuned on saillab/alpaca-slovak-cleaned, a Slovak Alpaca-style instruction dataset.

Dataset preparation converted each example into a chat-style conversation with system, user, and assistant messages. Empty instruction/output examples were excluded, and duplicate instruction/input/output triples were removed across the prepared splits.

Split	Rows
Train	41,601
Held-out	10,401

The data preparation used dataset revision 058172466eb1d6a28b161f29c74350911d154161.

Each training conversation used this system prompt:

Si užitočný asistent. Riaď sa jazykom a požadovaným formátom používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky.

Training Recipe

The model was trained with supervised fine-tuning using PEFT LoRA. Only the assistant turns were included in the training loss.

Setting	Value
LoRA rank / alpha / dropout	32 / 64 / 0.05
Target modules	linear projection modules
Sequence length	4096
Sample packing	enabled
Epochs	1
Effective batch size	16
Micro batch size	2
Gradient accumulation	8
Learning rate	1e-4
Scheduler	cosine
Warmup ratio	0.06
Optimizer	fused AdamW
Precision	bfloat16
Gradient checkpointing	enabled
Seed	42

After training, the LoRA adapter was merged into the base model with PEFT and saved as a standalone safetensors checkpoint with a Mistral chat template in the tokenizer.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mrshu/mistral-sk-7b-alpaca-slovak-it"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "system",
        "content": (
            "Si užitočný asistent. Riaď sa jazykom a požadovaným formátom "
            "používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky."
        ),
    },
    {"role": "user", "content": "Stručne vysvetli, čo je LoRA."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
)

print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Limitations

• The supervised fine-tuning data is translated instruction data, so the model may inherit translation artifacts, unnatural phrasing, or source-dataset biases.
• The model is strongly biased toward Slovak responses. Explicit English-only or strict-format requests may not be followed reliably.
• Strict JSON, exact labels, citations, and other constrained formats should be validated outside the model.
• The model may hallucinate facts, produce unsafe content, or follow malicious instructions without additional safeguards.