---
library_name: transformers
license: apache-2.0
language:
- sk
- en
base_model:
- slovak-nlp/Qwen3-14B-sk
datasets:
- saillab/alpaca-slovak-cleaned
tags:
- slovak
- instruction-tuned
- qwen3
- lora
- merged-lora
- text-generation
pipeline_tag: text-generation
---

# Qwen3-14B-sk Alpaca Slovak IT

Qwen3-14B-sk Alpaca Slovak IT is an instruction-tuned Slovak assistant model
derived from `slovak-nlp/Qwen3-14B-sk`. It was trained with a LoRA supervised
fine-tuning recipe on Slovak Alpaca-style instruction data and then merged back
into the base model, so the released artifact can be loaded directly with
`transformers`.

The model is designed to answer in Slovak by default while still attempting to
follow explicit user requests for another language or a specific output format.

## Intended Use

This model is intended for Slovak instruction following, Slovak question
answering, drafting, rewriting, summarization-style prompts, and general
assistant workflows where Slovak is the primary language.

It can also respond to English prompts and translation-style requests, but
language control is not guaranteed. Use additional application-level checks for
strict formatting, policy compliance, or high-reliability translation.

Do not use this model as the sole source for medical, legal, financial, safety,
or other high-stakes decisions. It has not been safety aligned, red-teamed, or
moderated for production deployment.

## Model Details

| Field | Value |
| --- | --- |
| Base model | `slovak-nlp/Qwen3-14B-sk` |
| Base revision | `61bc40572b2fa4720e75a7a9de457151aefce396` |
| Release repository | [`mrshu/qwen3-14b-sk-alpaca-slovak-it`](https://huggingface.co/mrshu/qwen3-14b-sk-alpaca-slovak-it) |
| Architecture | Qwen3 causal language model |
| Tuned artifact | Merged full-weight checkpoint |
| Adapter used during training | LoRA |
| Weight dtype | bfloat16 |
| License | Apache-2.0 |

## Training Data

The model was instruction-tuned on
[`saillab/alpaca-slovak-cleaned`](https://huggingface.co/datasets/saillab/alpaca-slovak-cleaned),
a Slovak Alpaca-style instruction dataset.

Dataset preparation converted each example into a chat-style conversation with
system, user, and assistant messages. Empty instruction/output examples were
excluded, and duplicate instruction/input/output triples were removed across
the prepared splits.

| Split | Rows |
| --- | ---: |
| Train | 41,601 |
| Held-out | 10,401 |

The data preparation used dataset revision
`058172466eb1d6a28b161f29c74350911d154161`.

Each training conversation used this system prompt:

```text
Si užitočný asistent. Riaď sa jazykom a požadovaným formátom používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky.
```

## Training Recipe

The model was trained with supervised fine-tuning using PEFT LoRA. Only the
assistant turns were included in the training loss.

| Setting | Value |
| --- | --- |
| LoRA rank / alpha / dropout | `32` / `64` / `0.05` |
| Target modules | linear projection modules |
| Sequence length | `4096` |
| Sample packing | enabled |
| Epochs | `1` |
| Effective batch size | `16` |
| Micro batch size | `1` |
| Gradient accumulation | `16` |
| Learning rate | `1e-4` |
| Scheduler | cosine |
| Warmup ratio | `0.06` |
| Optimizer | fused AdamW |
| Precision | bfloat16 |
| Gradient checkpointing | enabled |
| Seed | `42` |

After training, the LoRA adapter was merged into the base model with PEFT and
saved as a standalone safetensors checkpoint.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mrshu/qwen3-14b-sk-alpaca-slovak-it"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {
        "role": "system",
        "content": (
            "Si užitočný asistent. Riaď sa jazykom a požadovaným formátom "
            "používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky."
        ),
    },
    {"role": "user", "content": "Stručne vysvetli, čo je LoRA."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.6,
    top_p=0.95,
)

print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
```

## Limitations

- The supervised fine-tuning data is translated instruction data, so the model
  may inherit translation artifacts, unnatural phrasing, or source-dataset
  biases.
- The model is biased toward Slovak responses. Explicit requests for another
  language may work, but they are not guaranteed.
- Strict JSON, exact labels, citations, and other constrained formats should be
  validated outside the model.
- The model may hallucinate facts, produce unsafe content, or follow malicious
  instructions without additional safeguards.