165 lines
5.0 KiB
Markdown
165 lines
5.0 KiB
Markdown
---
|
|
library_name: transformers
|
|
license: apache-2.0
|
|
language:
|
|
- sk
|
|
- en
|
|
base_model:
|
|
- slovak-nlp/Qwen3-14B-sk
|
|
datasets:
|
|
- saillab/alpaca-slovak-cleaned
|
|
tags:
|
|
- slovak
|
|
- instruction-tuned
|
|
- qwen3
|
|
- lora
|
|
- merged-lora
|
|
- text-generation
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# Qwen3-14B-sk Alpaca Slovak IT
|
|
|
|
Qwen3-14B-sk Alpaca Slovak IT is an instruction-tuned Slovak assistant model
|
|
derived from `slovak-nlp/Qwen3-14B-sk`. It was trained with a LoRA supervised
|
|
fine-tuning recipe on Slovak Alpaca-style instruction data and then merged back
|
|
into the base model, so the released artifact can be loaded directly with
|
|
`transformers`.
|
|
|
|
The model is designed to answer in Slovak by default while still attempting to
|
|
follow explicit user requests for another language or a specific output format.
|
|
|
|
## Intended Use
|
|
|
|
This model is intended for Slovak instruction following, Slovak question
|
|
answering, drafting, rewriting, summarization-style prompts, and general
|
|
assistant workflows where Slovak is the primary language.
|
|
|
|
It can also respond to English prompts and translation-style requests, but
|
|
language control is not guaranteed. Use additional application-level checks for
|
|
strict formatting, policy compliance, or high-reliability translation.
|
|
|
|
Do not use this model as the sole source for medical, legal, financial, safety,
|
|
or other high-stakes decisions. It has not been safety aligned, red-teamed, or
|
|
moderated for production deployment.
|
|
|
|
## Model Details
|
|
|
|
| Field | Value |
|
|
| --- | --- |
|
|
| Base model | `slovak-nlp/Qwen3-14B-sk` |
|
|
| Base revision | `61bc40572b2fa4720e75a7a9de457151aefce396` |
|
|
| Release repository | [`mrshu/qwen3-14b-sk-alpaca-slovak-it`](https://huggingface.co/mrshu/qwen3-14b-sk-alpaca-slovak-it) |
|
|
| Architecture | Qwen3 causal language model |
|
|
| Tuned artifact | Merged full-weight checkpoint |
|
|
| Adapter used during training | LoRA |
|
|
| Weight dtype | bfloat16 |
|
|
| License | Apache-2.0 |
|
|
|
|
## Training Data
|
|
|
|
The model was instruction-tuned on
|
|
[`saillab/alpaca-slovak-cleaned`](https://huggingface.co/datasets/saillab/alpaca-slovak-cleaned),
|
|
a Slovak Alpaca-style instruction dataset.
|
|
|
|
Dataset preparation converted each example into a chat-style conversation with
|
|
system, user, and assistant messages. Empty instruction/output examples were
|
|
excluded, and duplicate instruction/input/output triples were removed across
|
|
the prepared splits.
|
|
|
|
| Split | Rows |
|
|
| --- | ---: |
|
|
| Train | 41,601 |
|
|
| Held-out | 10,401 |
|
|
|
|
The data preparation used dataset revision
|
|
`058172466eb1d6a28b161f29c74350911d154161`.
|
|
|
|
Each training conversation used this system prompt:
|
|
|
|
```text
|
|
Si užitočný asistent. Riaď sa jazykom a požadovaným formátom používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky.
|
|
```
|
|
|
|
## Training Recipe
|
|
|
|
The model was trained with supervised fine-tuning using PEFT LoRA. Only the
|
|
assistant turns were included in the training loss.
|
|
|
|
| Setting | Value |
|
|
| --- | --- |
|
|
| LoRA rank / alpha / dropout | `32` / `64` / `0.05` |
|
|
| Target modules | linear projection modules |
|
|
| Sequence length | `4096` |
|
|
| Sample packing | enabled |
|
|
| Epochs | `1` |
|
|
| Effective batch size | `16` |
|
|
| Micro batch size | `1` |
|
|
| Gradient accumulation | `16` |
|
|
| Learning rate | `1e-4` |
|
|
| Scheduler | cosine |
|
|
| Warmup ratio | `0.06` |
|
|
| Optimizer | fused AdamW |
|
|
| Precision | bfloat16 |
|
|
| Gradient checkpointing | enabled |
|
|
| Seed | `42` |
|
|
|
|
After training, the LoRA adapter was merged into the base model with PEFT and
|
|
saved as a standalone safetensors checkpoint.
|
|
|
|
## Usage
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "mrshu/qwen3-14b-sk-alpaca-slovak-it"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
trust_remote_code=True,
|
|
)
|
|
|
|
messages = [
|
|
{
|
|
"role": "system",
|
|
"content": (
|
|
"Si užitočný asistent. Riaď sa jazykom a požadovaným formátom "
|
|
"používateľa. Ak používateľ nežiada iný jazyk, odpovedaj po slovensky."
|
|
),
|
|
},
|
|
{"role": "user", "content": "Stručne vysvetli, čo je LoRA."},
|
|
]
|
|
|
|
inputs = tokenizer.apply_chat_template(
|
|
messages,
|
|
add_generation_prompt=True,
|
|
return_tensors="pt",
|
|
).to(model.device)
|
|
|
|
outputs = model.generate(
|
|
inputs,
|
|
max_new_tokens=512,
|
|
do_sample=True,
|
|
temperature=0.6,
|
|
top_p=0.95,
|
|
)
|
|
|
|
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
|
|
```
|
|
|
|
## Limitations
|
|
|
|
- The supervised fine-tuning data is translated instruction data, so the model
|
|
may inherit translation artifacts, unnatural phrasing, or source-dataset
|
|
biases.
|
|
- The model is biased toward Slovak responses. Explicit requests for another
|
|
language may work, but they are not guaranteed.
|
|
- Strict JSON, exact labels, citations, and other constrained formats should be
|
|
validated outside the model.
|
|
- The model may hallucinate facts, produce unsafe content, or follow malicious
|
|
instructions without additional safeguards.
|