初始化项目,由ModelHub XC社区提供模型
Model: nuinashco/gemma-3-1b-it-xlsum-ua-sft Source: Original Platform
This commit is contained in:
72
README.md
Normal file
72
README.md
Normal file
@@ -0,0 +1,72 @@
|
||||
---
|
||||
language:
|
||||
- uk
|
||||
license: gemma
|
||||
base_model: unsloth/gemma-3-1b-it
|
||||
datasets:
|
||||
- csebuetnlp/xlsum
|
||||
tags:
|
||||
- summarization
|
||||
- ukrainian
|
||||
- unsloth
|
||||
- gemma3
|
||||
- sft
|
||||
metrics:
|
||||
- rouge
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# gemma-3-1b-ua-safe-summarization
|
||||
|
||||
Full fine-tune of [**unsloth/gemma-3-1b-it**](https://huggingface.co/unsloth/gemma-3-1b-it) on Ukrainian news summarization.
|
||||
Trained with [Unsloth](https://github.com/unslothai/unsloth) + TRL SFT on the
|
||||
[**csebuetnlp/xlsum**](https://huggingface.co/datasets/csebuetnlp/xlsum) dataset
|
||||
(Ukrainian split of XL-Sum with safety filtering).
|
||||
|
||||
## Training details
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Base model | [unsloth/gemma-3-1b-it](https://huggingface.co/unsloth/gemma-3-1b-it) |
|
||||
| Dataset | [csebuetnlp/xlsum](https://huggingface.co/datasets/csebuetnlp/xlsum) |
|
||||
| Fine-tuning | Full SFT (no LoRA) |
|
||||
| Framework | [Unsloth](https://github.com/unslothai/unsloth) + TRL |
|
||||
| Epochs | ~1.48 (checkpoint-4000, best val ROUGE-L) |
|
||||
| Max seq length | 3072 |
|
||||
| Batch size | 8 per device |
|
||||
| Learning rate | 2e-5 (cosine decay, warmup 3 %) |
|
||||
| Precision | bfloat16 |
|
||||
| Optimizer | adamw_8bit |
|
||||
| Best eval ROUGE-L | **22.23** |
|
||||
|
||||
Response-only masking was applied — loss is computed on the model turn only.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
import torch
|
||||
|
||||
model_id = "nuinashco/gemma-3-1b-it-xlsum-ua-sft"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id, torch_dtype=torch.bfloat16, device_map="auto"
|
||||
)
|
||||
|
||||
article = "Ваш текст новини тут..."
|
||||
prompt = [
|
||||
{"role": "user", "content": f"Зроби короткий переказ наступного тексту:\n{article}"}
|
||||
]
|
||||
inputs = tokenizer.apply_chat_template(
|
||||
prompt, add_generation_prompt=True, return_tensors="pt"
|
||||
).to(model.device)
|
||||
|
||||
out = model.generate(inputs, max_new_tokens=128, do_sample=False)
|
||||
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- Trained for **Ukrainian** only; performance on other languages is undefined.
|
||||
- Inherits any biases present in the base model and training corpus.
|
||||
- Summaries may occasionally be factually inaccurate; always verify against the source.
|
||||
Reference in New Issue
Block a user