147 lines
4.7 KiB
Markdown
147 lines
4.7 KiB
Markdown
---
|
|
language:
|
|
- gn
|
|
license: "gpl-3.0"
|
|
tags:
|
|
- causal-lm
|
|
- text-generation
|
|
- guarani
|
|
- quality-assessment
|
|
- perplexity
|
|
- gemma2
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
# CoreguapaLM
|
|
|
|
This model is intended to be used to validate the quality of Guarani text. It was
|
|
trained on the [COREGUAPA corpus](https://corpus.spl.gov.py), a restricted dataset manually compiled and curated by the Paraguayan Secretary of Linguistic Policies. The corpus contains high-quality documents, including primarily copyrighted materials.
|
|
|
|
⚠️ Although the model is based on a transformer-based architecture (`Gemma2-9b-it`), **it was not developed as a generative tool** — its primary use is to compute the perplexity score of Guarani documents. Lower perplexity may indicate text that is more predictable by the model and more similar to the reference high-quality corpus.
|
|
|
|
## 📌 Summary
|
|
|
|
- **Model type:** Gemma2 For Causal LM
|
|
- **Base model:** `princeton-nlp/gemma-2-9b-it-SimPO`
|
|
- **Fine-tuning method:** Full fine-tuning (all model weights updated)
|
|
- **Dataset:** [COREGUAPA](https://corpus.spl.gov.py) (restricted)
|
|
- **Primary task:** Perplexity computation
|
|
|
|
## 🏗️ Model Details
|
|
|
|
- **Architecture:** `Gemma2ForCausalLM`
|
|
- **Number of layers:** `42`
|
|
- **Hidden size:** `3584`
|
|
- **Attention heads:** `16`
|
|
- **Feedforward intermediate size:** `14336`
|
|
- **Vocabulary size:** `256000`
|
|
- **Maximum context length:** `8192` tokens
|
|
- **Precision:** `float16`
|
|
- **Tokenizer:** saved in this folder via `tokenizer.json` and `tokenizer_config.json`
|
|
- **Generation config:** saved in `generation_config.json`
|
|
- **Prompt template:** `chat_template.jinja`
|
|
|
|
## ⚙️ Training Details
|
|
|
|
- **Batch size:** `1`
|
|
- **Gradient accumulation:** `1`
|
|
- **Learning rate:** `2e-5`
|
|
- **Weight decay:** `0.01`
|
|
- **Warmup steps:** `100`
|
|
- **Optimizer:** `paged_adamw_8bit`
|
|
- **Scheduler:** `linear`
|
|
- **Epochs:** `6`
|
|
- **Precision mode:** `bf16`
|
|
- **Gradient checkpointing:** enabled
|
|
|
|
## 🗃️ Dataset and Preprocessing
|
|
|
|
- **Raw source file:** private
|
|
- **Split strategy:** train / validation / test
|
|
- **Sequence length used for tokenization:** `2048`
|
|
- **Train dataset size**: `892` records (`1826816` tokens)
|
|
- **Validation dataset size**: `111` records (`227328` tokens)
|
|
- **Test dataset size**: `112` records (`229376` tokens)
|
|
- **Tokenizer:** `princeton-nlp/gemma-2-9b-it-SimPO`
|
|
|
|
## 🚀 Usage
|
|
|
|
Compute perplexity for a given Guarani text using the fine-tuned model:
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
import torch
|
|
import math
|
|
|
|
model_id = 'guaran-ia/coreguapa-lm'
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(model_id)
|
|
model.eval()
|
|
|
|
def perplexity(text: str) -> float:
|
|
inputs = tokenizer(text, return_tensors='pt')
|
|
with torch.no_grad():
|
|
outputs = model(**inputs, labels=inputs['input_ids'])
|
|
loss = outputs.loss
|
|
return math.exp(loss.item())
|
|
|
|
text = "Your Guarani text here."
|
|
print(f"Perplexity: {perplexity(text):.4f}")
|
|
```
|
|
|
|
## Perplexity for long texts
|
|
|
|
If input length exceeds the model/tokenizer maximum (8192 tokens), you can follow
|
|
the next recipe to compute perplexity over sliding chunks and average per-token loss.
|
|
|
|
```python
|
|
import torch, math
|
|
|
|
def perplexity_sliding(text: str, model, tokenizer, max_len: int = 8192, stride: int = 4096):
|
|
"""Compute perplexity over long text by slicing into overlapping chunks.
|
|
|
|
- `max_len` should be <= model.config.max_position_embeddings (8192).
|
|
- `stride` controls overlap; larger overlap gives smoother per-token averaging.
|
|
"""
|
|
enc = tokenizer(text, return_tensors='pt')['input_ids'][0]
|
|
n = enc.size(0)
|
|
if n == 0:
|
|
return float('nan')
|
|
|
|
total_nll = 0.0
|
|
total_tokens = 0
|
|
start = 0
|
|
while start < n:
|
|
end = min(start + max_len, n)
|
|
input_ids = enc[start:end].unsqueeze(0)
|
|
with torch.no_grad():
|
|
outputs = model(input_ids, labels=input_ids)
|
|
# outputs.loss is the average NLL for the chunk
|
|
loss = outputs.loss.item()
|
|
chunk_len = end - start
|
|
total_nll += loss * chunk_len
|
|
total_tokens += chunk_len
|
|
if end == n:
|
|
break
|
|
start += stride
|
|
|
|
avg_nll = total_nll / total_tokens
|
|
return math.exp(avg_nll)
|
|
|
|
# Example usage:
|
|
text = open('some_guarani.txt', encoding='utf-8').read()
|
|
tokenizer.model_max_length = 8192
|
|
print(f"Perplexity (sliding): {perplexity_sliding(text, model, tokenizer):.4f}")
|
|
```
|
|
|
|
## ❗ Limitations and Notes
|
|
|
|
- The training data is drawn from a private Guarani corpus.
|
|
- The model may reflect biases present in the source corpus.
|
|
- License metadata is provided in this folder.
|
|
|
|
## 📜 License
|
|
|
|
This model checkpoint and accompanying files are released under the GNU General Public License v3 (GPLv3).
|
|
See the `LICENSE` file in this directory for the full license text. |