---
language:
  - gn
license: "gpl-3.0"
tags:
  - causal-lm
  - text-generation
  - guarani
  - quality-assessment
  - perplexity
  - gemma2
library_name: transformers
pipeline_tag: text-generation
---

# GNTweetsLM 

The model GNTweetsLM is intended to be used to validate the quality of Guarani text. It was
trained on a publicly available [corpus of tweets](https://github.com/sgongora27/giossa-gongora-guarani-2021/blob/main/Tweets_set/tweets_gn_october.csv) written in Guarani and Jopara (Góngora et al. [2021](https://aclanthology.org/2022.computel-1.16)). 

⚠️ Although the model is based on a transformer-based architecture (`Gemma2-9b-it`), **it was not developed as a generative tool** — its primary use is to compute the perplexity score of Guarani documents. Lower perplexity may indicate text that is more predictable by the model and more similar to the reference high-quality corpus. 

## 📌 Summary 

- **Model type:**  Gemma2 For Causal LM
- **Base model:** `princeton-nlp/gemma-2-9b-it-SimPO`
- **Fine-tuning method:** Full fine-tuning (all model weights updated)
- **Primary task:** Perplexity computation
- **Dataset (HF):** `guaran-ia/gntweets`

## 🏗️ Model Details 

- **Architecture:** `Gemma2ForCausalLM`
- **Number of layers:** `42`
- **Hidden size:** `3584`
- **Attention heads:** `16`
- **Feedforward intermediate size:** `14336`
- **Vocabulary size:** `256000`
- **Maximum context length:** `8192` tokens
- **Precision:** `float16`
- **Tokenizer:** saved in this folder via `tokenizer.json` and `tokenizer_config.json`
- **Generation config:** saved in `generation_config.json`
- **Prompt template:** `chat_template.jinja`

## ⚙️ Training Details

- **Batch size:** `1`
- **Gradient accumulation:** `1`
- **Learning rate:** `2e-5`
- **Weight decay:** `0.01`
- **Warmup steps:** `100`
- **Optimizer:** `paged_adamw_8bit`
- **Scheduler:** `linear`
- **Epochs:** `6`
- **Precision mode:** `bf16`
- **Gradient checkpointing:** enabled

## 🗃️ Dataset and Preprocessing 

- **Split strategy:** train / validation / test
- **Sequence length used for tokenization:** `2048`
- **Train dataset size**: `936` records (`1916928` tokens)
- **Validation dataset size**: `117` records (`239616` tokens)
- **Test dataset size**: `117` records (`239616` tokens)
- **Tokenizer:** `princeton-nlp/gemma-2-9b-it-SimPO`
- **HF ID:** `guaran-ia/gntweets`

## 🚀 Usage

Compute perplexity for a given Guarani text using the fine-tuned model:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import math

model_id = 'guaran-ia/gntweets-lm'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
model.eval()

def perplexity(text: str) -> float:
  inputs = tokenizer(text, return_tensors='pt')
  with torch.no_grad():
    outputs = model(**inputs, labels=inputs['input_ids'])
    loss = outputs.loss
  return math.exp(loss.item())

text = "Your Guarani text here."
print(f"Perplexity: {perplexity(text):.4f}")
```

## Perplexity for long texts

If input length exceeds the model/tokenizer maximum (8192 tokens), you can follow
the next recipe to  compute perplexity over sliding chunks and average per-token loss.

```python
import torch, math

def perplexity_sliding(text: str, model, tokenizer, max_len: int = 8192, stride: int = 4096):
  """Compute perplexity over long text by slicing into overlapping chunks.

  - `max_len` should be <= model.config.max_position_embeddings (8192).
  - `stride` controls overlap; larger overlap gives smoother per-token averaging.
  """
  enc = tokenizer(text, return_tensors='pt')['input_ids'][0]
  n = enc.size(0)
  if n == 0:
    return float('nan')

  total_nll = 0.0
  total_tokens = 0
  start = 0
  while start < n:
    end = min(start + max_len, n)
    input_ids = enc[start:end].unsqueeze(0)
    with torch.no_grad():
      outputs = model(input_ids, labels=input_ids)
      # outputs.loss is the average NLL for the chunk
      loss = outputs.loss.item()
    chunk_len = end - start
    total_nll += loss * chunk_len
    total_tokens += chunk_len
    if end == n:
      break
    start += stride

  avg_nll = total_nll / total_tokens
  return math.exp(avg_nll)

# Example usage:
text = open('some_guarani.txt', encoding='utf-8').read()
tokenizer.model_max_length = 8192
print(f"Perplexity (sliding): {perplexity_sliding(text, model, tokenizer):.4f}")
```

## ❗ Limitations and Notes 

- The model may reflect biases present in the source corpus.
- License metadata is provided in this folder.

## 📜 License

This model checkpoint and accompanying files are released under the GNU General Public License v3 (GPLv3).
See the `LICENSE` file in this directory for the full license text.