Yuna-130M-Conversation/README.md

---
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- causal-lm
- gqa
- conversational
- roleplay
- fine-tuned
- yuna
---

# YunaGPT-124M V1 Conversation

**A compact, single-turn conversation and character role-play model.**

![Parameters](https://img.shields.io/badge/parameters-124%2C445%2C376-35c7e8)
![Context](https://img.shields.io/badge/context-2%2C048-9b72e8)
![Stage](https://img.shields.io/badge/stage-conversation_SFT-9b72e8)
![SFT tokens](https://img.shields.io/badge/conversation_tokens-~3.79M-9b72e8)
![Status](https://img.shields.io/badge/status-experimental-orange)

> **Important:** This model was trained to produce one assistant or character reply from one user turn. It has no built-in memory and is not a reliable factual or safety-aligned assistant.

<p align="center">
  <img src="Assets/info.png" alt="YunaGPT-124M V1 architecture and training infographic" width="600">
</p>

## Run on google Colab for free online:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](
https://colab.research.google.com/github/jordanmckin/Yuna_130m_Colab/blob/main/Yuna_130M_Conversation_Colab.ipynb
)


## Overview

YunaGPT-124M V1 Conversation continues from the final Story checkpoint and adds response-only conversation SFT. Its specialty is short dialogue and character role-play: an optional character description and one user message are provided as context, and the model generates one assistant reply.

The training data was deliberately flattened into independent user/assistant exchanges. The model was not trained to predict both sides of an ongoing transcript, and it does not remember earlier calls unless an application supplies context manually.

## Project background

Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's *Build a Large Language Model (From Scratch)*. It later moved to Hugging Face's native LLaMA implementation and became a home-hardware experiment using an RTX 3090. The broader project explores instruction following, creative writing, role-play, synthetic Final Fantasy X knowledge, and preference optimization.

The model may write plausible franchise dialogue or lore while inventing details. Do not rely on it as a factual source.

## Model summary

| Item | Value |
|---|---:|
| Parameters | **124,445,376** |
| Model class | `LlamaForCausalLM` |
| Training stage | Character/conversation SFT |
| Lineage | Base → Story → Conversation |
| Training exchanges | **47,625** |
| Validation exchanges | **5,298** |
| Recorded conversation tokens | Approximately **3.79 million** |
| Context length | **2,048 tokens** |
| Maximum trained user turn | 600 tokens |
| Maximum trained response | 512 tokens |
| Vocabulary | **24,000 tokens** |
| Tokenizer | Byte-level BPE |
| Weight format | `safetensors`, FP32 |
| Primary language | English |

## Prompt format

This checkpoint uses a custom role-play format, not the instruction format used by the Instruct and Story variants.

With a character description:

```text
### Character:
{character_or_persona}

### User:
{message}

### Assistant:
```

Without a character description:

```text
### User:
{message}

### Assistant:
```

The headings and blank lines are part of the training format. A standard `tokenizer.apply_chat_template(...)` call is not equivalent for this release.

## Run it yourself

```bash
pip install torch transformers
```

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Conversation"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()


def format_conversation_prompt(message: str, character: str = "") -> str:
    parts = []
    if character.strip():
        parts.append(f"### Character:\n{character.strip()}")
    parts.append(f"### User:\n{message.strip()}")
    parts.append("### Assistant:\n")
    return "\n\n".join(parts)


prompt = format_conversation_prompt(
    message="I found this strange key outside the library. Do you recognize it?",
    character=(
        "Mira is a friendly village librarian who is curious, patient, "
        "and fond of old adventure stories."
    ),
)
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=220,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())
```

Replace the placeholder repository name with the final model ID or a local folder. Keeping replies below roughly 512 new tokens is closest to the response lengths seen during training.

## Architecture

| Component | Configuration |
|---|---:|
| Architecture | Decoder-only Transformer |
| Attention | Grouped-Query Attention (GQA) |
| Hidden size | 576 |
| Intermediate size | 2,048 |
| Layers | 25 |
| Attention heads | 9 |
| Key/value heads | 3 |
| Activation | SiLU / SwiGLU feed-forward blocks |
| Normalization | RMSNorm, epsilon `1e-6` |
| Position encoding | RoPE, theta `10,000` |
| Maximum positions | 2,048 |
| Input/output embeddings | Tied |

## Conversation training

This checkpoint was fine-tuned for one epoch on `youndukn/ROLE_PLAY_INSTRUCT` using:

- 47,625 flattened training exchanges;
- 5,298 held-out test exchanges;
- optional character/persona context;
- one user turn followed by one assistant response;
- response-only loss with an explicit EOS target;
- a learning rate of `2.5e-5` and batch size 1.

The run recorded approximately 3.79 million processed tokens. Internal validation loss reached approximately **1.65** near the end of training. This is a dataset-specific next-token metric, not evidence of general conversational quality or safety.

The uploaded weights exactly match the final epoch-1 conversation checkpoint. Source dataset terms and attribution requirements remain applicable.

## Intended uses

- Single-turn character and role-play responses.
- Short conversational experiments with human supervision.
- Educational study of response-only dialogue fine-tuning.
- Local prototyping of character interfaces.

## Limitations and safety

Expected limitations include:

- no persistent memory and limited multi-turn consistency;
- character drift, repetition, role-label leakage, or speaking for the user;
- hallucinated facts and unreliable real-world advice;
- weak reasoning and instruction following outside role-play dialogue;
- sensitivity to persona wording and sampling settings;
- generation of stereotypes, manipulative dialogue, sexual content, offensive language, or other unsafe material inherited from training data;
- possible reproduction of phrases or information from source data.

Do not use this model for emotional dependency, impersonation, medical or mental-health support, high-impact advice, moderation, surveillance, or unsupervised public deployment. Make fictional framing clear to users and provide human oversight.

## Evaluation status

No standardized dialogue, role-play, factuality, bias, toxicity, privacy, or safety benchmarks are included. The internal validation loss only measures prediction on the held-out source split.

## Related variants

- **Base:** raw next-token completion checkpoint.
- **Instruct:** general single-turn instruction SFT branch with a different prompt wrapper.
- **Story:** creative-writing checkpoint that directly precedes this variant.

## License and attribution

No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' licenses, terms, or attribution requirements.

---

**YunaGPT-124M V1 Conversation is an experimental role-play model. Keep its use fictional, supervised, and transparent.**