初始化项目，由ModelHub XC社区提供模型
Model: meadbee/Yuna-130M-Conversation Source: Original Platform
This commit is contained in:
ModelHub XC
218
README.md
Normal file
218
README.md
Normal file
@@ -0,0 +1,218 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- llama
|
||||
- causal-lm
|
||||
- gqa
|
||||
- conversational
|
||||
- roleplay
|
||||
- fine-tuned
|
||||
- yuna
|
||||
---
|
||||
|
||||
# YunaGPT-124M V1 Conversation
|
||||
|
||||
**A compact, single-turn conversation and character role-play model.**
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
> **Important:** This model was trained to produce one assistant or character reply from one user turn. It has no built-in memory and is not a reliable factual or safety-aligned assistant.
|
||||
|
||||
<p align="center">
|
||||
<img src="Assets/info.png" alt="YunaGPT-124M V1 architecture and training infographic" width="600">
|
||||
</p>
|
||||
|
||||
## Run on google Colab for free online:
|
||||
[](
|
||||
https://colab.research.google.com/github/jordanmckin/Yuna_130m_Colab/blob/main/Yuna_130M_Conversation_Colab.ipynb
|
||||
)
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
YunaGPT-124M V1 Conversation continues from the final Story checkpoint and adds response-only conversation SFT. Its specialty is short dialogue and character role-play: an optional character description and one user message are provided as context, and the model generates one assistant reply.
|
||||
|
||||
The training data was deliberately flattened into independent user/assistant exchanges. The model was not trained to predict both sides of an ongoing transcript, and it does not remember earlier calls unless an application supplies context manually.
|
||||
|
||||
## Project background
|
||||
|
||||
Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's *Build a Large Language Model (From Scratch)*. It later moved to Hugging Face's native LLaMA implementation and became a home-hardware experiment using an RTX 3090. The broader project explores instruction following, creative writing, role-play, synthetic Final Fantasy X knowledge, and preference optimization.
|
||||
|
||||
The model may write plausible franchise dialogue or lore while inventing details. Do not rely on it as a factual source.
|
||||
|
||||
## Model summary
|
||||
|
||||
| Item | Value |
|
||||
|---|---:|
|
||||
| Parameters | **124,445,376** |
|
||||
| Model class | `LlamaForCausalLM` |
|
||||
| Training stage | Character/conversation SFT |
|
||||
| Lineage | Base → Story → Conversation |
|
||||
| Training exchanges | **47,625** |
|
||||
| Validation exchanges | **5,298** |
|
||||
| Recorded conversation tokens | Approximately **3.79 million** |
|
||||
| Context length | **2,048 tokens** |
|
||||
| Maximum trained user turn | 600 tokens |
|
||||
| Maximum trained response | 512 tokens |
|
||||
| Vocabulary | **24,000 tokens** |
|
||||
| Tokenizer | Byte-level BPE |
|
||||
| Weight format | `safetensors`, FP32 |
|
||||
| Primary language | English |
|
||||
|
||||
## Prompt format
|
||||
|
||||
This checkpoint uses a custom role-play format, not the instruction format used by the Instruct and Story variants.
|
||||
|
||||
With a character description:
|
||||
|
||||
```text
|
||||
### Character:
|
||||
{character_or_persona}
|
||||
|
||||
### User:
|
||||
{message}
|
||||
|
||||
### Assistant:
|
||||
```
|
||||
|
||||
Without a character description:
|
||||
|
||||
```text
|
||||
### User:
|
||||
{message}
|
||||
|
||||
### Assistant:
|
||||
```
|
||||
|
||||
The headings and blank lines are part of the training format. A standard `tokenizer.apply_chat_template(...)` call is not equivalent for this release.
|
||||
|
||||
## Run it yourself
|
||||
|
||||
```bash
|
||||
pip install torch transformers
|
||||
```
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Conversation"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
|
||||
model.eval()
|
||||
|
||||
|
||||
def format_conversation_prompt(message: str, character: str = "") -> str:
|
||||
parts = []
|
||||
if character.strip():
|
||||
parts.append(f"### Character:\n{character.strip()}")
|
||||
parts.append(f"### User:\n{message.strip()}")
|
||||
parts.append("### Assistant:\n")
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
prompt = format_conversation_prompt(
|
||||
message="I found this strange key outside the library. Do you recognize it?",
|
||||
character=(
|
||||
"Mira is a friendly village librarian who is curious, patient, "
|
||||
"and fond of old adventure stories."
|
||||
),
|
||||
)
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
with torch.no_grad():
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=220,
|
||||
do_sample=True,
|
||||
temperature=0.8,
|
||||
top_p=0.9,
|
||||
repetition_penalty=1.1,
|
||||
pad_token_id=tokenizer.pad_token_id,
|
||||
eos_token_id=tokenizer.eos_token_id,
|
||||
)
|
||||
|
||||
new_tokens = output[0, inputs["input_ids"].shape[1]:]
|
||||
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())
|
||||
```
|
||||
|
||||
Replace the placeholder repository name with the final model ID or a local folder. Keeping replies below roughly 512 new tokens is closest to the response lengths seen during training.
|
||||
|
||||
## Architecture
|
||||
|
||||
| Component | Configuration |
|
||||
|---|---:|
|
||||
| Architecture | Decoder-only Transformer |
|
||||
| Attention | Grouped-Query Attention (GQA) |
|
||||
| Hidden size | 576 |
|
||||
| Intermediate size | 2,048 |
|
||||
| Layers | 25 |
|
||||
| Attention heads | 9 |
|
||||
| Key/value heads | 3 |
|
||||
| Activation | SiLU / SwiGLU feed-forward blocks |
|
||||
| Normalization | RMSNorm, epsilon `1e-6` |
|
||||
| Position encoding | RoPE, theta `10,000` |
|
||||
| Maximum positions | 2,048 |
|
||||
| Input/output embeddings | Tied |
|
||||
|
||||
## Conversation training
|
||||
|
||||
This checkpoint was fine-tuned for one epoch on `youndukn/ROLE_PLAY_INSTRUCT` using:
|
||||
|
||||
- 47,625 flattened training exchanges;
|
||||
- 5,298 held-out test exchanges;
|
||||
- optional character/persona context;
|
||||
- one user turn followed by one assistant response;
|
||||
- response-only loss with an explicit EOS target;
|
||||
- a learning rate of `2.5e-5` and batch size 1.
|
||||
|
||||
The run recorded approximately 3.79 million processed tokens. Internal validation loss reached approximately **1.65** near the end of training. This is a dataset-specific next-token metric, not evidence of general conversational quality or safety.
|
||||
|
||||
The uploaded weights exactly match the final epoch-1 conversation checkpoint. Source dataset terms and attribution requirements remain applicable.
|
||||
|
||||
## Intended uses
|
||||
|
||||
- Single-turn character and role-play responses.
|
||||
- Short conversational experiments with human supervision.
|
||||
- Educational study of response-only dialogue fine-tuning.
|
||||
- Local prototyping of character interfaces.
|
||||
|
||||
## Limitations and safety
|
||||
|
||||
Expected limitations include:
|
||||
|
||||
- no persistent memory and limited multi-turn consistency;
|
||||
- character drift, repetition, role-label leakage, or speaking for the user;
|
||||
- hallucinated facts and unreliable real-world advice;
|
||||
- weak reasoning and instruction following outside role-play dialogue;
|
||||
- sensitivity to persona wording and sampling settings;
|
||||
- generation of stereotypes, manipulative dialogue, sexual content, offensive language, or other unsafe material inherited from training data;
|
||||
- possible reproduction of phrases or information from source data.
|
||||
|
||||
Do not use this model for emotional dependency, impersonation, medical or mental-health support, high-impact advice, moderation, surveillance, or unsupervised public deployment. Make fictional framing clear to users and provide human oversight.
|
||||
|
||||
## Evaluation status
|
||||
|
||||
No standardized dialogue, role-play, factuality, bias, toxicity, privacy, or safety benchmarks are included. The internal validation loss only measures prediction on the held-out source split.
|
||||
|
||||
## Related variants
|
||||
|
||||
- **Base:** raw next-token completion checkpoint.
|
||||
- **Instruct:** general single-turn instruction SFT branch with a different prompt wrapper.
|
||||
- **Story:** creative-writing checkpoint that directly precedes this variant.
|
||||
|
||||
## License and attribution
|
||||
|
||||
No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' licenses, terms, or attribution requirements.
|
||||
|
||||
---
|
||||
|
||||
**YunaGPT-124M V1 Conversation is an experimental role-play model. Keep its use fictional, supervised, and transparent.**
|
||||
Reference in New Issue
Block a user