初始化项目，由ModelHub XC社区提供模型
Model: meadbee/Yuna-130M-Conversation Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
Assets/info.png filter=lfs diff=lfs merge=lfs -text
|
||||||
3
Assets/info.png
Normal file
3
Assets/info.png
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:c2101769c75a627f728f920120bf5822486f1f29be765079a088ffd460cf4318
|
||||||
|
size 1800014
|
||||||
218
README.md
Normal file
218
README.md
Normal file
@@ -0,0 +1,218 @@
|
|||||||
|
---
|
||||||
|
language:
|
||||||
|
- en
|
||||||
|
library_name: transformers
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
tags:
|
||||||
|
- llama
|
||||||
|
- causal-lm
|
||||||
|
- gqa
|
||||||
|
- conversational
|
||||||
|
- roleplay
|
||||||
|
- fine-tuned
|
||||||
|
- yuna
|
||||||
|
---
|
||||||
|
|
||||||
|
# YunaGPT-124M V1 Conversation
|
||||||
|
|
||||||
|
**A compact, single-turn conversation and character role-play model.**
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
> **Important:** This model was trained to produce one assistant or character reply from one user turn. It has no built-in memory and is not a reliable factual or safety-aligned assistant.
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="Assets/info.png" alt="YunaGPT-124M V1 architecture and training infographic" width="600">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
## Run on google Colab for free online:
|
||||||
|
[](
|
||||||
|
https://colab.research.google.com/github/jordanmckin/Yuna_130m_Colab/blob/main/Yuna_130M_Conversation_Colab.ipynb
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
YunaGPT-124M V1 Conversation continues from the final Story checkpoint and adds response-only conversation SFT. Its specialty is short dialogue and character role-play: an optional character description and one user message are provided as context, and the model generates one assistant reply.
|
||||||
|
|
||||||
|
The training data was deliberately flattened into independent user/assistant exchanges. The model was not trained to predict both sides of an ongoing transcript, and it does not remember earlier calls unless an application supplies context manually.
|
||||||
|
|
||||||
|
## Project background
|
||||||
|
|
||||||
|
Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's *Build a Large Language Model (From Scratch)*. It later moved to Hugging Face's native LLaMA implementation and became a home-hardware experiment using an RTX 3090. The broader project explores instruction following, creative writing, role-play, synthetic Final Fantasy X knowledge, and preference optimization.
|
||||||
|
|
||||||
|
The model may write plausible franchise dialogue or lore while inventing details. Do not rely on it as a factual source.
|
||||||
|
|
||||||
|
## Model summary
|
||||||
|
|
||||||
|
| Item | Value |
|
||||||
|
|---|---:|
|
||||||
|
| Parameters | **124,445,376** |
|
||||||
|
| Model class | `LlamaForCausalLM` |
|
||||||
|
| Training stage | Character/conversation SFT |
|
||||||
|
| Lineage | Base → Story → Conversation |
|
||||||
|
| Training exchanges | **47,625** |
|
||||||
|
| Validation exchanges | **5,298** |
|
||||||
|
| Recorded conversation tokens | Approximately **3.79 million** |
|
||||||
|
| Context length | **2,048 tokens** |
|
||||||
|
| Maximum trained user turn | 600 tokens |
|
||||||
|
| Maximum trained response | 512 tokens |
|
||||||
|
| Vocabulary | **24,000 tokens** |
|
||||||
|
| Tokenizer | Byte-level BPE |
|
||||||
|
| Weight format | `safetensors`, FP32 |
|
||||||
|
| Primary language | English |
|
||||||
|
|
||||||
|
## Prompt format
|
||||||
|
|
||||||
|
This checkpoint uses a custom role-play format, not the instruction format used by the Instruct and Story variants.
|
||||||
|
|
||||||
|
With a character description:
|
||||||
|
|
||||||
|
```text
|
||||||
|
### Character:
|
||||||
|
{character_or_persona}
|
||||||
|
|
||||||
|
### User:
|
||||||
|
{message}
|
||||||
|
|
||||||
|
### Assistant:
|
||||||
|
```
|
||||||
|
|
||||||
|
Without a character description:
|
||||||
|
|
||||||
|
```text
|
||||||
|
### User:
|
||||||
|
{message}
|
||||||
|
|
||||||
|
### Assistant:
|
||||||
|
```
|
||||||
|
|
||||||
|
The headings and blank lines are part of the training format. A standard `tokenizer.apply_chat_template(...)` call is not equivalent for this release.
|
||||||
|
|
||||||
|
## Run it yourself
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install torch transformers
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
|
||||||
|
model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Conversation"
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
|
||||||
|
model.eval()
|
||||||
|
|
||||||
|
|
||||||
|
def format_conversation_prompt(message: str, character: str = "") -> str:
|
||||||
|
parts = []
|
||||||
|
if character.strip():
|
||||||
|
parts.append(f"### Character:\n{character.strip()}")
|
||||||
|
parts.append(f"### User:\n{message.strip()}")
|
||||||
|
parts.append("### Assistant:\n")
|
||||||
|
return "\n\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
prompt = format_conversation_prompt(
|
||||||
|
message="I found this strange key outside the library. Do you recognize it?",
|
||||||
|
character=(
|
||||||
|
"Mira is a friendly village librarian who is curious, patient, "
|
||||||
|
"and fond of old adventure stories."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||||||
|
|
||||||
|
with torch.no_grad():
|
||||||
|
output = model.generate(
|
||||||
|
**inputs,
|
||||||
|
max_new_tokens=220,
|
||||||
|
do_sample=True,
|
||||||
|
temperature=0.8,
|
||||||
|
top_p=0.9,
|
||||||
|
repetition_penalty=1.1,
|
||||||
|
pad_token_id=tokenizer.pad_token_id,
|
||||||
|
eos_token_id=tokenizer.eos_token_id,
|
||||||
|
)
|
||||||
|
|
||||||
|
new_tokens = output[0, inputs["input_ids"].shape[1]:]
|
||||||
|
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())
|
||||||
|
```
|
||||||
|
|
||||||
|
Replace the placeholder repository name with the final model ID or a local folder. Keeping replies below roughly 512 new tokens is closest to the response lengths seen during training.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
| Component | Configuration |
|
||||||
|
|---|---:|
|
||||||
|
| Architecture | Decoder-only Transformer |
|
||||||
|
| Attention | Grouped-Query Attention (GQA) |
|
||||||
|
| Hidden size | 576 |
|
||||||
|
| Intermediate size | 2,048 |
|
||||||
|
| Layers | 25 |
|
||||||
|
| Attention heads | 9 |
|
||||||
|
| Key/value heads | 3 |
|
||||||
|
| Activation | SiLU / SwiGLU feed-forward blocks |
|
||||||
|
| Normalization | RMSNorm, epsilon `1e-6` |
|
||||||
|
| Position encoding | RoPE, theta `10,000` |
|
||||||
|
| Maximum positions | 2,048 |
|
||||||
|
| Input/output embeddings | Tied |
|
||||||
|
|
||||||
|
## Conversation training
|
||||||
|
|
||||||
|
This checkpoint was fine-tuned for one epoch on `youndukn/ROLE_PLAY_INSTRUCT` using:
|
||||||
|
|
||||||
|
- 47,625 flattened training exchanges;
|
||||||
|
- 5,298 held-out test exchanges;
|
||||||
|
- optional character/persona context;
|
||||||
|
- one user turn followed by one assistant response;
|
||||||
|
- response-only loss with an explicit EOS target;
|
||||||
|
- a learning rate of `2.5e-5` and batch size 1.
|
||||||
|
|
||||||
|
The run recorded approximately 3.79 million processed tokens. Internal validation loss reached approximately **1.65** near the end of training. This is a dataset-specific next-token metric, not evidence of general conversational quality or safety.
|
||||||
|
|
||||||
|
The uploaded weights exactly match the final epoch-1 conversation checkpoint. Source dataset terms and attribution requirements remain applicable.
|
||||||
|
|
||||||
|
## Intended uses
|
||||||
|
|
||||||
|
- Single-turn character and role-play responses.
|
||||||
|
- Short conversational experiments with human supervision.
|
||||||
|
- Educational study of response-only dialogue fine-tuning.
|
||||||
|
- Local prototyping of character interfaces.
|
||||||
|
|
||||||
|
## Limitations and safety
|
||||||
|
|
||||||
|
Expected limitations include:
|
||||||
|
|
||||||
|
- no persistent memory and limited multi-turn consistency;
|
||||||
|
- character drift, repetition, role-label leakage, or speaking for the user;
|
||||||
|
- hallucinated facts and unreliable real-world advice;
|
||||||
|
- weak reasoning and instruction following outside role-play dialogue;
|
||||||
|
- sensitivity to persona wording and sampling settings;
|
||||||
|
- generation of stereotypes, manipulative dialogue, sexual content, offensive language, or other unsafe material inherited from training data;
|
||||||
|
- possible reproduction of phrases or information from source data.
|
||||||
|
|
||||||
|
Do not use this model for emotional dependency, impersonation, medical or mental-health support, high-impact advice, moderation, surveillance, or unsupervised public deployment. Make fictional framing clear to users and provide human oversight.
|
||||||
|
|
||||||
|
## Evaluation status
|
||||||
|
|
||||||
|
No standardized dialogue, role-play, factuality, bias, toxicity, privacy, or safety benchmarks are included. The internal validation loss only measures prediction on the held-out source split.
|
||||||
|
|
||||||
|
## Related variants
|
||||||
|
|
||||||
|
- **Base:** raw next-token completion checkpoint.
|
||||||
|
- **Instruct:** general single-turn instruction SFT branch with a different prompt wrapper.
|
||||||
|
- **Story:** creative-writing checkpoint that directly precedes this variant.
|
||||||
|
|
||||||
|
## License and attribution
|
||||||
|
|
||||||
|
No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' licenses, terms, or attribution requirements.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**YunaGPT-124M V1 Conversation is an experimental role-play model. Keep its use fictional, supervised, and transparent.**
|
||||||
33
config.json
Normal file
33
config.json
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"LlamaForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"dtype": "float32",
|
||||||
|
"eos_token_id": 1,
|
||||||
|
"head_dim": 64,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 576,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 2048,
|
||||||
|
"max_position_embeddings": 2048,
|
||||||
|
"mlp_bias": false,
|
||||||
|
"model_type": "llama",
|
||||||
|
"num_attention_heads": 9,
|
||||||
|
"num_hidden_layers": 25,
|
||||||
|
"num_key_value_heads": 3,
|
||||||
|
"pad_token_id": 3,
|
||||||
|
"pretraining_tp": 1,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_parameters": {
|
||||||
|
"rope_theta": 10000.0,
|
||||||
|
"rope_type": "default"
|
||||||
|
},
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"transformers_version": "5.13.1",
|
||||||
|
"unk_token_id": 2,
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 24000
|
||||||
|
}
|
||||||
10
generation_config.json
Normal file
10
generation_config.json
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 1,
|
||||||
|
"output_attentions": false,
|
||||||
|
"output_hidden_states": false,
|
||||||
|
"pad_token_id": 3,
|
||||||
|
"transformers_version": "5.13.1",
|
||||||
|
"use_cache": false
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:6a183fb3d3294dcafdf044cf08aa330f35b5754716e49fcbb80d59a1d2202308
|
||||||
|
size 497806792
|
||||||
119272
tokenizer.json
Normal file
119272
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
11
tokenizer_config.json
Normal file
11
tokenizer_config.json
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
{
|
||||||
|
"backend": "tokenizers",
|
||||||
|
"bos_token": "<|begin_of_text|>",
|
||||||
|
"eos_token": "<|end_of_text|>",
|
||||||
|
"is_local": true,
|
||||||
|
"local_files_only": false,
|
||||||
|
"model_max_length": 2048,
|
||||||
|
"pad_token": "<|pad|>",
|
||||||
|
"tokenizer_class": "TokenizersBackend",
|
||||||
|
"unk_token": "<|unk|>"
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user