初始化项目，由ModelHub XC社区提供模型
Model: meadbee/Yuna-130M-Conversation Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
Assets/info.png filter=lfs diff=lfs merge=lfs -text
|
||||
3
Assets/info.png
Normal file
3
Assets/info.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c2101769c75a627f728f920120bf5822486f1f29be765079a088ffd460cf4318
|
||||
size 1800014
|
||||
218
README.md
Normal file
218
README.md
Normal file
@@ -0,0 +1,218 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- llama
|
||||
- causal-lm
|
||||
- gqa
|
||||
- conversational
|
||||
- roleplay
|
||||
- fine-tuned
|
||||
- yuna
|
||||
---
|
||||
|
||||
# YunaGPT-124M V1 Conversation
|
||||
|
||||
**A compact, single-turn conversation and character role-play model.**
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
> **Important:** This model was trained to produce one assistant or character reply from one user turn. It has no built-in memory and is not a reliable factual or safety-aligned assistant.
|
||||
|
||||
<p align="center">
|
||||
<img src="Assets/info.png" alt="YunaGPT-124M V1 architecture and training infographic" width="600">
|
||||
</p>
|
||||
|
||||
## Run on google Colab for free online:
|
||||
[](
|
||||
https://colab.research.google.com/github/jordanmckin/Yuna_130m_Colab/blob/main/Yuna_130M_Conversation_Colab.ipynb
|
||||
)
|
||||
|
||||
|
||||
## Overview
|
||||
|
||||
YunaGPT-124M V1 Conversation continues from the final Story checkpoint and adds response-only conversation SFT. Its specialty is short dialogue and character role-play: an optional character description and one user message are provided as context, and the model generates one assistant reply.
|
||||
|
||||
The training data was deliberately flattened into independent user/assistant exchanges. The model was not trained to predict both sides of an ongoing transcript, and it does not remember earlier calls unless an application supplies context manually.
|
||||
|
||||
## Project background
|
||||
|
||||
Yuna began as a 30M-parameter educational language-model project inspired by Sebastian Raschka's *Build a Large Language Model (From Scratch)*. It later moved to Hugging Face's native LLaMA implementation and became a home-hardware experiment using an RTX 3090. The broader project explores instruction following, creative writing, role-play, synthetic Final Fantasy X knowledge, and preference optimization.
|
||||
|
||||
The model may write plausible franchise dialogue or lore while inventing details. Do not rely on it as a factual source.
|
||||
|
||||
## Model summary
|
||||
|
||||
| Item | Value |
|
||||
|---|---:|
|
||||
| Parameters | **124,445,376** |
|
||||
| Model class | `LlamaForCausalLM` |
|
||||
| Training stage | Character/conversation SFT |
|
||||
| Lineage | Base → Story → Conversation |
|
||||
| Training exchanges | **47,625** |
|
||||
| Validation exchanges | **5,298** |
|
||||
| Recorded conversation tokens | Approximately **3.79 million** |
|
||||
| Context length | **2,048 tokens** |
|
||||
| Maximum trained user turn | 600 tokens |
|
||||
| Maximum trained response | 512 tokens |
|
||||
| Vocabulary | **24,000 tokens** |
|
||||
| Tokenizer | Byte-level BPE |
|
||||
| Weight format | `safetensors`, FP32 |
|
||||
| Primary language | English |
|
||||
|
||||
## Prompt format
|
||||
|
||||
This checkpoint uses a custom role-play format, not the instruction format used by the Instruct and Story variants.
|
||||
|
||||
With a character description:
|
||||
|
||||
```text
|
||||
### Character:
|
||||
{character_or_persona}
|
||||
|
||||
### User:
|
||||
{message}
|
||||
|
||||
### Assistant:
|
||||
```
|
||||
|
||||
Without a character description:
|
||||
|
||||
```text
|
||||
### User:
|
||||
{message}
|
||||
|
||||
### Assistant:
|
||||
```
|
||||
|
||||
The headings and blank lines are part of the training format. A standard `tokenizer.apply_chat_template(...)` call is not equivalent for this release.
|
||||
|
||||
## Run it yourself
|
||||
|
||||
```bash
|
||||
pip install torch transformers
|
||||
```
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_id = "YOUR_USERNAME/YunaGPT-124M-V1-Conversation"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
|
||||
model.eval()
|
||||
|
||||
|
||||
def format_conversation_prompt(message: str, character: str = "") -> str:
|
||||
parts = []
|
||||
if character.strip():
|
||||
parts.append(f"### Character:\n{character.strip()}")
|
||||
parts.append(f"### User:\n{message.strip()}")
|
||||
parts.append("### Assistant:\n")
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
prompt = format_conversation_prompt(
|
||||
message="I found this strange key outside the library. Do you recognize it?",
|
||||
character=(
|
||||
"Mira is a friendly village librarian who is curious, patient, "
|
||||
"and fond of old adventure stories."
|
||||
),
|
||||
)
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
with torch.no_grad():
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=220,
|
||||
do_sample=True,
|
||||
temperature=0.8,
|
||||
top_p=0.9,
|
||||
repetition_penalty=1.1,
|
||||
pad_token_id=tokenizer.pad_token_id,
|
||||
eos_token_id=tokenizer.eos_token_id,
|
||||
)
|
||||
|
||||
new_tokens = output[0, inputs["input_ids"].shape[1]:]
|
||||
print(tokenizer.decode(new_tokens, skip_special_tokens=True).strip())
|
||||
```
|
||||
|
||||
Replace the placeholder repository name with the final model ID or a local folder. Keeping replies below roughly 512 new tokens is closest to the response lengths seen during training.
|
||||
|
||||
## Architecture
|
||||
|
||||
| Component | Configuration |
|
||||
|---|---:|
|
||||
| Architecture | Decoder-only Transformer |
|
||||
| Attention | Grouped-Query Attention (GQA) |
|
||||
| Hidden size | 576 |
|
||||
| Intermediate size | 2,048 |
|
||||
| Layers | 25 |
|
||||
| Attention heads | 9 |
|
||||
| Key/value heads | 3 |
|
||||
| Activation | SiLU / SwiGLU feed-forward blocks |
|
||||
| Normalization | RMSNorm, epsilon `1e-6` |
|
||||
| Position encoding | RoPE, theta `10,000` |
|
||||
| Maximum positions | 2,048 |
|
||||
| Input/output embeddings | Tied |
|
||||
|
||||
## Conversation training
|
||||
|
||||
This checkpoint was fine-tuned for one epoch on `youndukn/ROLE_PLAY_INSTRUCT` using:
|
||||
|
||||
- 47,625 flattened training exchanges;
|
||||
- 5,298 held-out test exchanges;
|
||||
- optional character/persona context;
|
||||
- one user turn followed by one assistant response;
|
||||
- response-only loss with an explicit EOS target;
|
||||
- a learning rate of `2.5e-5` and batch size 1.
|
||||
|
||||
The run recorded approximately 3.79 million processed tokens. Internal validation loss reached approximately **1.65** near the end of training. This is a dataset-specific next-token metric, not evidence of general conversational quality or safety.
|
||||
|
||||
The uploaded weights exactly match the final epoch-1 conversation checkpoint. Source dataset terms and attribution requirements remain applicable.
|
||||
|
||||
## Intended uses
|
||||
|
||||
- Single-turn character and role-play responses.
|
||||
- Short conversational experiments with human supervision.
|
||||
- Educational study of response-only dialogue fine-tuning.
|
||||
- Local prototyping of character interfaces.
|
||||
|
||||
## Limitations and safety
|
||||
|
||||
Expected limitations include:
|
||||
|
||||
- no persistent memory and limited multi-turn consistency;
|
||||
- character drift, repetition, role-label leakage, or speaking for the user;
|
||||
- hallucinated facts and unreliable real-world advice;
|
||||
- weak reasoning and instruction following outside role-play dialogue;
|
||||
- sensitivity to persona wording and sampling settings;
|
||||
- generation of stereotypes, manipulative dialogue, sexual content, offensive language, or other unsafe material inherited from training data;
|
||||
- possible reproduction of phrases or information from source data.
|
||||
|
||||
Do not use this model for emotional dependency, impersonation, medical or mental-health support, high-impact advice, moderation, surveillance, or unsupervised public deployment. Make fictional framing clear to users and provide human oversight.
|
||||
|
||||
## Evaluation status
|
||||
|
||||
No standardized dialogue, role-play, factuality, bias, toxicity, privacy, or safety benchmarks are included. The internal validation loss only measures prediction on the held-out source split.
|
||||
|
||||
## Related variants
|
||||
|
||||
- **Base:** raw next-token completion checkpoint.
|
||||
- **Instruct:** general single-turn instruction SFT branch with a different prompt wrapper.
|
||||
- **Story:** creative-writing checkpoint that directly precedes this variant.
|
||||
|
||||
## License and attribution
|
||||
|
||||
No model-weight license was declared in the project metadata when this card was prepared. Add an explicit license before public distribution. A model license does not override the source datasets' licenses, terms, or attribution requirements.
|
||||
|
||||
---
|
||||
|
||||
**YunaGPT-124M V1 Conversation is an experimental role-play model. Keep its use fictional, supervised, and transparent.**
|
||||
33
config.json
Normal file
33
config.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 0,
|
||||
"dtype": "float32",
|
||||
"eos_token_id": 1,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 576,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 2048,
|
||||
"max_position_embeddings": 2048,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 9,
|
||||
"num_hidden_layers": 25,
|
||||
"num_key_value_heads": 3,
|
||||
"pad_token_id": 3,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 10000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "5.13.1",
|
||||
"unk_token_id": 2,
|
||||
"use_cache": true,
|
||||
"vocab_size": 24000
|
||||
}
|
||||
10
generation_config.json
Normal file
10
generation_config.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 1,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"pad_token_id": 3,
|
||||
"transformers_version": "5.13.1",
|
||||
"use_cache": false
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6a183fb3d3294dcafdf044cf08aa330f35b5754716e49fcbb80d59a1d2202308
|
||||
size 497806792
|
||||
119272
tokenizer.json
Normal file
119272
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
11
tokenizer_config.json
Normal file
11
tokenizer_config.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<|begin_of_text|>",
|
||||
"eos_token": "<|end_of_text|>",
|
||||
"is_local": true,
|
||||
"local_files_only": false,
|
||||
"model_max_length": 2048,
|
||||
"pad_token": "<|pad|>",
|
||||
"tokenizer_class": "TokenizersBackend",
|
||||
"unk_token": "<|unk|>"
|
||||
}
|
||||
Reference in New Issue
Block a user