初始化项目，由ModelHub XC社区提供模型
Model: mario-rc/emotional-gpt2-medium Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
166
README.md
Normal file
166
README.md
Normal file
@@ -0,0 +1,166 @@
|
||||
---
|
||||
license: mit
|
||||
base_model: gpt2-medium
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- text-generation
|
||||
- gpt2
|
||||
- dialogue
|
||||
- emotion
|
||||
- causal-lm
|
||||
language:
|
||||
- en
|
||||
datasets:
|
||||
- DailyDialog
|
||||
metrics:
|
||||
- perplexity
|
||||
---
|
||||
|
||||
# Emotional GPT-2 Medium
|
||||
|
||||
`emotional-gpt2-medium` is a [GPT-2 Medium](https://huggingface.co/openai-community/gpt2-medium)
|
||||
causal language model fine-tuned for emotion-conditioned dialogue generation
|
||||
with DailyDialog-derived data.
|
||||
|
||||
GitHub repository: [`Mario-RC/emotional-gpt`](https://github.com/Mario-RC/emotional-gpt)
|
||||
|
||||
## Model Details
|
||||
|
||||
- Base model: [`gpt2-medium`](https://huggingface.co/openai-community/gpt2-medium)
|
||||
- Architecture: `GPT2LMHeadModel`
|
||||
- Task: text generation
|
||||
- Context length: 1024 tokens
|
||||
- Parameters: 354.8M
|
||||
- Evaluation perplexity: 10.0080
|
||||
|
||||
## Model Comparison
|
||||
|
||||
| Model | Base model | Parameters | Evaluation perplexity |
|
||||
| :--- | :---: | :---: | :---: |
|
||||
| [Emotional DistilGPT2](https://huggingface.co/mario-rc/emotional-distilgpt2) | [`distilgpt2`](https://huggingface.co/distilgpt2) | 81.9M | 15.3322 |
|
||||
| [Emotional GPT-2](https://huggingface.co/mario-rc/emotional-gpt2) | [`gpt2`](https://huggingface.co/openai-community/gpt2) | 124.4M | 12.9404 |
|
||||
| [Emotional GPT-2 Medium](https://huggingface.co/mario-rc/emotional-gpt2-medium) | [`gpt2-medium`](https://huggingface.co/openai-community/gpt2-medium) | 354.8M | 10.0080 |
|
||||
| [Emotional GPT-2 Large](https://huggingface.co/mario-rc/emotional-gpt2-large) | [`gpt2-large`](https://huggingface.co/openai-community/gpt2-large) | 774.0M | 7.4115 |
|
||||
| [Emotional DialoGPT Small](https://huggingface.co/mario-rc/emotional-dialogpt-small) | [`microsoft/DialoGPT-small`](https://huggingface.co/microsoft/DialoGPT-small) | 124.4M | 13.0488 |
|
||||
| [Emotional DialoGPT Medium](https://huggingface.co/mario-rc/emotional-dialogpt-medium) | [`microsoft/DialoGPT-medium`](https://huggingface.co/microsoft/DialoGPT-medium) | 354.8M | 10.5130 |
|
||||
| [Emotional DialoGPT Large](https://huggingface.co/mario-rc/emotional-dialogpt-large) | [`microsoft/DialoGPT-large`](https://huggingface.co/microsoft/DialoGPT-large) | 774.0M | 8.6719 |
|
||||
|
||||
## Training
|
||||
|
||||
The fine-tuning run used the following setup:
|
||||
|
||||
- Framework: Hugging Face Transformers
|
||||
- Training data: `data/gpt-dialogues/train.txt`; evaluation data: `data/gpt-dialogues/dev.txt`, built from DailyDialog CSV resources
|
||||
- Epochs: 4
|
||||
- Train/eval batch size per GPU: 6 / 6
|
||||
- Gradient accumulation steps: 1
|
||||
- Effective training batch size: 6
|
||||
- Learning rate: `1e-5`
|
||||
- Max gradient norm: `1.0`
|
||||
- Objective: line-by-line causal language modeling
|
||||
- Seed: `42`
|
||||
- Checkpointing/logging: every 5000 optimizer steps; last checkpoint kept
|
||||
- Memory optimization: gradient checkpointing not used
|
||||
|
||||
## Training Format
|
||||
|
||||
Training examples use adjacent DailyDialog utterance pairs with explicit source
|
||||
and target emotion labels:
|
||||
|
||||
```text
|
||||
<bos><source_emotion>source utterance<sep><target_emotion>target utterance<|endoftext|>
|
||||
```
|
||||
|
||||
## Prompt Format
|
||||
|
||||
At generation time, the prompt should include the source utterance and the
|
||||
desired target emotion:
|
||||
|
||||
```text
|
||||
<bos><source_emotion>source utterance<sep><target_emotion>
|
||||
```
|
||||
|
||||
Prompt and training tags:
|
||||
|
||||
- `<bos>` marks the beginning of one formatted dialogue example.
|
||||
- `<source_emotion>` is a placeholder for one emotion label describing the input/source utterance, for example `<fear>`.
|
||||
- `source utterance` is the user/input text.
|
||||
- `<sep>` separates the source side from the response side.
|
||||
- `<target_emotion>` is a placeholder for the emotion you want the generated response to follow, for example `<happiness>`.
|
||||
- `target utterance` is the response text generated by the model.
|
||||
- `<|endoftext|>` marks the end of one example. GPT-2 uses this as its native end-of-text/eos token, and generation can stop when this token is produced.
|
||||
|
||||
Emotion conditioning: replace `<source_emotion>` and `<target_emotion>` in the
|
||||
template with one of the model's literal emotion tokens in each position.
|
||||
|
||||
Supported emotion labels:
|
||||
|
||||
- `<no emotion>`
|
||||
- `<anger>`
|
||||
- `<disgust>`
|
||||
- `<fear>`
|
||||
- `<happiness>`
|
||||
- `<sadness>`
|
||||
- `<surprise>`
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
<bos><fear>I just started a new job and I am a bit nervous.<sep><happiness>
|
||||
```
|
||||
|
||||
This means: the source utterance expresses `fear`, and the requested response
|
||||
should be conditioned toward `happiness`.
|
||||
|
||||
## How to Use
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
repo_id = "mario-rc/emotional-gpt2-medium"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(repo_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(repo_id)
|
||||
model.config.pad_token_id = tokenizer.pad_token_id
|
||||
|
||||
prompt = "<bos><fear>I just started a new job and I am a bit nervous.<sep><happiness>"
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
do_sample=True,
|
||||
max_new_tokens=80,
|
||||
temperature=0.8,
|
||||
top_p=0.95,
|
||||
pad_token_id=tokenizer.pad_token_id,
|
||||
eos_token_id=tokenizer.eos_token_id,
|
||||
)
|
||||
|
||||
generated = outputs[0][inputs["input_ids"].shape[-1]:]
|
||||
response = tokenizer.decode(generated, skip_special_tokens=False)
|
||||
response = response.split(tokenizer.eos_token, 1)[0].strip()
|
||||
|
||||
emotion_labels = [
|
||||
"<no emotion>",
|
||||
"<anger>",
|
||||
"<disgust>",
|
||||
"<fear>",
|
||||
"<happiness>",
|
||||
"<sadness>",
|
||||
"<surprise>",
|
||||
]
|
||||
|
||||
for label in emotion_labels:
|
||||
if response.startswith(label):
|
||||
response = response[len(label):].strip()
|
||||
break
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
The model is intended for experimental dialogue/text generation. Generated text
|
||||
may be inaccurate, biased, repetitive, or emotionally inappropriate, and should
|
||||
be reviewed before user-facing use.
|
||||
12
added_tokens.json
Normal file
12
added_tokens.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"<anger>": 50259,
|
||||
"<bos>": 50266,
|
||||
"<disgust>": 50260,
|
||||
"<fear>": 50261,
|
||||
"<happiness>": 50262,
|
||||
"<no emotion>": 50258,
|
||||
"<pad>": 50265,
|
||||
"<sadness>": 50263,
|
||||
"<sep>": 50257,
|
||||
"<surprise>": 50264
|
||||
}
|
||||
40
config.json
Normal file
40
config.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"activation_function": "gelu_new",
|
||||
"architectures": [
|
||||
"GPT2LMHeadModel"
|
||||
],
|
||||
"attn_pdrop": 0.1,
|
||||
"bos_token_id": 50256,
|
||||
"dtype": "float32",
|
||||
"embd_pdrop": 0.1,
|
||||
"eos_token_id": 50256,
|
||||
"initializer_range": 0.02,
|
||||
"layer_norm_epsilon": 1e-05,
|
||||
"model_type": "gpt2",
|
||||
"n_ctx": 1024,
|
||||
"n_embd": 1024,
|
||||
"n_head": 16,
|
||||
"n_inner": null,
|
||||
"n_layer": 24,
|
||||
"n_positions": 1024,
|
||||
"n_special": 0,
|
||||
"predict_special_tokens": true,
|
||||
"reorder_and_upcast_attn": false,
|
||||
"resid_pdrop": 0.1,
|
||||
"scale_attn_by_inverse_layer_idx": false,
|
||||
"scale_attn_weights": true,
|
||||
"summary_activation": null,
|
||||
"summary_first_dropout": 0.1,
|
||||
"summary_proj_to_labels": true,
|
||||
"summary_type": "cls_index",
|
||||
"summary_use_proj": true,
|
||||
"task_specific_params": {
|
||||
"text-generation": {
|
||||
"do_sample": true,
|
||||
"max_length": 50
|
||||
}
|
||||
},
|
||||
"transformers_version": "4.57.3",
|
||||
"use_cache": true,
|
||||
"vocab_size": 50267
|
||||
}
|
||||
1
eval_results.txt
Normal file
1
eval_results.txt
Normal file
@@ -0,0 +1 @@
|
||||
perplexity = tensor(10.0080)
|
||||
6
generation_config.json
Normal file
6
generation_config.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 50256,
|
||||
"eos_token_id": 50256,
|
||||
"transformers_version": "4.57.3"
|
||||
}
|
||||
50001
merges.txt
Normal file
50001
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:29720a55d45eda85ab835104deb1d5481563b0dc6f102c177671d9af91043fce
|
||||
size 1419363840
|
||||
88
special_tokens_map.json
Normal file
88
special_tokens_map.json
Normal file
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
{
|
||||
"content": "<sep>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
{
|
||||
"content": "<no emotion>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
{
|
||||
"content": "<anger>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
{
|
||||
"content": "<disgust>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
{
|
||||
"content": "<fear>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
{
|
||||
"content": "<happiness>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
{
|
||||
"content": "<sadness>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
{
|
||||
"content": "<surprise>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
],
|
||||
"bos_token": {
|
||||
"content": "<bos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
113
tokenizer_config.json
Normal file
113
tokenizer_config.json
Normal file
@@ -0,0 +1,113 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"50256": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50257": {
|
||||
"content": "<sep>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50258": {
|
||||
"content": "<no emotion>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50259": {
|
||||
"content": "<anger>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50260": {
|
||||
"content": "<disgust>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50261": {
|
||||
"content": "<fear>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50262": {
|
||||
"content": "<happiness>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50263": {
|
||||
"content": "<sadness>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50264": {
|
||||
"content": "<surprise>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50265": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"50266": {
|
||||
"content": "<bos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"<sep>",
|
||||
"<no emotion>",
|
||||
"<anger>",
|
||||
"<disgust>",
|
||||
"<fear>",
|
||||
"<happiness>",
|
||||
"<sadness>",
|
||||
"<surprise>"
|
||||
],
|
||||
"bos_token": "<bos>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|endoftext|>",
|
||||
"errors": "replace",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 1024,
|
||||
"pad_token": "<pad>",
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": "<|endoftext|>"
|
||||
}
|
||||
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1989130aa5af585ff1413f5b72e0ee61c7926747dbe48ca5f82936b2e52afc08
|
||||
size 2641
|
||||
50259
vocab.json
Normal file
50259
vocab.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user