Deeokay/GPT2-medium-custom-v1.0
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Deeokay/GPT2-medium-custom-v1.0
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-07-25 11:09:11 +08:00
commit c75def83bf
13 changed files with 100775 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

38
.gitattributes vendored Normal file
View File

@@ -0,0 +1,38 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
gpt2-medium-custom-v1.0_bf32.gguf filter=lfs diff=lfs merge=lfs -text
gpt2-medium-custom-v1.0_f16.gguf filter=lfs diff=lfs merge=lfs -text
gpt2-medium-custom-v1.0_Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

241
README.md Normal file
View File

@@ -0,0 +1,241 @@
---
library_name: transformers
tags: []
---
# SUMMARY
Just a model using to learn Fine Tuning of 'gpt2-medium'
- on a self made datasets
- on a self made special tokens
- on a multiple fine tuned with ~15K dataset (in progress mode)
If interested in how I got to this point and how I created the datasets you can visit:
[Crafting GPT2 for Personalized AI-Preparing Data the Long Way](https://medium.com/@deeokay/the-soul-in-the-machine-crafting-gpt2-for-personalized-ai-9d38be3f635f)
<!-- Provide a quick summary of what the model is/does. -->
# FINE TUNED - BASE MODEL
I would consider this [GPT2-medium-custom-v1.0](https://huggingface.co/Deeokay/GPT2-medium-custom-v1.0) a the base model to start my Fine Tuning 2.0 on specific Datasets.
- Previous models of this: gpt-special-tokens-medium(1~4) are consider beta check-points to this
This model is available to test on Ollama [Deeokay/mediumgpt2](https://ollama.com/deeokay/mediumgpt2) it is not perfect and I am still working out some stuff, but I am quite proud that I was able to make it this far.
Please note, the acutal GGUF file is also included in this repository if you would like to create your own versions (templates etc.)
## DECLARING NEW SPECIAL TOKENS
```python
special_tokens_dict = {
'eos_token': '<|STOP|>',
'bos_token': '<|STOP|>',
'pad_token': '<|PAD|>',
'additional_special_tokens': ['<|BEGIN_QUERY|>', '<|BEGIN_QUERY|>',
'<|BEGIN_ANALYSIS|>', '<|END_ANALYSIS|>',
'<|BEGIN_RESPONSE|>', '<|END_RESPONSE|>',
'<|BEGIN_SENTIMENT|>', '<|END_SENTIMENT|>',
'<|BEGIN_CLASSIFICATION|>', '<|END_CLASSIFICATION|>',]
}
tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))
tokenizer.eos_token_id = tokenizer.convert_tokens_to_ids('<|STOP|>')
tokenizer.bos_token_id = tokenizer.convert_tokens_to_ids('<|STOP|>')
tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids('<|PAD|>')
```
The order of tokens is as follows:
```python
def combine_text(user_prompt, analysis, sentiment, new_response, classification):
user_q = f"<|STOP|><|BEGIN_QUERY|>{user_prompt}<|END_QUERY|>"
analysis = f"<|BEGIN_ANALYSIS|>{analysis}<|END_ANALYSIS|>"
new_response = f"<|BEGIN_RESPONSE|>{new_response}<|END_RESPONSE|>"
classification = f"<|BEGIN_CLASSIFICATION|>{classification}<|END_CLASSIFICATION|>"
sentiment = f"<|BEGIN_SENTIMENT|>Sentiment: {sentiment}<|END_SENTIMENT|><|STOP|>"
return user_q + analysis + new_response + classification + sentiment
```
## INFERANCING
I am currently testing two ways, if anyone knows a better one, please let me know!
```python
import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer
models_folder = "Deeokay/gpt2-medium-custom-v1.0"
model = GPT2LMHeadModel.from_pretrained(models_folder)
tokenizer = GPT2Tokenizer.from_pretrained(models_folder)
# Device configuration <<change as needed>>
device = torch.device("cpu")
model.to(device)
```
### OPTION 1 INFERFENCE
```python
import time
class Stopwatch:
def __init__(self):
self.start_time = None
self.end_time = None
def start(self):
self.start_time = time.time()
def stop(self):
self.end_time = time.time()
def elapsed_time(self):
if self.start_time is None:
return "Stopwatch hasn't been started"
if self.end_time is None:
return "Stopwatch hasn't been stopped"
return self.end_time - self.start_time
stopwatch1 = Stopwatch()
def generate_response(input_text, max_length=250):
stopwatch1.start()
# Prepare the input
# input_text = f"<|BEGIN_QUERY|>{input_text}<|END_QUERY|><|BEGIN_ANALYSIS|>{input_text}<|END_ANALYSIS|><|BEGIN_RESPONSE|>"
input_text = f"<|BEGIN_QUERY|>{input_text}<|END_QUERY|><|BEGIN_ANALYSIS|>"
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(device)
# Create attention mask
attention_mask = torch.ones_like(input_ids).to(device)
# Generate
output = model.generate(
input_ids,
max_new_tokens=max_length,
num_return_sequences=1,
no_repeat_ngram_size=2,
attention_mask=attention_mask,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.convert_tokens_to_ids('<|STOP|>'),
)
stopwatch1.stop()
return tokenizer.decode(output[0], skip_special_tokens=False)
```
### OPTION 2 INFERNCE
```python
import time
class Stopwatch:
def __init__(self):
self.start_time = None
self.end_time = None
def start(self):
self.start_time = time.time()
def stop(self):
self.end_time = time.time()
def elapsed_time(self):
if self.start_time is None:
return "Stopwatch hasn't been started"
if self.end_time is None:
return "Stopwatch hasn't been stopped"
return self.end_time - self.start_time
stopwatch2 = Stopwatch()
def generate_response2(input_text, max_length=250):
stopwatch2.start()
# Prepare the input
# input_text = f"<|BEGIN_QUERY|>{input_text}<|END_QUERY|><|BEGIN_ANALYSIS|>{input_text}<|END_ANALYSIS|><|BEGIN_RESPONSE|>"
input_text = f"<|BEGIN_QUERY|>{input_text}<|END_QUERY|><|BEGIN_ANALYSIS|>"
input_ids = tokenizer.encode(input_text, return_tensors="pt").to(device)
# Create attention mask
attention_mask = torch.ones_like(input_ids).to(device)
# # 2ND OPTION FOR : Generate
output = model.generate(
input_ids,
max_new_tokens=max_length,
attention_mask=attention_mask,
do_sample=True,
temperature=0.4, # this can be played around
top_k=60, # this can be played around
no_repeat_ngram_size=2,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
stopwatch2.stop()
return tokenizer.decode(output[0], skip_special_tokens=False)
```
### DECODING ANSWER
When I need just the response
```python
def decode(text):
full_text = text
# Extract the response part
start_token = "<|BEGIN_RESPONSE|>"
end_token = "<|END_RESPONSE|>"
start_idx = full_text.find(start_token)
end_idx = full_text.find(end_token)
if start_idx != -1 and end_idx != -1:
response = full_text[start_idx + len(start_token):end_idx].strip()
else:
response = full_text.strip()
return response
```
### MY SETUP
I use the stopwatch to time the responses and I use both inference to see the difference
```python
input_text = "Who is Steve Jobs and what was contribution?"
response1_full = generate_response(input_text)
#response1 = decode(response1_full)
print(f"Input: {input_text}")
print("=======================================")
print(f"Response1: {response1_full}")
elapsed1 = stopwatch1.elapsed_time()
print(f"Process took {elapsed1:.4f} seconds")
print("=======================================")
response2_full = generate_response2(input_text)
#response2 = decode(response2_full)
print(f"Response2: {response2_full}")
elapsed2 = stopwatch2.elapsed_time()
print(f"Process took {elapsed2:.4f} seconds")
print("=======================================")
```
### Out-of-Scope Use
Well everything that has a factual data.. trust at your own risk!
Never tested on mathamatical knowledge.
I quite enjoy how the response feels closer to what I had in mind..

13
added_tokens.json Normal file
View File

@@ -0,0 +1,13 @@
{
"<|BEGIN_ANALYSIS|>": 50260,
"<|BEGIN_CLASSIFICATION|>": 50266,
"<|BEGIN_QUERY|>": 50259,
"<|BEGIN_RESPONSE|>": 50262,
"<|BEGIN_SENTIMENT|>": 50264,
"<|END_ANALYSIS|>": 50261,
"<|END_CLASSIFICATION|>": 50267,
"<|END_RESPONSE|>": 50263,
"<|END_SENTIMENT|>": 50265,
"<|PAD|>": 50258,
"<|STOP|>": 50257
}

41
config.json Normal file
View File

@@ -0,0 +1,41 @@
{
"_name_or_path": "/Users/deeokay/Documents/LLMs/GPT2/hf_models/GPT2-medium-v1.2",
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 1024,
"n_head": 16,
"n_inner": null,
"n_layer": 24,
"n_positions": 1024,
"n_special": 0,
"predict_special_tokens": true,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.41.0",
"use_cache": true,
"vocab_size": 50268
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"transformers_version": "4.41.0"
}

3
gpt2-medium-custom-v1.0_Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ae5aebd5bf5e54a76d04a263025b2452383e6c5157af20d681713a5e0b5cf334
size 437511616

3
gpt2-medium-custom-v1.0_bf32.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:54849e8212984113c2a5811a40750859acdfb7372829f9fabb81fd70fb6b5f77
size 1627019200

3
gpt2-medium-custom-v1.0_f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2772c9523e44a7d5b2e309278e1707e4c3bdc3bc8bce811d6d7c94e4878ca373
size 817141696

50001
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b2337475b5d9d2dc627ce2e5cff2b4710318807f9e43ea6783bd0599eb12cdff
size 1419367936

42
special_tokens_map.json Normal file
View File

@@ -0,0 +1,42 @@
{
"additional_special_tokens": [
"<|BEGIN_QUERY|>",
"<|BEGIN_QUERY|>",
"<|BEGIN_ANALYSIS|>",
"<|END_ANALYSIS|>",
"<|BEGIN_RESPONSE|>",
"<|END_RESPONSE|>",
"<|BEGIN_SENTIMENT|>",
"<|END_SENTIMENT|>",
"<|BEGIN_CLASSIFICATION|>",
"<|END_CLASSIFICATION|>"
],
"bos_token": {
"content": "<|STOP|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|STOP|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|PAD|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
}
}

122
tokenizer_config.json Normal file
View File

@@ -0,0 +1,122 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"50256": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false,
"special": true
},
"50257": {
"content": "<|STOP|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50258": {
"content": "<|PAD|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50259": {
"content": "<|BEGIN_QUERY|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50260": {
"content": "<|BEGIN_ANALYSIS|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50261": {
"content": "<|END_ANALYSIS|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50262": {
"content": "<|BEGIN_RESPONSE|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50263": {
"content": "<|END_RESPONSE|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50264": {
"content": "<|BEGIN_SENTIMENT|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50265": {
"content": "<|END_SENTIMENT|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50266": {
"content": "<|BEGIN_CLASSIFICATION|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"50267": {
"content": "<|END_CLASSIFICATION|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|BEGIN_QUERY|>",
"<|BEGIN_QUERY|>",
"<|BEGIN_ANALYSIS|>",
"<|END_ANALYSIS|>",
"<|BEGIN_RESPONSE|>",
"<|END_RESPONSE|>",
"<|BEGIN_SENTIMENT|>",
"<|END_SENTIMENT|>",
"<|BEGIN_CLASSIFICATION|>",
"<|END_CLASSIFICATION|>"
],
"bos_token": "<|STOP|>",
"clean_up_tokenization_spaces": true,
"eos_token": "<|STOP|>",
"errors": "replace",
"model_max_length": 1024,
"pad_token": "<|PAD|>",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}

50259
vocab.json Normal file
View File

File diff suppressed because it is too large Load Diff