AlexWortega/instruct_rugptlarge
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: AlexWortega/instruct_rugptlarge
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-04 03:19:18 +08:00
commit 986cbb2bfd
12 changed files with 150631 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
.gitattributes vendored Normal file
View File

@@ -0,0 +1,34 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

136
README.md Normal file
View File

@@ -0,0 +1,136 @@
---
datasets:
- IlyaGusev/ru_turbo_alpaca
inference:
parameters:
min_length: 20
max_new_tokens: 250
top_k: 50
top_p: 0.9
early_stopping: true
no_repeat_ngram_size: 2
use_cache: true
repetition_penalty: 1.5
length_penalty: 0.8
num_beams: 2
license: apache-2.0
language:
- ru
pipeline_tag: text-generation
widget:
- text: Может ли встретиться пингвин и белый медведь?
example_title: Question Answering
- text: Как зарабатывать много денег обучая модели? <instructionS>
example_title: Open domain Knoweledge
- text: Напиши на python код который выведет привет мир <code>
example_title: Code writing
- text: 'Переведи на русский и укажи язык оригинала: My name is Arthur.'
example_title: Zero shor translate
- text: >-
Квадратный корень из x равен кубическому корню из y. Чему равно y в степени
2, если x = 4?
example_title: Math example
library_name: transformers
tags:
- finance
- code
---
<h1 style="font-size: 42px">Instructions ruGPT large v0.11_25к_a<h1/>
# Model Summary
> Это ruGPTlarge дообученная в инструктивно-флановом сетапе, она более ли менее ZSшотиться и FSшотиться и работает лучше чем XGLM1.7b, mgpt на русском языке
# Quick Start
```python
from transformers import GPT2TokenizerFast,GPT2LMHeadModel
tokenizer = GPT2TokenizerFast.from_pretrained("AlexWortega/instruct_rugptlarge")
special_tokens_dict = {'additional_special_tokens': ['<code>', '</code>', '<instructionS>', '<instructionE>', '<next>']}
tokenizer.add_special_tokens(special_tokens_dict)
device = 'cuda'
model = GPT2LMHeadModel.from_pretrained("AlexWortega/instruct_rugptlarge")
model.to(device)
model.resize_token_embeddings(len(tokenizer))
def generate_seqs(q,model, k=2):
gen_kwargs = {
"min_length": 20,
"max_new_tokens": 100,
"top_k": 50,
"top_p": 0.7,
"do_sample": True,
"early_stopping": True,
"no_repeat_ngram_size": 2,
"eos_token_id": tokenizer.eos_token_id,
"pad_token_id": tokenizer.eos_token_id,
"use_cache": True,
"repetition_penalty": 1.5,
"length_penalty": 1.2,
"num_beams": 4,
"num_return_sequences": k
}
q = q + '<instructionS>'
t = tokenizer.encode(q, return_tensors='pt').to(device)
g = model.generate(t, **gen_kwargs)
generated_sequences = tokenizer.batch_decode(g, skip_special_tokens=True)
return generated_sequences
```
обратите внимание, что лучшие параметры для генерации
```
gen_kwargs = {
"min_length": 20,
"max_new_tokens": 100,
"top_k": 50,
"top_p": 0.9,
"do_sample": True,
"early_stopping": True,
"no_repeat_ngram_size": 2,
"eos_token_id": tokenizer.eos_token_id,
"pad_token_id": tokenizer.eos_token_id,
"use_cache": True,
"repetition_penalty": 1.5,
"length_penalty": 0.8,
"num_beams": 4,
"num_return_sequences": k
}
```
# License
The weights of Instructions ruGPT Small v0.1a are licensed under version 2.0 of the Apache License.
## Hyperparameters
I used Novograd with a learning rate of 2e-5 and global batch size of 6 (3 for each data parallel worker).
I use both data parallelism and pipeline parallelism to conduct training.
During training, we truncate the input sequence to 1024 tokens, and for input sequence that contains less than 1024 tokens, we concatenate multiple sequences into one long sequence to improve the data efficiency.
# References
#Metrics
ван дей пипл, ван дееей
## BibTeX entry and citation info
```bibtex
@article{
title={GPT2xl is underrated task solver},
author={Nickolich Aleksandr, 5Q, datascience, Ilya Gusev, Alex Kukushkin, Karina Romanova, Arseniy Shahmatov, Maksim Gersimenko},
year={2023}
}
```

8
added_tokens.json Normal file
View File

@@ -0,0 +1,8 @@
{
"</code>": 50259,
"<code>": 50258,
"<instructionE>": 50261,
"<instructionS>": 50260,
"<next>": 50262,
"<|endoftext|>": 50257
}

34
config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"_name_or_path": "ckpts19/sft_0_5000/",
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"gradient_checkpointing": false,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 2048,
"n_embd": 1536,
"n_head": 16,
"n_inner": null,
"n_layer": 24,
"n_positions": 2048,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"torch_dtype": "float16",
"transformers_version": "4.26.1",
"use_cache": true,
"vocab_size": 50263
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 50256,
"eos_token_id": 50256,
"transformers_version": "4.26.1"
}

49997
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6d7d892c4b3cf445c26876f48f8b610dd3660cf55d9d4193585d19cd5379ef52
size 3141935904

3
pytorch_model.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5acb8ac75dbf6ec5f534ab36662296b8923b98b0a0ddfdddab7e8fdd6c77f188
size 1621392669

30
special_tokens_map.json Normal file
View File

@@ -0,0 +1,30 @@
{
"additional_special_tokens": [
"<code>",
"</code>",
"<instructionS>",
"<instructionE>",
"<next>"
],
"bos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
}
}

100345
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

34
tokenizer_config.json Normal file
View File

@@ -0,0 +1,34 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"bos_token": {
"__type": "AddedToken",
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"eos_token": {
"__type": "AddedToken",
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
},
"errors": "replace",
"model_max_length": 1000000000000000019884624838656,
"name_or_path": "sberbank-ai/rugpt3large_based_on_gpt2",
"pad_token": null,
"special_tokens_map_file": null,
"tokenizer_class": "GPT2Tokenizer",
"unk_token": {
"__type": "AddedToken",
"content": "<|endoftext|>",
"lstrip": false,
"normalized": true,
"rstrip": false,
"single_word": false
}
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long