初始化项目，由ModelHub XC社区提供模型
Model: AI4PD/ProtGPT3-1.3B-dpo Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
57
README.md
Normal file
57
README.md
Normal file
@@ -0,0 +1,57 @@
|
||||
---
|
||||
library_name: transformers
|
||||
tags:
|
||||
- biology
|
||||
- protein-language-model
|
||||
- protein-generation
|
||||
- causal-lm
|
||||
- mixture-of-experts
|
||||
- transformers
|
||||
- dpo
|
||||
- alignment
|
||||
- protein-design
|
||||
---
|
||||
|
||||
# Model Card for ProtGPT3-1.3B-dpo
|
||||
|
||||
|
||||
## Model Description
|
||||
ProtGPT3-1.3M-dpo is the DPO-aligned version of [ProtGPT3-1.3B](https://huggingface.co/AI4PD/ProtGPT3-1.3B). It is part of the [ProtGPT3 family](https://huggingface.co/collections/AI4PD/protgpt3-family), an open-source suite of promptable and aligned protein language models for protein design.
|
||||
|
||||
ProtGPT3-1.3M-dpo was further aligned with Direct Preference Optimization (DPO) to improve generation quality. The alignment procedure shifts the model toward protein sequences with higher predicted structural confidence and reduced low-complexity content, while preserving sequence diversity.
|
||||
For protein generation, each model -dpo version is recommended over the base model.
|
||||
|
||||
For more info and guidance on how to generate sequences with ProtGPT3-1.3B-dpo check out the extensive description provided in [ProtGPT3-1.3B](https://huggingface.co/AI4PD/ProtGPT3-1.3B), just replacing the model name (i.e., `model_name=AI4PD/ProtGPT3-1.3B-dpo`).
|
||||
|
||||
## Out-of-Scope Use
|
||||
|
||||
The model should not be used as the sole basis for experimental, clinical, environmental, or safety-critical decisions. Generated sequences require downstream computational and experimental validation. The model is not guaranteed to generate functional, soluble, safe, synthesizable, or experimentally successful proteins.
|
||||
|
||||
The model should not be used for irresponsible or harmful biological design applications.
|
||||
|
||||
## Bias, Risks, and Limitations
|
||||
|
||||
ProtGPT3-1.3B-dpo learns from public protein sequence datasets and may reproduce biases present in those datasets. Although DPO alignment reduces low-complexity generations and improves generation quality according to the alignment objectives (pLDDT and reduction of lcr, as a binary objective, see main manuscript), generated sequences may still be nonfunctional, unstable, insoluble, repetitive, biologically implausible, or unsuitable for a user’s intended application.
|
||||
|
||||
The DPO alignment objective uses predicted structural confidence and low-complexity filtering as proxy objectives. These proxies do not guarantee biological function, experimental success, safety, solubility, or manufacturability.
|
||||
|
||||
As with other generative protein models, ProtGPT3-1.3B-dpo may present dual-use risks if applied irresponsibly.
|
||||
|
||||
|
||||
## Citation
|
||||
|
||||
**BibTeX:**
|
||||
|
||||
```bibtex
|
||||
@article{protgpt3,
|
||||
title={ProtGPT3: an Open-source family of Promptable and Aligned Protein Language Models},
|
||||
author={Anonymous Authors},
|
||||
year={2026}
|
||||
}
|
||||
```
|
||||
|
||||
## More Information
|
||||
|
||||
For more info and guidance on how to generate sequences with ProtGPT3-112M-dpo check out the extensive description provided in [ProtGPT3-1.3B](https://huggingface.co/AI4PD/ProtGPT3-1.3B), just replacing the model name (i.e., `model_name=AI4PD/ProtGPT3-1.3B-dpo`).
|
||||
|
||||
|
||||
33
config.json
Normal file
33
config.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"architectures": [
|
||||
"MixtralForCausalLM"
|
||||
],
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 2,
|
||||
"head_dim": null,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 1024,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 3072,
|
||||
"max_position_embeddings": 16384,
|
||||
"model_type": "mixtral",
|
||||
"num_attention_heads": 16,
|
||||
"num_experts_per_tok": 2,
|
||||
"num_hidden_layers": 17,
|
||||
"num_key_value_heads": 4,
|
||||
"num_local_experts": 8,
|
||||
"output_router_logits": false,
|
||||
"pad_token_id": 0,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_theta": 1000000.0,
|
||||
"router_aux_loss_coef": 0.001,
|
||||
"router_jitter_noise": 0.0,
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.53.3",
|
||||
"use_cache": true,
|
||||
"vocab_size": 31
|
||||
}
|
||||
9
generation_config.json
Normal file
9
generation_config.json
Normal file
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": [
|
||||
2
|
||||
],
|
||||
"pad_token_id": 0,
|
||||
"transformers_version": "4.53.3"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:048f0dc74bd6b3cf6aa00aa7f91b00b8a23174f309bd2b898f593442ad2b5235
|
||||
size 2656587200
|
||||
30
special_tokens_map.json
Normal file
30
special_tokens_map.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<|bos|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|eos|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
143
tokenizer.json
Normal file
143
tokenizer.json
Normal file
@@ -0,0 +1,143 @@
|
||||
{
|
||||
"version": "1.0",
|
||||
"truncation": null,
|
||||
"padding": null,
|
||||
"added_tokens": [
|
||||
{
|
||||
"id": 0,
|
||||
"content": "<|pad|>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false,
|
||||
"special": true
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"content": "<|bos|>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false,
|
||||
"special": true
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"content": "<|eos|>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false,
|
||||
"special": true
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"content": "[UNK]",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false,
|
||||
"special": true
|
||||
},
|
||||
{
|
||||
"id": 31,
|
||||
"content": "<s>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false,
|
||||
"special": true
|
||||
},
|
||||
{
|
||||
"id": 32,
|
||||
"content": "</s>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false,
|
||||
"special": true
|
||||
},
|
||||
{
|
||||
"id": 33,
|
||||
"content": "<unk>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false,
|
||||
"special": true
|
||||
}
|
||||
],
|
||||
"normalizer": null,
|
||||
"pre_tokenizer": {
|
||||
"type": "Split",
|
||||
"pattern": {
|
||||
"String": ""
|
||||
},
|
||||
"behavior": "Isolated",
|
||||
"invert": false
|
||||
},
|
||||
"post_processor": {
|
||||
"type": "TemplateProcessing",
|
||||
"single": [
|
||||
{
|
||||
"Sequence": {
|
||||
"id": "A",
|
||||
"type_id": 0
|
||||
}
|
||||
}
|
||||
],
|
||||
"pair": [
|
||||
{
|
||||
"Sequence": {
|
||||
"id": "A",
|
||||
"type_id": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"Sequence": {
|
||||
"id": "B",
|
||||
"type_id": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"special_tokens": {}
|
||||
},
|
||||
"decoder": null,
|
||||
"model": {
|
||||
"type": "WordLevel",
|
||||
"vocab": {
|
||||
"<|pad|>": 0,
|
||||
"<|bos|>": 1,
|
||||
"<|eos|>": 2,
|
||||
"[UNK]": 3,
|
||||
"1": 4,
|
||||
"2": 5,
|
||||
"A": 6,
|
||||
"B": 7,
|
||||
"C": 8,
|
||||
"D": 9,
|
||||
"E": 10,
|
||||
"F": 11,
|
||||
"G": 12,
|
||||
"H": 13,
|
||||
"I": 14,
|
||||
"K": 15,
|
||||
"L": 16,
|
||||
"M": 17,
|
||||
"N": 18,
|
||||
"O": 19,
|
||||
"P": 20,
|
||||
"Q": 21,
|
||||
"R": 22,
|
||||
"S": 23,
|
||||
"T": 24,
|
||||
"U": 25,
|
||||
"V": 26,
|
||||
"W": 27,
|
||||
"X": 28,
|
||||
"Y": 29,
|
||||
"Z": 30
|
||||
},
|
||||
"unk_token": "[UNK]"
|
||||
}
|
||||
}
|
||||
73
tokenizer_config.json
Normal file
73
tokenizer_config.json
Normal file
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_eos_token": false,
|
||||
"add_prefix_space": null,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<|pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<|bos|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "<|eos|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"3": {
|
||||
"content": "[UNK]",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"31": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"33": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"bos_token": "<|bos|>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|eos|>",
|
||||
"extra_special_tokens": {},
|
||||
"legacy": true,
|
||||
"model_max_length": 1025,
|
||||
"pad_token": "<|pad|>",
|
||||
"tokenizer_class": "LlamaTokenizerFast",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user