102 lines
5.0 KiB
Markdown
102 lines
5.0 KiB
Markdown
---
|
||
license: apache-2.0
|
||
pipeline_tag: text-generation
|
||
widget:
|
||
- text: <|endoftext|>
|
||
inference:
|
||
parameters:
|
||
top_k: 950
|
||
repetition_penalty: 1.2
|
||
---
|
||
|
||
# **GPepT: A Language Model for Peptides and Peptidomimetics**
|
||
|
||
|
||
GPepT is a cutting-edge language model designed to understand and generate sequences in the specialized domain of peptides and peptidomimetics. It serves as a powerful tool for _de novo_ protein design and engineering. As demonstrated in our research, the incorporation of peptidomimetics significantly broadens the chemical space accessible through generated sequences, enabling innovative approaches to peptide-based therapeutics.
|
||
|
||
## **Model Overview**
|
||
GPepT builds upon the GPT-2 Transformer architecture, comprising 36 layers and a model dimensionality of 1280, with a total of 738 million parameters. This decoder-only model has been pre-trained on a curated dataset of peptides and peptidomimetics mined from bioactivity-labeled chemical formulas in ChEMBL.
|
||
|
||
To leverage GPepT’s pre-trained weights, input molecules must be converted into a standardized sequence-like representation of peptidomimetics using [**Monomerizer**](https://github.com/tsudalab/Monomerizer/tree/main). Detailed insights into the training process and datasets are provided in our accompanying publication.
|
||
|
||
Unlike traditional protein design models, GPepT is trained in a self-supervised manner, using raw sequence data without explicit annotation. This design enables the model to generalize across diverse sequence spaces, producing functional antimicrobial peptidomimetics upon fine-tuning.
|
||
|
||
SMILES representation, and selected chemical properties of each token, which corresponds to a non-canonical amino acid or terminal modification.
|
||
|
||
---
|
||
|
||
## **Using GPepT for Sequence Generation**
|
||
GPepT is fully compatible with the HuggingFace Transformers Python library. Installation instructions can be found [here](https://huggingface.co/docs/transformers/installation).
|
||
|
||
The model excels at generating peptidomimetic sequences in a zero-shot fashion, but it can also be fine-tuned on custom datasets to generate sequences tailored to specific requirements.
|
||
|
||
|
||
### **Example 1: Zero-Shot Sequence Generation**
|
||
GPepT generates sequences that extend from a specified input token (e.g., `<|endoftext|>`). If no input is provided, it selects the start token automatically and generates likely sequences. Here’s a Python example:
|
||
|
||
```python
|
||
from transformers import pipeline
|
||
|
||
# Initialize GPepT for text generation
|
||
GPepT = pipeline('text-generation', model="Playingyoyo/GPepT")
|
||
|
||
# Generate sequences
|
||
sequences = GPepT("<|endoftext|>",
|
||
max_length=25,
|
||
do_sample=True,
|
||
top_k=950,
|
||
repetition_penalty=1.5,
|
||
num_return_sequences=5,
|
||
eos_token_id=0)
|
||
|
||
# Print generated sequences
|
||
for seq in sequences:
|
||
print(seq['generated_text'])
|
||
```
|
||
|
||
Sample output:
|
||
```
|
||
<|endoftext|>R K A L E Z1649
|
||
<|endoftext|>G K A L Z341
|
||
<|endoftext|>G V A G K X4097 V A P
|
||
```
|
||
|
||
---
|
||
|
||
### **Example 2: Fine-Tuning for Directed Sequence Generation**
|
||
Fine-tuning enables GPepT to generate sequences with user-defined properties. To prepare training data:
|
||
1. ```git clone https://github.com/tsudalab/Monomerizer/tree/main```
|
||
2. ```cd Monomerizer```
|
||
3. ```python3 Monomerizer/run_pipeline.py --input_file path_to_your_smiles_file.txt```. Check the repo for the required format.
|
||
4. 3. will monomerize the SMILES and split the resulting sequences into training (`output/datetime/for_GPepT/train90.txt`) and validation (`output/datetime/for_GPepT/val10.txt`) files.
|
||
|
||
To fine-tune the model:
|
||
|
||
```bash
|
||
python run_clm.py --model_name_or_path Playingyoyo/GPepT \
|
||
--train_file path_to_train90.txt \
|
||
--validation_file path_to_val10.txt \
|
||
--tokenizer_name Playingyoyo/GPepT \
|
||
--do_train \
|
||
--do_eval \
|
||
--output_dir ./output \
|
||
--learning_rate 1e-5
|
||
```
|
||
|
||
Refer to the HuggingFace [script run_clm.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_clm.py) and [requirements.txt](https://huggingface.co/Playingyoyo/GPepT/blob/main/requirements.txt).
|
||
Note that train90.txt and val10.txt need to be at least 50 samples long.
|
||
|
||
The fine-tuned model will be saved in the `./output` directory, ready to generate tailored sequences.
|
||
|
||
---
|
||
|
||
## **Selecting Valid Sequences**
|
||
While GPepT generates diverse peptidomimetic sequences, not all are chemically valid. For example:
|
||
- **Invalid Sequences:** Those with terminal modifications (e.g., `Z`) embedded within the sequence.
|
||
- **Valid Sequences:** Should adhere to standard peptidomimetic rules.
|
||
|
||
By filtering out invalid sequences, GPepT users can ensure the generation of high-quality candidates for further study.
|
||
|
||
---
|
||
|
||
GPepT stands as a powerful tool for researchers at the forefront of peptide and peptidomimetic innovation, enabling both exploration and application in vast chemical and biological spaces. |