84 lines
3.6 KiB
Markdown
84 lines
3.6 KiB
Markdown
|
---
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
license: apache-2.0
|
|||
|
|
library_name: transformers
|
|||
|
|
model_type: qwen2
|
|||
|
|
tags:
|
|||
|
|
- biology
|
|||
|
|
- protein-language-model
|
|||
|
|
- saprot
|
|||
|
|
- 3Di
|
|||
|
|
- enzymeml
|
|||
|
|
- reinforcement-learning
|
|||
|
|
datasets:
|
|||
|
|
- westlake-repl/AF2_UniRef50
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# Qwen2 SaPROT-3Di CLM for GH114
|
|||
|
|
|
|||
|
|
## Model Description
|
|||
|
|
This is a **Qwen2-style** protein language model trained on **SaPROT 3Di-aware** protein sequences. Unlike SaPROT it is a CLM rather than a MLM, so it's generative (This becomes useful for DPO and the TRL-trainer from HF).
|
|||
|
|
|
|||
|
|
This model serves as a specialized base model designed for **GH114 reinforcement alignment**. It captures the structural and sequence properties of glycoside hydrolase family 114 (GH114) enzymes and their structural neighbors.
|
|||
|
|
|
|||
|
|
This model was specifically developed for the **AMLD Intelligence Summit 2026 EnzymeML workshop**.
|
|||
|
|
|
|||
|
|
## Training Details
|
|||
|
|
|
|||
|
|
### Pre-training
|
|||
|
|
The model was pre-trained on the [westlake-repl/AF2_UniRef50](https://huggingface.co/datasets/westlake-repl/AF2_UniRef50) dataset. This provides a robust foundation of protein structure-sequence understanding using the SaPROT 3Di alphabet. Batch Size 896 with 512 sequence length @ 10k Steps (smol training run). 4.58 billion-tokens. Final Train Loss 3.3809 Validation Loss 3.4621.
|
|||
|
|
|
|||
|
|
### Fine-tuning
|
|||
|
|
Following pre-training, the model was fine-tuned on a curated dataset of **≈700,000 structural homologs**. These homologs were selected based on shared **InterPro domains** with the GH114 dataset (IPR004352, IPR017853, IPR013785, IPR000254), ensuring the model is highly sensitive to the structural motifs relevant to this specific enzyme family. Anything within 90% sequence identity from the 55 GH114 sequences was removed from the training set. Two validation sets were used concurrently to monitor distribution overfitting (i.i.d) and the out-of-distribution generalization on the homologs of interest.
|
|||
|
|
4k Steps. 896 batch size, 512 max len. Train Loss 1.7648 Validation Loss 1.8568.
|
|||
|
|
|
|||
|
|
## Intended Use
|
|||
|
|
* **Primary Use:** As a base model for Reinforcement Learning (RL) alignment tasks targeting the FLOPP GH114 enzymes. log p(x).
|
|||
|
|
* **Context:** AMLD Intelligence Summit 2026 (EnzymeML Workshop).
|
|||
|
|
* **Input:** 3Di-encoded protein sequences (structure-aware tokens).
|
|||
|
|
|
|||
|
|
## How to Use
|
|||
|
|
You can load this model using the Hugging Face `transformers` library.
|
|||
|
|
|
|||
|
|
*Note: Ensure your input sequences are converted to the 3Di format (Foldseek alphabet) before passing them to the model.*
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
# Load model and tokenizer
|
|||
|
|
model_name = "NorseDrunkenSailor/Qwen_smol_GH114"
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)
|
|||
|
|
|
|||
|
|
# Example input (3Di sequence)
|
|||
|
|
sequence = "M#L#HdSdLdLdAdAdSdFdAd"
|
|||
|
|
inputs = tokenizer(sequence, return_tensors="pt")
|
|||
|
|
|
|||
|
|
# Generate continuation or embeddings
|
|||
|
|
outputs = model.generate(**inputs, max_new_tokens=200)
|
|||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Acknowledgements & Citations
|
|||
|
|
|
|||
|
|
This model relies on the 3Di alphabet from Foldeek and the SaProt idea of using these concatenated 3Di-sequence tokens in a PLM.
|
|||
|
|
|
|||
|
|
'''bibtex
|
|||
|
|
@article{su2023saprot,
|
|||
|
|
title={SaProt: Protein Language Modeling with Structure-aware Vocabulary},
|
|||
|
|
author={Su, Jin and Han, Chenchen and Zhou, Yuyang and Shan, Junjie and Zhou, Xibin and Yuan, Fajie},
|
|||
|
|
journal={bioRxiv},
|
|||
|
|
year={2023},
|
|||
|
|
publisher={Cold Spring Harbor Laboratory}
|
|||
|
|
}
|
|||
|
|
|
|||
|
|
@article{van2023foldseek,
|
|||
|
|
title={Foldseek: fast and accurate protein structure search},
|
|||
|
|
author={van Kempen, Michel and et al.},
|
|||
|
|
journal={Nature Biotechnology},
|
|||
|
|
year={2024}
|
|||
|
|
}
|
|||
|
|
'''
|