---
language:
- en
license: apache-2.0
library_name: transformers
model_type: qwen2
tags:
- biology
- protein-language-model
- saprot
- 3Di
- enzymeml
- reinforcement-learning
datasets:
- westlake-repl/AF2_UniRef50
pipeline_tag: text-generation
---

# Qwen2 SaPROT-3Di CLM for GH114

## Model Description
This is a **Qwen2-style** protein language model trained on **SaPROT 3Di-aware** protein sequences. Unlike SaPROT it is a CLM rather than a MLM, so it's generative (This becomes useful for DPO and the TRL-trainer from HF).

This model serves as a specialized base model designed for **GH114 reinforcement alignment**. It captures the structural and sequence properties of glycoside hydrolase family 114 (GH114) enzymes and their structural neighbors.

This model was specifically developed for the **AMLD Intelligence Summit 2026 EnzymeML workshop**.

## Training Details

### Pre-training
The model was pre-trained on the [westlake-repl/AF2_UniRef50](https://huggingface.co/datasets/westlake-repl/AF2_UniRef50) dataset. This provides a robust foundation of protein structure-sequence understanding using the SaPROT 3Di alphabet. Batch Size 896 with 512 sequence length @ 10k Steps (smol training run). 4.58 billion-tokens. Final Train Loss 3.3809 Validation Loss 3.4621.

### Fine-tuning
Following pre-training, the model was fine-tuned on a curated dataset of **≈700,000 structural homologs**. These homologs were selected based on shared **InterPro domains** with the GH114 dataset (IPR004352, IPR017853, IPR013785, IPR000254), ensuring the model is highly sensitive to the structural motifs relevant to this specific enzyme family. Anything within 90% sequence identity from the 55 GH114 sequences was removed from the training set. Two validation sets were used concurrently to monitor distribution overfitting (i.i.d) and the out-of-distribution generalization on the homologs of interest.
4k Steps. 896 batch size, 512 max len. Train Loss 1.7648 Validation Loss 1.8568.

## Intended Use
* **Primary Use:** As a base model for Reinforcement Learning (RL) alignment tasks targeting the FLOPP GH114 enzymes. log p(x).
* **Context:** AMLD Intelligence Summit 2026 (EnzymeML Workshop).
* **Input:** 3Di-encoded protein sequences (structure-aware tokens).

## How to Use
You can load this model using the Hugging Face `transformers` library. 

*Note: Ensure your input sequences are converted to the 3Di format (Foldseek alphabet) before passing them to the model.*

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model_name = "NorseDrunkenSailor/Qwen_smol_GH114"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)

# Example input (3Di sequence)
sequence = "M#L#HdSdLdLdAdAdSdFdAd" 
inputs = tokenizer(sequence, return_tensors="pt")

# Generate continuation or embeddings
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Acknowledgements & Citations

This model relies on the 3Di alphabet from Foldeek and the SaProt idea of using these concatenated 3Di-sequence tokens in a PLM.

'''bibtex
@article{su2023saprot,
  title={SaProt: Protein Language Modeling with Structure-aware Vocabulary},
  author={Su, Jin and Han, Chenchen and Zhou, Yuyang and Shan, Junjie and Zhou, Xibin and Yuan, Fajie},
  journal={bioRxiv},
  year={2023},
  publisher={Cold Spring Harbor Laboratory}
}

@article{van2023foldseek,
  title={Foldseek: fast and accurate protein structure search},
  author={van Kempen, Michel and et al.},
  journal={Nature Biotechnology},
  year={2024}
}
'''