76 lines
3.7 KiB
Markdown
76 lines
3.7 KiB
Markdown
|
---
|
||
|
|
license: cc-by-nc-4.0
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
base_model: facebook/galactica-125m
|
||
|
|
tags:
|
||
|
|
- chemistry
|
||
|
|
- molecular-generation
|
||
|
|
- molecular-property-prediction
|
||
|
|
- arxiv:2407.18897
|
||
|
|
---
|
||
|
|
|
||
|
|
# Chemlactica-125M
|
||
|
|
|
||
|
|
Chemlactica-125M is a 125M-parameter causal language model for molecular generation, property conditioning, property prediction, and optimization. It continues pretraining from [Galactica-125M](https://huggingface.co/facebook/galactica-125m) on [PubChemForLM](https://huggingface.co/datasets/yerevann/PubChemForLM): 40 billion tokens describing more than 110 million PubChem molecules, computed properties, and molecular similarities.
|
||
|
|
|
||
|
|
## Model details
|
||
|
|
|
||
|
|
- Architecture: `OPTForCausalLM`; 12 layers, hidden size 768, 12 attention heads
|
||
|
|
- Context length: 2,048 tokens
|
||
|
|
- Training objective: causal language modeling
|
||
|
|
- Training-data cutoff: January 26, 2023
|
||
|
|
- Molecular preprocessing: RDKit-canonicalized SMILES; numeric properties rounded to two decimals
|
||
|
|
|
||
|
|
## Prompt format
|
||
|
|
|
||
|
|
Chemlactica prompts begin with `</s>`. Generate a molecule by ending a structured property prefix with `[START_SMILES]`:
|
||
|
|
|
||
|
|
```text
|
||
|
|
</s>[SAS]2.25[/SAS][SIMILAR]CC(=O)OC1=CC=CC=C1C(=O)O 0.62[/SIMILAR][START_SMILES]
|
||
|
|
```
|
||
|
|
|
||
|
|
The similarity field is `reference SMILES`, a space, then the target ECFP Tanimoto similarity. To predict a property, provide the molecule and open the desired property tag:
|
||
|
|
|
||
|
|
```text
|
||
|
|
</s>[START_SMILES]CC(=O)OC1=CC=CC=C1C(=O)O[END_SMILES][SAS]
|
||
|
|
```
|
||
|
|
|
||
|
|
Common trained tags include `[SAS]`, `[QED]`, `[WEIGHT]`, `[TPSA]`, `[CLOGP]`, `[FORMULA]`, `[RINGCOUNT]`, and `[SIMILAR]`. Use only tags present in the training format; arbitrary natural-language instructions or oracle names are unsupported.
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model_id = "yerevann/chemlactica-125m"
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
||
|
|
model_id, device_map="auto", torch_dtype="auto"
|
||
|
|
)
|
||
|
|
|
||
|
|
prompt = "</s>[QED]0.80[/QED][START_SMILES]"
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
|
|
outputs = model.generate(
|
||
|
|
**inputs, max_new_tokens=128, do_sample=True, temperature=0.8, top_p=0.95
|
||
|
|
)
|
||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
|
||
|
|
```
|
||
|
|
|
||
|
|
For optimization, use the [Mol-E implementation and canonical PMO prompt registry](https://github.com/YerevaNN/ChemLactica/tree/main/chemlactica/mol_opt). In the five-seed PMO-10K reproduction, this checkpoint obtains summed Top-10 AUC scores of 16.944598 (task-agnostic) and 20.429421 (task-informed) across 23 tasks.
|
||
|
|
|
||
|
|
## Intended use and limitations
|
||
|
|
|
||
|
|
This checkpoint is intended for research on small-molecule language modeling and optimization. Generated strings may be invalid, duplicated, chemically implausible, difficult to synthesize, unsafe, or outside the requested property range. Reported properties are predictions or optimization-oracle outputs, not experimental measurements. PubChem-derived data may contain errors or biases, and the model has no knowledge of changes after the cutoff date. Do not use it for clinical decisions or as a substitute for experimental and expert review.
|
||
|
|
|
||
|
|
## Resources and license
|
||
|
|
|
||
|
|
- Paper: [Small Molecule Optimization with Large Language Models](https://arxiv.org/abs/2407.18897)
|
||
|
|
- Code: [YerevaNN/ChemLactica](https://github.com/YerevaNN/ChemLactica)
|
||
|
|
- Data: [yerevann/PubChemForLM](https://huggingface.co/datasets/yerevann/PubChemForLM)
|
||
|
|
- Family: [Chemlactica-1.3B](https://huggingface.co/yerevann/chemlactica-1.3b), [Chemma-2B](https://huggingface.co/yerevann/chemma-2b)
|
||
|
|
|
||
|
|
The checkpoint is released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). Users must also comply with applicable terms of the base model and source data.
|