soilfm-qwen2.5-14b-literature-cpt/README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-14B-Instruct
library_name: transformers
tags:
  - soil-science
  - soil-microbiome
  - domain-adaptation
  - continued-pretraining
  - qlora
  - qwen2
pipeline_tag: text-generation
language:
  - en
model-index:
  - name: soilfm-qwen2.5-14b-literature-cpt
    results: []
---

# SoilFM Language Tower — Qwen2.5-14B Literature CPT

A domain-adapted large language model for **soil science and soil microbiology**, created by continued pretraining of [Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) on 200,000 curated soil science text passages.

This model is the **Language Tower** component of [SoilFM2](https://github.com/northenlab), a multi-modal foundation model for soil microbiome analysis developed at Lawrence Berkeley National Laboratory.

## Model Details

| | |
|---|---|
| **Base model** | Qwen/Qwen2.5-14B-Instruct (14.2B parameters) |
| **Method** | Continued pretraining via QLoRA (4-bit NF4) |
| **Format** | Full merged model (LoRA weights merged into base) |
| **Precision** | BF16 |
| **Context length** | 32,768 tokens |
| **Size on disk** | ~28 GB |
| **LoRA adapter** | Also available at [northenlab/soilfm-qwen2.5-14b-qlora](https://huggingface.co/northenlab/soilfm-qwen2.5-14b-qlora) (263 MB) |

## Intended Uses

- Generating explanations of soil microbial processes, rhizosphere ecology, and plant-microbe interactions
- Providing domain-grounded context within the SoilFM2 multi-modal pipeline (prebiotic recommendation, substrate preference prediction)
- Serving as a soil-science-aware backbone for downstream fine-tuning or RAG systems
- Research and educational applications in soil microbiology

## Training Data

The training corpus was assembled from four sources of soil science domain knowledge, stratified-sampled to 200,000 training examples and 10,000 validation examples (seed = 42):

| Source | Description | Proportion | Train | Val |
|--------|-------------|:----------:|------:|----:|
| **PubMed Central** | Full-text soil microbiology papers (39,853 articles) | 55% | 110,000 | 5,500 |
| **Wikipedia** | Soil science articles | 20% | 40,000 | 2,000 |
| **USDA Soil Survey Manual** | Official USDA technical reference | 10% | 20,000 | 1,000 |
| **Wikipedia General Biology** | Broad biology context to prevent catastrophic forgetting | 15% | 30,000 | 1,500 |
| **Total** | | **100%** | **200,000** | **10,000** |

Text was chunked to 1,024 tokens with 100-token overlap. The full corpus contained 329M tokens across 388,563 chunks; the 200K stratified subsample was used for this training run.

### Preprocessing

- PubMed Central articles retrieved via BioC JSON API, cleaned of XML artifacts
- Soil Survey Manual cleaned of page headers, footers, and index content (57 of 435 chunks removed)
- All sources standardized to JSONL format with `text` and `source` fields

## Training Procedure

### Configuration

| Parameter | Value |
|-----------|-------|
| Quantization | 4-bit NF4 (BitsAndBytes, double quantization) |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Rank-stabilized LoRA (rsLoRA) | Yes |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Trainable parameters | ~263M (1.85% of total) |
| Optimizer | PagedAdamW8bit |
| Learning rate | 2e-5 (cosine schedule, 10% warmup) |
| Effective batch size | 128 (micro-batch 2, gradient accumulation 64) |
| Max gradient norm | 1.0 |
| Weight decay | 0.01 |
| Precision | BF16 mixed precision |
| Flash Attention 2 | Enabled |
| Epochs | 1 |
| Total steps | 1,500 |

### Infrastructure

- **GPU**: NVIDIA A100 PCIe 80GB (RunPod)
- **Training time**: ~42 hours
- **Peak VRAM**: 54 GB (67% utilization)

### Training Script

A custom manual PyTorch training loop was used (rather than HuggingFace Trainer) for compatibility and control. The script is available in the project repository as `train_soilfm_cpt_MANUAL.py`.

## Results

### Validation Loss

| Step | Validation Loss |
|-----:|:---------------:|
| 500 | 1.7369 |
| 1,000 | 1.6281 |
| 1,500 | **1.6130** |

**Total improvement: 7.2%** over the course of training. Loss was still decreasing at the end of training with no signs of overfitting. Gradient norms remained stable in the 0.2–0.8 range throughout.

### Qualitative Evaluation

**Prompt:** *"The role of root exudates in shaping rhizosphere microbial communities involves"*

**Output:** The model produces coherent, technically accurate continuations using appropriate domain terminology (root exudates, rhizosphere, primary/secondary metabolites, phytohormones), demonstrating successful domain adaptation.

## Usage

### Direct Loading (Recommended)

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "northenlab/soilfm-qwen2.5-14b-literature-cpt",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "northenlab/soilfm-qwen2.5-14b-literature-cpt"
)

prompt = "The role of mycorrhizal fungi in soil nutrient cycling"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### With vLLM (Production)

```bash
python3 -m vllm.entrypoints.openai.api_server \
  --model northenlab/soilfm-qwen2.5-14b-literature-cpt \
  --host 0.0.0.0 --port 8001
```

### LoRA Adapter Only

If you prefer to load the adapter separately (e.g., for 4-bit inference):

```python
from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-14B-Instruct",
    load_in_4bit=True,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "northenlab/soilfm-qwen2.5-14b-qlora")
```

## Part of SoilFM2

This Language Tower works alongside other SoilFM2 components:

| Component | Description | HuggingFace |
|-----------|-------------|-------------|
| **Language Tower** (this model) | Domain-adapted LLM | [northenlab/soilfm-qwen2.5-14b-literature-cpt](https://huggingface.co/northenlab/soilfm-qwen2.5-14b-literature-cpt) |
| **Graph Tower** | Heterogeneous GNN on 2.39M-node knowledge graph | [northenlab/soilfm2-graph-tower-joint-v0.1](https://huggingface.co/northenlab/soilfm2-graph-tower-joint-v0.1) |
| **BSPR** | Bayesian substrate preference model (AUC 0.94) | — |

Together these components power a prebiotic recommendation pipeline that takes 16S microbiome profiles as input and suggests soil amendments to steer community function.

## Use Restrictions

This model is intended for **research and non-commercial use only**. The training corpus includes PubMed Central Open Access articles under various Creative Commons licenses, some of which may carry non-commercial (CC BY-NC) terms. Users should ensure their use complies with the underlying data licenses.

The base model (Qwen2.5-14B-Instruct) is released under the Apache 2.0 license.

## Limitations

- Trained for 1 epoch on a 200K subsample of the full 667K corpus; additional training may further improve performance
- Domain adaptation was evaluated primarily via validation loss and qualitative generation; systematic benchmarking on soil science Q&A tasks is ongoing
- The model inherits the base Qwen2.5-14B-Instruct capabilities and limitations
- Not intended for medical, agricultural, or regulatory decision-making without expert review

## Citation

```bibtex
@misc{soilfm-language-tower-2025,
  title={SoilFM Language Tower: Domain Adaptation of Qwen2.5-14B for Soil Science},
  author={Northen Lab, Lawrence Berkeley National Laboratory},
  year={2025},
  publisher={HuggingFace},
  url={https://huggingface.co/northenlab/soilfm-qwen2.5-14b-literature-cpt},
  note={Continued pretraining on 200K soil science literature examples via QLoRA}
}
```

## License

Apache 2.0 (inherited from the base Qwen2.5-14B-Instruct model). Training data includes PubMed Central Open Access articles under various CC licenses — see **Use Restrictions** above.