初始化项目，由ModelHub XC社区提供模型
Model: northenlab/soilfm-qwen2.5-14b-literature-cpt Source: Original Platform
This commit is contained in:
ModelHub XC
203
README.md
Normal file
203
README.md
Normal file
@@ -0,0 +1,203 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model: Qwen/Qwen2.5-14B-Instruct
|
||||
library_name: transformers
|
||||
tags:
|
||||
- soil-science
|
||||
- soil-microbiome
|
||||
- domain-adaptation
|
||||
- continued-pretraining
|
||||
- qlora
|
||||
- qwen2
|
||||
pipeline_tag: text-generation
|
||||
language:
|
||||
- en
|
||||
model-index:
|
||||
- name: soilfm-qwen2.5-14b-literature-cpt
|
||||
results: []
|
||||
---
|
||||
|
||||
# SoilFM Language Tower — Qwen2.5-14B Literature CPT
|
||||
|
||||
A domain-adapted large language model for **soil science and soil microbiology**, created by continued pretraining of [Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) on 200,000 curated soil science text passages.
|
||||
|
||||
This model is the **Language Tower** component of [SoilFM2](https://github.com/northenlab), a multi-modal foundation model for soil microbiome analysis developed at Lawrence Berkeley National Laboratory.
|
||||
|
||||
## Model Details
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Base model** | Qwen/Qwen2.5-14B-Instruct (14.2B parameters) |
|
||||
| **Method** | Continued pretraining via QLoRA (4-bit NF4) |
|
||||
| **Format** | Full merged model (LoRA weights merged into base) |
|
||||
| **Precision** | BF16 |
|
||||
| **Context length** | 32,768 tokens |
|
||||
| **Size on disk** | ~28 GB |
|
||||
| **LoRA adapter** | Also available at [northenlab/soilfm-qwen2.5-14b-qlora](https://huggingface.co/northenlab/soilfm-qwen2.5-14b-qlora) (263 MB) |
|
||||
|
||||
## Intended Uses
|
||||
|
||||
- Generating explanations of soil microbial processes, rhizosphere ecology, and plant-microbe interactions
|
||||
- Providing domain-grounded context within the SoilFM2 multi-modal pipeline (prebiotic recommendation, substrate preference prediction)
|
||||
- Serving as a soil-science-aware backbone for downstream fine-tuning or RAG systems
|
||||
- Research and educational applications in soil microbiology
|
||||
|
||||
## Training Data
|
||||
|
||||
The training corpus was assembled from four sources of soil science domain knowledge, stratified-sampled to 200,000 training examples and 10,000 validation examples (seed = 42):
|
||||
|
||||
| Source | Description | Proportion | Train | Val |
|
||||
|--------|-------------|:----------:|------:|----:|
|
||||
| **PubMed Central** | Full-text soil microbiology papers (39,853 articles) | 55% | 110,000 | 5,500 |
|
||||
| **Wikipedia** | Soil science articles | 20% | 40,000 | 2,000 |
|
||||
| **USDA Soil Survey Manual** | Official USDA technical reference | 10% | 20,000 | 1,000 |
|
||||
| **Wikipedia General Biology** | Broad biology context to prevent catastrophic forgetting | 15% | 30,000 | 1,500 |
|
||||
| **Total** | | **100%** | **200,000** | **10,000** |
|
||||
|
||||
Text was chunked to 1,024 tokens with 100-token overlap. The full corpus contained 329M tokens across 388,563 chunks; the 200K stratified subsample was used for this training run.
|
||||
|
||||
### Preprocessing
|
||||
|
||||
- PubMed Central articles retrieved via BioC JSON API, cleaned of XML artifacts
|
||||
- Soil Survey Manual cleaned of page headers, footers, and index content (57 of 435 chunks removed)
|
||||
- All sources standardized to JSONL format with `text` and `source` fields
|
||||
|
||||
## Training Procedure
|
||||
|
||||
### Configuration
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Quantization | 4-bit NF4 (BitsAndBytes, double quantization) |
|
||||
| LoRA rank | 16 |
|
||||
| LoRA alpha | 32 |
|
||||
| LoRA dropout | 0.05 |
|
||||
| Rank-stabilized LoRA (rsLoRA) | Yes |
|
||||
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
|
||||
| Trainable parameters | ~263M (1.85% of total) |
|
||||
| Optimizer | PagedAdamW8bit |
|
||||
| Learning rate | 2e-5 (cosine schedule, 10% warmup) |
|
||||
| Effective batch size | 128 (micro-batch 2, gradient accumulation 64) |
|
||||
| Max gradient norm | 1.0 |
|
||||
| Weight decay | 0.01 |
|
||||
| Precision | BF16 mixed precision |
|
||||
| Flash Attention 2 | Enabled |
|
||||
| Epochs | 1 |
|
||||
| Total steps | 1,500 |
|
||||
|
||||
### Infrastructure
|
||||
|
||||
- **GPU**: NVIDIA A100 PCIe 80GB (RunPod)
|
||||
- **Training time**: ~42 hours
|
||||
- **Peak VRAM**: 54 GB (67% utilization)
|
||||
|
||||
### Training Script
|
||||
|
||||
A custom manual PyTorch training loop was used (rather than HuggingFace Trainer) for compatibility and control. The script is available in the project repository as `train_soilfm_cpt_MANUAL.py`.
|
||||
|
||||
## Results
|
||||
|
||||
### Validation Loss
|
||||
|
||||
| Step | Validation Loss |
|
||||
|-----:|:---------------:|
|
||||
| 500 | 1.7369 |
|
||||
| 1,000 | 1.6281 |
|
||||
| 1,500 | **1.6130** |
|
||||
|
||||
**Total improvement: 7.2%** over the course of training. Loss was still decreasing at the end of training with no signs of overfitting. Gradient norms remained stable in the 0.2–0.8 range throughout.
|
||||
|
||||
### Qualitative Evaluation
|
||||
|
||||
**Prompt:** *"The role of root exudates in shaping rhizosphere microbial communities involves"*
|
||||
|
||||
**Output:** The model produces coherent, technically accurate continuations using appropriate domain terminology (root exudates, rhizosphere, primary/secondary metabolites, phytohormones), demonstrating successful domain adaptation.
|
||||
|
||||
## Usage
|
||||
|
||||
### Direct Loading (Recommended)
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
"northenlab/soilfm-qwen2.5-14b-literature-cpt",
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(
|
||||
"northenlab/soilfm-qwen2.5-14b-literature-cpt"
|
||||
)
|
||||
|
||||
prompt = "The role of mycorrhizal fungi in soil nutrient cycling"
|
||||
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(**inputs, max_new_tokens=256)
|
||||
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
### With vLLM (Production)
|
||||
|
||||
```bash
|
||||
python3 -m vllm.entrypoints.openai.api_server \
|
||||
--model northenlab/soilfm-qwen2.5-14b-literature-cpt \
|
||||
--host 0.0.0.0 --port 8001
|
||||
```
|
||||
|
||||
### LoRA Adapter Only
|
||||
|
||||
If you prefer to load the adapter separately (e.g., for 4-bit inference):
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM
|
||||
from peft import PeftModel
|
||||
|
||||
base = AutoModelForCausalLM.from_pretrained(
|
||||
"Qwen/Qwen2.5-14B-Instruct",
|
||||
load_in_4bit=True,
|
||||
device_map="auto",
|
||||
)
|
||||
model = PeftModel.from_pretrained(base, "northenlab/soilfm-qwen2.5-14b-qlora")
|
||||
```
|
||||
|
||||
## Part of SoilFM2
|
||||
|
||||
This Language Tower works alongside other SoilFM2 components:
|
||||
|
||||
| Component | Description | HuggingFace |
|
||||
|-----------|-------------|-------------|
|
||||
| **Language Tower** (this model) | Domain-adapted LLM | [northenlab/soilfm-qwen2.5-14b-literature-cpt](https://huggingface.co/northenlab/soilfm-qwen2.5-14b-literature-cpt) |
|
||||
| **Graph Tower** | Heterogeneous GNN on 2.39M-node knowledge graph | [northenlab/soilfm2-graph-tower-joint-v0.1](https://huggingface.co/northenlab/soilfm2-graph-tower-joint-v0.1) |
|
||||
| **BSPR** | Bayesian substrate preference model (AUC 0.94) | — |
|
||||
|
||||
Together these components power a prebiotic recommendation pipeline that takes 16S microbiome profiles as input and suggests soil amendments to steer community function.
|
||||
|
||||
## Use Restrictions
|
||||
|
||||
This model is intended for **research and non-commercial use only**. The training corpus includes PubMed Central Open Access articles under various Creative Commons licenses, some of which may carry non-commercial (CC BY-NC) terms. Users should ensure their use complies with the underlying data licenses.
|
||||
|
||||
The base model (Qwen2.5-14B-Instruct) is released under the Apache 2.0 license.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Trained for 1 epoch on a 200K subsample of the full 667K corpus; additional training may further improve performance
|
||||
- Domain adaptation was evaluated primarily via validation loss and qualitative generation; systematic benchmarking on soil science Q&A tasks is ongoing
|
||||
- The model inherits the base Qwen2.5-14B-Instruct capabilities and limitations
|
||||
- Not intended for medical, agricultural, or regulatory decision-making without expert review
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{soilfm-language-tower-2025,
|
||||
title={SoilFM Language Tower: Domain Adaptation of Qwen2.5-14B for Soil Science},
|
||||
author={Northen Lab, Lawrence Berkeley National Laboratory},
|
||||
year={2025},
|
||||
publisher={HuggingFace},
|
||||
url={https://huggingface.co/northenlab/soilfm-qwen2.5-14b-literature-cpt},
|
||||
note={Continued pretraining on 200K soil science literature examples via QLoRA}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0 (inherited from the base Qwen2.5-14B-Instruct model). Training data includes PubMed Central Open Access articles under various CC licenses — see **Use Restrictions** above.
|
||||
Reference in New Issue
Block a user