136 lines
5.1 KiB
Markdown
136 lines
5.1 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model: Qwen/Qwen2.5-7B-Instruct
|
|||
|
|
library_name: transformers
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
tags:
|
|||
|
|
- clinical-trials
|
|||
|
|
- healthcare
|
|||
|
|
- biomedical
|
|||
|
|
- information-extraction
|
|||
|
|
- qlora
|
|||
|
|
- peft
|
|||
|
|
- trl
|
|||
|
|
- grpo
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# ClinTrial-LM — Qwen2.5-7B fine-tuned for clinical-trial understanding
|
|||
|
|
|
|||
|
|
A **QLoRA**-fine-tuned [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)
|
|||
|
|
specialised for clinical-trial tasks, trained on ~26k instruction examples derived from the
|
|||
|
|
[ClinicalTrials.gov](https://clinicaltrials.gov) registry.
|
|||
|
|
|
|||
|
|
📦 **Code, full pipeline & write-up:** https://github.com/omkar-droid/clintrial-lm
|
|||
|
|
|
|||
|
|
> ⚕️ **Research/education only. Not medical advice.** Do not use for clinical decision-making.
|
|||
|
|
|
|||
|
|
## What it does
|
|||
|
|
|
|||
|
|
| Task | Description |
|
|||
|
|
|---|---|
|
|||
|
|
| **Eligibility extraction** | Free-text inclusion/exclusion criteria → structured JSON |
|
|||
|
|
| **Plain-language summary** | Trial description → patient-friendly summary |
|
|||
|
|
| **Condition Q&A** | "What conditions does this trial study?" |
|
|||
|
|
| **Phase classification** | Identify the trial phase |
|
|||
|
|
|
|||
|
|
## Results
|
|||
|
|
|
|||
|
|
Evaluated on a **held-out test set split by trial ID** (no trial appears in both train and test),
|
|||
|
|
500 sampled examples.
|
|||
|
|
|
|||
|
|
| Task | Metric | Base Qwen2.5-7B | **This model (SFT)** |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| Eligibility extraction | criterion F1 | 0.840 | **0.968** |
|
|||
|
|
| Eligibility output | **JSON validity** | 0.986 | **1.000** |
|
|||
|
|
| Phase classification | exact match | 0.000 | **0.794** |
|
|||
|
|
| Condition Q&A | token F1 | 0.298 | **0.742** |
|
|||
|
|
| Plain-language summary | ROUGE-L | 0.195 | **0.290** |
|
|||
|
|
|
|||
|
|
Eligibility extraction reaches **precision 0.962 / recall 0.978** with **100% schema-valid JSON**.
|
|||
|
|
|
|||
|
|
The base model scores 0.000 on phase classification not because it lacks the knowledge, but because
|
|||
|
|
it won't answer in the required format — fine-tuning buys **format discipline and faithfulness**.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import torch
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
|
|||
|
|
MODEL_ID = "OmkarShewale/clintrial-qwen2.5-7b-sft"
|
|||
|
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
|||
|
|
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16, device_map="auto")
|
|||
|
|
|
|||
|
|
SYSTEM = (
|
|||
|
|
"You are a clinical research assistant. You help patients and clinicians understand "
|
|||
|
|
"clinical trials. Answer only from the information provided, be precise, and never "
|
|||
|
|
"invent eligibility criteria, conditions, or outcomes."
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
criteria = """Inclusion Criteria:
|
|||
|
|
1. Adults aged 18 years or older with confirmed type 2 diabetes
|
|||
|
|
2. HbA1c between 7.0% and 10.5% at screening
|
|||
|
|
|
|||
|
|
Exclusion Criteria:
|
|||
|
|
1. History of severe hypoglycaemia within the last 6 months
|
|||
|
|
2. Pregnancy or breastfeeding
|
|||
|
|
"""
|
|||
|
|
|
|||
|
|
messages = [
|
|||
|
|
{"role": "system", "content": SYSTEM},
|
|||
|
|
{"role": "user", "content":
|
|||
|
|
'Extract the eligibility criteria from the trial text below into JSON with two lists, '
|
|||
|
|
'"inclusion" and "exclusion". Copy each criterion verbatim; do not add any.\n\n'
|
|||
|
|
f"Trial eligibility text:\n{criteria}"},
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
enc = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
|
|||
|
|
return_tensors="pt", return_dict=True).to(model.device)
|
|||
|
|
out = model.generate(**enc, max_new_tokens=1024, do_sample=False)
|
|||
|
|
print(tokenizer.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Note:** eligibility answers can be long (up to ~1.7k tokens). Use `max_new_tokens >= 1024` or the
|
|||
|
|
JSON will be truncated and fail to parse.
|
|||
|
|
|
|||
|
|
## Training
|
|||
|
|
|
|||
|
|
| | |
|
|||
|
|
|---|---|
|
|||
|
|
| Base model | Qwen2.5-7B-Instruct |
|
|||
|
|
| Method | QLoRA — 4-bit NF4 + LoRA (r=16, α=32, dropout 0.05) |
|
|||
|
|
| Target modules | q/k/v/o_proj, gate/up/down_proj |
|
|||
|
|
| Trainable params | ~0.5% of total |
|
|||
|
|
| Precision | bf16 compute, gradient checkpointing |
|
|||
|
|
| Effective batch | 32 (8 × 4 grad-accum) |
|
|||
|
|
| LR / schedule | 2e-4, cosine, 3% warmup |
|
|||
|
|
| Hardware | 1× NVIDIA H100 NVL (95 GB), ~19 GB used |
|
|||
|
|
| Framework | HuggingFace TRL + PEFT + Transformers |
|
|||
|
|
|
|||
|
|
**Best-checkpoint selection matters here:** eval loss bottomed around step 500 and then *rose* while
|
|||
|
|
train loss kept falling (overfitting). `load_best_model_at_end` on `eval_loss` selected the step-500
|
|||
|
|
checkpoint — this model. Training for 3 epochs was more than necessary.
|
|||
|
|
|
|||
|
|
A **GRPO / RLVR** variant (RL against a programmatic reward: JSON validity + criterion F1) was also
|
|||
|
|
trained; it matched this SFT model but did not beat it, because SFT had already saturated the reward.
|
|||
|
|
Details in the [repo](https://github.com/omkar-droid/clintrial-lm).
|
|||
|
|
|
|||
|
|
## Data
|
|||
|
|
|
|||
|
|
Built from 8,000 real ClinicalTrials.gov studies. Targets come from the registry's **own structured
|
|||
|
|
fields** (not a teacher LLM), so labels are auditable. Splits are partitioned **by trial**, not by
|
|||
|
|
example, to prevent leakage.
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- **Not medical advice.** Outputs may be wrong; a human expert must review anything clinical.
|
|||
|
|
- Trained on English registry text only; performance on other formats/languages is unknown.
|
|||
|
|
- Summary quality (ROUGE-L 0.29) is the weakest task.
|
|||
|
|
- Only 8k of ~500k available trials were used — more data would likely improve generalisation.
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
Apache-2.0, inherited from Qwen2.5. Source registry data is public-domain U.S. government work.
|