license, base_model, library_name, pipeline_tag, language, tags

license	base_model	library_name	pipeline_tag	language	tags
apache-2.0	Qwen/Qwen2.5-7B-Instruct	transformers	text-generation	en	clinical-trials healthcare biomedical information-extraction qlora peft trl grpo

ClinTrial-LM — Qwen2.5-7B fine-tuned for clinical-trial understanding

A QLoRA-fine-tuned Qwen2.5-7B-Instruct specialised for clinical-trial tasks, trained on ~26k instruction examples derived from the ClinicalTrials.gov registry.

📦 Code, full pipeline & write-up: https://github.com/omkar-droid/clintrial-lm

⚕️ Research/education only. Not medical advice. Do not use for clinical decision-making.

What it does

Task	Description
Eligibility extraction	Free-text inclusion/exclusion criteria → structured JSON
Plain-language summary	Trial description → patient-friendly summary
Condition Q&A	"What conditions does this trial study?"
Phase classification	Identify the trial phase

Results

Evaluated on a held-out test set split by trial ID (no trial appears in both train and test), 500 sampled examples.

Task	Metric	Base Qwen2.5-7B	This model (SFT)
Eligibility extraction	criterion F1	0.840	0.968
Eligibility output	JSON validity	0.986	1.000
Phase classification	exact match	0.000	0.794
Condition Q&A	token F1	0.298	0.742
Plain-language summary	ROUGE-L	0.195	0.290

Eligibility extraction reaches precision 0.962 / recall 0.978 with 100% schema-valid JSON.

The base model scores 0.000 on phase classification not because it lacks the knowledge, but because it won't answer in the required format — fine-tuning buys format discipline and faithfulness.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "OmkarShewale/clintrial-qwen2.5-7b-sft"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16, device_map="auto")

SYSTEM = (
    "You are a clinical research assistant. You help patients and clinicians understand "
    "clinical trials. Answer only from the information provided, be precise, and never "
    "invent eligibility criteria, conditions, or outcomes."
)

criteria = """Inclusion Criteria:
1. Adults aged 18 years or older with confirmed type 2 diabetes
2. HbA1c between 7.0% and 10.5% at screening

Exclusion Criteria:
1. History of severe hypoglycaemia within the last 6 months
2. Pregnancy or breastfeeding
"""

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content":
        'Extract the eligibility criteria from the trial text below into JSON with two lists, '
        '"inclusion" and "exclusion". Copy each criterion verbatim; do not add any.\n\n'
        f"Trial eligibility text:\n{criteria}"},
]

enc = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
                                    return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**enc, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Note: eligibility answers can be long (up to ~1.7k tokens). Use max_new_tokens >= 1024 or the JSON will be truncated and fail to parse.

Training

Base model	Qwen2.5-7B-Instruct
Method	QLoRA — 4-bit NF4 + LoRA (r=16, α=32, dropout 0.05)
Target modules	q/k/v/o_proj, gate/up/down_proj
Trainable params	~0.5% of total
Precision	bf16 compute, gradient checkpointing
Effective batch	32 (8 × 4 grad-accum)
LR / schedule	2e-4, cosine, 3% warmup
Hardware	1× NVIDIA H100 NVL (95 GB), ~19 GB used
Framework	HuggingFace TRL + PEFT + Transformers

Best-checkpoint selection matters here: eval loss bottomed around step 500 and then rose while train loss kept falling (overfitting). load_best_model_at_end on eval_loss selected the step-500 checkpoint — this model. Training for 3 epochs was more than necessary.

A GRPO / RLVR variant (RL against a programmatic reward: JSON validity + criterion F1) was also trained; it matched this SFT model but did not beat it, because SFT had already saturated the reward. Details in the repo.

Data

Built from 8,000 real ClinicalTrials.gov studies. Targets come from the registry's own structured fields (not a teacher LLM), so labels are auditable. Splits are partitioned by trial, not by example, to prevent leakage.

Limitations

• Not medical advice. Outputs may be wrong; a human expert must review anything clinical.
• Trained on English registry text only; performance on other formats/languages is unknown.
• Summary quality (ROUGE-L 0.29) is the weakest task.
• Only 8k of ~500k available trials were used — more data would likely improve generalisation.

License

Apache-2.0, inherited from Qwen2.5. Source registry data is public-domain U.S. government work.