初始化项目，由ModelHub XC社区提供模型

Model: OmkarShewale/clintrial-qwen2.5-7b-sft Source: Original Platform
2026-08-08 09:37:17 +08:00
commit 3c68119d92
8 changed files with 336 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,135 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-7B-Instruct
+library_name: transformers
+pipeline_tag: text-generation
+language:
+  - en
+tags:
+  - clinical-trials
+  - healthcare
+  - biomedical
+  - information-extraction
+  - qlora
+  - peft
+  - trl
+  - grpo
+---
+
+# ClinTrial-LM — Qwen2.5-7B fine-tuned for clinical-trial understanding
+
+A **QLoRA**-fine-tuned [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)
+specialised for clinical-trial tasks, trained on ~26k instruction examples derived from the
+[ClinicalTrials.gov](https://clinicaltrials.gov) registry.
+
+📦 **Code, full pipeline & write-up:** https://github.com/omkar-droid/clintrial-lm
+
+> ⚕️ **Research/education only. Not medical advice.** Do not use for clinical decision-making.
+
+## What it does
+
+| Task | Description |
+|---|---|
+| **Eligibility extraction** | Free-text inclusion/exclusion criteria → structured JSON |
+| **Plain-language summary** | Trial description → patient-friendly summary |
+| **Condition Q&A** | "What conditions does this trial study?" |
+| **Phase classification** | Identify the trial phase |
+
+## Results
+
+Evaluated on a **held-out test set split by trial ID** (no trial appears in both train and test),
+500 sampled examples.
+
+| Task | Metric | Base Qwen2.5-7B | **This model (SFT)** |
+|---|---|---|---|
+| Eligibility extraction | criterion F1 | 0.840 | **0.968** |
+| Eligibility output | **JSON validity** | 0.986 | **1.000** |
+| Phase classification | exact match | 0.000 | **0.794** |
+| Condition Q&A | token F1 | 0.298 | **0.742** |
+| Plain-language summary | ROUGE-L | 0.195 | **0.290** |
+
+Eligibility extraction reaches **precision 0.962 / recall 0.978** with **100% schema-valid JSON**.
+
+The base model scores 0.000 on phase classification not because it lacks the knowledge, but because
+it won't answer in the required format — fine-tuning buys **format discipline and faithfulness**.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_ID = "OmkarShewale/clintrial-qwen2.5-7b-sft"
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16, device_map="auto")
+
+SYSTEM = (
+    "You are a clinical research assistant. You help patients and clinicians understand "
+    "clinical trials. Answer only from the information provided, be precise, and never "
+    "invent eligibility criteria, conditions, or outcomes."
+)
+
+criteria = """Inclusion Criteria:
+1. Adults aged 18 years or older with confirmed type 2 diabetes
+2. HbA1c between 7.0% and 10.5% at screening
+
+Exclusion Criteria:
+1. History of severe hypoglycaemia within the last 6 months
+2. Pregnancy or breastfeeding
+"""
+
+messages = [
+    {"role": "system", "content": SYSTEM},
+    {"role": "user", "content":
+        'Extract the eligibility criteria from the trial text below into JSON with two lists, '
+        '"inclusion" and "exclusion". Copy each criterion verbatim; do not add any.\n\n'
+        f"Trial eligibility text:\n{criteria}"},
+]
+
+enc = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
+                                    return_tensors="pt", return_dict=True).to(model.device)
+out = model.generate(**enc, max_new_tokens=1024, do_sample=False)
+print(tokenizer.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+**Note:** eligibility answers can be long (up to ~1.7k tokens). Use `max_new_tokens >= 1024` or the
+JSON will be truncated and fail to parse.
+
+## Training
+
+| | |
+|---|---|
+| Base model | Qwen2.5-7B-Instruct |
+| Method | QLoRA — 4-bit NF4 + LoRA (r=16, α=32, dropout 0.05) |
+| Target modules | q/k/v/o_proj, gate/up/down_proj |
+| Trainable params | ~0.5% of total |
+| Precision | bf16 compute, gradient checkpointing |
+| Effective batch | 32 (8 × 4 grad-accum) |
+| LR / schedule | 2e-4, cosine, 3% warmup |
+| Hardware | 1× NVIDIA H100 NVL (95 GB), ~19 GB used |
+| Framework | HuggingFace TRL + PEFT + Transformers |
+
+**Best-checkpoint selection matters here:** eval loss bottomed around step 500 and then *rose* while
+train loss kept falling (overfitting). `load_best_model_at_end` on `eval_loss` selected the step-500
+checkpoint — this model. Training for 3 epochs was more than necessary.
+
+A **GRPO / RLVR** variant (RL against a programmatic reward: JSON validity + criterion F1) was also
+trained; it matched this SFT model but did not beat it, because SFT had already saturated the reward.
+Details in the [repo](https://github.com/omkar-droid/clintrial-lm).
+
+## Data
+
+Built from 8,000 real ClinicalTrials.gov studies. Targets come from the registry's **own structured
+fields** (not a teacher LLM), so labels are auditable. Splits are partitioned **by trial**, not by
+example, to prevent leakage.
+
+## Limitations
+
+- **Not medical advice.** Outputs may be wrong; a human expert must review anything clinical.
+- Trained on English registry text only; performance on other formats/languages is unknown.
+- Summary quality (ROUGE-L 0.29) is the weakest task.
+- Only 8k of ~500k available trials were used — more data would likely improve generalisation.
+
+## License
+
+Apache-2.0, inherited from Qwen2.5. Source registry data is public-domain U.S. government work.