73 lines
3.1 KiB
Markdown
73 lines
3.1 KiB
Markdown
|
---
|
|||
|
|
license: llama3.1
|
|||
|
|
datasets:
|
|||
|
|
- biodatlab/ec-raft-dataset
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# EC-RAFT: Automated Generation of Clinical Trial Eligibility Criteria
|
|||
|
|
|
|||
|
|
## Model Description
|
|||
|
|
|
|||
|
|
**EC-RAFT** is a fine-tuned Retrieval-Augmented Fine-Tuning (RAFT) model based on **LLaMA-3.1-8B-Instruct** architecture.
|
|||
|
|
It is designed to automatically generate **structured, high-quality clinical trial eligibility criteria (EC)** directly from trial titles and descriptions.
|
|||
|
|
|
|||
|
|
EC-RAFT integrates **domain-specific retrieval** with **synthesized intermediate reasoning** steps, enabling it to produce **clinically relevant** and **contextually appropriate** EC sets.
|
|||
|
|
|
|||
|
|
## Fine-tuning Details
|
|||
|
|
|
|||
|
|
- **Original Model:** LLaMA-3.1-8B-Instruct
|
|||
|
|
- **Datasets used for fine-tuning:**
|
|||
|
|
- ClinicalTrials.gov (267,347 trials, 2000–2024) [biodatlab/ec-raft-dataset](https://huggingface.co/datasets/biodatlab/ec-raft-dataset)
|
|||
|
|
- Retrieval corpus constructed using **SciNCL model**
|
|||
|
|
- Intermediate reasoning steps **R** generated using **Gemini-1.5-flash-002**
|
|||
|
|
- Fine-tuning method:
|
|||
|
|
- **Retrieval-Augmented Fine-Tuning (RAFT)**
|
|||
|
|
- **Low-Rank Adaptation (LoRA)**
|
|||
|
|
|
|||
|
|
## Model Performance
|
|||
|
|
|
|||
|
|
Evaluated on a held-out ClinicalTrials.gov test split:
|
|||
|
|
|
|||
|
|
| Metric | Score |
|
|||
|
|
|-----------------------------------|---------|
|
|||
|
|
| **BERTScore** (semantic similarity) | **86.23** |
|
|||
|
|
| **Precision** (LLM-guided evaluation) | **78.84%** |
|
|||
|
|
| **Recall** (LLM-guided evaluation) | **75.89%** |
|
|||
|
|
| **Mean LLM-as-a-Judge Score** (0–3) | **1.7150** |
|
|||
|
|
| **Mean Pair-BERTScore** | **67.76** |
|
|||
|
|
|
|||
|
|
- **Outperforms zero-shot LLaMA-3.1 and Gemini-1.5-flash baselines**
|
|||
|
|
- **Outperforms fine-tuned LLaMA and Meditron baselines**
|
|||
|
|
- **Clinically validated:** LLM-as-a-Judge scores highly correlated with human physician evaluation
|
|||
|
|
|
|||
|
|
## Intended Use
|
|||
|
|
|
|||
|
|
- Assist **researchers**, **trial designers**, and **sponsors** in drafting clinical trial eligibility criteria.
|
|||
|
|
- **Automate** EC generation to reduce manual effort and improve consistency.
|
|||
|
|
- Support **clinical trial design** transparency and quality.
|
|||
|
|
- Enable integration with **trial registry platforms**, **clinical trial matching systems**, and **EC recommendation tools**.
|
|||
|
|
|
|||
|
|
## Limitations
|
|||
|
|
|
|||
|
|
- Requires **human validation** of generated EC before clinical use.
|
|||
|
|
- Trained on **public ClinicalTrials.gov data** — may not generalize well to:
|
|||
|
|
- Rare or novel diseases
|
|||
|
|
- Specialized or non-standard trial designs
|
|||
|
|
- Non-public trial data
|
|||
|
|
- Optimized for **English-language clinical trials**.
|
|||
|
|
- As with any LLM-based system, risks include hallucination, subtle errors, and domain shifts.
|
|||
|
|
- Evaluation metrics (BERTScore, LLM-as-a-Judge) are proxies — not full substitutes for domain expert review.
|
|||
|
|
|
|||
|
|
## Acknowledgments
|
|||
|
|
|
|||
|
|
This model was developed using resources provided by:
|
|||
|
|
|
|||
|
|
- **RAVIS Technology** for feedback and collaboration.
|
|||
|
|
- **Faculty of Medicine Ramathibodi Hospital**
|
|||
|
|
- **NSTDA Supercomputer Center (ThaiSC), Project \#pv814001**
|
|||
|
|
|
|||
|
|
We also acknowledge the contributions of the broader open-source community whose tools and prior works on **RAFT**, **SciNCL**, **LoRA**, **LLaMA-3**, and **biomedical NLP** made this project possible.
|