189 lines
4.5 KiB
Markdown
189 lines
4.5 KiB
Markdown
---
|
||
library_name: transformers
|
||
tags:
|
||
- unsloth
|
||
- trl
|
||
- sft
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
base_model:
|
||
- meta-llama/Llama-3.2-3B-Instruct
|
||
pipeline_tag: text-generation
|
||
metrics:
|
||
- accuracy
|
||
- bleu
|
||
- rouge
|
||
---
|
||
|
||
# Model Card for MediLlama-3.2
|
||
|
||
A fine-tuned version of Meta's LLaMA 3.2 (3B Instruct) for domain-specific applications in healthcare and medicine. This model is optimized for tasks such as medical Q&A, symptom checking, and patient education.
|
||
|
||
## Model Details
|
||
|
||
### Model Description
|
||
|
||
This model is a domain-adapted version of LLaMA 3.2 3B Instruct. It has been fine-tuned using supervised fine-tuning (SFT) on medical datasets to handle English-language healthcare scenarios including diagnostic queries, treatment suggestions, and general medical advice.
|
||
|
||
- **Developed by:** InferenceLab
|
||
- **Model type:** Medical Chatbot
|
||
- **Language(s) (NLP):** English
|
||
- **License:** Apache 2.0
|
||
- **Finetuned from model:** meta-llama/Llama-3.2-3B-Instruct
|
||
|
||
## Uses
|
||
|
||
### Direct Use
|
||
|
||
MediLlama-3.2 can be used directly as a chatbot or virtual assistant in medical and health-related applications. Ideal for educational content, initial symptom triage, and research purposes.
|
||
|
||
### Downstream Use
|
||
|
||
Can be integrated into larger telehealth systems, clinical documentation tools, or diagnostic assistants after further task-specific fine-tuning.
|
||
|
||
### Out-of-Scope Use
|
||
|
||
- Should not be used for real-time diagnosis or treatment decisions without expert validation.
|
||
- Not suitable for high-risk or life-threatening emergency response.
|
||
- Not trained on pediatric or highly specialized medical domains.
|
||
|
||
## Bias, Risks, and Limitations
|
||
|
||
While the model is trained on medical data, it may still exhibit:
|
||
- Biases from source data
|
||
- Hallucinations or incorrect suggestions
|
||
- Outdated or non-region-specific medical advice
|
||
|
||
### Recommendations
|
||
|
||
Users should validate outputs with certified medical professionals. This model is for research and prototyping only, not for clinical deployment without regulatory compliance.
|
||
|
||
## How to Get Started with the Model
|
||
|
||
```python
|
||
import torch
|
||
from transformers import pipeline
|
||
|
||
model_id = "InferenceLab/MediLlama-3.2"
|
||
pipe = pipeline(
|
||
"text-generation",
|
||
model=model_id,
|
||
torch_dtype=torch.bfloat16,
|
||
device_map="auto",
|
||
)
|
||
|
||
messages = [
|
||
{"role": "system", "content": "You are a helpful Medical assistant."},
|
||
{"role": "user", "content": "Hi! How are you?"},
|
||
]
|
||
outputs = pipe(
|
||
messages,
|
||
max_new_tokens=256,
|
||
)
|
||
print(outputs[0]["generated_text"][-1])
|
||
|
||
````
|
||
|
||
## Training Details
|
||
|
||
### Training Data
|
||
|
||
Model trained using cleaned and preprocessed medical QA datasets, synthetic doctor-patient conversations, and publicly available health forums. Protected health information (PHI) was removed.
|
||
|
||
### Training Procedure
|
||
|
||
Supervised fine-tuning (SFT) using TRL and Unsloth libraries.
|
||
|
||
#### Preprocessing
|
||
|
||
Tokenization using LLaMA tokenizer with special medical instruction formatting.
|
||
|
||
#### Training Hyperparameters
|
||
|
||
* **Training regime:** bf16 mixed precision
|
||
* **Learning rate:** 1e-5
|
||
|
||
#### Speeds, Sizes, Times
|
||
|
||
* **Training time:** \~12 hours on 4×A100 GPUs
|
||
|
||
## Evaluation
|
||
|
||
### Testing Data, Factors & Metrics
|
||
|
||
#### Testing Data
|
||
|
||
Subset of unseen medical QA pairs, synthetic test cases, and MedQA-derived examples.
|
||
|
||
#### Factors
|
||
|
||
* Input prompt complexity
|
||
* Use of medical terminology
|
||
* Chat length
|
||
|
||
#### Metrics
|
||
|
||
* **Accuracy:** 81.3%
|
||
* **BLEU:** 34.5
|
||
* **ROUGE-L:** 62.2
|
||
|
||
### Results
|
||
|
||
#### Summary
|
||
|
||
Model shows good generalization to unseen prompts and performs competitively for general medical dialogue. Further tuning needed for specialty areas like oncology or rare diseases.
|
||
|
||
## Model Examination
|
||
|
||
Explainability tools like LLaMA-MedLens (if available) are suggested to interpret model decisions.
|
||
|
||
## Environmental Impact
|
||
|
||
* **Hardware Type:** 4×NVIDIA A100 40GB
|
||
* **Hours used:** 12
|
||
* **Cloud Provider:** AWS
|
||
* **Compute Region:** us-west-2
|
||
* **Carbon Emitted:** \~35.8 kg CO2eq (estimated)
|
||
|
||
## Technical Specifications
|
||
|
||
### Model Architecture and Objective
|
||
|
||
* Based on Meta LLaMA 3.2 3B Instruct
|
||
* Decoder-only transformer
|
||
* Objective: Causal Language Modeling (CLM) with instruction fine-tuning
|
||
|
||
### Compute Infrastructure
|
||
|
||
#### Hardware
|
||
|
||
* 4×NVIDIA A100 40GB
|
||
|
||
#### Software
|
||
|
||
* Python 3.10
|
||
* Transformers (v4.40+)
|
||
* TRL
|
||
* Unsloth
|
||
* PyTorch 2.1
|
||
|
||
|
||
## Glossary
|
||
|
||
* **SFT**: Supervised Fine-Tuning
|
||
* **BLEU**: Bilingual Evaluation Understudy
|
||
* **ROUGE**: Recall-Oriented Understudy for Gisting Evaluation
|
||
|
||
## More Information
|
||
|
||
For collaborations, deployment help, or fine-tuning extensions, please contact the developers.
|
||
|
||
## Model Card Authors
|
||
|
||
* InferenceLab Team
|
||
|
||
|
||
|
||
|