64 lines
1.6 KiB
Markdown
64 lines
1.6 KiB
Markdown
---
|
|
library_name: transformers
|
|
tags:
|
|
- medical
|
|
license: apache-2.0
|
|
language:
|
|
- fr
|
|
- en
|
|
base_model:
|
|
- ik-ram28/MedMistralInstruct-CPT-7B
|
|
- mistralai/Mistral-7B-Instruct-v0.1
|
|
---
|
|
|
|
|
|
## MedMistralInstruct-CPT-SFT-7B
|
|
|
|
### Model Description
|
|
|
|
MedMistralInstruct-CPT-SFT-7B is a French medical language model based on Mistral-7B-Instruct-v0.1, adapted through Continual Pre-Training followed by Supervised Fine-Tuning.
|
|
|
|
### Model Details
|
|
|
|
- **Model Type**: Causal Language Model
|
|
- **Base Model**: Mistral-7B-Instruct-v0.1
|
|
- **Language**: French
|
|
- **Domain**: Medical/Healthcare
|
|
- **Parameters**: 7 billion
|
|
- **License**: Apache 2.0
|
|
|
|
### Training Details
|
|
|
|
**Continual Pre-Training (CPT)**
|
|
- **Dataset**: NACHOS corpus (7.4 GB French medical texts)
|
|
- **Training Duration**: 2.8 epochs
|
|
- **Hardware**: 32 NVIDIA A100 80GB GPUs
|
|
- **Training Time**: ~40 hours
|
|
|
|
**Supervised Fine-Tuning (SFT)**
|
|
- **Dataset**: 30K French medical question-answer pairs
|
|
- **Method**: DoRA (Weight-Decomposed Low-Rank Adaptation)
|
|
- **Training Duration**: 10 epochs
|
|
- **Hardware**: 1 NVIDIA H100 80GB GPU
|
|
- **Training Time**: ~42 hours
|
|
|
|
|
|
### Computational Requirements
|
|
- **Carbon Emissions**: 33.96 kgCO2e (CPT+SFT)
|
|
- **Training Time**: 82 hours total (CPT+SFT)
|
|
|
|
|
|
|
|
### Ethical Considerations
|
|
- **Medical Accuracy**: For research and educational purposes only
|
|
- **Professional Oversight**: Requires verification by qualified medical professionals
|
|
- **Bias Awareness**: May contain biases from training data
|
|
- **Privacy**: Do not input private health information
|
|
|
|
### Citation
|
|
```bibtex
|
|
|
|
```
|
|
|
|
### Contact
|
|
For questions about these models, please contact: ikram.belmadani@lis-lab.fr |