77 lines
2.7 KiB
Markdown
77 lines
2.7 KiB
Markdown
|
---
|
||
|
|
library_name: transformers
|
||
|
|
tags: []
|
||
|
|
pipeline_tag: image-text-to-text
|
||
|
|
---
|
||
|
|
|
||
|
|
# Fine-Grained Visual Classification on Plant Leaf Diseases
|
||
|
|
|
||
|
|
Project Page: [SelfSynthX](https://github.com/sycny/SelfSynthX).
|
||
|
|
|
||
|
|
Paper on arXiv: [Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data](https://arxiv.org/abs/2502.14044)
|
||
|
|
|
||
|
|
|
||
|
|
This model is a fine-tuned multimodal foundation model based on [LLaVA-1.5-7B-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf), optimized for detecting and explaining plant leaf diseases using the Plant disease dataset.
|
||
|
|
|
||
|
|
## Key Details
|
||
|
|
|
||
|
|
- **Base Model:** LLaVA-1.5-7B
|
||
|
|
- **Dataset:** Healthy and diseased leaves across multiple plant species
|
||
|
|
- **Innovation:**
|
||
|
|
- **Self-Synthesized Data:** Extracts and describes disease-specific visual symptoms using the Information Bottleneck principle.
|
||
|
|
- **Iterative Fine-Tuning:** Uses reward model-free rejection sampling to improve classification accuracy and explanation quality.
|
||
|
|
- **Intended Use:** Identification of plant leaf diseases with human-verifiable symptom descriptions.
|
||
|
|
|
||
|
|
## How to Use
|
||
|
|
|
||
|
|
```python
|
||
|
|
import requests
|
||
|
|
from PIL import Image
|
||
|
|
import torch
|
||
|
|
from transformers import AutoProcessor, LlavaForConditionalGeneration
|
||
|
|
|
||
|
|
model_id = "YuchengShi/LLaVA-v1.5-7B-Plant-Leaf-Diseases-Detection"
|
||
|
|
model = LlavaForConditionalGeneration.from_pretrained(
|
||
|
|
model_id,
|
||
|
|
torch_dtype=torch.float16,
|
||
|
|
low_cpu_mem_usage=True,
|
||
|
|
).to("cuda")
|
||
|
|
processor = AutoProcessor.from_pretrained(model_id)
|
||
|
|
|
||
|
|
conversation = [
|
||
|
|
{
|
||
|
|
"role": "user",
|
||
|
|
"content": [
|
||
|
|
{"type": "text", "text": "What disease does this leaf have?"},
|
||
|
|
{"type": "image"},
|
||
|
|
],
|
||
|
|
},
|
||
|
|
]
|
||
|
|
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
|
||
|
|
image_file = "plant-disease/test1.png"
|
||
|
|
raw_image = Image.open(requests.get(image_file, stream=True).raw)
|
||
|
|
inputs = processor(images=raw_image, text=prompt, return_tensors='pt').to("cuda", torch.float16)
|
||
|
|
|
||
|
|
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
|
||
|
|
print(processor.decode(output[0][2:], skip_special_tokens=True))
|
||
|
|
```
|
||
|
|
|
||
|
|
## Training & Evaluation
|
||
|
|
|
||
|
|
- **Training:** Fine-tuned using LoRA on PlantVillage with iterative rejection sampling.
|
||
|
|
- **Evaluation:** Demonstrates superior accuracy and robust, interpretable explanations compared to baseline models.
|
||
|
|
|
||
|
|
## Citation
|
||
|
|
|
||
|
|
If you use this model, please cite:
|
||
|
|
|
||
|
|
```bibtex
|
||
|
|
@inproceedings{
|
||
|
|
shi2025enhancing,
|
||
|
|
title={Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data},
|
||
|
|
author={Yucheng Shi and Quanzheng Li and Jin Sun and Xiang Li and Ninghao Liu},
|
||
|
|
booktitle={The Thirteenth International Conference on Learning Representations},
|
||
|
|
year={2025},
|
||
|
|
url={https://openreview.net/forum?id=lHbLpwbEyt}
|
||
|
|
}
|
||
|
|
```
|