90 lines
3.1 KiB
Markdown
90 lines
3.1 KiB
Markdown
---
|
|
license: apache-2.0
|
|
datasets:
|
|
- FreedomIntelligence/PubMedVision
|
|
language:
|
|
- en
|
|
- zh
|
|
pipeline_tag: text-generation
|
|
base_model:
|
|
- Qwen/Qwen2.5-VL-7B-Instruct
|
|
---
|
|
<div align="center">
|
|
<h1>
|
|
HuatuoGPT-Vision-7B (Qwen2.5-VL Architecture)
|
|
</h1>
|
|
</div>
|
|
|
|
<div align="center">
|
|
<a href="https://github.com/FreedomIntelligence/HuatuoGPT-Vision" target="_blank">GitHub</a> | <a href="https://arxiv.org/abs/2406.19280" target="_blank">Paper</a>
|
|
</div>
|
|
|
|
> This is the HuatuoGPT-Vision model based on the [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) architecture. It was trained on the PubMedVision dataset using the Qwen2.5-VL framework. For details, please refer to the training code at [HuatuoGPT-Vision Code](https://github.com/FreedomIntelligence/HuatuoGPT-Vision).
|
|
|
|
# <span id="Start">Introduction</span>
|
|
HuatuoGPT-Vision is a multimodal LLM for medical applications, built with the [PubMedVision dataset](https://huggingface.co/datasets/FreedomIntelligence/PubMedVision). HuatuoGPT-Vision-7B is trained based on Qwen2-7B using the LLaVA-v1.5 architecture.
|
|
|
|
|
|
|
|
# <span id="Start">Quick Start</span>
|
|
**For usage, please refer to [Qwen2.5-VL-Instruction](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct), as it shares the same model architecture and usage pattern.**
|
|
|
|
Here we show a code snippet to show you how to use the chat model with `transformers` and `qwen_vl_utils`:
|
|
|
|
```python
|
|
from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
|
|
from qwen_vl_utils import process_vision_info
|
|
|
|
# default processer
|
|
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
|
|
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{
|
|
"type": "image",
|
|
"image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
|
|
},
|
|
{"type": "text", "text": "Describe this image."},
|
|
],
|
|
}
|
|
]
|
|
|
|
text = processor.apply_chat_template(
|
|
messages, tokenize=False, add_generation_prompt=True
|
|
)
|
|
image_inputs, video_inputs = process_vision_info(messages)
|
|
inputs = processor(
|
|
text=[text],
|
|
images=image_inputs,
|
|
videos=video_inputs,
|
|
padding=True,
|
|
return_tensors="pt",
|
|
)
|
|
inputs = inputs.to("cuda")
|
|
|
|
# Inference: Generation of the output
|
|
generated_ids = model.generate(**inputs, max_new_tokens=128)
|
|
generated_ids_trimmed = [
|
|
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
|
|
]
|
|
output_text = processor.batch_decode(
|
|
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
|
|
)
|
|
print(output_text)
|
|
```
|
|
|
|
# <span id="Start">Citation</span>
|
|
|
|
```
|
|
@misc{chen2024huatuogptvisioninjectingmedicalvisual,
|
|
title={HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale},
|
|
author={Junying Chen and Ruyi Ouyang and Anningzhe Gao and Shunian Chen and Guiming Hardy Chen and Xidong Wang and Ruifei Zhang and Zhenyang Cai and Ke Ji and Guangjun Yu and Xiang Wan and Benyou Wang},
|
|
year={2024},
|
|
eprint={2406.19280},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CV},
|
|
url={https://arxiv.org/abs/2406.19280},
|
|
}
|
|
``` |