初始化项目，由ModelHub XC社区提供模型
Model: swift/llama3-llava-next-8b-hf Source: Original Platform
This commit is contained in:
ModelHub XC
160
README.md
Normal file
160
README.md
Normal file
@@ -0,0 +1,160 @@
|
||||
---
|
||||
license: llama3
|
||||
tags:
|
||||
- vision
|
||||
- image-text-to-text
|
||||
language:
|
||||
- en
|
||||
pipeline_tag: image-text-to-text
|
||||
---
|
||||
|
||||
# LLaVa-Next Model Card
|
||||
|
||||
The LLaVA-NeXT model was proposed in [LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild
|
||||
](https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/) by Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, Chunyuan Li.
|
||||
These LLaVa-NeXT series improves upon [LLaVa-1.6](https://llava-vl.github.io/blog/2024-01-30-llava-next/) by training with stringer language backbones, improving the
|
||||
performance.
|
||||
|
||||
Disclaimer: The team releasing LLaVa-NeXT did not write a model card for this model so this model card has been written by the Hugging Face team.
|
||||
|
||||
## Model description
|
||||
|
||||
LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases. LLaVA NeXT Llama3 improves on LLaVA 1.6 BY:
|
||||
- More diverse and high quality data mixture
|
||||
- Better and bigger language backbone
|
||||
|
||||
Base LLM: [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)
|
||||
|
||||
|
||||
|
||||
## Intended uses & limitations
|
||||
|
||||
You can use the raw model for tasks like image captioning, visual question answering, multimodal chatbot use cases. See the [model hub](https://huggingface.co/models?search=llava-hf) to look for
|
||||
other versions on a task that interests you.
|
||||
|
||||
### How to use
|
||||
|
||||
To run the model with the `pipeline`, see the below example:
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
pipe = pipeline("image-text-to-text", model="llava-hf/llama3-llava-next-8b-hf")
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"},
|
||||
{"type": "text", "text": "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
out = pipe(text=messages, max_new_tokens=20)
|
||||
print(out)
|
||||
>>> [{'input_text': [{'role': 'user', 'content': [{'type': 'image', 'url': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg'}, {'type': 'text', 'text': 'What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud'}]}], 'generated_text': 'Lava'}]
|
||||
```
|
||||
|
||||
|
||||
You can also load and use the model like following:
|
||||
|
||||
```python
|
||||
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
|
||||
import torch
|
||||
from PIL import Image
|
||||
import requests
|
||||
|
||||
processor = LlavaNextProcessor.from_pretrained("llava-hf/llama3-llava-next-8b-hf")
|
||||
model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llama3-llava-next-8b-hf", torch_dtype=torch.float16, device_map="auto")
|
||||
|
||||
# prepare image and text prompt, using the appropriate prompt template
|
||||
url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
|
||||
image = Image.open(requests.get(url, stream=True).raw)
|
||||
|
||||
# Define a chat histiry and use `apply_chat_template` to get correctly formatted prompt
|
||||
# Each value in "content" has to be a list of dicts with types ("text", "image")
|
||||
conversation = [
|
||||
{
|
||||
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What is shown in this image?"},
|
||||
{"type": "image"},
|
||||
],
|
||||
},
|
||||
]
|
||||
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
|
||||
|
||||
inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)
|
||||
|
||||
# autoregressively complete prompt
|
||||
output = model.generate(**inputs, max_new_tokens=100)
|
||||
|
||||
print(processor.decode(output[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
-----------
|
||||
From transformers>=v4.48, you can also pass image url or local path to the conversation history, and let the chat template handle the rest.
|
||||
Chat template will load the image for you and return inputs in `torch.Tensor` which you can pass directly to `model.generate()`
|
||||
|
||||
```python
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}
|
||||
{"type": "text", "text": "What is shown in this image?"},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors"pt")
|
||||
output = model.generate(**inputs, max_new_tokens=50)
|
||||
```
|
||||
|
||||
### Model optimization
|
||||
|
||||
#### 4-bit quantization through `bitsandbytes` library
|
||||
|
||||
First make sure to install `bitsandbytes`, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:
|
||||
|
||||
```diff
|
||||
model = LlavaNextForConditionalGeneration.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.float16,
|
||||
low_cpu_mem_usage=True,
|
||||
+ load_in_4bit=True
|
||||
)
|
||||
```
|
||||
|
||||
#### Use Flash-Attention 2 to further speed-up generation
|
||||
|
||||
First make sure to install `flash-attn`. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:
|
||||
|
||||
```diff
|
||||
model = LlavaNextForConditionalGeneration.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.float16,
|
||||
low_cpu_mem_usage=True,
|
||||
+ use_flash_attention_2=True
|
||||
).to(0)
|
||||
```
|
||||
### Training Data
|
||||
- 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP.
|
||||
- 158K GPT-generated multimodal instruction-following data.
|
||||
- 500K academic-task-oriented VQA data mixture.
|
||||
- 50K GPT-4V data mixture.
|
||||
- 40K ShareGPT data.
|
||||
|
||||
|
||||
### BibTeX entry and citation info
|
||||
|
||||
```bibtex
|
||||
@misc{li2024llavanext-strong,
|
||||
title={LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild},
|
||||
url={https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/},
|
||||
author={Li, Bo and Zhang, Kaichen and Zhang, Hao and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Yuanhan and Liu, Ziwei and Li, Chunyuan},
|
||||
month={May},
|
||||
year={2024}
|
||||
}
|
||||
```
|
||||
Reference in New Issue
Block a user