104 lines
4.6 KiB
Markdown
104 lines
4.6 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
base_model:
|
|||
|
|
- microsoft/llava-med-v1.5-mistral-7b
|
|||
|
|
pipeline_tag: visual-question-answering
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
tags:
|
|||
|
|
- medical
|
|||
|
|
- biology
|
|||
|
|
---
|
|||
|
|
# llava-med-v1.5-mistral-7b-hf
|
|||
|
|
|
|||
|
|
This repository contains a **drop-in, Hugging Face–compatible** checkpoint converted from
|
|||
|
|
[https://huggingface.co/microsoft/llava-med-v1.5-mistral-7b](https://huggingface.co/microsoft/llava-med-v1.5-mistral-7b).
|
|||
|
|
You can load it with the **exact same code** you use for the original model—no extra conversion steps required.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
## ✅ Clarification on Vocab Size Expansion & Weight Integrity
|
|||
|
|
You can refer to the original conversion code we provided to access the relevant implementation details and code snippets.
|
|||
|
|
You may wonder if the vocab size mismatch (e.g., 32000 vs. 32064) breaks the original weights by causing dimension mismatches between embed_tokens/lm_head and the original weights.
|
|||
|
|
|
|||
|
|
The answer is no. Below are key clarifications:
|
|||
|
|
|
|||
|
|
### 1. Does expanding embed_tokens affect the attention mechanism?
|
|||
|
|
No. The embed_tokens layer is simply an embedding lookup table (nn.Embedding) that maps token IDs to vectors.
|
|||
|
|
The attention mechanism (e.g., nn.MultiheadAttention or LlamaAttention) does not operate on this lookup table directly. It only cares if the dimension of input hidden states is consistent.
|
|||
|
|
Thus, even if you add more tokens, as long as the hidden_size remains unchanged, the weight shape of the attention layer is not affected at all.
|
|||
|
|
|
|||
|
|
### 2. Which layers actually change after expansion?
|
|||
|
|
| Module | Weight Shape Change | Affects Attention Calculation? |
|
|||
|
|
|----------------------|--------------------------------------|---------------------------------|
|
|||
|
|
| embed_tokens | [vocab_size+2, hidden_size] | ❌ No |
|
|||
|
|
| lm_head | [vocab_size+2, hidden_size] | ❌ No |
|
|||
|
|
| All attention layers | No shape change | ✅ Completely unchanged |
|
|||
|
|
|
|||
|
|
#### Analogy for easier understanding
|
|||
|
|
Think of the model as a dictionary:
|
|||
|
|
- embed_tokens is the "new word list": You add two new words, but the length of each word’s explanation (hidden_size) stays the same.
|
|||
|
|
- Attention is the "reading rule": It only focuses on how the vectors of each word in a sentence interact with each other, not how many words are in the dictionary.
|
|||
|
|
|
|||
|
|
#### The only "change" lies in input distribution
|
|||
|
|
- The original model never encountered embeddings for tokens like `<image>` or `<pad>`.
|
|||
|
|
- Now these tokens are initialized, and their new vectors appear during the first forward pass.
|
|||
|
|
- This is a data-level change, not damage to the model’s parameters.
|
|||
|
|
|
|||
|
|
#### Summary
|
|||
|
|
Expanding the vocab only changes the "dictionary size", not the "reading rules". The weight shape and calculation logic of the attention mechanism remain completely unchanged. The output layer (lm_head) changes in shape but not in functional logic.
|
|||
|
|
|
|||
|
|
### 3. What exactly changes in the output layer (lm_head)?
|
|||
|
|
The line below modifies the weight matrix of lm_head (the language modeling head):
|
|||
|
|
```python
|
|||
|
|
model.resize_token_embeddings(config.text_config.vocab_size + 2, pad_shape)
|
|||
|
|
```
|
|||
|
|
## Quick Start
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import LlavaForConditionalGeneration, AutoProcessor
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
model_path = "chaoyinshe/llava-med-v1.5-mistral-7b-hf"
|
|||
|
|
|
|||
|
|
model = LlavaForConditionalGeneration.from_pretrained(
|
|||
|
|
model_path,
|
|||
|
|
torch_dtype=torch.bfloat16,
|
|||
|
|
attn_implementation="flash_attention_2", # requires FA2
|
|||
|
|
device_map="auto" # multi-GPU ready
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
processor = AutoProcessor.from_pretrained(model_path)
|
|||
|
|
|
|||
|
|
# Example inference
|
|||
|
|
messages = [
|
|||
|
|
{
|
|||
|
|
"role": "user",
|
|||
|
|
"content": [
|
|||
|
|
{"type": "image"},
|
|||
|
|
{"type": "text", "text": "What is the main finding in this chest X-ray?"}
|
|||
|
|
]
|
|||
|
|
}
|
|||
|
|
]
|
|||
|
|
|
|||
|
|
prompt = processor.tokenizer.apply_chat_template(
|
|||
|
|
messages, tokenize=False, add_generation_prompt=True
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
inputs = processor(
|
|||
|
|
images=[image], text=prompt, return_tensors="pt"
|
|||
|
|
).to(model.device, torch.bfloat16)
|
|||
|
|
|
|||
|
|
with torch.inference_mode():
|
|||
|
|
out = model.generate(**inputs, max_new_tokens=256)
|
|||
|
|
|
|||
|
|
print(processor.decode(out[0], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
## ✅ Training Screenshot
|
|||
|
|
(Note: The image below is for illustrative purposes only — actual training metrics may vary.)
|
|||
|
|
🤗
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|