初始化项目，由ModelHub XC社区提供模型
Model: chaoyinshe/llava-med-v1.5-mistral-7b-hf Source: Original Platform
This commit is contained in:
ModelHub XC
104
README.md
Normal file
104
README.md
Normal file
@@ -0,0 +1,104 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
base_model:
|
||||
- microsoft/llava-med-v1.5-mistral-7b
|
||||
pipeline_tag: visual-question-answering
|
||||
language:
|
||||
- en
|
||||
tags:
|
||||
- medical
|
||||
- biology
|
||||
---
|
||||
# llava-med-v1.5-mistral-7b-hf
|
||||
|
||||
This repository contains a **drop-in, Hugging Face–compatible** checkpoint converted from
|
||||
[https://huggingface.co/microsoft/llava-med-v1.5-mistral-7b](https://huggingface.co/microsoft/llava-med-v1.5-mistral-7b).
|
||||
You can load it with the **exact same code** you use for the original model—no extra conversion steps required.
|
||||
|
||||
---
|
||||
## ✅ Clarification on Vocab Size Expansion & Weight Integrity
|
||||
You can refer to the original conversion code we provided to access the relevant implementation details and code snippets.
|
||||
You may wonder if the vocab size mismatch (e.g., 32000 vs. 32064) breaks the original weights by causing dimension mismatches between embed_tokens/lm_head and the original weights.
|
||||
|
||||
The answer is no. Below are key clarifications:
|
||||
|
||||
### 1. Does expanding embed_tokens affect the attention mechanism?
|
||||
No. The embed_tokens layer is simply an embedding lookup table (nn.Embedding) that maps token IDs to vectors.
|
||||
The attention mechanism (e.g., nn.MultiheadAttention or LlamaAttention) does not operate on this lookup table directly. It only cares if the dimension of input hidden states is consistent.
|
||||
Thus, even if you add more tokens, as long as the hidden_size remains unchanged, the weight shape of the attention layer is not affected at all.
|
||||
|
||||
### 2. Which layers actually change after expansion?
|
||||
| Module | Weight Shape Change | Affects Attention Calculation? |
|
||||
|----------------------|--------------------------------------|---------------------------------|
|
||||
| embed_tokens | [vocab_size+2, hidden_size] | ❌ No |
|
||||
| lm_head | [vocab_size+2, hidden_size] | ❌ No |
|
||||
| All attention layers | No shape change | ✅ Completely unchanged |
|
||||
|
||||
#### Analogy for easier understanding
|
||||
Think of the model as a dictionary:
|
||||
- embed_tokens is the "new word list": You add two new words, but the length of each word’s explanation (hidden_size) stays the same.
|
||||
- Attention is the "reading rule": It only focuses on how the vectors of each word in a sentence interact with each other, not how many words are in the dictionary.
|
||||
|
||||
#### The only "change" lies in input distribution
|
||||
- The original model never encountered embeddings for tokens like `<image>` or `<pad>`.
|
||||
- Now these tokens are initialized, and their new vectors appear during the first forward pass.
|
||||
- This is a data-level change, not damage to the model’s parameters.
|
||||
|
||||
#### Summary
|
||||
Expanding the vocab only changes the "dictionary size", not the "reading rules". The weight shape and calculation logic of the attention mechanism remain completely unchanged. The output layer (lm_head) changes in shape but not in functional logic.
|
||||
|
||||
### 3. What exactly changes in the output layer (lm_head)?
|
||||
The line below modifies the weight matrix of lm_head (the language modeling head):
|
||||
```python
|
||||
model.resize_token_embeddings(config.text_config.vocab_size + 2, pad_shape)
|
||||
```
|
||||
## Quick Start
|
||||
|
||||
```python
|
||||
from transformers import LlavaForConditionalGeneration, AutoProcessor
|
||||
import torch
|
||||
|
||||
model_path = "chaoyinshe/llava-med-v1.5-mistral-7b-hf"
|
||||
|
||||
model = LlavaForConditionalGeneration.from_pretrained(
|
||||
model_path,
|
||||
torch_dtype=torch.bfloat16,
|
||||
attn_implementation="flash_attention_2", # requires FA2
|
||||
device_map="auto" # multi-GPU ready
|
||||
)
|
||||
|
||||
processor = AutoProcessor.from_pretrained(model_path)
|
||||
|
||||
# Example inference
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image"},
|
||||
{"type": "text", "text": "What is the main finding in this chest X-ray?"}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
prompt = processor.tokenizer.apply_chat_template(
|
||||
messages, tokenize=False, add_generation_prompt=True
|
||||
)
|
||||
|
||||
inputs = processor(
|
||||
images=[image], text=prompt, return_tensors="pt"
|
||||
).to(model.device, torch.bfloat16)
|
||||
|
||||
with torch.inference_mode():
|
||||
out = model.generate(**inputs, max_new_tokens=256)
|
||||
|
||||
print(processor.decode(out[0], skip_special_tokens=True))
|
||||
```
|
||||
## ✅ Training Screenshot
|
||||
(Note: The image below is for illustrative purposes only — actual training metrics may vary.)
|
||||
🤗
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user