llava-hf/llava-onevision-qwen2-0.5b-ov-hf
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: llava-hf/llava-onevision-qwen2-0.5b-ov-hf
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-10 00:08:14 +08:00
commit db7797d0e4
42 changed files with 455424 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

258
README.md Normal file
View File

@@ -0,0 +1,258 @@
---
language:
- en
- zh
license: apache-2.0
tags:
- vision
- image-text-to-text
- transformers.js
datasets:
- lmms-lab/LLaVA-OneVision-Data
pipeline_tag: image-text-to-text
arxiv: 2408.03326
library_name: transformers
---
# LLaVA-Onevision Model Card
![image/png](llava_onevision_arch.png)
Check out also the Google Colab demo to run Llava on a free-tier Google Colab instance: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1-4AtYjR8UMtCALV0AswU1kiNkWCLTALT?usp=sharing)
Below is the model card of 0.5B LLaVA-Onevision model which is copied from the original LLaVA-Onevision model card that you can find [here](https://huggingface.co/lmms-lab/llava-onevision-qwen2-0.5b-si).
## Model details
**Model type:**
LLaVA-Onevision is an open-source multimodal LLM trained by fine-tuning Qwen2 on GPT-generated multimodal instruction-following data.
LLaVA-OneVision is the first single model that can simultaneously push the performance boundaries of open LMMs in three important computer
vision scenarios: single-image, multi-image, and video scenarios. Importantly, the design of LLaVA-OneVision allows strong transfer learning
across different modalities/scenarios, yielding new emerging capabilities. In particular, strong video understanding and cross-scenario
capabilities are demonstrated through task transfer from images to videos.
**Model date:**
LLaVA-Onevision-0.5-ov was added in August 2024.
**Paper or resources for more information:**
https://llava-vl.github.io/
- **Architecture:** SO400M + Qwen2
- **Pretraining Stage:** LCS-558K, 1 epoch, projector
- **Mid Stage:** A mixture of 4.7M high-quality synthetic data, 1 epoch, full model
- **Final-Image Stage:** A mixture of 3.6M single-image data, 1 epoch, full model
- **OneVision Stage:** A mixture of 1.6M single-image/multi-image/video data, 1 epoch, full model
- **Precision:** bfloat16
## How to use the model
First, make sure to have `transformers` installed from [branch](https://github.com/huggingface/transformers/pull/32673) or `transformers >= 4.45.0`.
The model supports multi-image and multi-prompt generation. Meaning that you can pass multiple images in your prompt. Make sure also to follow the correct prompt template by applying chat template:
### Using `pipeline`:
Below we used [`"llava-hf/llava-onevision-qwen2-0.5b-ov-hf"`](https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf) checkpoint.
```python
from transformers import pipeline
pipe = pipeline("image-text-to-text", model="llava-onevision-qwen2-0.5b-ov-hf")
messages = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"},
{"type": "text", "text": "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"},
],
},
]
out = pipe(text=messages, max_new_tokens=20)
print(out)
>>> [{'input_text': [{'role': 'user', 'content': [{'type': 'image', 'url': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg'}, {'type': 'text', 'text': 'What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud'}]}], 'generated_text': 'Lava'}]
```
### Using pure `transformers`:
Below is an example script to run generation in `float16` precision on a GPU device:
```python
import requests
from PIL import Image
import torch
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration
model_id = "llava-hf/llava-onevision-qwen2-0.5b-ov-hf"
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
).to(0)
processor = AutoProcessor.from_pretrained(model_id)
# Define a chat history and use `apply_chat_template` to get correctly formatted prompt
# Each value in "content" has to be a list of dicts with types ("text", "image")
conversation = [
{
"role": "user",
"content": [
{"type": "text", "text": "What are these?"},
{"type": "image"},
],
},
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(images=raw_image, text=prompt, return_tensors='pt').to(0, torch.float16)
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(output[0][2:], skip_special_tokens=True))
```
-----------
From transformers>=v4.48, you can also pass image/video url or local path to the conversation history, and let the chat template handle the rest.
Chat template will load the image for you and return inputs in `torch.Tensor` which you can pass directly to `model.generate()`
```python
messages = [
{
"role": "user",
"content": [
{"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"}
{"type": "text", "text": "What is shown in this image?"},
],
},
]
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors"pt")
output = model.generate(**inputs, max_new_tokens=50)
```
### Model optimization
#### 4-bit quantization through `bitsandbytes` library
First make sure to install `bitsandbytes`, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with:
```diff
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
+ load_in_4bit=True
)
```
#### Use Flash-Attention 2 to further speed-up generation
First make sure to install `flash-attn`. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with:
```diff
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
+ use_flash_attention_2=True
).to(0)
```
### Usage w/ Transformers.js
If you haven't already, you can install the [Transformers.js](https://huggingface.co/docs/transformers.js) JavaScript library from [NPM](https://www.npmjs.com/package/@huggingface/transformers) using:
```bash
npm i @huggingface/transformers
```
**Example:** Multi-round conversations w/ PKV caching
```js
import { AutoProcessor, AutoTokenizer, LlavaOnevisionForConditionalGeneration, RawImage } from '@huggingface/transformers';
// Load tokenizer, processor and model
const model_id = 'llava-hf/llava-onevision-qwen2-0.5b-ov-hf';
const tokenizer = await AutoTokenizer.from_pretrained(model_id);
const processor = await AutoProcessor.from_pretrained(model_id);
const model = await LlavaOnevisionForConditionalGeneration.from_pretrained(model_id, {
dtype: {
embed_tokens: 'fp16', // or 'fp32' or 'q8'
vision_encoder: 'fp16', // or 'fp32' or 'q8'
decoder_model_merged: 'q4', // or 'q8'
},
// device: 'webgpu',
});
// Prepare text inputs
const prompt = 'What does the text say?';
const messages = [
{ role: 'system', content: 'Answer the question.' },
{ role: 'user', content: `<image>\n${prompt}` }
]
const text = tokenizer.apply_chat_template(messages, { tokenize: false, add_generation_prompt: true });
const text_inputs = tokenizer(text);
// Prepare vision inputs
const url = 'https://huggingface.co/qnguyen3/nanoLLaVA/resolve/main/example_1.png';
const image = await RawImage.fromURL(url);
const vision_inputs = await processor(image);
// Generate response
const { past_key_values, sequences } = await model.generate({
...text_inputs,
...vision_inputs,
do_sample: false,
max_new_tokens: 64,
return_dict_in_generate: true,
});
// Decode output
const answer = tokenizer.decode(
sequences.slice(0, [text_inputs.input_ids.dims[1], null]),
{ skip_special_tokens: true },
);
console.log(answer);
// The text says "small but mighty" in a playful font.
const new_messages = [
...messages,
{ role: 'assistant', content: answer },
{ role: 'user', content: 'How does the text correlate to the context of the image?' }
]
const new_text = tokenizer.apply_chat_template(new_messages, { tokenize: false, add_generation_prompt: true });
const new_text_inputs = tokenizer(new_text);
// Generate another response
const output = await model.generate({
...new_text_inputs,
past_key_values,
do_sample: false,
max_new_tokens: 256,
});
const new_answer = tokenizer.decode(
output.slice(0, [new_text_inputs.input_ids.dims[1], null]),
{ skip_special_tokens: true },
);
console.log(new_answer);
// The text "small but mighty" is likely a playful or humorous reference to the image of the blue mouse with the orange dumbbell. It could be used as a motivational phrase or a playful way to express the idea that even small things can be impressive or powerful.
```
# Citation
```
@misc{li2024llavaonevisioneasyvisualtask,
title={LLaVA-OneVision: Easy Visual Task Transfer},
author={Bo Li and Yuanhan Zhang and Dong Guo and Renrui Zhang and Feng Li and Hao Zhang and Kaichen Zhang and Yanwei Li and Ziwei Liu and Chunyuan Li},
year={2024},
eprint={2408.03326},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2408.03326},
}
```

7
added_tokens.json Normal file
View File

@@ -0,0 +1,7 @@
{
"<image>": 151646,
"<video>": 151647,
"<|endoftext|>": 151643,
"<|im_end|>": 151645,
"<|im_start|>": 151644
}

3
chat_template.json Normal file
View File

@@ -0,0 +1,3 @@
{
"chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + ' '}}{# Render all images first #}{% for content in message['content'] | selectattr('type', 'equalto', 'image') %}{{ '<image>' }}{% endfor %}{# Render all video then #}{% for content in message['content'] | selectattr('type', 'equalto', 'video') %}{{ '<video>' }}{% endfor %}{# Render all text next #}{% if message['role'] != 'assistant' %}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{{ '\n' + content['text'] }}{% endfor %}{% else %}{% for content in message['content'] | selectattr('type', 'equalto', 'text') %}{% generation %}{{ '\n' + content['text'] }}{% endgeneration %}{% endfor %}{% endif %}{{'<|im_end|>'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
}

193
config.json Normal file
View File

@@ -0,0 +1,193 @@
{
"_name_or_path": "/raid/raushan/ov-500",
"architectures": [
"LlavaOnevisionForConditionalGeneration"
],
"ignore_index": -100,
"image_grid_pinpoints": [
[
384,
384
],
[
384,
768
],
[
384,
1152
],
[
384,
1536
],
[
384,
1920
],
[
384,
2304
],
[
768,
384
],
[
768,
768
],
[
768,
1152
],
[
768,
1536
],
[
768,
1920
],
[
768,
2304
],
[
1152,
384
],
[
1152,
768
],
[
1152,
1152
],
[
1152,
1536
],
[
1152,
1920
],
[
1152,
2304
],
[
1536,
384
],
[
1536,
768
],
[
1536,
1152
],
[
1536,
1536
],
[
1536,
1920
],
[
1536,
2304
],
[
1920,
384
],
[
1920,
768
],
[
1920,
1152
],
[
1920,
1536
],
[
1920,
1920
],
[
1920,
2304
],
[
2304,
384
],
[
2304,
768
],
[
2304,
1152
],
[
2304,
1536
],
[
2304,
1920
],
[
2304,
2304
]
],
"image_token_index": 151646,
"model_type": "llava_onevision",
"projector_hidden_act": "gelu",
"text_config": {
"_name_or_path": "Qwen/Qwen2-0.5B-Instruct",
"architectures": [
"Qwen2ForCausalLM"
],
"bos_token_id": 151643,
"eos_token_id": 151645,
"hidden_size": 896,
"intermediate_size": 4864,
"max_window_layers": 24,
"model_type": "qwen2",
"num_attention_heads": 14,
"num_hidden_layers": 24,
"num_key_value_heads": 2,
"rope_theta": 1000000.0,
"tie_word_embeddings": true,
"torch_dtype": "bfloat16",
"vocab_size": 152000
},
"tie_word_embeddings": false,
"torch_dtype": "float16",
"transformers_version": "4.45.0.dev0",
"use_image_newline_parameter": true,
"video_token_index": 151647,
"vision_aspect_ratio": "anyres_max_9",
"vision_config": {
"hidden_size": 1152,
"image_size": 384,
"intermediate_size": 4304,
"model_type": "siglip_vision_model",
"num_attention_heads": 16,
"num_hidden_layers": 26,
"patch_size": 14,
"vision_use_head": false
},
"vision_feature_layer": -1,
"vision_feature_select_strategy": "full"
}

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "image-text-to-text", "allow_remote": true}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 151643,
"eos_token_id": 151645,
"transformers_version": "4.45.0.dev0"
}

BIN
llava_onevision_arch.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 204 KiB

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:07b3362c3412de79baf2379e44e5b0b2a8f4b965ebebd11d7b5b3eb4450fe96e
size 1787445680

3
onnx/decoder_model_merged.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:28f22cb3162626a3a4f052b72ef5652cc16d935fbc03f9dbe44834295fe5b686
size 1994025808

3
onnx/decoder_model_merged_bnb4.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a4a0dccf252c1d2f5d6e10a839544a2947903b09f97e03ba67789874780be561
size 295863516

3
onnx/decoder_model_merged_fp16.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:be46b9b128a2765cd02dfe0164fba18340920814b5480789b819ffc748215f8f
size 997472723

3
onnx/decoder_model_merged_int8.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc674946412447fa76df18686c32541b1388c0fa62cbf53c36dccd1a90649c3f
size 512154211

3
onnx/decoder_model_merged_q4.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:76b912a695768539fd63a155ccf7ee4eeffeb7e82e56d008593c7577ac4e05c8
size 326738324

3
onnx/decoder_model_merged_q4f16.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0ec93a115c9756adc8a9e7e8adf6b031c24a4ab16b63b8dc17a56810fe03b12c
size 287346295

3
onnx/decoder_model_merged_quantized.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:cc674946412447fa76df18686c32541b1388c0fa62cbf53c36dccd1a90649c3f
size 512154211

3
onnx/decoder_model_merged_uint8.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:39f89681a425c8d55c437f6864080f3795e7c3394f1db334dcc3602ba4ea5046
size 512154296

3
onnx/embed_tokens.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9a4c7906766ba2fe21448715e58fd10ed3bfb6f7949d5a1d801afff691b33e4e
size 544768302

3
onnx/embed_tokens_bnb4.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b60e16c36e6b9f66d9768a65bab8285ad5d10ae84954f62f6ed2a952e2334128
size 544768321

3
onnx/embed_tokens_fp16.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e76a118db87244aa315d8a8585ca0f3f58c1aec91b1ca44bc81bb7b7d7448aa1
size 272384452

3
onnx/embed_tokens_int8.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4b4dec69949d75a775d871c5e1dc3db6bd4fd6e8ceffb3deafe64e8f16a8323d
size 136192544

3
onnx/embed_tokens_q4.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b60e16c36e6b9f66d9768a65bab8285ad5d10ae84954f62f6ed2a952e2334128
size 544768321

3
onnx/embed_tokens_q4f16.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8e293817d606044bc377d25639e5f3f1911ddeeba92312e3d9fecf1bb7945367
size 272384471

3
onnx/embed_tokens_quantized.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4b4dec69949d75a775d871c5e1dc3db6bd4fd6e8ceffb3deafe64e8f16a8323d
size 136192544

3
onnx/embed_tokens_uint8.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4054003183ee1f97514b34ab1c59ae2a6b54d1162f29a9440caeebbec486bc8a
size 136192544

3
onnx/vision_encoder.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:06cf8f4eefdea6cb8f095724e37da8fa0358820a3506e1c85915d5d2bdadab43
size 1598932026

3
onnx/vision_encoder_bnb4.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1b1f4cffc59c992d2af6a6ce0c14d828130ba179e8a7c31080308dcefc4cddd9
size 231929646

3
onnx/vision_encoder_fp16.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:31b66d6b44dd56c7d3a94b6a0414eebb85daa5d07c64c9a149968e1d5b880f66
size 799767232

3
onnx/vision_encoder_int8.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d284a45f927b2dbf8a61622444b1c622275c323d95cbd1d07bd3a38c72caf1bb
size 404097518

3
onnx/vision_encoder_q4.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:02509d861e604a9501f8a160b87ab1446e6e0d53e514cab483eafa7bacbd37ab
size 257082918

3
onnx/vision_encoder_q4f16.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a921a473839f50aa2cdb09d77e41bcd899dac788489e4d1ff9a05478aac37e08
size 228394416

3
onnx/vision_encoder_quantized.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0191aa785ce1a65b6e45de14641dbcb37a65f49773e322029118803592cb28b2
size 404097602

3
onnx/vision_encoder_uint8.onnx Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0191aa785ce1a65b6e45de14641dbcb37a65f49773e322029118803592cb28b2
size 404097602

171
preprocessor_config.json Normal file
View File

@@ -0,0 +1,171 @@
{
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": true,
"do_rescale": true,
"do_resize": true,
"image_grid_pinpoints": [
[
384,
384
],
[
384,
768
],
[
384,
1152
],
[
384,
1536
],
[
384,
1920
],
[
384,
2304
],
[
768,
384
],
[
768,
768
],
[
768,
1152
],
[
768,
1536
],
[
768,
1920
],
[
768,
2304
],
[
1152,
384
],
[
1152,
768
],
[
1152,
1152
],
[
1152,
1536
],
[
1152,
1920
],
[
1152,
2304
],
[
1536,
384
],
[
1536,
768
],
[
1536,
1152
],
[
1536,
1536
],
[
1536,
1920
],
[
1536,
2304
],
[
1920,
384
],
[
1920,
768
],
[
1920,
1152
],
[
1920,
1536
],
[
1920,
1920
],
[
1920,
2304
],
[
2304,
384
],
[
2304,
768
],
[
2304,
1152
],
[
2304,
1536
],
[
2304,
1920
],
[
2304,
2304
]
],
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "LlavaOnevisionImageProcessor",
"image_std": [
0.5,
0.5,
0.5
],
"processor_class": "LlavaOnevisionProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"size": {
"height": 384,
"width": 384
}
}

7
processor_config.json Normal file
View File

@@ -0,0 +1,7 @@
{
"image_token": "<image>",
"num_image_tokens": 729,
"processor_class": "LlavaOnevisionProcessor",
"video_token": "<video>",
"vision_feature_select_strategy": "full"
}

20
special_tokens_map.json Normal file
View File

@@ -0,0 +1,20 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

303137
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

64
tokenizer_config.json Normal file
View File

@@ -0,0 +1,64 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<image>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<video>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"bos_token": null,
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"max_length": null,
"model_max_length": 32768,
"pad_to_multiple_of": null,
"pad_token": "<|endoftext|>",
"pad_token_type_id": 0,
"padding_side": "right",
"processor_class": "LlavaOnevisionProcessor",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

33
video_preprocessor_config.json Normal file
View File

@@ -0,0 +1,33 @@
{
"crop_size": null,
"default_to_square": false,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": true,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": false,
"fps": null,
"image_mean": [
0.5,
0.5,
0.5
],
"image_std": [
0.5,
0.5,
0.5
],
"num_frames": null,
"processor_class": "LlavaOnevisionProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"size": {
"height": 384,
"width": 384
},
"size_divisor": null,
"video_metadata": null,
"video_processor_type": "LlavaOnevisionVideoProcessor"
}

25
video_processor/preprocessor_config.json Normal file
View File

@@ -0,0 +1,25 @@
{
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": true,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_type": "LlavaOnevisionVideoProcessor",
"image_std": [
0.5,
0.5,
0.5
],
"processor_class": "LlavaOnevisionProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"size": {
"height": 384,
"width": 384
}
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long