初始化项目，由ModelHub XC社区提供模型
Model: LifetimeMistake/Qwen3-VL-Embedding-2B-AWQ-4bit Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
129
README.md
Normal file
129
README.md
Normal file
@@ -0,0 +1,129 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
library_name: transformers
|
||||
base_model: Qwen/Qwen3-VL-Embedding-2B
|
||||
base_model_relation: quantized
|
||||
pipeline_tag: feature-extraction
|
||||
tags:
|
||||
- transformers
|
||||
- qwen3_vl
|
||||
- multimodal embedding
|
||||
- embedding
|
||||
- feature-extraction
|
||||
- quantized
|
||||
- awq
|
||||
- 4bit
|
||||
- compressed-tensors
|
||||
- custom_code
|
||||
language:
|
||||
- en
|
||||
- zh
|
||||
- multilingual
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<img src="https://model-demo.oss-cn-hangzhou.aliyuncs.com/Qwen3-VL-Embedding.png" width="400"/>
|
||||
</p>
|
||||
|
||||
# Qwen3-VL-Embedding-2B-AWQ-4bit
|
||||
|
||||
[](https://arxiv.org/abs/2601.04720)
|
||||
[](https://qwen.ai/blog?id=qwen3-vl-embedding)
|
||||
[](https://github.com/QwenLM/Qwen3-VL-Embedding)
|
||||
[](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B)
|
||||
|
||||
## Quantized Model Overview
|
||||
|
||||
This repository contains a 4-bit AWQ derivative of [`Qwen/Qwen3-VL-Embedding-2B`](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) prepared for direct vLLM deployment through the `compressed-tensors` backend.
|
||||
|
||||
### What Was Quantized
|
||||
|
||||
- **Quantization method:** `llm-compressor` AWQ (`W4A16_ASYM`)
|
||||
- **Export format:** `compressed-tensors`
|
||||
- **Runtime backend:** vLLM `compressed-tensors`
|
||||
- **Weight format:** 4-bit grouped asymmetric integer weights
|
||||
- **Group size:** `128`
|
||||
- **Calibration pipeline:** `layer_sequential`
|
||||
- **Quantized modules:** text-side `Linear` layers in the Qwen3-VL decoder
|
||||
- **Left unquantized:** all `model.visual*` modules and `lm_head`
|
||||
|
||||
### Calibration Data
|
||||
|
||||
This checkpoint was built from the same **1000-sample mixed retrieval manifest** as the FP16 and NVFP4 workflow, but the final AWQ pass used **876 text-only samples** and skipped **124 image-bearing rows** because the vision stack remained excluded from quantization.
|
||||
|
||||
Calibration sources:
|
||||
|
||||
- Polish text retrieval: `mteb/MSMARCO-PL`, `mteb/NQ-PL`, `mteb/FiQA-PL`
|
||||
- Multilingual text retrieval: MIRACL hard-negative slices for `en`, `de`, `es`, `fr`, `ja`
|
||||
- Multimodal retrieval in the master manifest: `vidore/colpali_train_set` and `lmms-lab/flickr30k`
|
||||
- Hard-negative augmentation: MIRACL-derived negatives
|
||||
|
||||
### Local Benchmark Setup
|
||||
|
||||
The numbers below are from local full benchmark runs using the same harness for stock FP16 and quantized checkpoints.
|
||||
|
||||
Benchmark tasks:
|
||||
|
||||
- `mteb/MSMARCO-PL`
|
||||
- `mteb/NQ-PL`
|
||||
- MIRACL hard-negative slices: `en`, `de`, `es`, `fr`, `ja`
|
||||
- `vidore/vidore_v3_industrial`
|
||||
- `vidore/vidore_v3_computer_science`
|
||||
|
||||
Metrics:
|
||||
|
||||
- `nDCG@10`
|
||||
- `Recall@10`
|
||||
- `MRR@10`
|
||||
|
||||
### Baseline Comparison
|
||||
|
||||
Compared with the stock FP16 `Qwen/Qwen3-VL-Embedding-2B` checkpoint on the local full benchmark:
|
||||
|
||||
| Metric | Stock FP16 | AWQ 4-bit | Delta |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| `nDCG@10` | `0.56222` | `0.54474` | `-0.01748` |
|
||||
| `Recall@10` | `0.64934` | `0.63544` | `-0.01390` |
|
||||
| `MRR@10` | `0.78883` | `0.80040` | `+0.01157` |
|
||||
| Benchmark wall time | `434.853 s` | `435.140 s` | `0.07% slower` |
|
||||
| Average request latency | `0.332726 s` | `0.333469 s` | `+0.000743 s` |
|
||||
| Throughput | `18.4338 rps` | `18.4217 rps` | `-0.0121 rps` |
|
||||
|
||||
Notes:
|
||||
|
||||
- This was the **better multimodal** quantized checkpoint of the two we tested.
|
||||
- It preserved the ViDoRe image benchmarks substantially better than NVFP4 and improved `vidore_v3_computer_science` over the FP16 baseline.
|
||||
- It did **not** produce a meaningful runtime speedup versus the FP16 checkpoint in this harness.
|
||||
- The AWQ export is larger than the NVFP4 export and took much longer to build.
|
||||
|
||||
### vLLM Usage
|
||||
|
||||
```bash
|
||||
HF_TOKEN=hf_xxx \
|
||||
vllm serve LifetimeMistake/Qwen3-VL-Embedding-2B-AWQ-4bit \
|
||||
--runner pooling \
|
||||
--convert embed \
|
||||
--trust-remote-code \
|
||||
--quantization compressed-tensors \
|
||||
--limit-mm-per-prompt '{"image":1}'
|
||||
```
|
||||
|
||||
If your vLLM build does not automatically pick up the bundled `chat_template.jinja`, download the repo locally and pass `--chat-template /path/to/chat_template.jinja`.
|
||||
|
||||
## Base Model Introduction
|
||||
|
||||
This model is a quantized derivative of [`Qwen/Qwen3-VL-Embedding-2B`](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B), the 2B member of Qwen’s multimodal embedding series.
|
||||
|
||||
Upstream model highlights:
|
||||
|
||||
- Multimodal inputs: text, images, screenshots, video, and mixed text+vision inputs
|
||||
- 30+ language support
|
||||
- 32k context length
|
||||
- Output dimension up to `2048`, with support for smaller embedding dimensions
|
||||
- Instruction-aware retrieval behavior, with English instructions recommended even for multilingual tasks
|
||||
|
||||
For the full base model card, broader benchmark tables, and upstream usage examples, see:
|
||||
|
||||
- Base model: https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B
|
||||
- GitHub: https://github.com/QwenLM/Qwen3-VL-Embedding
|
||||
- Technical report: https://arxiv.org/abs/2601.04720
|
||||
28
added_tokens.json
Normal file
28
added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"</think>": 151668,
|
||||
"</tool_call>": 151658,
|
||||
"</tool_response>": 151666,
|
||||
"<think>": 151667,
|
||||
"<tool_call>": 151657,
|
||||
"<tool_response>": 151665,
|
||||
"<|box_end|>": 151649,
|
||||
"<|box_start|>": 151648,
|
||||
"<|endoftext|>": 151643,
|
||||
"<|file_sep|>": 151664,
|
||||
"<|fim_middle|>": 151660,
|
||||
"<|fim_pad|>": 151662,
|
||||
"<|fim_prefix|>": 151659,
|
||||
"<|fim_suffix|>": 151661,
|
||||
"<|im_end|>": 151645,
|
||||
"<|im_start|>": 151644,
|
||||
"<|image_pad|>": 151655,
|
||||
"<|object_ref_end|>": 151647,
|
||||
"<|object_ref_start|>": 151646,
|
||||
"<|quad_end|>": 151651,
|
||||
"<|quad_start|>": 151650,
|
||||
"<|repo_name|>": 151663,
|
||||
"<|video_pad|>": 151656,
|
||||
"<|vision_end|>": 151653,
|
||||
"<|vision_pad|>": 151654,
|
||||
"<|vision_start|>": 151652
|
||||
}
|
||||
125
chat_template.jinja
Normal file
125
chat_template.jinja
Normal file
@@ -0,0 +1,125 @@
|
||||
{%- set default_system_message = 'Represent the user\'s input.' -%}
|
||||
{%- if tools %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{%- if messages[0].content is string %}
|
||||
{{- messages[0].content }}
|
||||
{%- else %}
|
||||
{%- for content in messages[0].content %}
|
||||
{%- if 'text' in content %}
|
||||
{{- content.text }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '\n\n' }}
|
||||
{%- else %}
|
||||
{{- default_system_message + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||
{%- else %}
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if messages[0].content is string %}
|
||||
{{- messages[0].content }}
|
||||
{%- else %}
|
||||
{%- for content in messages[0].content %}
|
||||
{%- if 'text' in content %}
|
||||
{{- content.text }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>system\n' + default_system_message + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- for message in messages %}
|
||||
{%- if message.role == "user" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' }}
|
||||
{%- if message.content is string %}
|
||||
{{- message.content }}
|
||||
{%- else %}
|
||||
{%- for content in message.content %}
|
||||
{%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
|
||||
<|vision_start|><|image_pad|><|vision_end|>
|
||||
{%- elif content.type == 'video' or 'video' in content %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
|
||||
<|vision_start|><|video_pad|><|vision_end|>
|
||||
{%- elif 'text' in content %}
|
||||
{{- content.text }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' }}
|
||||
{%- if message.content is string %}
|
||||
{{- message.content }}
|
||||
{%- else %}
|
||||
{%- for content_item in message.content %}
|
||||
{%- if 'text' in content_item %}
|
||||
{{- content_item.text }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if (loop.first and message.content) or (not loop.first) %}
|
||||
{{- '\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.function %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n{"name": "' }}
|
||||
{{- tool_call.name }}
|
||||
{{- '", "arguments": ' }}
|
||||
{%- if tool_call.arguments is string %}
|
||||
{{- tool_call.arguments }}
|
||||
{%- else %}
|
||||
{{- tool_call.arguments | tojson }}
|
||||
{%- endif %}
|
||||
{{- '}\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{%- if message.content is string %}
|
||||
{{- message.content }}
|
||||
{%- else %}
|
||||
{%- for content in message.content %}
|
||||
{%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
|
||||
<|vision_start|><|image_pad|><|vision_end|>
|
||||
{%- elif content.type == 'video' or 'video' in content %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
|
||||
<|vision_start|><|video_pad|><|vision_end|>
|
||||
{%- elif 'text' in content %}
|
||||
{{- content.text }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- endif %}
|
||||
205
config.json
Normal file
205
config.json
Normal file
@@ -0,0 +1,205 @@
|
||||
{
|
||||
"architectures": [
|
||||
"Qwen3VLForConditionalGeneration"
|
||||
],
|
||||
"dtype": "bfloat16",
|
||||
"image_token_id": 151655,
|
||||
"model_type": "qwen3_vl",
|
||||
"text_config": {
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 151643,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 151645,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2048,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 6144,
|
||||
"max_position_embeddings": 262144,
|
||||
"model_type": "qwen3_vl_text",
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 28,
|
||||
"num_key_value_heads": 8,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": {
|
||||
"mrope_interleaved": true,
|
||||
"mrope_section": [
|
||||
24,
|
||||
20,
|
||||
20
|
||||
],
|
||||
"rope_type": "default"
|
||||
},
|
||||
"rope_theta": 5000000,
|
||||
"tie_word_embeddings": true,
|
||||
"use_cache": true,
|
||||
"vocab_size": 151936
|
||||
},
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "4.57.1",
|
||||
"use_cache": false,
|
||||
"video_token_id": 151656,
|
||||
"vision_config": {
|
||||
"deepstack_visual_indexes": [
|
||||
5,
|
||||
11,
|
||||
17
|
||||
],
|
||||
"depth": 24,
|
||||
"dtype": "bfloat16",
|
||||
"hidden_act": "gelu_pytorch_tanh",
|
||||
"hidden_size": 1024,
|
||||
"in_channels": 3,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"model_type": "qwen3_vl",
|
||||
"num_heads": 16,
|
||||
"num_position_embeddings": 2304,
|
||||
"out_hidden_size": 2048,
|
||||
"patch_size": 16,
|
||||
"spatial_merge_size": 2,
|
||||
"temporal_patch_size": 2
|
||||
},
|
||||
"vision_end_token_id": 151653,
|
||||
"vision_start_token_id": 151652,
|
||||
"quantization_config": {
|
||||
"config_groups": {
|
||||
"group_0": {
|
||||
"format": "pack-quantized",
|
||||
"input_activations": null,
|
||||
"output_activations": null,
|
||||
"targets": [
|
||||
"Linear"
|
||||
],
|
||||
"weights": {
|
||||
"actorder": null,
|
||||
"block_structure": null,
|
||||
"dynamic": false,
|
||||
"group_size": 128,
|
||||
"num_bits": 4,
|
||||
"observer": "minmax",
|
||||
"observer_kwargs": {},
|
||||
"strategy": "group",
|
||||
"symmetric": false,
|
||||
"type": "int"
|
||||
}
|
||||
}
|
||||
},
|
||||
"format": "pack-quantized",
|
||||
"global_compression_ratio": null,
|
||||
"ignore": [
|
||||
"lm_head",
|
||||
"model.visual.blocks.0.attn.proj",
|
||||
"model.visual.blocks.0.attn.qkv",
|
||||
"model.visual.blocks.0.mlp.linear_fc1",
|
||||
"model.visual.blocks.0.mlp.linear_fc2",
|
||||
"model.visual.blocks.1.attn.proj",
|
||||
"model.visual.blocks.1.attn.qkv",
|
||||
"model.visual.blocks.1.mlp.linear_fc1",
|
||||
"model.visual.blocks.1.mlp.linear_fc2",
|
||||
"model.visual.blocks.10.attn.proj",
|
||||
"model.visual.blocks.10.attn.qkv",
|
||||
"model.visual.blocks.10.mlp.linear_fc1",
|
||||
"model.visual.blocks.10.mlp.linear_fc2",
|
||||
"model.visual.blocks.11.attn.proj",
|
||||
"model.visual.blocks.11.attn.qkv",
|
||||
"model.visual.blocks.11.mlp.linear_fc1",
|
||||
"model.visual.blocks.11.mlp.linear_fc2",
|
||||
"model.visual.blocks.12.attn.proj",
|
||||
"model.visual.blocks.12.attn.qkv",
|
||||
"model.visual.blocks.12.mlp.linear_fc1",
|
||||
"model.visual.blocks.12.mlp.linear_fc2",
|
||||
"model.visual.blocks.13.attn.proj",
|
||||
"model.visual.blocks.13.attn.qkv",
|
||||
"model.visual.blocks.13.mlp.linear_fc1",
|
||||
"model.visual.blocks.13.mlp.linear_fc2",
|
||||
"model.visual.blocks.14.attn.proj",
|
||||
"model.visual.blocks.14.attn.qkv",
|
||||
"model.visual.blocks.14.mlp.linear_fc1",
|
||||
"model.visual.blocks.14.mlp.linear_fc2",
|
||||
"model.visual.blocks.15.attn.proj",
|
||||
"model.visual.blocks.15.attn.qkv",
|
||||
"model.visual.blocks.15.mlp.linear_fc1",
|
||||
"model.visual.blocks.15.mlp.linear_fc2",
|
||||
"model.visual.blocks.16.attn.proj",
|
||||
"model.visual.blocks.16.attn.qkv",
|
||||
"model.visual.blocks.16.mlp.linear_fc1",
|
||||
"model.visual.blocks.16.mlp.linear_fc2",
|
||||
"model.visual.blocks.17.attn.proj",
|
||||
"model.visual.blocks.17.attn.qkv",
|
||||
"model.visual.blocks.17.mlp.linear_fc1",
|
||||
"model.visual.blocks.17.mlp.linear_fc2",
|
||||
"model.visual.blocks.18.attn.proj",
|
||||
"model.visual.blocks.18.attn.qkv",
|
||||
"model.visual.blocks.18.mlp.linear_fc1",
|
||||
"model.visual.blocks.18.mlp.linear_fc2",
|
||||
"model.visual.blocks.19.attn.proj",
|
||||
"model.visual.blocks.19.attn.qkv",
|
||||
"model.visual.blocks.19.mlp.linear_fc1",
|
||||
"model.visual.blocks.19.mlp.linear_fc2",
|
||||
"model.visual.blocks.2.attn.proj",
|
||||
"model.visual.blocks.2.attn.qkv",
|
||||
"model.visual.blocks.2.mlp.linear_fc1",
|
||||
"model.visual.blocks.2.mlp.linear_fc2",
|
||||
"model.visual.blocks.20.attn.proj",
|
||||
"model.visual.blocks.20.attn.qkv",
|
||||
"model.visual.blocks.20.mlp.linear_fc1",
|
||||
"model.visual.blocks.20.mlp.linear_fc2",
|
||||
"model.visual.blocks.21.attn.proj",
|
||||
"model.visual.blocks.21.attn.qkv",
|
||||
"model.visual.blocks.21.mlp.linear_fc1",
|
||||
"model.visual.blocks.21.mlp.linear_fc2",
|
||||
"model.visual.blocks.22.attn.proj",
|
||||
"model.visual.blocks.22.attn.qkv",
|
||||
"model.visual.blocks.22.mlp.linear_fc1",
|
||||
"model.visual.blocks.22.mlp.linear_fc2",
|
||||
"model.visual.blocks.23.attn.proj",
|
||||
"model.visual.blocks.23.attn.qkv",
|
||||
"model.visual.blocks.23.mlp.linear_fc1",
|
||||
"model.visual.blocks.23.mlp.linear_fc2",
|
||||
"model.visual.blocks.3.attn.proj",
|
||||
"model.visual.blocks.3.attn.qkv",
|
||||
"model.visual.blocks.3.mlp.linear_fc1",
|
||||
"model.visual.blocks.3.mlp.linear_fc2",
|
||||
"model.visual.blocks.4.attn.proj",
|
||||
"model.visual.blocks.4.attn.qkv",
|
||||
"model.visual.blocks.4.mlp.linear_fc1",
|
||||
"model.visual.blocks.4.mlp.linear_fc2",
|
||||
"model.visual.blocks.5.attn.proj",
|
||||
"model.visual.blocks.5.attn.qkv",
|
||||
"model.visual.blocks.5.mlp.linear_fc1",
|
||||
"model.visual.blocks.5.mlp.linear_fc2",
|
||||
"model.visual.blocks.6.attn.proj",
|
||||
"model.visual.blocks.6.attn.qkv",
|
||||
"model.visual.blocks.6.mlp.linear_fc1",
|
||||
"model.visual.blocks.6.mlp.linear_fc2",
|
||||
"model.visual.blocks.7.attn.proj",
|
||||
"model.visual.blocks.7.attn.qkv",
|
||||
"model.visual.blocks.7.mlp.linear_fc1",
|
||||
"model.visual.blocks.7.mlp.linear_fc2",
|
||||
"model.visual.blocks.8.attn.proj",
|
||||
"model.visual.blocks.8.attn.qkv",
|
||||
"model.visual.blocks.8.mlp.linear_fc1",
|
||||
"model.visual.blocks.8.mlp.linear_fc2",
|
||||
"model.visual.blocks.9.attn.proj",
|
||||
"model.visual.blocks.9.attn.qkv",
|
||||
"model.visual.blocks.9.mlp.linear_fc1",
|
||||
"model.visual.blocks.9.mlp.linear_fc2",
|
||||
"model.visual.deepstack_merger_list.0.linear_fc1",
|
||||
"model.visual.deepstack_merger_list.0.linear_fc2",
|
||||
"model.visual.deepstack_merger_list.1.linear_fc1",
|
||||
"model.visual.deepstack_merger_list.1.linear_fc2",
|
||||
"model.visual.deepstack_merger_list.2.linear_fc1",
|
||||
"model.visual.deepstack_merger_list.2.linear_fc2",
|
||||
"model.visual.merger.linear_fc1",
|
||||
"model.visual.merger.linear_fc2"
|
||||
],
|
||||
"kv_cache_scheme": null,
|
||||
"quant_method": "compressed-tensors",
|
||||
"quantization_status": "compressed",
|
||||
"sparsity_config": {},
|
||||
"transform_config": {},
|
||||
"version": "0.12.2"
|
||||
}
|
||||
}
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 151643,
|
||||
"eos_token_id": 151645,
|
||||
"transformers_version": "4.57.6",
|
||||
"use_cache": false
|
||||
}
|
||||
151388
merges.txt
Normal file
151388
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2e1b04df208947055a5fd46db833460c02108fdf1859535de84e1bb8802a6e77
|
||||
size 2791145616
|
||||
39
preprocessor_config.json
Normal file
39
preprocessor_config.json
Normal file
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"crop_size": null,
|
||||
"data_format": "channels_first",
|
||||
"default_to_square": true,
|
||||
"device": null,
|
||||
"disable_grouping": null,
|
||||
"do_center_crop": null,
|
||||
"do_convert_rgb": true,
|
||||
"do_normalize": true,
|
||||
"do_pad": null,
|
||||
"do_rescale": true,
|
||||
"do_resize": true,
|
||||
"image_mean": [
|
||||
0.5,
|
||||
0.5,
|
||||
0.5
|
||||
],
|
||||
"image_processor_type": "Qwen2VLImageProcessorFast",
|
||||
"image_std": [
|
||||
0.5,
|
||||
0.5,
|
||||
0.5
|
||||
],
|
||||
"input_data_format": null,
|
||||
"max_pixels": 1310720,
|
||||
"merge_size": 2,
|
||||
"min_pixels": 4096,
|
||||
"pad_size": null,
|
||||
"patch_size": 16,
|
||||
"processor_class": "Qwen3VLProcessor",
|
||||
"resample": 3,
|
||||
"rescale_factor": 0.00392156862745098,
|
||||
"return_tensors": null,
|
||||
"size": {
|
||||
"longest_edge": 1310720,
|
||||
"shortest_edge": 4096
|
||||
},
|
||||
"temporal_patch_size": 2
|
||||
}
|
||||
46
quantize_summary.json
Normal file
46
quantize_summary.json
Normal file
@@ -0,0 +1,46 @@
|
||||
{
|
||||
"timestamp": 1774921447,
|
||||
"model": "Qwen/Qwen3-VL-Embedding-2B",
|
||||
"output_dir": "/home/szymon/source/repos/Qwen3-VL-Embedding-2B-quant/artifacts/checkpoints/qwen3_vl_embedding_2b_awq_v1",
|
||||
"quant_mode": "awq",
|
||||
"quantizer_impl": "llmcompressor",
|
||||
"modelopt_cfg": null,
|
||||
"llmcompressor_scheme": "W4A16_ASYM",
|
||||
"llmcompressor_pipeline": "layer_sequential",
|
||||
"llmcompressor_sequential_targets": [
|
||||
"Qwen3VLTextDecoderLayer"
|
||||
],
|
||||
"vllm_quantization_backend": "compressed-tensors",
|
||||
"vllm_dtype": null,
|
||||
"manifest": "/home/szymon/source/repos/Qwen3-VL-Embedding-2B-quant/data/calibration/master/manifests/calibration_manifest.jsonl",
|
||||
"num_calib_samples": 876,
|
||||
"torch_dtype": "bfloat16",
|
||||
"device": "cuda",
|
||||
"disable_globs": [
|
||||
"*visual*",
|
||||
"*vision*"
|
||||
],
|
||||
"skipped_image_rows": 124,
|
||||
"elapsed_s": 936.56,
|
||||
"copied_remote_files": [
|
||||
"added_tokens.json",
|
||||
"chat_template.jinja",
|
||||
"config.json",
|
||||
"merges.txt",
|
||||
"preprocessor_config.json",
|
||||
"special_tokens_map.json",
|
||||
"tokenizer.json",
|
||||
"tokenizer_config.json",
|
||||
"video_preprocessor_config.json",
|
||||
"vocab.json"
|
||||
],
|
||||
"export_checks": {
|
||||
"config_json": true,
|
||||
"weights": true,
|
||||
"tokenizer_config_json": true,
|
||||
"tokenizer_json_or_vocab": true,
|
||||
"chat_template": true,
|
||||
"processor_definition": true,
|
||||
"compressed_tensors_config": true
|
||||
}
|
||||
}
|
||||
17
recipe.yaml
Normal file
17
recipe.yaml
Normal file
@@ -0,0 +1,17 @@
|
||||
default_stage:
|
||||
default_modifiers:
|
||||
AWQModifier:
|
||||
targets: [Linear]
|
||||
ignore: ['re:.*lm_head.*', 're:.*vision.*', 're:.*visual.*']
|
||||
scheme: W4A16_ASYM
|
||||
mappings:
|
||||
- smooth_layer: re:.*input_layernorm$
|
||||
balance_layers: ['re:.*q_proj$', 're:.*k_proj$', 're:.*v_proj$']
|
||||
- smooth_layer: re:.*v_proj$
|
||||
balance_layers: ['re:.*o_proj$']
|
||||
- smooth_layer: re:.*post_attention_layernorm$
|
||||
balance_layers: ['re:.*gate_proj$', 're:.*up_proj$']
|
||||
- smooth_layer: re:.*up_proj$
|
||||
balance_layers: ['re:.*down_proj$']
|
||||
offload_device: !!python/object/apply:torch.device [cpu]
|
||||
duo_scaling: false
|
||||
337
scripts/qwen3_vl_embedding.py
Normal file
337
scripts/qwen3_vl_embedding.py
Normal file
@@ -0,0 +1,337 @@
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
import unicodedata
|
||||
import numpy as np
|
||||
import logging
|
||||
|
||||
from PIL import Image
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, List, Union, Dict, Any
|
||||
from transformers.models.qwen3_vl.modeling_qwen3_vl import Qwen3VLPreTrainedModel, Qwen3VLModel, Qwen3VLConfig
|
||||
from transformers.models.qwen3_vl.processing_qwen3_vl import Qwen3VLProcessor
|
||||
from transformers.modeling_outputs import ModelOutput
|
||||
from transformers.processing_utils import Unpack
|
||||
from transformers.utils import TransformersKwargs
|
||||
from transformers.cache_utils import Cache
|
||||
from transformers.utils.generic import check_model_inputs
|
||||
from qwen_vl_utils.vision_process import process_vision_info
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Constants for configuration
|
||||
MAX_LENGTH = 8192
|
||||
IMAGE_BASE_FACTOR = 16
|
||||
IMAGE_FACTOR = IMAGE_BASE_FACTOR * 2
|
||||
MIN_PIXELS = 4 * IMAGE_FACTOR * IMAGE_FACTOR
|
||||
MAX_PIXELS = 1800 * IMAGE_FACTOR * IMAGE_FACTOR
|
||||
FPS = 1
|
||||
MAX_FRAMES = 64
|
||||
FRAME_MAX_PIXELS = 768 * IMAGE_FACTOR * IMAGE_FACTOR
|
||||
MAX_TOTAL_PIXELS = 10 * FRAME_MAX_PIXELS
|
||||
PAD_TOKEN = "<|endoftext|>"
|
||||
|
||||
# Define output structure for embeddings
|
||||
@dataclass
|
||||
class Qwen3VLForEmbeddingOutput(ModelOutput):
|
||||
last_hidden_state: Optional[torch.FloatTensor] = None
|
||||
attention_mask: Optional[torch.Tensor] = None
|
||||
|
||||
# Define model class to compute embeddings
|
||||
class Qwen3VLForEmbedding(Qwen3VLPreTrainedModel):
|
||||
_checkpoint_conversion_mapping = {}
|
||||
accepts_loss_kwargs = False
|
||||
config: Qwen3VLConfig
|
||||
|
||||
def __init__(self, config):
|
||||
super().__init__(config)
|
||||
self.model = Qwen3VLModel(config)
|
||||
self.post_init()
|
||||
|
||||
def get_input_embeddings(self):
|
||||
return self.model.get_input_embeddings()
|
||||
|
||||
def set_input_embeddings(self, value):
|
||||
self.model.set_input_embeddings(value)
|
||||
|
||||
def set_decoder(self, decoder):
|
||||
self.model.set_decoder(decoder)
|
||||
|
||||
def get_decoder(self):
|
||||
return self.model.get_decoder()
|
||||
|
||||
# Extract video features from model
|
||||
def get_video_features(self, pixel_values_videos: torch.FloatTensor,
|
||||
video_grid_thw: Optional[torch.LongTensor] = None):
|
||||
return self.model.get_video_features(pixel_values_videos, video_grid_thw)
|
||||
|
||||
# Extract image features from model
|
||||
def get_image_features(self, pixel_values: torch.FloatTensor,
|
||||
image_grid_thw: Optional[torch.LongTensor] = None):
|
||||
return self.model.get_image_features(pixel_values, image_grid_thw)
|
||||
|
||||
# Make modules accessible through properties
|
||||
@property
|
||||
def language_model(self):
|
||||
return self.model.language_model
|
||||
|
||||
@property
|
||||
def visual(self):
|
||||
return self.model.visual
|
||||
|
||||
# Forward pass through model with input parameters
|
||||
# @check_model_inputs
|
||||
def forward(self,
|
||||
input_ids: torch.LongTensor = None,
|
||||
attention_mask: Optional[torch.Tensor] = None,
|
||||
position_ids: Optional[torch.LongTensor] = None,
|
||||
past_key_values: Optional[Cache] = None,
|
||||
inputs_embeds: Optional[torch.FloatTensor] = None,
|
||||
pixel_values: Optional[torch.Tensor] = None,
|
||||
pixel_values_videos: Optional[torch.FloatTensor] = None,
|
||||
image_grid_thw: Optional[torch.LongTensor] = None,
|
||||
video_grid_thw: Optional[torch.LongTensor] = None,
|
||||
cache_position: Optional[torch.LongTensor] = None,
|
||||
logits_to_keep: Union[int, torch.Tensor] = 0,
|
||||
**kwargs: Unpack[TransformersKwargs],
|
||||
) -> Union[tuple, Qwen3VLForEmbeddingOutput]:
|
||||
# Pass inputs through the model
|
||||
outputs = self.model(
|
||||
input_ids=input_ids,
|
||||
pixel_values=pixel_values,
|
||||
pixel_values_videos=pixel_values_videos,
|
||||
image_grid_thw=image_grid_thw,
|
||||
video_grid_thw=video_grid_thw,
|
||||
position_ids=position_ids,
|
||||
attention_mask=attention_mask,
|
||||
past_key_values=past_key_values,
|
||||
inputs_embeds=inputs_embeds,
|
||||
cache_position=cache_position,
|
||||
**kwargs,
|
||||
)
|
||||
# Return the model output
|
||||
return Qwen3VLForEmbeddingOutput(
|
||||
last_hidden_state=outputs.last_hidden_state,
|
||||
attention_mask=attention_mask,
|
||||
)
|
||||
|
||||
def sample_frames(frames: List[Union[str, Image.Image]], num_segments: int, max_segments: int) -> List[str]:
|
||||
duration = len(frames)
|
||||
frame_id_array = np.linspace(0, duration - 1, num_segments, dtype=int)
|
||||
frame_id_list = frame_id_array.tolist()
|
||||
last_frame_id = frame_id_list[-1]
|
||||
|
||||
# Create a list of sampled frames
|
||||
sampled_frames = []
|
||||
for frame_idx in frame_id_list:
|
||||
try:
|
||||
sampled_frames.append(frames[frame_idx])
|
||||
except:
|
||||
break
|
||||
# Ensure the sampled list meets the required segment count
|
||||
while len(sampled_frames) < num_segments:
|
||||
sampled_frames.append(frames[last_frame_id])
|
||||
return sampled_frames[:max_segments]
|
||||
|
||||
# Define embedder class for processing inputs and generating embeddings
|
||||
class Qwen3VLEmbedder():
|
||||
def __init__(
|
||||
self,
|
||||
model_name_or_path: str,
|
||||
max_length: int = MAX_LENGTH,
|
||||
min_pixels: int = MIN_PIXELS,
|
||||
max_pixels: int = MAX_PIXELS,
|
||||
total_pixels: int = MAX_TOTAL_PIXELS,
|
||||
fps: float = FPS,
|
||||
num_frames: int = MAX_FRAMES,
|
||||
max_frames: int = MAX_FRAMES,
|
||||
default_instruction: str = "Represent the user's input.",
|
||||
**kwargs
|
||||
):
|
||||
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
|
||||
self.max_length = max_length
|
||||
self.min_pixels = min_pixels
|
||||
self.max_pixels = max_pixels
|
||||
self.total_pixels = total_pixels
|
||||
self.fps = fps
|
||||
self.num_frames = num_frames
|
||||
self.max_frames = max_frames
|
||||
|
||||
self.default_instruction = default_instruction
|
||||
|
||||
self.model = Qwen3VLForEmbedding.from_pretrained(
|
||||
model_name_or_path, trust_remote_code=True, **kwargs
|
||||
).to(device)
|
||||
self.processor = Qwen3VLProcessor.from_pretrained(
|
||||
model_name_or_path, padding_side='right'
|
||||
)
|
||||
self.model.eval()
|
||||
|
||||
@torch.no_grad()
|
||||
def forward(self, inputs: Dict[str, Any]) -> Dict[str, torch.Tensor]:
|
||||
outputs = self.model(**inputs)
|
||||
return {
|
||||
'last_hidden_state': outputs.last_hidden_state,
|
||||
'attention_mask': inputs.get('attention_mask')
|
||||
}
|
||||
|
||||
# Truncate token sequence to a specified max length
|
||||
def _truncate_tokens(self, token_ids: List[int], max_length: int) -> List[int]:
|
||||
if len(token_ids) <= max_length:
|
||||
return token_ids
|
||||
|
||||
special_token_ids = set(self.processor.tokenizer.all_special_ids)
|
||||
num_special = sum(1 for token_idx in token_ids if token_idx in special_token_ids)
|
||||
num_non_special_to_keep = max_length - num_special
|
||||
|
||||
final_token_ids = []
|
||||
non_special_kept_count = 0
|
||||
# Ensure retention of special tokens while truncating the rest
|
||||
for token_idx in token_ids:
|
||||
if token_idx in special_token_ids:
|
||||
final_token_ids.append(token_idx)
|
||||
elif non_special_kept_count < num_non_special_to_keep:
|
||||
final_token_ids.append(token_idx)
|
||||
non_special_kept_count += 1
|
||||
return final_token_ids
|
||||
|
||||
# Format input based on provided text, image, video, and instruction
|
||||
def format_model_input(
|
||||
self, text: Optional[str] = None,
|
||||
image: Optional[Union[str, Image.Image]] = None,
|
||||
video: Optional[Union[str, List[Union[str, Image.Image]]]] = None,
|
||||
instruction: Optional[str] = None,
|
||||
fps: Optional[float] = None,
|
||||
max_frames: Optional[int] = None
|
||||
) -> List[Dict]:
|
||||
|
||||
# Ensure instruction ends with punctuation
|
||||
if instruction:
|
||||
instruction = instruction.strip()
|
||||
if instruction and not unicodedata.category(instruction[-1]).startswith('P'):
|
||||
instruction = instruction + '.'
|
||||
|
||||
# Initialize conversation with system prompts
|
||||
content = []
|
||||
conversation = [
|
||||
{"role": "system", "content": [{"type": "text", "text": instruction or self.default_instruction}]},
|
||||
{"role": "user", "content": content}
|
||||
]
|
||||
|
||||
# Add text, image, or video content to conversation
|
||||
if not text and not image and not video:
|
||||
content.append({'type': 'text', 'text': "NULL"})
|
||||
return conversation
|
||||
|
||||
if video:
|
||||
video_content = None
|
||||
video_kwargs = { 'total_pixels': self.total_pixels }
|
||||
if isinstance(video, list):
|
||||
video_content = video
|
||||
if self.num_frames is not None or self.max_frames is not None:
|
||||
video_content = sample_frames(video_content, self.num_frames, self.max_frames)
|
||||
video_content = [
|
||||
('file://' + ele if isinstance(ele, str) else ele)
|
||||
for ele in video_content
|
||||
]
|
||||
elif isinstance(video, str):
|
||||
video_content = video if video.startswith(('http://', 'https://')) else 'file://' + video
|
||||
video_kwargs = {'fps': fps or self.fps, 'max_frames': max_frames or self.max_frames,}
|
||||
else:
|
||||
raise TypeError(f"Unrecognized video type: {type(video)}")
|
||||
|
||||
# Add video input details to content
|
||||
if video_content:
|
||||
content.append({
|
||||
'type': 'video', 'video': video_content,
|
||||
**video_kwargs
|
||||
})
|
||||
|
||||
if image:
|
||||
image_content = None
|
||||
if isinstance(image, Image.Image):
|
||||
image_content = image
|
||||
elif isinstance(image, str):
|
||||
image_content = image if image.startswith(('http', 'oss')) else 'file://' + image
|
||||
else:
|
||||
raise TypeError(f"Unrecognized image type: {type(image)}")
|
||||
|
||||
# Add image input details to content
|
||||
if image_content:
|
||||
content.append({
|
||||
'type': 'image', 'image': image_content,
|
||||
"min_pixels": self.min_pixels,
|
||||
"max_pixels": self.max_pixels
|
||||
})
|
||||
|
||||
if text:
|
||||
content.append({'type': 'text', 'text': text})
|
||||
|
||||
return conversation
|
||||
|
||||
# Preprocess input conversations for model consumption
|
||||
def _preprocess_inputs(self, conversations: List[List[Dict]]) -> Dict[str, torch.Tensor]:
|
||||
text = self.processor.apply_chat_template(
|
||||
conversations, add_generation_prompt=True, tokenize=False
|
||||
)
|
||||
|
||||
try:
|
||||
images, video_inputs, video_kwargs = process_vision_info(
|
||||
conversations, image_patch_size=16,
|
||||
return_video_metadata=True, return_video_kwargs=True
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error in processing vision info: {e}")
|
||||
images = None
|
||||
video_inputs = None
|
||||
video_kwargs = {'do_sample_frames': False}
|
||||
text = self.processor.apply_chat_template(
|
||||
[{'role': 'user', 'content': [{'type': 'text', 'text': 'NULL'}]}],
|
||||
add_generation_prompt=True, tokenize=False
|
||||
)
|
||||
|
||||
if video_inputs is not None:
|
||||
videos, video_metadata = zip(*video_inputs)
|
||||
videos = list(videos)
|
||||
video_metadata = list(video_metadata)
|
||||
else:
|
||||
videos, video_metadata = None, None
|
||||
|
||||
inputs = self.processor(
|
||||
text=text, images=images, videos=videos, video_metadata=video_metadata, truncation=True,
|
||||
max_length=self.max_length, padding=True, do_resize=False, return_tensors='pt',
|
||||
**video_kwargs
|
||||
)
|
||||
return inputs
|
||||
|
||||
# Pool the last hidden state by attention mask for embeddings
|
||||
@staticmethod
|
||||
def _pooling_last(hidden_state: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
|
||||
flipped_tensor = attention_mask.flip(dims=[1])
|
||||
last_one_positions = flipped_tensor.argmax(dim=1)
|
||||
col = attention_mask.shape[1] - last_one_positions - 1
|
||||
row = torch.arange(hidden_state.shape[0], device=hidden_state.device)
|
||||
return hidden_state[row, col]
|
||||
|
||||
# Process inputs to generate normalized embeddings
|
||||
def process(self, inputs: List[Dict[str, Any]], normalize: bool = True) -> tuple:
|
||||
conversations = [self.format_model_input(
|
||||
text=ele.get('text'),
|
||||
image=ele.get('image'),
|
||||
video=ele.get('video'),
|
||||
instruction=ele.get('instruction'),
|
||||
fps=ele.get('fps'),
|
||||
max_frames=ele.get('max_frames')
|
||||
) for ele in inputs]
|
||||
|
||||
processed_inputs = self._preprocess_inputs(conversations)
|
||||
processed_inputs = {k: v.to(self.model.device) for k, v in processed_inputs.items()}
|
||||
|
||||
outputs = self.forward(processed_inputs)
|
||||
embeddings = self._pooling_last(outputs['last_hidden_state'], outputs['attention_mask'])
|
||||
|
||||
# Normalize the embeddings if specified
|
||||
if normalize:
|
||||
embeddings = F.normalize(embeddings, p=2, dim=-1)
|
||||
|
||||
return embeddings
|
||||
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"eos_token": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:def76fb086971c7867b829c23a26261e38d9d74e02139253b38aeb9df8b4b50a
|
||||
size 11423705
|
||||
239
tokenizer_config.json
Normal file
239
tokenizer_config.json
Normal file
@@ -0,0 +1,239 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"151643": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151644": {
|
||||
"content": "<|im_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151645": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151646": {
|
||||
"content": "<|object_ref_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151647": {
|
||||
"content": "<|object_ref_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151648": {
|
||||
"content": "<|box_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151649": {
|
||||
"content": "<|box_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151650": {
|
||||
"content": "<|quad_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151651": {
|
||||
"content": "<|quad_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151652": {
|
||||
"content": "<|vision_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151653": {
|
||||
"content": "<|vision_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151654": {
|
||||
"content": "<|vision_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151655": {
|
||||
"content": "<|image_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151656": {
|
||||
"content": "<|video_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151657": {
|
||||
"content": "<tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151658": {
|
||||
"content": "</tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151659": {
|
||||
"content": "<|fim_prefix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151660": {
|
||||
"content": "<|fim_middle|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151661": {
|
||||
"content": "<|fim_suffix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151662": {
|
||||
"content": "<|fim_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151663": {
|
||||
"content": "<|repo_name|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151664": {
|
||||
"content": "<|file_sep|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151665": {
|
||||
"content": "<tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151666": {
|
||||
"content": "</tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151667": {
|
||||
"content": "<think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151668": {
|
||||
"content": "</think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"bos_token": null,
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|im_end|>",
|
||||
"errors": "replace",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 262144,
|
||||
"pad_token": "<|endoftext|>",
|
||||
"split_special_tokens": false,
|
||||
"tokenizer_class": "Qwen2Tokenizer",
|
||||
"unk_token": null
|
||||
}
|
||||
41
video_preprocessor_config.json
Normal file
41
video_preprocessor_config.json
Normal file
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"crop_size": null,
|
||||
"data_format": "channels_first",
|
||||
"default_to_square": true,
|
||||
"device": null,
|
||||
"do_center_crop": null,
|
||||
"do_convert_rgb": true,
|
||||
"do_normalize": true,
|
||||
"do_rescale": true,
|
||||
"do_resize": true,
|
||||
"do_sample_frames": true,
|
||||
"fps": 2,
|
||||
"image_mean": [
|
||||
0.5,
|
||||
0.5,
|
||||
0.5
|
||||
],
|
||||
"image_std": [
|
||||
0.5,
|
||||
0.5,
|
||||
0.5
|
||||
],
|
||||
"input_data_format": null,
|
||||
"max_frames": 768,
|
||||
"merge_size": 2,
|
||||
"min_frames": 4,
|
||||
"num_frames": null,
|
||||
"pad_size": null,
|
||||
"patch_size": 16,
|
||||
"processor_class": "Qwen3VLProcessor",
|
||||
"resample": 3,
|
||||
"rescale_factor": 0.00392156862745098,
|
||||
"return_metadata": false,
|
||||
"size": {
|
||||
"longest_edge": 25165824,
|
||||
"shortest_edge": 4096
|
||||
},
|
||||
"temporal_patch_size": 2,
|
||||
"video_metadata": null,
|
||||
"video_processor_type": "Qwen3VLVideoProcessor"
|
||||
}
|
||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user