初始化项目，由ModelHub XC社区提供模型
Model: microsoft/GUI-Actor-Verifier-2B Source: Original Platform
This commit is contained in:
ModelHub XC
51
.gitattributes
vendored
Normal file
51
.gitattributes
vendored
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zstandard filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.db* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ark* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
|
||||||
|
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gguf* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ggml filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.llamafile* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
|
||||||
|
merges.txt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
|
vocab.json filter=lfs diff=lfs merge=lfs -text
|
||||||
201
README.md
Normal file
201
README.md
Normal file
@@ -0,0 +1,201 @@
|
|||||||
|
---
|
||||||
|
base_model:
|
||||||
|
- ByteDance-Seed/UI-TARS-2B-SFT
|
||||||
|
datasets:
|
||||||
|
- OS-Copilot/OS-Atlas-data
|
||||||
|
license: mit
|
||||||
|
pipeline_tag: image-text-to-text
|
||||||
|
library_name: transformers
|
||||||
|
---
|
||||||
|
|
||||||
|
# GUI-Actor-Verifier-2B
|
||||||
|
|
||||||
|
|
||||||
|
This model was introduced in the paper [**GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents**](https://huggingface.co/papers/2506.03143).
|
||||||
|
It is developed based on [UI-TARS-2B-SFT](https://huggingface.co/ByteDance-Seed/UI-TARS-2B-SFT) and is designed to predict the correctness of an action position given a language instruction. This model is well-suited for **GUI-Actor**, as its attention map effectively provides diverse candidates for verification with only a single inference.
|
||||||
|
|
||||||
|
|
||||||
|
For more details on model design and evaluation, please check: [🏠 Project Page](https://microsoft.github.io/GUI-Actor/) | [💻 Github Repo](https://github.com/microsoft/GUI-Actor) | [📑 Paper](https://huggingface.co/papers/2506.03143).
|
||||||
|
|
||||||
|
|
||||||
|
| Model List | Hugging Face Link |
|
||||||
|
|--------------------------------------------|--------------------------------------------|
|
||||||
|
| **GUI-Actor-7B-Qwen2-VL** | [🤗 Hugging Face](https://huggingface.co/microsoft/GUI-Actor-7B-Qwen2-VL) |
|
||||||
|
| **GUI-Actor-2B-Qwen2-VL** | [🤗 Hugging Face](https://huggingface.co/microsoft/GUI-Actor-2B-Qwen2-VL) |
|
||||||
|
| **GUI-Actor-7B-Qwen2.5-VL** | [🤗 Hugging Face](https://huggingface.co/microsoft/GUI-Actor-7B-Qwen2.5-VL) |
|
||||||
|
| **GUI-Actor-3B-Qwen2.5-VL** | [🤗 Hugging Face](https://huggingface.co/microsoft/GUI-Actor-3B-Qwen2.5-VL) |
|
||||||
|
| **GUI-Actor-Verifier-2B** | [🤗 Hugging Face](https://huggingface.co/microsoft/GUI-Actor-Verifier-2B) |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## 📊 Performance Comparison on GUI Grounding Benchmarks
|
||||||
|
Table 1. Main results on ScreenSpot-Pro, ScreenSpot, and ScreenSpot-v2 with **Qwen2-VL** as the backbone. † indicates scores obtained from our own evaluation of the official models on Huggingface.
|
||||||
|
| Method | Backbone VLM | ScreenSpot-Pro | ScreenSpot | ScreenSpot-v2 |
|
||||||
|
|------------------|--------------|----------------|------------|----------------|
|
||||||
|
| **_72B models:_**
|
||||||
|
| AGUVIS-72B | Qwen2-VL | - | 89.2 | - |
|
||||||
|
| UGround-V1-72B | Qwen2-VL | 34.5 | **89.4** | - |
|
||||||
|
| UI-TARS-72B | Qwen2-VL | **38.1** | 88.4 | **90.3** |
|
||||||
|
| **_7B models:_**
|
||||||
|
| OS-Atlas-7B | Qwen2-VL | 18.9 | 82.5 | 84.1 |
|
||||||
|
| AGUVIS-7B | Qwen2-VL | 22.9 | 84.4 | 86.0† |
|
||||||
|
| UGround-V1-7B | Qwen2-VL | 31.1 | 86.3 | 87.6† |
|
||||||
|
| UI-TARS-7B | Qwen2-VL | 35.7 | 89.5 | **91.6** |
|
||||||
|
| GUI-Actor-7B | Qwen2-VL | 40.7 | 88.3 | 89.5 |
|
||||||
|
| GUI-Actor-7B + Verifier | Qwen2-VL | **44.2** | **89.7** | 90.9 |
|
||||||
|
| **_2B models:_**
|
||||||
|
| UGround-V1-2B | Qwen2-VL | 26.6 | 77.1 | - |
|
||||||
|
| UI-TARS-2B | Qwen2-VL | 27.7 | 82.3 | 84.7 |
|
||||||
|
| GUI-Actor-2B | Qwen2-VL | 36.7 | 86.5 | 88.6 |
|
||||||
|
| GUI-Actor-2B + Verifier | Qwen2-VL | **41.8** | **86.9** | **89.3** |
|
||||||
|
|
||||||
|
Table 2. Main results on the ScreenSpot-Pro and ScreenSpot-v2 with **Qwen2.5-VL** as the backbone.
|
||||||
|
| Method | Backbone VLM | ScreenSpot-Pro | ScreenSpot-v2 |
|
||||||
|
|----------------|---------------|----------------|----------------|
|
||||||
|
| **_7B models:_**
|
||||||
|
| Qwen2.5-VL-7B | Qwen2.5-VL | 27.6 | 88.8 |
|
||||||
|
| Jedi-7B | Qwen2.5-VL | 39.5 | 91.7 |
|
||||||
|
| GUI-Actor-7B | Qwen2.5-VL | 44.6 | 92.1 |
|
||||||
|
| GUI-Actor-7B + Verifier | Qwen2.5-VL | **47.7** | **92.5** |
|
||||||
|
| **_3B models:_**
|
||||||
|
| Qwen2.5-VL-3B | Qwen2.5-VL | 25.9 | 80.9 |
|
||||||
|
| Jedi-3B | Qwen2.5-VL | 36.1 | 88.6 |
|
||||||
|
| GUI-Actor-3B | Qwen2.5-VL | 42.2 | 91.0 |
|
||||||
|
| GUI-Actor-3B + Verifier | Qwen2.5-VL | **45.9** | **92.4** |
|
||||||
|
|
||||||
|
## 🚀 Usage
|
||||||
|
The verifier takes a language instruction and an image with a red circle marking the target position as input. One example is shown below. It outputs either ‘True’ or ‘False’, and you can also use the probability of each label to score the sample.
|
||||||
|
|
||||||
|
For more detailed usage, please refer to our github repo.
|
||||||
|
|
||||||
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/64d45451c34a346181b130dd/1LTBORYJsO9Ru6B4q_SKl.png" alt="image" width="500"/>
|
||||||
|
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
|
||||||
|
from transformers.generation import GenerationConfig
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import os
|
||||||
|
import numpy as np
|
||||||
|
from PIL import Image, ImageDraw
|
||||||
|
from qwen_vl_utils import process_vision_info
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# load model
|
||||||
|
model_name_or_path = "microsoft/GUI-Actor-Verifier-2B"
|
||||||
|
model = Qwen2VLForConditionalGeneration.from_pretrained(
|
||||||
|
model_name_or_path,
|
||||||
|
device_map="cuda:0",
|
||||||
|
trust_remote_code=True,
|
||||||
|
torch_dtype=torch.bfloat16,
|
||||||
|
attn_implementation="flash_attention_2"
|
||||||
|
).eval()
|
||||||
|
output_len = 1
|
||||||
|
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
|
||||||
|
processor = AutoProcessor.from_pretrained(model_name_or_path)
|
||||||
|
|
||||||
|
def draw_annotations(img, point_in_pixel, bbox, output_path='test.png', color='red', size=1):
|
||||||
|
draw = ImageDraw.Draw(img)
|
||||||
|
|
||||||
|
# Draw the ground truth bounding box in green
|
||||||
|
if bbox:
|
||||||
|
# Assuming bbox format is [x1, y1, x2, y2]
|
||||||
|
draw.rectangle(bbox, outline="yellow", width=4)
|
||||||
|
|
||||||
|
# Draw a small circle around the predicted point in red
|
||||||
|
if point_in_pixel:
|
||||||
|
# Create a small rectangle around the point (5 pixels in each direction)
|
||||||
|
radius = np.ceil(8 * size).astype(int)
|
||||||
|
circle_bbox = [
|
||||||
|
point_in_pixel[0] - radius, # x1
|
||||||
|
point_in_pixel[1] - radius, # y1
|
||||||
|
point_in_pixel[0] + radius, # x2
|
||||||
|
point_in_pixel[1] + radius # y2
|
||||||
|
]
|
||||||
|
draw.ellipse(circle_bbox, outline=color, width=np.ceil(4 * size).astype(int))
|
||||||
|
|
||||||
|
return img
|
||||||
|
|
||||||
|
def ground_only_positive(model, tokenizer, processor, instruction, image, point):
|
||||||
|
if isinstance(image, str):
|
||||||
|
image_path = image
|
||||||
|
image = Image.open(image_path)
|
||||||
|
else:
|
||||||
|
image_path = image_to_temp_filename(image)
|
||||||
|
assert os.path.exists(image_path) and os.path.isfile(image_path), "Invalid input image path."
|
||||||
|
|
||||||
|
width, height = image.size
|
||||||
|
image = draw_annotations(image, point, None, output_path=None, size=height/1000 * 1.2)
|
||||||
|
|
||||||
|
prompt_origin = "Please observe the screenshot and exame whether the hollow red circle accurately placed on the intended position in the image: '{}'. Answer True or False."
|
||||||
|
full_prompt = prompt_origin.format(instruction)
|
||||||
|
|
||||||
|
messages = [
|
||||||
|
{
|
||||||
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{
|
||||||
|
"type": "image",
|
||||||
|
"image": image,
|
||||||
|
},
|
||||||
|
{"type": "text", "text": full_prompt},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
]
|
||||||
|
# Preparation for inference
|
||||||
|
text_input = processor.apply_chat_template(
|
||||||
|
messages, tokenize=False, add_generation_prompt=True
|
||||||
|
)
|
||||||
|
image_inputs, video_inputs = process_vision_info(messages)
|
||||||
|
inputs = processor(
|
||||||
|
text=[text_input],
|
||||||
|
images=image_inputs,
|
||||||
|
videos=video_inputs,
|
||||||
|
padding=True,
|
||||||
|
return_tensors="pt",
|
||||||
|
)
|
||||||
|
inputs = inputs.to("cuda:0")
|
||||||
|
|
||||||
|
generated_ids = model.generate(
|
||||||
|
**inputs,
|
||||||
|
max_new_tokens=output_len,
|
||||||
|
do_sample=False,
|
||||||
|
temperature=0.0
|
||||||
|
)
|
||||||
|
|
||||||
|
generated_ids_trimmed = [
|
||||||
|
out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
|
||||||
|
]
|
||||||
|
response = processor.batch_decode(
|
||||||
|
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
print(response)
|
||||||
|
matches = re.findall(r'\b(?:True|False)\b', response)
|
||||||
|
if not len(matches):
|
||||||
|
answer = 'Error Format'
|
||||||
|
else:
|
||||||
|
answer = matches[-1]
|
||||||
|
return answer
|
||||||
|
|
||||||
|
# given the image path and instruction and coorindate
|
||||||
|
instruction = 'close this window'
|
||||||
|
image = Image.open('test.png')
|
||||||
|
width, height = image.size
|
||||||
|
point = [int(0.9709 * width), int(0.1548, * height)] # The point should be in pixels
|
||||||
|
answer = ground_only_positive(model, tokenizer, processor, instruction, image, point) # output True or False
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📝 Citation
|
||||||
|
```
|
||||||
|
@article{wu2025gui,
|
||||||
|
title={GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents},
|
||||||
|
author={Wu, Qianhui and Cheng, Kanzhi and Yang, Rui and Zhang, Chaoyun and Yang, Jianwei and Jiang, Huiqiang and Mu, Jian and Peng, Baolin and Qiao, Bo and Tan, Reuben and others},
|
||||||
|
journal={arXiv preprint arXiv:2506.03143},
|
||||||
|
year={2025}
|
||||||
|
}
|
||||||
|
```
|
||||||
18
added_tokens.json
Normal file
18
added_tokens.json
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"<|box_end|>": 151649,
|
||||||
|
"<|box_start|>": 151648,
|
||||||
|
"<|diff_marker|>": 151658,
|
||||||
|
"<|endoftext|>": 151643,
|
||||||
|
"<|im_end|>": 151645,
|
||||||
|
"<|im_start|>": 151644,
|
||||||
|
"<|image_pad|>": 151655,
|
||||||
|
"<|object_ref_end|>": 151647,
|
||||||
|
"<|object_ref_start|>": 151646,
|
||||||
|
"<|quad_end|>": 151651,
|
||||||
|
"<|quad_start|>": 151650,
|
||||||
|
"<|recipient|>": 151657,
|
||||||
|
"<|video_pad|>": 151656,
|
||||||
|
"<|vision_end|>": 151653,
|
||||||
|
"<|vision_pad|>": 151654,
|
||||||
|
"<|vision_start|>": 151652
|
||||||
|
}
|
||||||
3
chat_template.json
Normal file
3
chat_template.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
{
|
||||||
|
"chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
|
||||||
|
}
|
||||||
59
config.json
Normal file
59
config.json
Normal file
@@ -0,0 +1,59 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen2VLForConditionalGeneration"
|
||||||
|
],
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 151643,
|
||||||
|
"eos_token_id": [
|
||||||
|
151658
|
||||||
|
],
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 1536,
|
||||||
|
"image_token_id": 151655,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 8960,
|
||||||
|
"max_position_embeddings": 32768,
|
||||||
|
"max_window_layers": 28,
|
||||||
|
"model_type": "qwen2_vl",
|
||||||
|
"num_attention_heads": 12,
|
||||||
|
"num_hidden_layers": 28,
|
||||||
|
"num_key_value_heads": 2,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": {
|
||||||
|
"mrope_section": [
|
||||||
|
16,
|
||||||
|
24,
|
||||||
|
24
|
||||||
|
],
|
||||||
|
"rope_type": "default",
|
||||||
|
"type": "default"
|
||||||
|
},
|
||||||
|
"rope_theta": 1000000.0,
|
||||||
|
"sliding_window": 32768,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.50.3",
|
||||||
|
"use_cache": false,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"video_token_id": 151656,
|
||||||
|
"vision_config": {
|
||||||
|
"depth": 32,
|
||||||
|
"embed_dim": 1280,
|
||||||
|
"hidden_act": "quick_gelu",
|
||||||
|
"hidden_size": 1536,
|
||||||
|
"in_channels": 3,
|
||||||
|
"in_chans": 3,
|
||||||
|
"mlp_ratio": 4,
|
||||||
|
"model_type": "qwen2_vl",
|
||||||
|
"num_heads": 16,
|
||||||
|
"patch_size": 14,
|
||||||
|
"spatial_merge_size": 2,
|
||||||
|
"spatial_patch_size": 14,
|
||||||
|
"temporal_patch_size": 2,
|
||||||
|
"torch_dtype": "bfloat16"
|
||||||
|
},
|
||||||
|
"vision_end_token_id": 151653,
|
||||||
|
"vision_start_token_id": 151652,
|
||||||
|
"vision_token_id": 151654,
|
||||||
|
"vocab_size": 151659
|
||||||
|
}
|
||||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
|||||||
|
{"framework": "pytorch", "task": "image-text-to-text", "allow_remote": true}
|
||||||
8
generation_config.json
Normal file
8
generation_config.json
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 151643,
|
||||||
|
"eos_token_id": [
|
||||||
|
151658
|
||||||
|
],
|
||||||
|
"transformers_version": "4.50.3"
|
||||||
|
}
|
||||||
BIN
merges.txt
(Stored with Git LFS)
Normal file
BIN
merges.txt
(Stored with Git LFS)
Normal file
Binary file not shown.
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:be70c29b1293b35d097cbb7d0d8bd21473b7392b3f99649e77d0792ff28e0587
|
||||||
|
size 4417199904
|
||||||
29
preprocessor_config.json
Normal file
29
preprocessor_config.json
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"do_convert_rgb": true,
|
||||||
|
"do_normalize": true,
|
||||||
|
"do_rescale": true,
|
||||||
|
"do_resize": true,
|
||||||
|
"image_mean": [
|
||||||
|
0.48145466,
|
||||||
|
0.4578275,
|
||||||
|
0.40821073
|
||||||
|
],
|
||||||
|
"image_processor_type": "Qwen2VLImageProcessor",
|
||||||
|
"image_std": [
|
||||||
|
0.26862954,
|
||||||
|
0.26130258,
|
||||||
|
0.27577711
|
||||||
|
],
|
||||||
|
"max_pixels": 2116800,
|
||||||
|
"merge_size": 2,
|
||||||
|
"min_pixels": 3136,
|
||||||
|
"patch_size": 14,
|
||||||
|
"processor_class": "Qwen2VLProcessor",
|
||||||
|
"resample": 3,
|
||||||
|
"rescale_factor": 0.00392156862745098,
|
||||||
|
"size": {
|
||||||
|
"longest_edge": 2116800,
|
||||||
|
"shortest_edge": 3136
|
||||||
|
},
|
||||||
|
"temporal_patch_size": 2
|
||||||
|
}
|
||||||
117
special_tokens_map.json
Normal file
117
special_tokens_map.json
Normal file
@@ -0,0 +1,117 @@
|
|||||||
|
{
|
||||||
|
"additional_special_tokens": [
|
||||||
|
{
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|object_ref_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|object_ref_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|box_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|box_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|quad_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|quad_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|vision_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|vision_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|vision_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|image_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|video_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|recipient|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"content": "<|diff_marker|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"eos_token": "<|diff_marker|>",
|
||||||
|
"pad_token": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false
|
||||||
|
}
|
||||||
|
}
|
||||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:b5e3b77495dbe08a59f0f76a2b9c8354d594e2c0cab76e88d9513f9facc9bb91
|
||||||
|
size 11420753
|
||||||
163
tokenizer_config.json
Normal file
163
tokenizer_config.json
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
{
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"added_tokens_decoder": {
|
||||||
|
"151643": {
|
||||||
|
"content": "<|endoftext|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151644": {
|
||||||
|
"content": "<|im_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151645": {
|
||||||
|
"content": "<|im_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151646": {
|
||||||
|
"content": "<|object_ref_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151647": {
|
||||||
|
"content": "<|object_ref_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151648": {
|
||||||
|
"content": "<|box_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151649": {
|
||||||
|
"content": "<|box_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151650": {
|
||||||
|
"content": "<|quad_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151651": {
|
||||||
|
"content": "<|quad_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151652": {
|
||||||
|
"content": "<|vision_start|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151653": {
|
||||||
|
"content": "<|vision_end|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151654": {
|
||||||
|
"content": "<|vision_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151655": {
|
||||||
|
"content": "<|image_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151656": {
|
||||||
|
"content": "<|video_pad|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151657": {
|
||||||
|
"content": "<|recipient|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
},
|
||||||
|
"151658": {
|
||||||
|
"content": "<|diff_marker|>",
|
||||||
|
"lstrip": false,
|
||||||
|
"normalized": false,
|
||||||
|
"rstrip": false,
|
||||||
|
"single_word": false,
|
||||||
|
"special": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"additional_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>",
|
||||||
|
"<|recipient|>",
|
||||||
|
"<|diff_marker|>"
|
||||||
|
],
|
||||||
|
"bos_token": null,
|
||||||
|
"chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|diff_marker|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"extra_special_tokens": {},
|
||||||
|
"model_max_length": 8192,
|
||||||
|
"pad_token": "<|endoftext|>",
|
||||||
|
"padding_side": "right",
|
||||||
|
"processor_class": "Qwen2VLProcessor",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"unk_token": null
|
||||||
|
}
|
||||||
4234
trainer_state.json
Normal file
4234
trainer_state.json
Normal file
File diff suppressed because it is too large
Load Diff
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:80c85a12c64e0ffc241d7d891ca83b4f2e33884a6c683accc2a9aa8bb8d87462
|
||||||
|
size 5816
|
||||||
BIN
vocab.json
(Stored with Git LFS)
Normal file
BIN
vocab.json
(Stored with Git LFS)
Normal file
Binary file not shown.
Reference in New Issue
Block a user