yaolily/TimeChat-Captioner-GRPO-7B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: yaolily/TimeChat-Captioner-GRPO-7B
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-23 13:17:42 +08:00
commit c19439e8dc
22 changed files with 162468 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

158
README.md Normal file
View File

@@ -0,0 +1,158 @@
---
base_model:
- Qwen/Qwen2.5-Omni-7B
library_name: transformers
pipeline_tag: video-text-to-text
---
# TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions
<div align="left">
[![Paper](https://img.shields.io/badge/arXiv-2602.08711-b31b1b)](https://arxiv.org/pdf/2602.08711)
[![Model](https://img.shields.io/badge/🤗%20Hugging%20Face-Model-blue)](https://huggingface.co/yaolily/TimeChat-Captioner-GRPO-7B)
[![Dataset](https://img.shields.io/badge/🤗%20Hugging%20Face-Dataset-green)](https://huggingface.co/datasets/yaolily/Timechat-OmniCaptioner-40K)
[![Benchmark](https://img.shields.io/badge/🤗%20Hugging%20Face-Benchmark-yellow)](https://huggingface.co/datasets/yaolily/OmniDenseCap-Benchmark)
</div>
---
## 🌟 Overview
**TimeChat-Captioner** is a multimodal model designed to generate detailed, time-aware, and structurally coherent captions for multi-scene videos. It effectively coordinates visual and audio information to provide comprehensive video descriptions.
- **🌐 Project Page:** [timechat-captioner.github.io](https://timechat-captioner.github.io/)
- **🏠 Model:** [TimeChat-Captioner (7B)](https://huggingface.co/yaolily/TimeChat-Captioner-GRPO-7B)
- **💻 Code:** [GitHub Repository](https://github.com/yaolinli/TimeChat-Captioner)
- **📚 Train Dataset:** [TimeChatCap-42K](https://huggingface.co/datasets/yaolily/Timechat-OmniCaptioner-42K)
- **🏆 Benchmark:** [OmniDCBench](https://huggingface.co/datasets/yaolily/OmniDenseCap-Benchmark)
<img width="1773" height="714" alt="image" src="https://github.com/user-attachments/assets/4234857e-5ba6-4b6e-bbb7-eabd0eac2244" />
---
## 🚀 Quick Start
Below, we provide simple examples to show how to use TimeChat-Captioner-GRPO-7B with 🤗 Transformers.
### Installation
```bash
conda create -n timechatcap python=3.12
conda activate timechatcap
pip install torch torchvision
pip install transformers==4.57.1
pip install accelerate
pip install flash-attn --no-build-isolation
# It's highly recommended to use `[decord]` feature for faster video loading.
pip install qwen-omni-utils[decord] -U
```
### Usage
> **Note:** To annotate high-quality timestamps and captions, limit video input to around 1 minute. Please segment longer videos into around 60-second clips before processing.
```python
import torch
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info
# 1. Configuration
MODEL_ID = "yaolily/TimeChat-Captioner-GRPO-7B"
VIDEO_PATH = "example_video.mp4" # <--- Replace with your video path
MAX_PIXELS = 297920
VIDEO_MAX_PIXELS = 297920
print(f"🚀 Processing video: {VIDEO_PATH}")
# 2. Load Model & Processor
print("⏳ Loading model...")
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
MODEL_ID,
torch_dtype=torch.bfloat16,
device_map="cuda",
attn_implementation="flash_attention_2"
)
processor = Qwen2_5OmniProcessor.from_pretrained(MODEL_ID)
model.disable_talker()
# 3. Construct Conversation
# The prompt encourages detailed, time-aware audio-visual description.
conversation = [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Thoroughly describe everything in the video, capturing every detail. Include as much information from the audio as possible, and ensure that the descriptions of both audio and video are well-coordinated."
},
{
"type": "video",
"video": VIDEO_PATH,
"max_pixels": MAX_PIXELS,
"max_frames": 160,
"fps": 2.0,
"video_max_pixels": VIDEO_MAX_PIXELS
}
],
},
]
# 4. Process Inputs
print("⚙️ Processing inputs...")
text = processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
audios, images, videos = process_mm_info(conversation, use_audio_in_video=True)
inputs = processor(
text=text,
audio=audios,
images=images,
videos=videos,
return_tensors="pt",
padding=True,
use_audio_in_video=True
)
inputs = inputs.to(model.device).to(model.dtype)
# 5. Generate Description
print("✨ Generating description...")
with torch.inference_mode():
text_ids = model.generate(
**inputs,
use_audio_in_video=True,
return_audio=False,
thinker_max_new_tokens=9216,
talker_max_tokens=9216
)
response = processor.decode(text_ids[0][inputs.input_ids[0].size(0):], skip_special_tokens=True)
print("
" + "="*50)
print("🎬 VIDEO DESCRIPTION:")
print("="*50)
print(response)
print("="*50)
```
---
## 📖 Citation
```bibtex
@misc{yao2026timechatcaptioner,
title={TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions},
author={Linli Yao and Yuancheng Wei and Yaojie Zhang and Lei Li and Xinlong Chen and Feifan Song and Ziyue Wang and Kun Ouyang and Yuanxin Liu and Lingpeng Kong and Qi Liu and Pengfei Wan and Kun Gai and Yuanxing Zhang and Xu Sun},
year={2026},
eprint={2602.08711},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2602.08711}
}
```

24
added_tokens.json Normal file
View File

@@ -0,0 +1,24 @@
{
"</tool_call>": 151658,
"<tool_call>": 151657,
"<|AUDIO|>": 151646,
"<|IMAGE|>": 151655,
"<|VIDEO|>": 151656,
"<|audio_bos|>": 151647,
"<|audio_eos|>": 151648,
"<|box_end|>": 151649,
"<|endoftext|>": 151643,
"<|file_sep|>": 151664,
"<|fim_middle|>": 151660,
"<|fim_pad|>": 151662,
"<|fim_prefix|>": 151659,
"<|fim_suffix|>": 151661,
"<|im_end|>": 151645,
"<|im_start|>": 151644,
"<|quad_end|>": 151651,
"<|quad_start|>": 151650,
"<|repo_name|>": 151663,
"<|vision_bos|>": 151652,
"<|vision_eos|>": 151653,
"<|vision_pad|>": 151654
}

521
args.json Normal file
View File

File diff suppressed because one or more lines are too long

7
chat_template.jinja Normal file
View File

@@ -0,0 +1,7 @@
{% set audio_count = namespace(value=0) %}{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system
You are a helpful assistant.<|im_end|>
{% endif %}<|im_start|>{{ message['role'] }}
{% if message['content'] is string %}{{ message['content'] }}<|im_end|>
{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_bos|><|IMAGE|><|vision_eos|>{% elif content['type'] == 'audio' or 'audio' in content or 'audio_url' in content %}{% set audio_count.value = audio_count.value + 1 %}{% if add_audio_id %}Audio {{ audio_count.value }}: {% endif %}<|audio_bos|><|AUDIO|><|audio_eos|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_bos|><|VIDEO|><|vision_eos|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>
{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
{% endif %}

631
config.json Normal file
View File

@@ -0,0 +1,631 @@
{
"architectures": [
"Qwen2_5OmniForConditionalGeneration"
],
"dtype": "bfloat16",
"enable_audio_output": false,
"enable_talker": true,
"eos_token_id": 151645,
"hidden_size": 3584,
"keys_to_ignore_at_inference": [
"past_key_values",
"hidden_states",
"attention_mask",
"hidden_states",
"attention_mask"
],
"model_type": "qwen2_5_omni",
"pad_token_id": 151643,
"talker_config": {
"_name_or_path": "Qwen2.5-Omni-7B/talker",
"architectures": [
"Qwen2OmniTalkerForConditionalGeneration"
],
"attention_dropout": 0.0,
"audio_end_token_id": 151648,
"audio_start_token_id": 151647,
"audio_token_index": 151646,
"dtype": "bfloat16",
"embedding_size": 3584,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 896,
"image_token_index": 151655,
"init_std": 0.02,
"initializer_range": 0.02,
"intermediate_size": 18944,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 32768,
"max_window_layers": 28,
"model_type": "qwen2_5_omni_talker",
"num_attention_heads": 12,
"num_hidden_layers": 24,
"num_key_value_heads": 4,
"position_id_per_seconds": 25,
"rms_norm_eps": 1e-06,
"rope_scaling": {
"mrope_section": [
16,
24,
24
],
"rope_type": "default",
"type": "default"
},
"rope_theta": 1000000.0,
"seconds_per_chunk": 2,
"sliding_window": null,
"spatial_merge_size": 2,
"tts_codec_end_token_id": 8294,
"tts_codec_mask_token_id": 8296,
"tts_codec_pad_token_id": 8292,
"tts_codec_start_token_id": 8293,
"tts_text_end_token_id": 151861,
"tts_text_pad_token_id": 151859,
"tts_text_start_token_id": 151860,
"use_cache": true,
"use_sliding_window": false,
"video_token_index": 151656,
"vision_end_token_id": 151653,
"vision_start_token_id": 151652,
"vocab_size": 8448
},
"thinker_config": {
"_name_or_path": "Qwen2.5-Omni-7B/thinker",
"architectures": [
"Qwen2OmniNaViTThinkerForConditionalGeneration"
],
"audio_config": {
"_name_or_path": "",
"activation_dropout": 0.0,
"activation_function": "gelu",
"add_cross_attention": false,
"architectures": null,
"attention_dropout": 0.0,
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": null,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"d_model": 1280,
"decoder_start_token_id": null,
"diversity_penalty": 0.0,
"do_sample": false,
"dropout": 0.0,
"dtype": "bfloat16",
"early_stopping": false,
"encoder_attention_heads": 20,
"encoder_ffn_dim": 5120,
"encoder_layerdrop": 0.0,
"encoder_layers": 32,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": null,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"init_std": 0.02,
"initializer_range": 0.02,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"length_penalty": 1.0,
"max_length": 20,
"max_source_positions": 1500,
"min_length": 0,
"model_type": "qwen2_5_omni_audio_encoder",
"n_window": 100,
"no_repeat_ngram_size": 0,
"num_beam_groups": 1,
"num_beams": 1,
"num_hidden_layers": 32,
"num_mel_bins": 128,
"num_return_sequences": 1,
"output_attentions": false,
"output_dim": 3584,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": 151643,
"prefix": null,
"problem_type": null,
"pruned_heads": {},
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"scale_embedding": false,
"sep_token_id": null,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torchscript": false,
"typical_p": 1.0,
"use_bfloat16": false
},
"audio_end_token_id": 151648,
"audio_start_token_id": 151647,
"audio_token_index": 151646,
"bos_token_id": 151644,
"dtype": "bfloat16",
"eos_token_id": 151645,
"ignore_index": -100,
"image_token_index": 151655,
"init_std": 0.02,
"initializer_range": 0.02,
"model_type": "qwen2_5_omni_thinker",
"pad_token_id": 151643,
"position_id_per_seconds": 25,
"seconds_per_chunk": 2,
"text_config": {
"_name_or_path": "",
"add_cross_attention": false,
"architectures": null,
"attention_dropout": 0.0,
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": null,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"diversity_penalty": 0.0,
"do_sample": false,
"dtype": "bfloat16",
"early_stopping": false,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": null,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"hidden_act": "silu",
"hidden_size": 3584,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"init_std": 0.02,
"initializer_range": 0.02,
"intermediate_size": 18944,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"length_penalty": 1.0,
"max_length": 20,
"max_position_embeddings": 32768,
"max_window_layers": 28,
"min_length": 0,
"model_type": "qwen2_5_omni_text",
"no_repeat_ngram_size": 0,
"num_attention_heads": 28,
"num_beam_groups": 1,
"num_beams": 1,
"num_hidden_layers": 28,
"num_key_value_heads": 4,
"num_return_sequences": 1,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": 151643,
"prefix": null,
"problem_type": null,
"pruned_heads": {},
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"rms_norm_eps": 1e-06,
"rope_scaling": {
"mrope_section": [
16,
24,
24
],
"rope_type": "default",
"type": "default"
},
"rope_theta": 1000000.0,
"sep_token_id": null,
"sliding_window": null,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": false,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torchscript": false,
"typical_p": 1.0,
"use_bfloat16": false,
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 152064
},
"user_token_id": 872,
"video_token_index": 151656,
"vision_config": {
"_name_or_path": "",
"add_cross_attention": false,
"architectures": null,
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": null,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"depth": 32,
"diversity_penalty": 0.0,
"do_sample": false,
"dtype": "bfloat16",
"early_stopping": false,
"embed_dim": 1280,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": null,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"fullatt_block_indexes": [
7,
15,
23,
31
],
"hidden_act": "silu",
"hidden_size": 1280,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"in_channels": 3,
"in_chans": 3,
"init_std": 0.02,
"initializer_range": 0.02,
"intermediate_size": 3420,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"length_penalty": 1.0,
"max_length": 20,
"min_length": 0,
"model_type": "qwen2_5_omni_vision_encoder",
"no_repeat_ngram_size": 0,
"num_beam_groups": 1,
"num_beams": 1,
"num_heads": 16,
"num_return_sequences": 1,
"out_hidden_size": 3584,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": 151643,
"patch_size": 14,
"prefix": null,
"problem_type": null,
"pruned_heads": {},
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"sep_token_id": null,
"spatial_merge_size": 2,
"spatial_patch_size": 14,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"temporal_patch_size": 2,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"tokens_per_second": 25,
"top_k": 50,
"top_p": 1.0,
"torchscript": false,
"typical_p": 1.0,
"use_bfloat16": false,
"window_size": 112
},
"vision_end_token_id": 151653,
"vision_start_token_id": 151652,
"vision_token_id": 151654
},
"token2wav_config": {
"bigvgan_config": {
"_name_or_path": "",
"add_cross_attention": false,
"architectures": null,
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": null,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"diversity_penalty": 0.0,
"do_sample": false,
"dtype": null,
"early_stopping": false,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": null,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"length_penalty": 1.0,
"max_length": 20,
"mel_dim": 80,
"min_length": 0,
"model_type": "qwen2_5_omni_bigvgan",
"no_repeat_ngram_size": 0,
"num_beam_groups": 1,
"num_beams": 1,
"num_return_sequences": 1,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": null,
"prefix": null,
"problem_type": null,
"pruned_heads": {},
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"resblock_dilation_sizes": [
[
1,
3,
5
],
[
1,
3,
5
],
[
1,
3,
5
]
],
"resblock_kernel_sizes": [
3,
7,
11
],
"return_dict": true,
"return_dict_in_generate": false,
"sep_token_id": null,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torchscript": false,
"typical_p": 1.0,
"upsample_initial_channel": 1536,
"upsample_kernel_sizes": [
11,
7,
4,
4,
4,
4
],
"upsample_rates": [
5,
3,
2,
2,
2,
2
],
"use_bfloat16": false,
"use_bias_at_final": false
},
"dit_config": {
"_name_or_path": "",
"add_cross_attention": false,
"architectures": null,
"bad_words_ids": null,
"begin_suppress_tokens": null,
"block_size": 24,
"bos_token_id": null,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"depth": 22,
"dim": 1024,
"diversity_penalty": 0.0,
"do_sample": false,
"dropout": 0.1,
"dtype": "float32",
"early_stopping": false,
"emb_dim": 512,
"enc_attention_channels": 64,
"enc_channels": [
256,
256,
256,
256,
768
],
"enc_dilations": [
1,
2,
3,
4,
1
],
"enc_dim": 128,
"enc_emb_dim": 192,
"enc_global_context": true,
"enc_kernel_sizes": [
5,
3,
3,
3,
1
],
"enc_lin_neurons": 192,
"enc_res2net_scale": 2,
"enc_se_channels": 64,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": null,
"exponential_decay_length_penalty": null,
"ff_mult": 2,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"head_dim": 64,
"heads": 16,
"hidden_size": 1024,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"length_penalty": 1.0,
"look_ahead_layers": [
10
],
"look_backward_layers": [
0,
20
],
"max_length": 20,
"max_position_embeddings": 32768,
"mel_dim": 80,
"min_length": 0,
"model_type": "qwen2_5_omni_dit",
"no_repeat_ngram_size": 0,
"num_attention_heads": 16,
"num_beam_groups": 1,
"num_beams": 1,
"num_embeds": 8193,
"num_hidden_layers": 22,
"num_return_sequences": 1,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": null,
"prefix": null,
"problem_type": null,
"pruned_heads": {},
"remove_invalid_values": false,
"repeats": 2,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"rope_theta": 10000.0,
"sep_token_id": null,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torchscript": false,
"typical_p": 1.0,
"use_bfloat16": false
},
"dtype": "bfloat16",
"model_type": "qwen2_5_omni_token2wav"
},
"transformers_version": "4.57.1"
}

8
generation_config.json Normal file
View File

@@ -0,0 +1,8 @@
{
"_from_model_config": true,
"eos_token_id": [
151645,
151643
],
"transformers_version": "4.57.1"
}

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:47e09b588950032ae29d1d87c7d680638792254da0f80b37122e7facb9c65b58
size 4985055536

3
model-00002-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1f7798d7a5baecc19ef0f27f7ace114fa4b0b2c62cae1671defffe88ed07c73d
size 4991496832

3
model-00003-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8a70d22d9d4a03cf4334e726c15bccc9556cb1e9f013bc7da1dca732b6888432
size 4991496936

3
model-00004-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:141b23df808e110c337a97888c3221e170f28e511a4e2df13466c8d7f619736c
size 2895740064

1354
model.safetensors.index.json Normal file
View File

File diff suppressed because it is too large Load Diff

31
preprocessor_config.json Normal file
View File

@@ -0,0 +1,31 @@
{
"chunk_length": 300,
"dither": 0.0,
"feature_extractor_type": "WhisperFeatureExtractor",
"feature_size": 128,
"hop_length": 160,
"image_mean": [
0.48145466,
0.4578275,
0.40821073
],
"image_processor_type": "Qwen2VLImageProcessor",
"image_std": [
0.26862954,
0.26130258,
0.27577711
],
"max_pixels": 12845056,
"merge_size": 2,
"min_pixels": 3136,
"n_fft": 400,
"n_samples": 4800000,
"nb_max_frames": 30000,
"padding_side": "right",
"padding_value": 0.0,
"patch_size": 14,
"processor_class": "Qwen2_5OmniProcessor",
"return_attention_mask": true,
"sampling_rate": 16000,
"temporal_patch_size": 2
}

38
special_tokens_map.json Normal file
View File

@@ -0,0 +1,38 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|AUDIO|>",
"<|audio_bos|>",
"<|audio_eos|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_bos|>",
"<|vision_eos|>",
"<|vision_pad|>",
"<|IMAGE|>",
"<|VIDEO|>"
],
"audio_bos_token": "<|audio_bos|>",
"audio_eos_token": "<|audio_eos|>",
"audio_token": "<|AUDIO|>",
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"image_token": "<|IMAGE|>",
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"video_token": "<|VIDEO|>",
"vision_bos_token": "<|vision_bos|>",
"vision_eos_token": "<|vision_eos|>"
}

3
spk_dict.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6a05609b28f5d42b7b748f0f07592545c8f1f6885b9ae8fff64baf56e86b2a18
size 259544

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8441917e39ae0244e06d704b95b3124795cec478e297f9afac39ba670d7e9d99
size 11421870

222
tokenizer_config.json Normal file
View File

@@ -0,0 +1,222 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|AUDIO|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|audio_bos|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|audio_eos|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_bos|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_eos|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|IMAGE|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|VIDEO|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|AUDIO|>",
"<|audio_bos|>",
"<|audio_eos|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_bos|>",
"<|vision_eos|>",
"<|vision_pad|>",
"<|IMAGE|>",
"<|VIDEO|>"
],
"audio_bos_token": "<|audio_bos|>",
"audio_eos_token": "<|audio_eos|>",
"audio_token": "<|AUDIO|>",
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": {
"audio_bos_token": "<|audio_bos|>",
"audio_eos_token": "<|audio_eos|>",
"audio_token": "<|AUDIO|>",
"image_token": "<|IMAGE|>",
"video_token": "<|VIDEO|>",
"vision_bos_token": "<|vision_bos|>",
"vision_eos_token": "<|vision_eos|>"
},
"image_token": "<|IMAGE|>",
"model_max_length": 32768,
"pad_token": "<|endoftext|>",
"processor_class": "Qwen2_5OmniProcessor",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null,
"video_token": "<|VIDEO|>",
"vision_bos_token": "<|vision_bos|>",
"vision_eos_token": "<|vision_eos|>"
}

7970
trainer_state.json Normal file
View File

File diff suppressed because it is too large Load Diff

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:519b67b2a28b20e37f307a47f64302afbd94079b8f5c5c5f576a935da39e0257
size 12241

58
video_preprocessor_config.json Normal file
View File

@@ -0,0 +1,58 @@
{
"chunk_length": 300,
"crop_size": null,
"data_format": "channels_first",
"default_to_square": true,
"device": null,
"dither": 0.0,
"do_center_crop": null,
"do_convert_rgb": true,
"do_normalize": true,
"do_pad": null,
"do_rescale": true,
"do_resize": true,
"do_sample_frames": false,
"feature_extractor_type": "WhisperFeatureExtractor",
"feature_size": 128,
"fps": null,
"hop_length": 160,
"image_mean": [
0.48145466,
0.4578275,
0.40821073
],
"image_processor_type": "Qwen2VLImageProcessor",
"image_std": [
0.26862954,
0.26130258,
0.27577711
],
"input_data_format": null,
"max_frames": 768,
"max_pixels": 12845056,
"merge_size": 2,
"min_frames": 4,
"min_pixels": 3136,
"n_fft": 400,
"n_samples": 4800000,
"nb_max_frames": 30000,
"num_frames": null,
"pad_size": null,
"padding_side": "right",
"padding_value": 0.0,
"patch_size": 14,
"processor_class": "Qwen2_5OmniProcessor",
"resample": 3,
"rescale_factor": 0.00392156862745098,
"return_attention_mask": true,
"return_metadata": false,
"sampling_rate": 16000,
"size": {
"longest_edge": 12845056,
"shortest_edge": 3136
},
"size_divisor": null,
"temporal_patch_size": 2,
"video_metadata": null,
"video_processor_type": "Qwen2VLVideoProcessor"
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long