243 lines
8.0 KiB
Markdown
243 lines
8.0 KiB
Markdown
---
|
||
language: en
|
||
license: apache-2.0
|
||
tags:
|
||
- vision-language-model
|
||
- visual-storytelling
|
||
- chain-of-thought
|
||
- grounded-text-generation
|
||
- cross-frame-consistency
|
||
- storytelling
|
||
- image-to-text
|
||
datasets:
|
||
- daniel3303/StoryReasoning
|
||
metrics:
|
||
- precision
|
||
- recall
|
||
- bleu
|
||
- meteor
|
||
- rouge
|
||
base_model:
|
||
- Qwen/Qwen2.5-VL-7B-Instruct
|
||
pipeline_tag: image-to-text
|
||
model-index:
|
||
- name: QwenStoryteller
|
||
results:
|
||
- task:
|
||
type: visual-storytelling
|
||
name: Visual Storytelling
|
||
dataset:
|
||
name: StoryReasoning
|
||
type: daniel3303/StoryReasoning
|
||
split: test
|
||
metrics:
|
||
- name: Character Precision
|
||
type: precision
|
||
value: 0.83
|
||
- name: Object Precision
|
||
type: precision
|
||
value: 0.46
|
||
- name: Total Precision
|
||
type: precision
|
||
value: 0.57
|
||
- name: mAP
|
||
type: mean_average_precision
|
||
value: 0.27
|
||
- name: Character Recall
|
||
type: recall
|
||
value: 0.62
|
||
- name: Object Recall
|
||
type: recall
|
||
value: 0.25
|
||
- name: Total Recall
|
||
type: recall
|
||
value: 0.40
|
||
- name: METEOR
|
||
type: meteor
|
||
value: 0.14
|
||
- name: ROUGE-L
|
||
type: rouge-l
|
||
value: 0.16
|
||
- name: BLEU-4
|
||
type: bleu-4
|
||
value: 0.054
|
||
- name: Description Accuracy
|
||
type: accuracy
|
||
value: 2.76
|
||
description: "Rating on a scale of 1-5"
|
||
- name: Average Hallucinations
|
||
type: error_rate
|
||
value: 3.56
|
||
description: "Average number of hallucinations per story"
|
||
library_name: transformers
|
||
---
|
||
|
||
# QwenStoryteller
|
||
|
||
QwenStoryteller is a fine-tuned version of Qwen2.5-VL 7B specialized for grounded visual storytelling with cross-frame consistency, capable of generating coherent narratives from multiple images while maintaining character and object identity throughout the story.
|
||
|
||
## Model Description
|
||
|
||
**Base Model:** Qwen2.5-VL 7B
|
||
**Training Method:** LoRA fine-tuning (rank 2048, alpha 4096)
|
||
**Training Dataset:** [StoryReasoning](https://huggingface.co/datasets/daniel3303/StoryReasoning)
|
||
|
||
QwenStoryteller processes sequences of images to perform:
|
||
- End-to-end object detection
|
||
- Cross-frame object re-identification
|
||
- Landmark detection
|
||
- Chain-of-thought reasoning for scene understanding
|
||
- Grounded story generation with explicit visual references
|
||
|
||
The model was fine-tuned on the StoryReasoning dataset using LoRA with a rank of 2048 and alpha scaling factor of 4096, targeting self-attention layers of the language components. Training used a peak learning rate of 1×10⁻⁴ with batch size 32, warmup for the first 3% of steps for 4 epochs, AdamW optimizer with weight decay 0.01, and bfloat16 precision.
|
||
|
||
## System Prompt
|
||
The model was trained with the following system prompt, and we recommend using it as it is for inference.
|
||
|
||
```
|
||
You are an AI storyteller that can analyze sequences of images and create creative narratives.
|
||
First think step-by-step to analyze characters, objects, settings, and narrative structure.
|
||
Then create a grounded story that maintains consistent character identity and object references across frames.
|
||
Use <think></think> tags to show your reasoning process before writing the final story.
|
||
```
|
||
|
||
## Key Features
|
||
|
||
- **Cross-Frame Consistency:** Maintains consistent character and object identity across multiple frames through visual similarity and face recognition techniques
|
||
- **Structured Reasoning:** Employs chain-of-thought reasoning to analyze scenes with explicit modeling of characters, objects, settings, and narrative structure
|
||
- **Grounded Storytelling:** Uses specialized XML tags to link narrative elements directly to visual entities
|
||
- **Reduced Hallucinations:** Achieves 12.3% fewer hallucinations compared to the non-fine-tuned base model
|
||
|
||
## Usage
|
||
|
||
```python
|
||
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
|
||
from qwen_vl_utils import process_vision_info
|
||
import torch
|
||
from PIL import Image
|
||
|
||
# Load the model
|
||
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
|
||
"daniel3303/QwenStoryteller", torch_dtype="auto", device_map="auto"
|
||
)
|
||
|
||
# Load processor
|
||
processor = AutoProcessor.from_pretrained("daniel3303/QwenStoryteller")
|
||
|
||
# Load images
|
||
images = [
|
||
Image.open("image1.jpg"),
|
||
Image.open("image2.jpg"),
|
||
Image.open("image3.jpg"),
|
||
Image.open("image4.jpg"),
|
||
Image.open("image5.jpg")
|
||
]
|
||
|
||
# Create image content list
|
||
image_content = []
|
||
for img in images:
|
||
image_content.append({
|
||
"type": "image",
|
||
"image": img,
|
||
})
|
||
|
||
# Add text prompt at the end
|
||
image_content.append({"type": "text", "text": "Generate a story based on these images."})
|
||
|
||
# Create messages with system prompt
|
||
messages = [
|
||
{
|
||
"role": "system",
|
||
"content": "You are an AI storyteller that can analyze sequences of images and create creative narratives. First think step-by-step to analyze characters, objects, settings, and narrative structure. Then create a grounded story that maintains consistent character identity and object references across frames. Use <think></think> tags to show your reasoning process before writing the final story."
|
||
},
|
||
{
|
||
"role": "user",
|
||
"content": image_content,
|
||
}
|
||
]
|
||
|
||
# Preparation for inference
|
||
text = processor.apply_chat_template(
|
||
messages, tokenize=False, add_generation_prompt=True
|
||
)
|
||
image_inputs, video_inputs = process_vision_info(messages)
|
||
inputs = processor(
|
||
text=[text],
|
||
images=image_inputs,
|
||
videos=video_inputs,
|
||
padding=True,
|
||
return_tensors="pt",
|
||
)
|
||
inputs = inputs.to(model.device)
|
||
|
||
# Inference: Generation of the output
|
||
generated_ids = model.generate(
|
||
**inputs,
|
||
max_new_tokens=4096,
|
||
do_sample=True,
|
||
temperature=0.7,
|
||
top_p=0.9
|
||
)
|
||
generated_ids_trimmed = [
|
||
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
|
||
]
|
||
story = processor.batch_decode(
|
||
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
|
||
)[0]
|
||
|
||
print(story)
|
||
```
|
||
|
||
### Using vLLM for faster inference
|
||
|
||
For significantly faster inference, you can use vLLM to serve the model. Simply install vLLM and run:
|
||
|
||
```bash
|
||
# Install vLLM
|
||
pip install vllm
|
||
|
||
# Serve the model with vLLM
|
||
vllm serve daniel3303/QwenStoryteller
|
||
```
|
||
|
||
## Output Format
|
||
|
||
QwenStoryteller produces two main outputs:
|
||
|
||
1. **Chain-of-Thought Analysis (`<think></think>`):** A structured analysis containing:
|
||
- Character tables with consistent identity references, emotions, actions, and spatial locations
|
||
- Object tables with functions, interactions, and spatial coordinates
|
||
- Setting tables categorizing environmental elements
|
||
- Narrative structure tables modeling story progression
|
||
|
||
2. **Grounded Story:** A narrative with specialized XML tags linking text to visual elements:
|
||
- `<gdi>`: Image tags for specific frames
|
||
- `<gdo>`: Entity reference tags for character and object mentions
|
||
- `<gda>`: Action tags for character actions
|
||
- `<gdl>`: Location/landmark tags for background elements
|
||
|
||
## Limitations
|
||
|
||
- Re-identification relies primarily on object appearance rather than overall context, which can lead to confusion with similar-looking objects/persons
|
||
- Movie-derived training data introduces biases from cinematic composition that may not generalize to candid visual sequences
|
||
- Low grounding rates for first-person pronouns as they primarily appear in character dialogues
|
||
- May still produce hallucinations, albeit at a reduced rate compared to the base model
|
||
|
||
## Citation
|
||
|
||
```
|
||
@misc{oliveira2025storyreasoningdatasetusingchainofthought,
|
||
title={StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation},
|
||
author={Daniel A. P. Oliveira and David Martins de Matos},
|
||
year={2025},
|
||
eprint={2505.10292},
|
||
archivePrefix={arXiv},
|
||
primaryClass={cs.CV},
|
||
url={https://arxiv.org/abs/2505.10292},
|
||
}
|
||
```
|
||
|
||
## Contact
|
||
|
||
For questions or feedback regarding this model, please contact:
|
||
- Daniel A. P. Oliveira (daniel.oliveira@inesc-id.pt) |