enginex-mlu370-any2any/transformers/docs/source/en/model_doc/ovis2.md

This model was released on 2024-05-31 and added to Hugging Face Transformers on 2025-08-18.

Ovis2

Overview

The Ovis2 is an updated version of the Ovis model developed by the AIDC-AI team at Alibaba International Digital Commerce Group.

Ovis2 is the latest advancement in multi-modal large language models (MLLMs), succeeding Ovis1.6. It retains the architectural design of the Ovis series, which focuses on aligning visual and textual embeddings, and introduces major improvements in data curation and training methods.

Ovis2 architecture.

This model was contributed by thisisiron.

Usage example

from PIL import Image
import requests
import torch
from torchvision import io
from typing import Dict
from transformers.image_utils import load_images, load_video
from transformers import AutoModelForVision2Seq, AutoTokenizer, AutoProcessor, infer_device

device = f"{infer_device()}:0"

model = AutoModelForVision2Seq.from_pretrained(
    "thisisiron/Ovis2-2B-hf",
    dtype=torch.bfloat16,
).eval().to(device)
processor = AutoProcessor.from_pretrained("thisisiron/Ovis2-2B-hf")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Describe the image."},
        ],
    },
]
url = "http://images.cocodataset.org/val2014/COCO_val2014_000000537955.jpg"
image = Image.open(requests.get(url, stream=True).raw)
messages = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(messages)

inputs = processor(
    images=[image],
    text=messages,
    return_tensors="pt",
)
inputs = inputs.to(model.device)
inputs['pixel_values'] = inputs['pixel_values'].to(torch.bfloat16)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
    generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
    output_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
    print(output_text)

Ovis2Config

autodoc Ovis2Config

Ovis2VisionConfig

autodoc Ovis2VisionConfig

Ovis2Model

autodoc Ovis2Model

Ovis2ForConditionalGeneration

autodoc Ovis2ForConditionalGeneration - forward

Ovis2ImageProcessor

autodoc Ovis2ImageProcessor

Ovis2ImageProcessorFast

autodoc Ovis2ImageProcessorFast

Ovis2Processor

autodoc Ovis2Processor