初始化项目，由ModelHub XC社区提供模型

Model: HuggingFaceTB/SmolVLM-500M-Instruct Source: Original Platform
2026-06-05 22:26:12 +08:00
commit 0c685c2b0d
39 changed files with 296967 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,175 @@
+---
+library_name: transformers
+license: apache-2.0
+datasets:
+- HuggingFaceM4/the_cauldron
+- HuggingFaceM4/Docmatix
+pipeline_tag: image-text-to-text
+language:
+- en
+base_model:
+- HuggingFaceTB/SmolLM2-360M-Instruct
+- google/siglip-base-patch16-512
+---
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/SmolVLM_256_banner.png" width="800" height="auto" alt="Image description">
+
+# SmolVLM-500M
+
+SmolVLM-500M is a tiny multimodal model, member of the SmolVLM family. It accepts arbitrary sequences of image and text inputs to produce text outputs. It's designed for efficiency. SmolVLM can answer questions about images, describe visual content, or transcribe text. Its lightweight architecture makes it suitable for on-device applications while maintaining strong performance on multimodal tasks. It can run inference on one image with 1.23GB of GPU RAM.
+
+## Model Summary
+
+- **Developed by:** Hugging Face 🤗
+- **Model type:** Multi-modal model (image+text)
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Architecture:** Based on [Idefics3](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) (see technical summary)
+
+## Resources
+
+- **Demo:** [SmolVLM-256 Demo](https://huggingface.co/spaces/HuggingFaceTB/SmolVLM-256M-Demo)
+- **Blog:** [Blog post](https://huggingface.co/blog/smolvlm)
+
+## Uses
+
+SmolVLM can be used for inference on multimodal (image + text) tasks where the input comprises text queries along with one or more images. Text and images can be interleaved arbitrarily, enabling tasks like image captioning, visual question answering, and storytelling based on visual content. The model does not support image generation.
+
+To fine-tune SmolVLM on a specific task, you can follow [the fine-tuning tutorial](https://github.com/huggingface/smollm/blob/main/vision/finetuning/Smol_VLM_FT.ipynb).
+
+## Evaluation
+
+
+<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smoller_vlm_benchmarks.png" alt="Benchmarks" style="width:90%;" />
+
+
+### Technical Summary
+
+SmolVLM leverages the lightweight SmolLM2 language model to provide a compact yet powerful multimodal experience. It introduces several changes compared to the larger SmolVLM 2.2B model:
+
+- **Image compression:** We introduce a more radical image compression compared to Idefics3 and SmolVLM-2.2B to enable the model to infer faster and use less RAM.
+- **Visual Token Encoding:** SmolVLM-256 uses 64 visual tokens to encode image patches of size 512×512. Larger images are divided into patches, each encoded separately, enhancing efficiency without compromising performance.
+- **New special tokens:** We added new special tokens to divide the subimages. This allows for more efficient tokenization of the images.
+- **Smoller vision encoder:** We went from a 400M parameter siglip vision encoder to a much smaller 93M encoder.
+- **Larger image patches:** We are now passing patches of 512x512 to the vision encoder, instead of 384x384 like the larger SmolVLM. This allows the information to be encoded more efficiently.
+
+More details about the training and architecture are available in our technical report.
+
+### How to get started
+
+You can use transformers to load, infer and fine-tune SmolVLM.
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoProcessor, AutoModelForVision2Seq
+from transformers.image_utils import load_image
+
+DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
+
+# Load images
+image = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
+
+# Initialize processor and model
+processor = AutoProcessor.from_pretrained("HuggingFaceTB/SmolVLM-500M-Instruct")
+model = AutoModelForVision2Seq.from_pretrained(
+    "HuggingFaceTB/SmolVLM-500M-Instruct",
+    torch_dtype=torch.bfloat16,
+    _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "eager",
+).to(DEVICE)
+
+# Create input messages
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image"},
+            {"type": "text", "text": "Can you describe this image?"}
+        ]
+    },
+]
+
+# Prepare inputs
+prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
+inputs = processor(text=prompt, images=[image], return_tensors="pt")
+inputs = inputs.to(DEVICE)
+
+# Generate outputs
+generated_ids = model.generate(**inputs, max_new_tokens=500)
+generated_texts = processor.batch_decode(
+    generated_ids,
+    skip_special_tokens=True,
+)
+
+print(generated_texts[0])
+"""
+Assistant: The image depicts a cityscape featuring a prominent landmark, the Statue of Liberty, prominently positioned on Liberty Island. The statue is a green, humanoid figure with a crown atop its head and is situated on a small island surrounded by water. The statue is characterized by its large, detailed structure, with a statue of a woman holding a torch above her head and a tablet in her left hand. The statue is surrounded by a small, rocky island, which is partially visible in the foreground.
+In the background, the cityscape is dominated by numerous high-rise buildings, which are densely packed and vary in height. The buildings are primarily made of glass and steel, reflecting the sunlight and creating a bright, urban skyline. The skyline is filled with various architectural styles, including modern skyscrapers and older, more traditional buildings.
+The water surrounding the island is calm, with a few small boats visible, indicating that the area is likely a popular tourist destination. The water is a deep blue, suggesting that it is a large body of water, possibly a river or a large lake.
+In the foreground, there is a small strip of land with trees and grass, which adds a touch of natural beauty to the urban landscape. The trees are green, indicating that it is likely spring or summer.
+The image captures a moment of tranquility and reflection, as the statue and the cityscape come together to create a harmonious and picturesque scene. The statue's presence in the foreground draws attention to the city's grandeur, while the calm water and natural elements in the background provide a sense of peace and serenity.
+In summary, the image showcases the Statue of Liberty, a symbol of freedom and democracy, set against a backdrop of a bustling cityscape. The statue is a prominent and iconic representation of human achievement, while the cityscape is a testament to human ingenuity and progress. The image captures the beauty and complexity of urban life, with the statue serving as a symbol of hope and freedom, while the cityscape provides a glimpse into the modern world.
+"""
+```
+
+
+### Model optimizations
+
+**Precision**: For better performance, load and run the model in half-precision (`torch.bfloat16`) if your hardware supports it.
+
+```python
+from transformers import AutoModelForVision2Seq
+import torch
+
+model = AutoModelForVision2Seq.from_pretrained(
+    "HuggingFaceTB/SmolVLM-Instruct",
+    torch_dtype=torch.bfloat16
+).to("cuda")
+```
+
+You can also load SmolVLM with 4/8-bit quantization using bitsandbytes, torchao or Quanto. Refer to [this page](https://huggingface.co/docs/transformers/en/main_classes/quantization) for other options.
+
+```python
+from transformers import AutoModelForVision2Seq, BitsAndBytesConfig
+import torch
+
+quantization_config = BitsAndBytesConfig(load_in_8bit=True)
+model = AutoModelForVision2Seq.from_pretrained(
+    "HuggingFaceTB/SmolVLM-Instruct",
+    quantization_config=quantization_config,
+)
+```
+
+**Vision Encoder Efficiency**: Adjust the image resolution by setting `size={"longest_edge": N*512}` when initializing the processor, where N is your desired value. The default `N=4` works well, which results in input images of
+size 2048×2048. Decreasing N can save GPU memory and is appropriate for lower-resolution images. This is also useful if you want to fine-tune on videos.
+
+
+## Misuse and Out-of-scope Use
+
+SmolVLM is not intended for high-stakes scenarios or critical decision-making processes that affect an individual's well-being or livelihood. The model may produce content that appears factual but may not be accurate. Misuse includes, but is not limited to:
+
+- Prohibited Uses:
+  - Evaluating or scoring individuals (e.g., in employment, education, credit)
+  - Critical automated decision-making
+  - Generating unreliable factual content
+- Malicious Activities:
+  - Spam generation
+  - Disinformation campaigns
+  - Harassment or abuse
+  - Unauthorized surveillance
+
+### License
+
+SmolVLM is built upon [SigLIP](https://huggingface.co/google/siglip-base-patch16-512) as image encoder and [SmolLM2](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) for text decoder part.
+
+We release the SmolVLM checkpoints under the Apache 2.0 license.
+
+## Training Details
+
+### Training Data
+
+The training data comes from [The Cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) and [Docmatix](https://huggingface.co/datasets/HuggingFaceM4/Docmatix) datasets, with emphasis on document understanding (25%) and image captioning (18%), while maintaining balanced coverage across other crucial capabilities like visual reasoning, chart comprehension, and general instruction following.
+<img src="https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct/resolve/main/mixture_the_cauldron.png" alt="Example Image" style="width:90%;" />
+
+
+
--- a/added_tokens.json
+++ b/added_tokens.json
@@ -0,0 +1,130 @@
+{
+  "<end_of_utterance>": 49279,
+  "<fake_token_around_image>": 49189,
+  "<global-img>": 49152,
+  "<image>": 49190,
+  "<row_1_col_1>": 49153,
+  "<row_1_col_2>": 49154,
+  "<row_1_col_3>": 49155,
+  "<row_1_col_4>": 49156,
+  "<row_1_col_5>": 49157,
+  "<row_1_col_6>": 49158,
+  "<row_2_col_1>": 49159,
+  "<row_2_col_2>": 49160,
+  "<row_2_col_3>": 49161,
+  "<row_2_col_4>": 49162,
+  "<row_2_col_5>": 49163,
+  "<row_2_col_6>": 49164,
+  "<row_3_col_1>": 49165,
+  "<row_3_col_2>": 49166,
+  "<row_3_col_3>": 49167,
+  "<row_3_col_4>": 49168,
+  "<row_3_col_5>": 49169,
+  "<row_3_col_6>": 49170,
+  "<row_4_col_1>": 49171,
+  "<row_4_col_2>": 49172,
+  "<row_4_col_3>": 49173,
+  "<row_4_col_4>": 49174,
+  "<row_4_col_5>": 49175,
+  "<row_4_col_6>": 49176,
+  "<row_5_col_1>": 49177,
+  "<row_5_col_2>": 49178,
+  "<row_5_col_3>": 49179,
+  "<row_5_col_4>": 49180,
+  "<row_5_col_5>": 49181,
+  "<row_5_col_6>": 49182,
+  "<row_6_col_1>": 49183,
+  "<row_6_col_2>": 49184,
+  "<row_6_col_3>": 49185,
+  "<row_6_col_4>": 49186,
+  "<row_6_col_5>": 49187,
+  "<row_6_col_6>": 49188,
+  "<|reserved_special_token_0|>": 49191,
+  "<|reserved_special_token_10|>": 49201,
+  "<|reserved_special_token_11|>": 49202,
+  "<|reserved_special_token_12|>": 49203,
+  "<|reserved_special_token_13|>": 49204,
+  "<|reserved_special_token_14|>": 49205,
+  "<|reserved_special_token_15|>": 49206,
+  "<|reserved_special_token_16|>": 49207,
+  "<|reserved_special_token_17|>": 49208,
+  "<|reserved_special_token_18|>": 49209,
+  "<|reserved_special_token_19|>": 49210,
+  "<|reserved_special_token_1|>": 49192,
+  "<|reserved_special_token_20|>": 49211,
+  "<|reserved_special_token_21|>": 49212,
+  "<|reserved_special_token_22|>": 49213,
+  "<|reserved_special_token_23|>": 49214,
+  "<|reserved_special_token_24|>": 49215,
+  "<|reserved_special_token_25|>": 49216,
+  "<|reserved_special_token_26|>": 49217,
+  "<|reserved_special_token_27|>": 49218,
+  "<|reserved_special_token_28|>": 49219,
+  "<|reserved_special_token_29|>": 49220,
+  "<|reserved_special_token_2|>": 49193,
+  "<|reserved_special_token_30|>": 49221,
+  "<|reserved_special_token_31|>": 49222,
+  "<|reserved_special_token_32|>": 49223,
+  "<|reserved_special_token_33|>": 49224,
+  "<|reserved_special_token_34|>": 49225,
+  "<|reserved_special_token_35|>": 49226,
+  "<|reserved_special_token_36|>": 49227,
+  "<|reserved_special_token_37|>": 49228,
+  "<|reserved_special_token_38|>": 49229,
+  "<|reserved_special_token_39|>": 49230,
+  "<|reserved_special_token_3|>": 49194,
+  "<|reserved_special_token_40|>": 49231,
+  "<|reserved_special_token_41|>": 49232,
+  "<|reserved_special_token_42|>": 49233,
+  "<|reserved_special_token_43|>": 49234,
+  "<|reserved_special_token_44|>": 49235,
+  "<|reserved_special_token_45|>": 49236,
+  "<|reserved_special_token_46|>": 49237,
+  "<|reserved_special_token_47|>": 49238,
+  "<|reserved_special_token_48|>": 49239,
+  "<|reserved_special_token_49|>": 49240,
+  "<|reserved_special_token_4|>": 49195,
+  "<|reserved_special_token_50|>": 49241,
+  "<|reserved_special_token_51|>": 49242,
+  "<|reserved_special_token_52|>": 49243,
+  "<|reserved_special_token_53|>": 49244,
+  "<|reserved_special_token_54|>": 49245,
+  "<|reserved_special_token_55|>": 49246,
+  "<|reserved_special_token_56|>": 49247,
+  "<|reserved_special_token_57|>": 49248,
+  "<|reserved_special_token_58|>": 49249,
+  "<|reserved_special_token_59|>": 49250,
+  "<|reserved_special_token_5|>": 49196,
+  "<|reserved_special_token_60|>": 49251,
+  "<|reserved_special_token_61|>": 49252,
+  "<|reserved_special_token_62|>": 49253,
+  "<|reserved_special_token_63|>": 49254,
+  "<|reserved_special_token_64|>": 49255,
+  "<|reserved_special_token_65|>": 49256,
+  "<|reserved_special_token_66|>": 49257,
+  "<|reserved_special_token_67|>": 49258,
+  "<|reserved_special_token_68|>": 49259,
+  "<|reserved_special_token_69|>": 49260,
+  "<|reserved_special_token_6|>": 49197,
+  "<|reserved_special_token_70|>": 49261,
+  "<|reserved_special_token_71|>": 49262,
+  "<|reserved_special_token_72|>": 49263,
+  "<|reserved_special_token_73|>": 49264,
+  "<|reserved_special_token_74|>": 49265,
+  "<|reserved_special_token_75|>": 49266,
+  "<|reserved_special_token_76|>": 49267,
+  "<|reserved_special_token_77|>": 49268,
+  "<|reserved_special_token_78|>": 49269,
+  "<|reserved_special_token_79|>": 49270,
+  "<|reserved_special_token_7|>": 49198,
+  "<|reserved_special_token_80|>": 49271,
+  "<|reserved_special_token_81|>": 49272,
+  "<|reserved_special_token_82|>": 49273,
+  "<|reserved_special_token_83|>": 49274,
+  "<|reserved_special_token_84|>": 49275,
+  "<|reserved_special_token_85|>": 49276,
+  "<|reserved_special_token_86|>": 49277,
+  "<|reserved_special_token_87|>": 49278,
+  "<|reserved_special_token_8|>": 49199,
+  "<|reserved_special_token_9|>": 49200
+}
--- a/chat_template.json
+++ b/chat_template.json
@@ -0,0 +1,3 @@
+{
+  "chat_template": "<|im_start|>{% for message in messages %}{{message['role'] | capitalize}}{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}{% for line in message['content'] %}{% if line['type'] == 'text' %}{{line['text']}}{% elif line['type'] == 'image' %}{{ '<image>' }}{% endif %}{% endfor %}<end_of_utterance>\n{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}"
+}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,271 @@
+{
+  "architectures": [
+    "Idefics3ForConditionalGeneration"
+  ],
+  "image_token_id": 49190,
+  "model_type": "idefics3",
+  "scale_factor": 4,
+  "text_config": {
+    "_attn_implementation_autoset": false,
+    "_flash_attn_2_enabled": true,
+    "_name_or_path": "None",
+    "add_cross_attention": false,
+    "architectures": [
+      "VLlama3ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 1,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 2,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_dim": 64,
+    "hidden_act": "silu",
+    "hidden_size": 960,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 2560,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "is_llama_config": true,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 8192,
+    "min_length": 0,
+    "mlp_bias": false,
+    "model_type": "llama",
+    "neftune_noise_alpha": 0.0,
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 15,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 5,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 2,
+    "perceiver_config": {
+      "_attn_implementation_autoset": false,
+      "_name_or_path": "",
+      "add_cross_attention": false,
+      "architectures": null,
+      "attention_dropout": 0.0,
+      "bad_words_ids": null,
+      "begin_suppress_tokens": null,
+      "bos_token_id": null,
+      "chunk_size_feed_forward": 0,
+      "cross_attention_hidden_size": null,
+      "decoder_start_token_id": null,
+      "diversity_penalty": 0.0,
+      "do_sample": false,
+      "early_stopping": false,
+      "encoder_no_repeat_ngram_size": 0,
+      "eos_token_id": null,
+      "exponential_decay_length_penalty": null,
+      "finetuning_task": null,
+      "forced_bos_token_id": null,
+      "forced_eos_token_id": null,
+      "hidden_act": "silu",
+      "id2label": {
+        "0": "LABEL_0",
+        "1": "LABEL_1"
+      },
+      "is_decoder": false,
+      "is_encoder_decoder": false,
+      "label2id": {
+        "LABEL_0": 0,
+        "LABEL_1": 1
+      },
+      "length_penalty": 1.0,
+      "max_length": 20,
+      "min_length": 0,
+      "model_type": "vllama3",
+      "no_repeat_ngram_size": 0,
+      "num_beam_groups": 1,
+      "num_beams": 1,
+      "num_key_value_heads": 1,
+      "num_return_sequences": 1,
+      "output_attentions": false,
+      "output_hidden_states": false,
+      "output_scores": false,
+      "pad_token_id": null,
+      "prefix": null,
+      "problem_type": null,
+      "pruned_heads": {},
+      "qk_layer_norms_perceiver": false,
+      "remove_invalid_values": false,
+      "repetition_penalty": 1.0,
+      "resampler_depth": 6,
+      "resampler_head_dim": 96,
+      "resampler_n_heads": 16,
+      "resampler_n_latents": 64,
+      "return_dict": true,
+      "return_dict_in_generate": false,
+      "sep_token_id": null,
+      "suppress_tokens": null,
+      "task_specific_params": null,
+      "temperature": 1.0,
+      "tf_legacy_loss": false,
+      "tie_encoder_decoder": false,
+      "tie_word_embeddings": true,
+      "tokenizer_class": null,
+      "top_k": 50,
+      "top_p": 1.0,
+      "torch_dtype": null,
+      "torchscript": false,
+      "transformers_version": "4.46.0",
+      "typical_p": 1.0,
+      "use_bfloat16": false
+    },
+    "pixel_shuffle_factor": 4,
+    "prefix": null,
+    "pretraining_tp": 1,
+    "problem_type": null,
+    "pruned_heads": {},
+    "qk_layer_norms": false,
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rms_norm_eps": 1e-05,
+    "rope_interleaved": false,
+    "rope_scaling": null,
+    "rope_theta": 100000,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": "bfloat16",
+    "torchscript": false,
+    "transformers.js_config": {
+      "kv_cache_dtype": {
+        "fp16": "float16",
+        "q4f16": "float16"
+      }
+    },
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "use_resampler": false,
+    "vocab_size": 49280
+  },
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.0",
+  "transformers.js_config": {
+    "kv_cache_dtype": {
+      "q4f16": "float16",
+      "fp16": "float16"
+    }
+  },
+  "use_cache": true,
+  "vision_config": {
+    "_attn_implementation_autoset": false,
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 768,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": 512,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-06,
+    "length_penalty": 1.0,
+    "max_image_size": {
+      "longest_edge": 512
+    },
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "idefics3",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 12,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_channels": 3,
+    "num_hidden_layers": 12,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_size": 16,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "size": {
+      "longest_edge": 2048
+    },
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_base_siglip": false,
+    "use_bfloat16": false
+  },
+  "vocab_size": 49280
+}
--- a/configuration.json
+++ b/configuration.json
@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "image-text-to-text", "allow_remote": true}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 49279,
+  "pad_token_id": 2,
+  "transformers_version": "4.46.0"
+}
--- a/merges.txt
+++ b/merges.txt
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d05b567eeaf534e83d375551f068ed57b5f52d37c657197f644af5ef9db091a2
+size 1015025832
--- a/onnx/decoder_model_merged.onnx
+++ b/onnx/decoder_model_merged.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d4dc5a8464c92cf151be233e23da809992e111f5596ba245fe1eca28e838b7b
+size 1450425745
--- a/onnx/decoder_model_merged_bnb4.onnx
+++ b/onnx/decoder_model_merged_bnb4.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3b311d0d34a7c067485ad9d440045c892a73885c7db52f13336b428d5c302e3
+size 206503276
--- a/onnx/decoder_model_merged_fp16.onnx
+++ b/onnx/decoder_model_merged_fp16.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5425225b824ba82d9586ea7c719cde2c86c26fa5aaa5f43a1500a4fd40a5ee41
+size 725489129
--- a/onnx/decoder_model_merged_int8.onnx
+++ b/onnx/decoder_model_merged_int8.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:36f53411b56c21868b0cf3d1fa0c1a7395ab88bcd1f7ae677bf91c54ab09cca0
+size 365038968
--- a/onnx/decoder_model_merged_q4.onnx
+++ b/onnx/decoder_model_merged_q4.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d64c0fdce1c3515c40a08ff8e398bfbc0b2e961ae46d1b84b91e6a7b269b9d1
+size 229119140
--- a/onnx/decoder_model_merged_q4f16.onnx
+++ b/onnx/decoder_model_merged_q4f16.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f828e9c1ae0290e766703677d1df2a8657ecadf24dfd55f5ff28ac23cf1cf41
+size 205328252
--- a/onnx/decoder_model_merged_quantized.onnx
+++ b/onnx/decoder_model_merged_quantized.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69f48213b59637eee88e2b1af3c6bb23677fce77774a7237e8c1516c6b78ce47
+size 365039089
--- a/onnx/decoder_model_merged_uint8.onnx
+++ b/onnx/decoder_model_merged_uint8.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69f48213b59637eee88e2b1af3c6bb23677fce77774a7237e8c1516c6b78ce47
+size 365039089
--- a/onnx/embed_tokens.onnx
+++ b/onnx/embed_tokens.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fe00a73de387cbe52cdb78f1b7c8bcda6bcb9974e40dc1b143315fc9f7e79af
+size 189235499
--- a/onnx/embed_tokens_bnb4.onnx
+++ b/onnx/embed_tokens_bnb4.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19f4fad7cb17c6dad11b4520d2abcba173fc91f755d71907d31d11a9e5761606
+size 189235518
--- a/onnx/embed_tokens_fp16.onnx
+++ b/onnx/embed_tokens_fp16.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b5f8cb6f78f90182bb60a3df22e9c47c93dc771d8d8172cce1f653f8da8a746
+size 94617986
--- a/onnx/embed_tokens_int8.onnx
+++ b/onnx/embed_tokens_int8.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5f14887bf9b756d9fff2e412d474f0b5403e91f33b23460c9411f0ca54d945b
+size 47309344
--- a/onnx/embed_tokens_q4.onnx
+++ b/onnx/embed_tokens_q4.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19f4fad7cb17c6dad11b4520d2abcba173fc91f755d71907d31d11a9e5761606
+size 189235518
--- a/onnx/embed_tokens_q4f16.onnx
+++ b/onnx/embed_tokens_q4f16.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76c3b8232c463118190d1c711f2a7bff00a72f2077b9b19998f3a2c6344eb035
+size 94618005
--- a/onnx/embed_tokens_quantized.onnx
+++ b/onnx/embed_tokens_quantized.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5f14887bf9b756d9fff2e412d474f0b5403e91f33b23460c9411f0ca54d945b
+size 47309344
--- a/onnx/embed_tokens_uint8.onnx
+++ b/onnx/embed_tokens_uint8.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5f14887bf9b756d9fff2e412d474f0b5403e91f33b23460c9411f0ca54d945b
+size 47309344
--- a/onnx/vision_encoder.onnx
+++ b/onnx/vision_encoder.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ecd87031065252c5e3419f9a75db938a92cd5d55d2c22024d8815efd980104f
+size 393190865
--- a/onnx/vision_encoder_bnb4.onnx
+++ b/onnx/vision_encoder_bnb4.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4af9028ddd3357b50a4d823987bb81bf2aca83eeba425bfc02aaa339eeca9f4
+size 60688947
--- a/onnx/vision_encoder_fp16.onnx
+++ b/onnx/vision_encoder_fp16.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2fca5aecb5fe29549ce0a1cc3091f64f7f33dafb5f5f215e2b774b5cf1eeec3
+size 196731877
--- a/onnx/vision_encoder_int8.onnx
+++ b/onnx/vision_encoder_int8.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50b814aa048a996bf5d5128a16c19046371a9a1aae24b930672f550af6c43646
+size 98966519
--- a/onnx/vision_encoder_q4.onnx
+++ b/onnx/vision_encoder_q4.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:179812d5b1315edbe3e7164ea3d847c52723692e83ac9c770a5234962663fcad
+size 66734107
--- a/onnx/vision_encoder_q4f16.onnx
+++ b/onnx/vision_encoder_q4f16.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5f3df9fd7113e564b1e167de20750572d25fd75c822365f496ff467bc9e7457
+size 57691792
--- a/onnx/vision_encoder_quantized.onnx
+++ b/onnx/vision_encoder_quantized.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0612873758c96d759910a45dddbd150ea26d40cffb1ae49f391ea39dcbf50c09
+size 98966563
--- a/onnx/vision_encoder_uint8.onnx
+++ b/onnx/vision_encoder_uint8.onnx
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0612873758c96d759910a45dddbd150ea26d40cffb1ae49f391ea39dcbf50c09
+size 98966563
--- a/preprocessor_config.json
+++ b/preprocessor_config.json
@@ -0,0 +1,28 @@
+{
+  "do_convert_rgb": true,
+  "do_image_splitting": true,
+  "do_normalize": true,
+  "do_pad": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Idefics3ImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "max_image_size": {
+    "longest_edge": 512
+  },
+  "processor_class": "Idefics3Processor",
+  "resample": 1,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 2048
+  }
+}
--- a/processor_config.json
+++ b/processor_config.json
@@ -0,0 +1,4 @@
+{
+  "image_seq_len": 64,
+  "processor_class": "Idefics3Processor"
+}
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,53 @@
+{
+  "additional_special_tokens": [
+    {
+      "content": "<fake_token_around_image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    },
+    {
+      "content": "<end_of_utterance>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false
+    }
+  ],
+  "bos_token": {
+    "content": "<|im_start|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
--- a/vocab.json
+++ b/vocab.json
				`@@ -0,0 +1 @@`
				`{"framework": "pytorch", "task": "image-text-to-text", "allow_remote": true}`