naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-16 19:08:17 +08:00
commit 0a1c6fdda0
22 changed files with 115757 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

51
.gitattributes vendored Normal file
View File

@@ -0,0 +1,51 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.tfevents* filter=lfs diff=lfs merge=lfs -text
*.db* filter=lfs diff=lfs merge=lfs -text
*.ark* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.gguf* filter=lfs diff=lfs merge=lfs -text
*.ggml filter=lfs diff=lfs merge=lfs -text
*.llamafile* filter=lfs diff=lfs merge=lfs -text
*.pt2 filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text
model-00001-of-00003.safetensors filter=lfs diff=lfs merge=lfs -text
model-00002-of-00003.safetensors filter=lfs diff=lfs merge=lfs -text
model-00003-of-00003.safetensors filter=lfs diff=lfs merge=lfs -text

62
LICENSE Normal file
View File

@@ -0,0 +1,62 @@
HyperCLOVA X SEED Model License Agreement
Model Release Date: April 24, 2025
This HyperCLOVA X SEED Model License Agreement (the “Agreement”) is a legal agreement between you and NAVER Corporation and NAVER Cloud Corporation (“NAVER”) and governs your use of the Models that NAVER provides to You under this Agreement.
NAVER Corp., as the holder of the intellectual property of the Model, and its affiliate, NAVER Cloud Corp., as the exclusive business operator of HyperCLOVA X, enter into this Agreement with you. NAVER and you are each a “party” and collectively the “parties.”
By using, reproducing, modifying, distributing, performing or displaying any portion or element of the Model or Derivative Model, or otherwise accepting the terms of this Agreement, you agree to be bound by this Agreement. You represent to us that you are lawfully able to enter into contracts, and if you are entering into this Agreement for an entity, that you have legal authority to bind that entity.
1. Definitions.
1.1. "Affiliate” means any entity directly or indirectly controlling, controlled by or under common control with either party, where “control” means the possession, directly or indirectly, of the power to independently direct or cause the direction of the management and policies of an entity, whether through ownership of more than fifty percent (50%) of the stock or other equity interests entitled to vote for representation on its board of directors, or body performing similar functions, by contract or otherwise.
1.2. “Derivative Model” means all (i) modifications to the Model, (ii) works based on the Model, or (iii) any other machine learning model which is created by transfer of patterns of the weights, parameters, operations, or Output of the Model, to that model in order to cause that model to perform similarly to the Model, including distillation methods that use intermediate data representations or methods based on the generation of synthetic data Outputs by the Model for training that Model. For clarity, Outputs are not deemed Derivative Model.
1.3. “Licensee” or “you” means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity’s behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
1.4. “Model” means the foundational large language models and software and algorithms, including machine-learning model code and trained model weights distributed by NAVER.
1.5. “Output” means the information content output of the Model or a Derivative Model that results from operating or otherwise using the Model or Derivative Models.
2. Conditions for Use, License Grant and Restrictions
2.1. Conditions for Use. The Model and any Derivative Model are subject to the terms of this Agreement and govern your use. If You institute copyright or patent litigation against any entity (including a crossclaim or counterclaim in a lawsuit) alleging that the Model or a Derivative Model constitutes direct or contributory copyright or patent infringement, then any license granted to you under this Agreement for that Model or Derivative Model will terminate as of the date such litigation is filed. NAVER may update this Agreement to comply with legal and regulatory requirements any time and You agree to either comply with any updated license or cease your copying, use, and distribution of the Model and any Derivative Model.
2.2. License Grant. Subject to the terms and conditions of this Agreement, NAVER hereby grants to you a non-exclusive, worldwide, non-transferable, revocable and royalty-free limited license under NAVER’s intellectual property or other rights owned by NAVER embodied in the Model to access, download, install, copy, use, reproduce, distribute, create derivative works of, and make modifications to the Model.
2.3. Prohibited Use Policy. NAVER is committed to safety, trust and transparency in AI development. NAVER encourages You to (i) ensure that the product or service you develop, use, offer as a service or distributes meets the legal and ethical requirements of the relevant industry or use case, (ii) take reasonable measures to address unintended bias and to mitigate harm to others, including underrepresented or vulnerable groups, and (iii) inform users of the nature and limitations of the product or service. NAVER expressly prohibits the use of its products or services for any purpose in violation of applicable law and regulation, including but not limited to (a) illegal surveillance, (b) illegal collection or processing of biometric information without the consent of the subject where required under applicable law, or (c) illegal harassment, abuse, threatening or bullying of individuals or groups of individuals or intentionally misleading or deceiving others.
3. Redistribution.
3.1. You may reproduce, distribute or make available the Model or Derivative Models thereof, or a product or service (including another AI model) that contains any of them, if you meet all of the following conditions: you must (i) include the Prohibited Use Policy referenced in Section 2.3. as an enforceable provision in any agreement (e.g., license agreement, terms of use, etc.) governing the use and/or distribution of the Model or Derivative Model and you must provide notice to subsequence users you distribute to the Model or Derivative Models are subject to the use restrictions in Section 2.3., (ii) provide all third party recipients of the Model or Derivative Models a copy of this Agreement, (iii) cause any modified files to carry prominent notices stating that you modified the files; (iv) include the following attribution notice within a “Notice” text file distributed as part of such copies: “HyperCLOVA X SEED Model is licensed under the HyperCLOVA X SEED Model License Agreement, Copyright © NAVER Corp. All Rights Reserved.”, and (v) prominently display “Powered by HyperCLOVA X” on a related website, user interface, blogpost, about page, or product documentation. If you use the Model or any Outputs of the Model to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include “HyperCLOVA X” at the beginning of any such AI model name.
3.2. You may add your own copyright statement to your modifications and, except as set forth in this Section, may provide additional or different license terms and conditions for use, reproduction, or distribution of your modifications, or for any such Derivative Models as a whole, provided your use, reproduction, and distribution of the Model or Derivative Models otherwise comply with the terms and conditions stated in this Agreement. Any additional or different terms and conditions you impose must not conflict with the terms of this Agreement.
4. Additional Commercial Terms. If (i) as of the Model Release Date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s Affiliates, is greater than 10 million monthly active users in the preceding calendar month, or (ii) the Licensee or its Affiliate distributes or makes available any product or service, which is substantially similar to or directly competes with any product and service provided by NAVER, then the Licensee must request a license from NAVER. Such license may be granted by NAVER at its sole discretion, and the Licensee is not authorized to exercise any rights under this Agreement unless and until NAVER expressly grants you such rights.
5. Generated Output. NAVER claims no rights in Outputs you generate using the Model. You and your use are solely responsible for Outputs and their subsequent uses.
6. DISCLAIMER OF WARRANTY. UNLESS REQUIRED BY APPLICABLE LAW, THE MODEL AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OR ANY KIND, AND NAVER DISCLAIMS ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE MODEL, DERIVATIVE MODELS, OUTPUTS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE MODEL AND ANY OUTPUTS AND RESULTS AND YOUR EXERCISE OF PERMISSION UNDER THIS AGREEMENT.
7. LIMITATION OF LIABILITY. IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW (SUCH AS IN CASES OF DELIBERATE AND GROSSLY NEGLIGENT ACTS), WILL NAVER BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY, OR PUNITIVE DAMAGES, OR LOST PROFITS OF ANY KIND, ARISING FROM OR RELATED TO THIS AGREEMENT, OR RESULTING FROMTHE USE OR INABILITY TO USE THE MODEL, DERIVATIVE MODELS OR, OUTPUTS (INCLUDING, BUT NOT LIMITED TO, DAMAGES FOR LOSS OF GOODWILL, WORK STOPPAGES, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER COMMERCIAL DAMAGES OR LOSSES), EVEN IF NAVER HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
8. Indemnity. You will indemnify and hold harmless NAVER from and against any claim by any third party arising out of or related to your use or distribution of the Model, Derivative Model or Outputs.
9. Intellectual Property.
9.1. This Agreement does not grant permission to use the trade names, trademarks, service marks, or product names of NAVER, except as required for reasonable and customary use in describing the origin of the Model and reproducing the content of the “Notice” text file.
9.2. NAVER Corp. owns the Model and any Derivative Model created by NAVER Corp. Except as expressively granted in this Agreement, NAVER Corp. reserves all rights, interests and remedies in connection with the Model and Derivative Model created by NAVER Corp. and no other license or right is granted to you by implication, estoppel or otherwise. Subject to NAVER Corp.’s ownership of the Model and any Derivative Model made by or for NAVER Corp., with respect to any derivative works and modifications of the Model that are made by you, as between you and NAVER Corp., you are and will be the owner of such derivative works and modifications.
10. Term and Termination. The term of this Agreement will commence upon your acceptance of this Agreement or access to the Model and will continue in full force and effect until terminated in accordance with the terms and conditions of this Agreement. NAVER may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you shall delete and cease use of the Model and Derivative Model. Section 5, 6, 7 and 10 shall survive the termination of this Agreement.
11. Governing Law and Jurisdiction.
11.1. This Agreement will be governed by and construed in accordance with the laws of the Republic of Korea, without regard to its conflicts of laws principles.
11.2. Any disputes, controversies, or claims arising out of or relating to this Agreement, including its existence, validity, interpretation, performance, breach, or termination, shall be referred to and finally resolved by arbitration administered by the Korean Commercial Arbitration Board (KCAB) in accordance with the International Arbitration Rules of the Korean Commercial Arbitration Board in force at the time of the commencement of the arbitration. The seat of arbitration shall be Seoul, Republic of Korea. The tribunal shall consist of one arbitrator. The language of the arbitration shall be English. Either party may seek interim or provisional relief from a court of competent jurisdiction, and doing so shall not be considered a waiver of any provision in this section. The arbitral tribunal also has the authority to issue orders for interim or provisional relief.
12. Modifications. NAVER reserves the right to modify or amend this Agreement at any time, in its sole discretion. Any modifications will be effective upon posting the updated Agreement on our website or through other means of communication. You are responsible for reviewing the Agreement periodically for changes.
13. No Waiver. NAVER will not be treated as having waived any rights by not exercising (or delaying the exercise of) any rights under this Agreement.

348
README.md Normal file
View File

@@ -0,0 +1,348 @@
---
license: other
license_name: hyperclovax-seed
license_link: LICENSE
library_name: transformers
---
![image/png](https://cdn-uploads.huggingface.co/production/uploads/6512d9827fccffe1e9e28fa7/Lra7yfdthGdKcNk7vP5RS.png)
## **Overview**
HyperCLOVAX-SEED-Vision-Instruct-3B is a model developed by NAVER, built upon its proprietary backbone model and fine-tuned through post-training. It is capable of understanding both text and images, as well as generating text.
The model is primarily designed with a focus on lightweight architecture, optimizing computational efficiency. In terms of visual understanding, it can handle visual question answering (VQA), chart and diagram interpretation, and even comprehend content. HyperCLOVAX-SEED-Vision-Instruct-3B aims for a Pareto-optimal balance specifically tuned for the Korean language, and it demonstrates competitive performance using fewer visual tokens compared to other models of similar size in inference scenarios.
Particularly, the model shows relative strengths in handling Korean-language inputs and outperforms similarly sized open-source models in related benchmarks. As the first open-source vision-language model in Korea capable of visual understanding, it is expected to significantly contribute to strengthening Korea's sovereign AI capabilities.
## **Updates**
- **(2025.07.25)**: vLLM engine is available with [our repository](https://github.com/NAVER-Cloud-HyperCLOVA-X/vllm/tree/v0.9.2rc2_hyperclovax_vision_seed)
- **(2025.07.08)**: Major code update for supporting vLLM engine ([link - related_discussion](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B/discussions/27))
- **(2025.04.22)**: Initial release of the repository.
## **Basic Information**
- **Model Architecture**: LLaVA-based Vision-Language Model
- **LLM Module**: Transformer-based architecture (Dense Model)
- **Vision Encoder** : SigLIP-based architecture with 378x378px input resolution per grid.
- **Vision-Language Connector** : C-Abstractor based architecture with AnyRes mechanism, supporting up to 1.29M total pixels across 9 grids.
- **Parameter Count**: 3.2B (LLM Module) + 0.43B (Vision Module)
- **Input/Output Format**: Text + Image + Video / Text
- **Context Length**: 16k
- **Knowledge Cutoff Date**: The model was trained on data collected before August 2024.
## **Training**
#### **Text**
Securing high-quality data is essential even during post-training, but having humans manually create or revise large-scale datasets posed significant limitations in terms of both cost and resources. Additionally, tasks requiring domain expertise were difficult to handle, and the risk of human error was high. To overcome these challenges, we utilized an automated validation system powered by HyperCLOVA X, which improved data quality and streamlined the training process — ultimately leading to enhanced overall model performance. As a result, the model showed significant improvements in areas with definitive answers, such as mathematics and coding.
While reducing the cost of data collection is important, finding efficient training strategies is equally critical. HyperCLOVAX-SEED-Vision-Instruct-3B was developed starting from the HyperCLOVAX-SEED-Text-Base-3B and applied both Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) based on an online reinforcement algorithm called GRPO.
#### **Vision**
The Vision Understanding feature — where the model receives images and questions as input and generates text-based answers — was not part of the initial design of HyperCLOVA X. Therefore, the model architecture was carefully designed to add capabilities for handling vision-related tasks, such as image-based question answering (VQA) and chart/diagram interpretation, without compromising the existing performance of the HCX LLM. Special attention was given to handling auxiliary information within the input, especially considering the context length.
Although HyperCLOVAX-SEED-Vision-Instruct-3B is a lightweight model, it is capable of performing basic image VQA tasks and even supports OCR-free processing. One of the key focus areas for this 3B model was optimizing the efficiency of video input tokens. Since input token length directly affects computational cost, the number of tokens extracted per frame was carefully adjusted to enable efficient video understanding with as few tokens as possible. Additionally, during the RLHF training phase, vision-specific V-RLHF data was used to enhance the model’s learning, just like in the text domain.
## Benchmark
#### Text
| **Model** | **KMMLU (5-shot, acc)** | **HAE-RAE (5-shot, acc)** | **CLiCK (5-shot, acc)** | **KoBEST (5-shot, acc)** |
|----------------------------|--------|---------|---------|-------|
| HyperCLOVAX-SEED-Text-Base-3B | 0.4847 | 0.7635 | 0.6386 | 0.7792 |
| HyperCLOVAX-SEED-Vision-Instruct-3B| 0.4422 | 0.6499 | 0.5599 | 0.7180 |
| Qwen2.5-3B-instruct | 0.4451 | 0.6031 | 0.5649 | 0.7053 |
| gemma-3-4b-it | 0.3895 | 0.6059 | 0.5303 | 0.7262 |
#### Vision
| Model Name | Max Token Count per Video | VideoMME (Ko) | NAVER-TV-CLIP (Ko) | VideoChatGPT (Ko) | PerceptionTest (En) | ActivityNet-QA (En) | KoNet (Ko) | MMBench-Val (En) | TextVQA-Val (En) | Korean VisIT-Bench (Ko) | Image (4 benchmarks) | Video (5 benchmarks) | All (9 benchmarks) |
|-----------------------------------|--------------------------------|----------------|---------------------|--------------------|-----------------------|----------------------|------------|-------------------|-------------------|--------------------------|------------------------|------------------------|----------------------|
| HyperCLOVAX-SEED-Vision-Instruct-3B | 1856 tokens, 108 frames | 48.2 | 61.0 | 53.6 | 55.2 | 50.6 | 69.2 | 81.8 | 79.2 | 37.0 | 46.68 | 53.70 | 59.54 |
| HyperCLOVAX-SEED-Vision-Instruct-3B (without OCR)| 1856 tokens, 108 frames | 48.2 | 61.0 | 53.6 | 55.2 | 50.6 | 36.6 | 80.7 | 76.0 | 43.5 | 56.74 | 53.70 | 55.05 |
| Qwen-2.5-VL-3B | 24576 tokens, 768 frames | 55.1 | 48.3 | 45.6 | 66.9 | 55.7 | 58.3 | 84.3 | 79.6 | 81.5 | 59.35 | 54.31 | 56.55 |
| Qwen-2.5-VL-3B (w/ 2000 tokens) | 2000 tokens, 128 frames | 50.3 | 43.9 | 44.3 | 58.3 | 54.2 | 58.5 | 84.3 | 79.3 | 15.7 | 59.50 | 50.18 | 54.33 |
| Qwen-2.5-VL-7B | 24576 tokens, 768 frames | 60.6 | 66.7 | 51.8 | 70.5 | 56.6 | 68.4 | 88.3 | 84.9 | 85.6 | 69.34 | 61.23 | 64.84 |
| Gemma-3-4B | 4096 tokens, 16 frames | 45.4 | 36.8 | 57.1 | 50.6 | 46.3 | 25.0 | 79.2 | 58.9 | 32.3 | 48.91 | 47.24 | 47.98 |
| GPT4V (gpt-4-turbo-2024-04-09) | Unknown, Original Image , 8 frames | 49.1 | 75.0 | 55.5 | 57.4 | 45.7 | 38.7 | 84.2 | 60.4 | 52.0 | 58.88 | 51.59 | 54.83 |
| GPT4o (gpt-4o-2024-08-06) | Unknown, 512 resize, 128 frames| 61.6 | 66.6 | 61.8 | 50.2 | 41.7 | 60.6 | 84.2 | 73.2 | 50.5 | 67.15 | 56.42 | 61.19 |
| InternV-2-2B | 4096 tokens, 16 frames | 28.9 | 21.1 | 40.2 | 50.5 | 50.3 | 3.3 | 79.3 | 75.1 | 51.1 | 39.74 | 38.19 | 38.88 |
| InternV-2-4B | 4096 tokens, 16 frames | 33.8 | 36.0 | 22.8 | 54.2 | 52.0 | 22.7 | 83.0 | 76.9 | 51.6 | 46.11 | 39.75 | 42.58 |
| InternV-2-8B | 4096 tokens, 16 frames | 43.7 | 41.2 | 32.4 | 58.5 | 53.2 | 28.5 | 86.6 | 79.0 | 97.0 | 50.32 | 45.79 | 47.81 |
## Dependencies
- [einops](https://einops.rocks/)
- [timm](https://github.com/huggingface/pytorch-image-models)
- [av](https://github.com/PyAV-Org/PyAV)
- [decord](https://github.com/dmlc/decord)
## Example
**(code & benchmark score) checked with transformers 4.52.4**
```python
from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer
model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True).to(device="cuda")
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# LLM Example
# It is recommended to use the chat template with HyperCLOVAX models.
# Using the chat template allows you to easily format your input in ChatML style.
llm_chat = [
{"role": "system", "content": [{"type": "text", "text": "you are helpful assistant!"}]},
{
"role": "user",
"content": [
{"type": "text", "text": "Hello, how are you?"},
{"type": "text", "text": "I said. Hello, how are you today?"},
]
},
{"role": "assistant", "content": [{"type": "text", "text": "I'm doing great. How can I help you today?"}]},
{"role": "user", "content": [{"type": "text", "text": "I'd like to show off how chat templating works!"}]},
]
model_inputs = processor.apply_chat_template(
llm_chat, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True
)
model_inputs = model_inputs.to(device="cuda")
# Please adjust parameters like top_p appropriately for your use case.
output_ids = model.generate(
**model_inputs,
max_new_tokens=64,
do_sample=True,
top_p=0.6,
temperature=0.5,
repetition_penalty=1.0,
)
print("=" * 80)
print("LLM EXAMPLE")
print(processor.batch_decode(output_ids)[0])
print("=" * 80)
# VLM Example
# For images and videos, you can use url, local_path, base64, or bytes as input sources.
vlm_chat = [
{"role": "system", "content": [{"text": "System Prompt", "type": "text"}]},
{"role": "user", "content": [{"text": "User Text Prompt 1", "type": "text"}]},
{
"role": "user",
"content": [{
"filename": "tradeoff_sota.png",
"image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff_sota.png?raw=true",
"lens_keywords": "Gucci Ophidia, cross bag, Ophidia small, GG, Supreme shoulder bag",
"lens_local_keywords": "[0.07, 0.21, 0.92, 0.90] Gucci Ophidia",
"ocr": "List the words in the image in raster order. Even if the word order feels unnatural for reading, the model will handle it as long as it follows raster order.", "type": "image",
}],
},
{
"role": "user",
"content": [{
"filename": "tradeoff.png",
"image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff.png?raw=true",
"type": "image",
}],
},
{"role": "assistant", "content": [{"text": "Assistant Text Prompt 1", "type": "text"}]},
{"role": "user", "content": [{"text": "User Text Prompt 2", "type": "text"}]},
{
"role": "user",
"content": [
{
"type": "video",
"video": "freenaturestock-rolling-mist-clouds.mp4",
"lens_keywords": "Prada re-edition, nylon bag, mini cross bag, logo strap, essential shoulder bag",
"lens_local_keywords": "[0.12, 0.34, 0.85, 0.76] Prada re-edition",
"speech_to_text": "Please enter the dialogue, voice, sound, lines, and words in the video in text format.",
},
{"text": "User Text Prompt 3", "type": "text"},
]
},
]
model_inputs = processor.apply_chat_template(
vlm_chat, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True,
)
model_inputs = model_inputs.to(device="cuda")
output_ids = model.generate(
**model_inputs,
max_new_tokens=64,
do_sample=True,
top_p=0.6,
temperature=0.5,
repetition_penalty=1.0,
)
print("=" * 80)
print("VLM EXAMPLE")
print(processor.batch_decode(output_ids)[0])
print("=" * 80)
```
## Example for v0.1.0
**(code & benchmark score) checked with transformers 4.45.0**
```python
from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer
model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B"
revision="v0.1.0"
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, revision=revision).to(device="cuda")
preprocessor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True, revision=revision)
tokenizer = AutoTokenizer.from_pretrained(model_name, revision=revision)
# LLM Example
# It is recommended to use the chat template with HyperCLOVAX models.
# Using the chat template allows you to easily format your input in ChatML style.
chat = [
{"role": "system", "content": "you are helpful assistant!"},
{"role": "user", "content": "Hello, how are you?"},
{"role": "assistant", "content": "I'm doing great. How can I help you today?"},
{"role": "user", "content": "I'd like to show off how chat templating works!"},
]
input_ids = tokenizer.apply_chat_template(chat, return_tensors="pt", tokenize=True)
input_ids = input_ids.to(device="cuda")
# Please adjust parameters like top_p appropriately for your use case.
output_ids = model.generate(
input_ids,
max_new_tokens=64,
do_sample=True,
top_p=0.6,
temperature=0.5,
repetition_penalty=1.0,
)
print("=" * 80)
print("LLM EXAMPLE")
print(tokenizer.batch_decode(output_ids)[0])
print("=" * 80)
# VLM Example
# For image and video inputs, you can use url, local_path, base64, or bytes.
vlm_chat = [
{"role": "system", "content": {"type": "text", "text": "System Prompt"}},
{"role": "user", "content": {"type": "text", "text": "User Text 1"}},
{
"role": "user",
"content": {
"type": "image",
"filename": "tradeoff_sota.png",
"image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff_sota.png?raw=true",
"ocr": "List the words in the image in raster order. Even if the word order feels unnatural for reading, the model will handle it as long as it follows raster order.",
"lens_keywords": "Gucci Ophidia, cross bag, Ophidia small, GG, Supreme shoulder bag",
"lens_local_keywords": "[0.07, 0.21, 0.92, 0.90] Gucci Ophidia",
}
},
{
"role": "user",
"content": {
"type": "image",
"filename": "tradeoff.png",
"image": "https://github.com/naver-ai/rdnet/blob/main/resources/images/tradeoff.png?raw=true",
}
},
{"role": "assistant", "content": {"type": "text", "text": "Assistant Text 1"}},
{"role": "user", "content": {"type": "text", "text": "User Text 2"}},
{
"role": "user",
"content": {
"type": "video",
"filename": "rolling-mist-clouds.mp4",
"video": "freenaturestock-rolling-mist-clouds.mp4",
}
},
{"role": "user", "content": {"type": "text", "text": "User Text 3"}},
]
new_vlm_chat, all_images, is_video_list = preprocessor.load_images_videos(vlm_chat)
preprocessed = preprocessor(all_images, is_video_list=is_video_list)
input_ids = tokenizer.apply_chat_template(
new_vlm_chat, return_tensors="pt", tokenize=True, add_generation_prompt=True,
)
output_ids = model.generate(
input_ids=input_ids.to(device="cuda"),
max_new_tokens=8192,
do_sample=True,
top_p=0.6,
temperature=0.5,
repetition_penalty=1.0,
**preprocessed,
)
print("=" * 80)
print("VLM EXAMPLE")
print(tokenizer.batch_decode(output_ids)[0])
print("=" * 80)
```
- To ensure the highest level of image understanding performance, it is recommended to include additional information such as Optical Character Recognition (OCR) results and entity recognition (Lens). The provided usage examples are written under the assumption that OCR and Lens results are available. If you input data in this format, you can expect significantly improved output quality.
## vLLM
To speed up your inference, you can use the vLLM engine from [our repository](https://github.com/NAVER-Cloud-HyperCLOVA-X/vllm/tree/v0.9.2rc2_hyperclovax_vision_seed).
Make sure to switch to the `v0.9.2rc2_hyperclovax_vision_seed` branch.
**Launch API server**:
```bash
pyenv virtualenv 3.10.2 .vllm
pyenv activate .vllm
sudo apt-get install -y kmod
pip install --upgrade setuptools wheel pip
pip install setuptools_scm
# install latest commit (e.g. v0.9.0)
VLLM_USE_PRECOMPILED=1 pip install -e .[serve] --cache-dir=/mnt/tmp
pip install -U pynvml
pip install timm av decord
# or install previous commit (e.g. v0.8.4)
pip install -r ./requirements/build.txt
pip install -r ./requirements/common.txt
pip install -r ./requirements/cuda.txt
pip install flash_attn==2.7.4.post1
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
export VLLM_COMMIT=dc1b4a6f1300003ae27f033afbdff5e2683721ce
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
VLLM_USE_PRECOMPILED=1 pip install -e .[serve] --cache-dir=/mnt/tmp
pip install -U pynvml
pip install timm av decord
# Then launch api
MODEL=your/mode/path
export ATTENTION_BACKEND=FLASH_ATTN_VLLM_V1
VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND=${ATTENTION_BACKEND} CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.openai.api_server \
--seed 20250525 \
--port ${PORT} \
--allowed-local-media-path $ALLOWED_LOCAL_MEDIA_PATH \
--max-model-len 8192 \
--max-num-batched-tokens 8192 \
--max-num-seqs 128 \
--max-parallel-loading-workers 128 \
--limit-mm-per-prompt.image="32" \
--limit-mm-per-prompt.viedo="32" \
--max-num-frames 256 \
--tensor-parallel-size 1 \
--data-parallel-size 1 \
--model ${MODEL} \
--dtype float16 \
--trust-remote-code \
--chat-template-content-format "openai" \
--download-dir $DONWLOAD_DIR
```
**Request Example**:
- https://github.com/vllm-project/vllm/pull/20931#issue-3229161410
**Offline Inference Examples**:
- https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language.py
- https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language_multi_image.py

35
added_tokens.json Normal file
View File

@@ -0,0 +1,35 @@
{
"<EMAIL>": 110521,
"<KEY>": 110522,
"<NAME>": 110520,
"<PASSWORD>": 110523,
"<code_to_intermediate>": 110502,
"<empty_output>": 110501,
"<file_sep>": 110492,
"<intermediate_to_code>": 110503,
"<issue_closed>": 110495,
"<issue_comment>": 110494,
"<issue_start>": 110493,
"<jupyter_code>": 110498,
"<jupyter_output>": 110499,
"<jupyter_script>": 110500,
"<jupyter_start>": 110496,
"<jupyter_text>": 110497,
"<pr>": 110504,
"<pr_base>": 110507,
"<pr_base_code>": 110509,
"<pr_comment>": 110512,
"<pr_diff>": 110510,
"<pr_diff_hunk>": 110511,
"<pr_diff_hunk_comment_line>": 110519,
"<pr_event_id>": 110513,
"<pr_file>": 110508,
"<pr_in_reply_to_comment_id>": 110518,
"<pr_in_reply_to_review_id>": 110517,
"<pr_is_merged>": 110506,
"<pr_review>": 110514,
"<pr_review_comment>": 110516,
"<pr_review_state>": 110515,
"<pr_status>": 110505,
"<repo_name>": 110491
}

65
chat_template.jinja Normal file
View File

@@ -0,0 +1,65 @@
<|im_start|>tool_list
<|im_end|>
{% for message in messages %}
{% set content = message['content'] %}
{% set role = message['role'] %}
{% if loop.first and role != 'system' %}
<|im_start|>system
You are a helpful assistant.<|im_end|>
{% endif %}
{% if message['content'] is string %}
<|im_start|>{{ role }}
{{ message['content'] }}<|im_end|>
{% elif message['content'] is mapping %}
{% if content['type'] == 'image' %}
<|im_start|>{{ role }} (mime)
{"type": "image/jpeg", "filename": "{{ content['filename'] }}"}<|im_end|>
<|im_start|>{{ role }} (vector)
<|dummy3|><|im_end|>
<|im_start|>image/aux
다음 중 ocr은 사진에서 검출된 글자이고, lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. 참고하여 답변하세요. {"ocr": "{{ content['ocr'] or '' }}", "lens_keywords": "{{ content['lens_keywords'] or '' }}", "lens_local_keywords": "{{ content['lens_local_keywords'] or '' }}"}<|im_end|>
{% elif content['type'] == 'video' %}
<|im_start|>{{ role }} (mime)
{"type": "video/mp4", "filename": "{{ content['filename'] }}"}<|im_end|>
<|im_start|>{{ role }} (vector)
<|_unuse_missing_100270|><|im_end|>
<|im_start|>image/aux
{% if content.get('is_final_grid') %}
다음 중 lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. speech_to_text는 비디오 속에서의 대화, 음성, 소리, 대사, 그리고 말을 전부 글로 받아 적은 것 입니다. 참고하여 답변하세요. {"video_time_stamp": "{{ content['video_time_stamp'] }}", "lens_keywords": "{{ content.get('lens_keywords', '') }}", "lens_local_keywords": "{{ content.get('lens_local_keywords', '') }}", "speech_to_text": "{{ content.get('speech_to_text', '') }}"}
{% else %}
다음 중 video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. 참고하여 답변하세요. {"video_time_stamp": "{{ content['video_time_stamp'] }}"}
{% endif %}<|im_end|>
{% elif content['type'] == 'text' %}
<|im_start|>{{ role }}
{{ content['text'] }}<|im_end|>
{% endif %}
{% elif message['content'] is sequence %}
{% for content in message['content'] %}
{% if content['type'] == 'image' %}
<|im_start|>{{ role }} (mime)
{"type": "image/jpeg", "filename": "{{ content['filename'] }}"}<|im_end|>
<|im_start|>{{ role }} (vector)
<|dummy3|><|im_end|>
<|im_start|>image/aux
다음 중 ocr은 사진에서 검출된 글자이고, lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. 참고하여 답변하세요. {"ocr": "{{ content['ocr'] or '' }}", "lens_keywords": "{{ content['lens_keywords'] or '' }}", "lens_local_keywords": "{{ content['lens_local_keywords'] or '' }}"}<|im_end|>
{% elif content['type'] == 'video' %}
<|im_start|>{{ role }} (mime)
{"type": "video/mp4", "filename": "{{ content['filename'] }}"}<|im_end|>
<|im_start|>{{ role }} (vector)
<|_unuse_missing_100270|><|im_end|>
<|im_start|>image/aux
{% if content.get('is_final_grid') %}
다음 중 lens_keyword는 사진에서 추출된 keyword와 bbox 위치입니다. bbox는 0~1 사이로 정규화된 [x1, y1, x2, y2]의 형태입니다. video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. speech_to_text는 비디오 속에서의 대화, 음성, 소리, 대사, 그리고 말을 전부 글로 받아 적은 것 입니다. 참고하여 답변하세요. {"video_time_stamp": "{{ content['video_time_stamp'] }}", "lens_keywords": "{{ content.get('lens_keywords', '') }}", "lens_local_keywords": "{{ content.get('lens_local_keywords', '') }}", "speech_to_text": "{{ content.get('speech_to_text', '') }}"}
{% else %}
다음 중 video_time_stamp는 비디오에서 해당 구간의 시간 정보입니다. 참고하여 답변하세요. {"video_time_stamp": "{{ content['video_time_stamp'] }}"}
{% endif %}<|im_end|>
{% elif content['type'] == 'text' %}
<|im_start|>{{ role }}
{{ content['text'] }}<|im_end|>
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}
<|im_start|>assistant
{% endif %}

202
config.json Normal file
View File

@@ -0,0 +1,202 @@
{
"anyres": true,
"architectures": [
"HCXVisionForCausalLM"
],
"auto_map": {
"AutoConfig": "configuration_hyperclovax.HCXVisionConfig",
"AutoModelForCausalLM": "modeling_hyperclovax.HCXVisionForCausalLM"
},
"decoder_max_length": 16384,
"freeze_decoder": false,
"freeze_encoder": true,
"freeze_mm_projector": false,
"hidden_size": 3072,
"ignore_index": -100,
"video_token_id": 100270,
"image_token_id": 100271,
"mm_projector_type": "cabstractor",
"text_config": {
"_attn_implementation_autoset": true,
"_name_or_path": "",
"add_cross_attention": false,
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": 100257,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"diversity_penalty": 0.0,
"do_sample": false,
"early_stopping": false,
"encoder_no_repeat_ngram_size": 0,
"end_token_id": 100257,
"eos_token_id": 100257,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 3072,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"initializer_range": 0.02,
"intermediate_size": 7168,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"length_penalty": 1.0,
"logits_scaling": 1.0,
"max_length": 20,
"max_position_embeddings": 131072,
"min_length": 0,
"mlp_bias": false,
"model_type": "llama",
"no_repeat_ngram_size": 0,
"num_attention_heads": 24,
"num_beam_groups": 1,
"num_beams": 1,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"num_return_sequences": 1,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": 100257,
"prefix": null,
"pretraining_tp": 1,
"problem_type": null,
"pruned_heads": {},
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"resid_pdrop": 0.2,
"return_dict": true,
"return_dict_in_generate": false,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 100000000,
"sep_token_id": null,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torch_dtype": "bfloat16",
"torchscript": false,
"transformers_version": "4.52.4",
"typical_p": 1.0,
"use_bfloat16": false,
"use_cache": true,
"vocab_size": 110592
},
"max_image_cnt": 12,
"max_num_grids": 9,
"model_type": "hyperclovax_vlm",
"num_queries_vis_abstractor_image": 81,
"num_queries_vis_abstractor_video_slow": 81,
"num_queries_vis_abstractor_video_fast": 9,
"first_last_frames_slow": false,
"proj_pos_emb": true,
"proj_prenorm": false,
"q_former_model_name_or_path": null,
"torch_dtype": "bfloat16",
"transformers_version": "4.52.4",
"unpad": true,
"use_1x1_grid": true,
"use_nth_layer": -2,
"vision_config": {
"_attn_implementation_autoset": true,
"_name_or_path": "",
"add_cross_attention": false,
"architectures": [
"SiglipVisionModel"
],
"attention_dropout": 0.0,
"auto_map": {},
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": null,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"diversity_penalty": 0.0,
"do_sample": false,
"early_stopping": false,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": null,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"hidden_act": "gelu_pytorch_tanh",
"hidden_size": 1152,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"image_size": 378,
"initializer_factor": 1.0,
"intermediate_size": 4304,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"layer_norm_eps": 1e-06,
"length_penalty": 1.0,
"max_length": 20,
"max_num_grids": 9,
"min_length": 0,
"model_type": "siglip_vision_model",
"no_repeat_ngram_size": 0,
"num_attention_heads": 16,
"num_beam_groups": 1,
"num_beams": 1,
"num_channels": 3,
"num_hidden_layers": 27,
"num_return_sequences": 1,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": null,
"patch_size": 14,
"prefix": null,
"problem_type": null,
"pruned_heads": {},
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"sep_token_id": null,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torch_dtype": "bfloat16",
"torchscript": false,
"transformers_version": "4.52.4",
"typical_p": 1.0,
"use_bfloat16": true
}
}

1
configuration.json Normal file
View File

@@ -0,0 +1 @@
{"framework": "pytorch", "task": "others", "allow_remote": true}

66
configuration_hyperclovax.py Normal file
View File

@@ -0,0 +1,66 @@
from transformers import AutoConfig
from transformers.configuration_utils import PretrainedConfig
from transformers.utils import logging
logger = logging.get_logger(__name__)
class HCXVisionConfig(PretrainedConfig):
model_type = "hyperclovax_vlm"
keys_to_ignore_at_inference = ["past_key_values"]
# The `gpt2` class has a different name, so it needs to be updated accordingly.
text_config_attribute_map = {
"n_embd": "hidden_size",
"n_positions": "max_position_embeddings",
"n_head": "num_attention_heads",
"n_layer": "num_hidden_layers",
}
def __init__(
self,
text_config=None,
vision_config=None,
use_nth_layer=-2,
img_start_id=100009, # <|dummy3|>
decoder_max_length=4096,
anyres=False,
unpad=False,
max_num_grids=-1,
num_queries_vis_abstractor=-1,
ignore_index=-100,
proj_pos_emb=True,
proj_prenorm=False,
use_1x1_grid=False,
**kwargs,
):
for key, val in self.text_config_attribute_map.items():
if text_config is not None and key in text_config:
text_config[val] = text_config.pop(key)
if text_config is not None:
_text_config = AutoConfig.for_model(text_config["model_type"])
self.text_config = _text_config.from_dict(text_config)
# In DeepSpeed ZeRO-3, the memory size is automatically determined based on the `hidden_size` specified in the config.
self.hidden_size = text_config["hidden_size"] if "hidden_size" in text_config else text_config["n_embd"]
if vision_config is not None:
_vision_config = AutoConfig.for_model(vision_config["model_type"])
self.vision_config = _vision_config.from_dict(vision_config)
# add VLM configs
self.use_nth_layer = use_nth_layer
self.decoder_max_length = decoder_max_length
self.anyres = anyres
self.unpad = unpad
self.max_num_grids = max_num_grids
self.num_queries_vis_abstractor = num_queries_vis_abstractor
self.img_start_id = img_start_id
self.ignore_index = ignore_index
self.proj_pos_emb = proj_pos_emb
self.proj_prenorm = proj_prenorm
self.use_1x1_grid = use_1x1_grid
super().__init__(**kwargs)
def get_text_config(self, decoder=False):
return self.text_config

789
image_processing_hyperclovax.py Normal file
View File

@@ -0,0 +1,789 @@
import copy
import math
import os
from typing import Dict, List, Optional, Union
import numpy as np
import torch
from PIL import Image
from transformers.feature_extraction_utils import BatchFeature
from transformers.image_processing_utils import (
BaseImageProcessor,
get_size_dict,
)
from transformers.image_transforms import (
convert_to_rgb,
get_resize_output_image_size,
resize,
to_channel_dimension_format,
)
from transformers.image_utils import (
OPENAI_CLIP_MEAN,
OPENAI_CLIP_STD,
ChannelDimension,
ImageInput,
PILImageResampling,
get_image_size,
infer_channel_dimension_format,
is_scaled_image,
make_list_of_images,
to_numpy_array,
valid_images,
)
from transformers.utils import TensorType, logging
logger = logging.get_logger(__name__)
class HCXImageProcessor(BaseImageProcessor):
r"""
Constructs a VLM image processor. Based on [`CLIPImageProcessor`] with incorporation of additional techniques for processing high resolution images.
Args:
anyres: (bool) anyres 기능을 사용할지 안할지
unpad: (bool) anyres 사용시, unpad 기능 (순수 pad 영역에 해당하는 visual tokens 은 LLM input 에서 제거) 을 사용할지 안할지
num_queries_vis_abstractor: (int) 각 grid 에 대해서 resampler 를 사용하는 경우, visual query 수
possible_resolutions: (List) anyres 기능 사용시, 가능한 resolution 조합, 예: [[336, 336], [336, 672], [672, 336]]
patch_size: (int) ViT patch size
pad_to_square: (bool) 정사각형으로 padding 을 수행할지, 안할지를 결정. False 이면 정사각형이 아니기 때문에 center crop 을 거쳐 ViT 의 입력으로 들어감
"""
model_input_names = ["pixel_values"]
def __init__(
self,
do_resize: bool = True,
size: Dict[str, int] = None,
anyres: bool = False,
unpad: bool = False,
num_queries_vis_abstractor_image: int = 81,
num_queries_vis_abstractor_video_slow: int = 81,
num_queries_vis_abstractor_video_fast: int = 9,
first_last_frames_slow_video: bool = False,
possible_resolutions: List = [],
patch_size: int = 14,
pad_to_square: bool = True,
resample: PILImageResampling = PILImageResampling.BICUBIC,
do_center_crop: bool = True,
crop_size: Dict[str, int] = None,
do_rescale: bool = True,
rescale_factor: Union[int, float] = 1 / 255,
do_normalize: bool = True,
image_mean: Optional[Union[float, List[float]]] = None,
image_std: Optional[Union[float, List[float]]] = None,
do_convert_rgb: bool = True,
**kwargs,
) -> None:
super().__init__(**kwargs)
size = size if size is not None else {"shortest_edge": 336}
size = get_size_dict(size, default_to_square=False)
crop_size = crop_size if crop_size is not None else {"height": 336, "width": 336}
crop_size = get_size_dict(crop_size, default_to_square=True, param_name="crop_size")
self.do_resize = do_resize
self.size = size
self.anyres = anyres
self.unpad = unpad
self.num_queries_vis_abstractor_image = num_queries_vis_abstractor_image
self.num_queries_vis_abstractor_video_slow = num_queries_vis_abstractor_video_slow
self.num_queries_vis_abstractor_video_fast = num_queries_vis_abstractor_video_fast
self.first_last_frames_slow_video = first_last_frames_slow_video
self.possible_resolutions = [_resolution for _resolution in possible_resolutions]
self.patch_size = patch_size
self.pad_to_square = pad_to_square
self.resample = resample
self.do_center_crop = do_center_crop
self.crop_size = crop_size
self.do_rescale = do_rescale
self.rescale_factor = rescale_factor
self.do_normalize = do_normalize
self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN
self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD
self.do_convert_rgb = do_convert_rgb
def resize(
self,
image: np.ndarray,
size: Dict[str, int],
resample: PILImageResampling = PILImageResampling.BICUBIC,
data_format: Optional[Union[str, ChannelDimension]] = None,
input_data_format: Optional[Union[str, ChannelDimension]] = None,
**kwargs,
) -> np.ndarray:
default_to_square = True
if "shortest_edge" in size:
size = size["shortest_edge"]
default_to_square = False
elif "height" in size and "width" in size:
size = (size["height"], size["width"])
else:
raise ValueError("Size must contain either 'shortest_edge' or 'height' and 'width'.")
output_size = get_resize_output_image_size(
image,
size=size,
default_to_square=default_to_square,
input_data_format=input_data_format,
)
return resize(
image,
size=output_size,
resample=resample,
data_format=data_format,
input_data_format=input_data_format,
**kwargs,
)
def _preprocess(
self,
images: ImageInput,
do_resize: bool = None,
size: Dict[str, int] = None,
resample: PILImageResampling = None,
do_center_crop: bool = None,
crop_size: int = None,
do_rescale: bool = None,
rescale_factor: float = None,
do_normalize: bool = None,
image_mean: Optional[Union[float, List[float]]] = None,
image_std: Optional[Union[float, List[float]]] = None,
data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
input_data_format: Optional[Union[str, ChannelDimension]] = None,
) -> Image.Image:
images = make_list_of_images(images)
if do_resize:
images = [
self.resize(image=image, size=size, resample=resample, input_data_format=input_data_format)
for image in images
]
if do_center_crop:
images = [
self.center_crop(image=image, size=crop_size, input_data_format=input_data_format) for image in images
]
if do_rescale:
images = [
self.rescale(image=image, scale=rescale_factor, input_data_format=input_data_format) for image in images
]
if do_normalize:
images = [
self.normalize(image=image, mean=image_mean, std=image_std, input_data_format=input_data_format)
for image in images
]
images = [
to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format) for image in images
]
return images
def _resize_for_local_grids(
self, image: np.array, target_resolution: tuple, resample, input_data_format: ChannelDimension
) -> np.array:
new_height, new_width = _get_local_grids_output_size(image, target_resolution, input_data_format)
# Resize the image
resized_image = resize(image, (new_height, new_width), resample=resample, input_data_format=input_data_format)
return resized_image
def _pad_for_patching(
self, image: np.array, target_resolution: tuple, input_data_format: ChannelDimension
) -> np.array:
"""
Pad an image to a target resolution while maintaining aspect ratio.
"""
target_height, target_width = target_resolution
background_color = tuple(int(x * 255) for x in self.image_mean)
padded_image = pad(
image,
target_size=(target_height, target_width),
background_color=background_color,
input_data_format=input_data_format,
)
return padded_image
def get_image_grids(
self,
image: np.array,
possible_resolutions,
grid_size: int,
resample: PILImageResampling,
data_format: ChannelDimension,
input_data_format: ChannelDimension,
) -> List[np.array]:
if not isinstance(possible_resolutions, list):
raise ValueError("possible_resolutions must be a list of possible resolutions.")
image_size = get_image_size(image, channel_dim=input_data_format)
best_resolution = select_best_resolution(image_size, possible_resolutions)
resized_image = self._resize_for_local_grids(
image, best_resolution, resample=resample, input_data_format=input_data_format
)
padded_image = self._pad_for_patching(resized_image, best_resolution, input_data_format=input_data_format)
local_grids = divide_to_grids(padded_image, grid_size=grid_size, input_data_format=input_data_format)
# make sure that all patches are in the input data format
local_grids = [
to_channel_dimension_format(grid, channel_dim=data_format, input_channel_dim=input_data_format)
for grid in local_grids
]
return local_grids
def preprocess(
self,
images: ImageInput,
do_resize: bool = None,
size: Dict[str, int] = None,
anyres: bool = None,
unpad: bool = None,
is_video: bool = False,
num_queries_vis_abstractor_image: int = None,
num_queries_vis_abstractor_video_slow: int = None,
num_queries_vis_abstractor_video_fast: int = None,
first_last_frames_slow_video: bool = None,
possible_resolutions: List = None,
patch_size: int = None,
pad_to_square: bool = None,
resample: PILImageResampling = None,
do_center_crop: bool = None,
crop_size: int = None,
do_rescale: bool = None,
rescale_factor: float = None,
do_normalize: bool = None,
image_mean: Optional[Union[float, List[float]]] = None,
image_std: Optional[Union[float, List[float]]] = None,
do_convert_rgb: bool = None,
return_tensors: Optional[Union[str, TensorType]] = None,
data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
input_data_format: Optional[Union[str, ChannelDimension]] = None,
return_dummy_image: bool = False,
first_last_frames_slow: bool = False,
is_first_or_last_frames: bool = False,
**kwargs,
):
"""
HCXVisionImageProcessor 로 image tensor, original image size (width, height), visual tokens
:return pixel_values: List of 4D tensor 로 image tensor
:return image_sizes: List of Dict 로 image width, height [{"width": image 1 의 width, "height": image 1 의 height}, {"width": image 2 의 width, "height": image 2 의 height}, ...]
:return vision_query_lengths: List of int 로 각 image 가 LLM 입력으로 전달될때 변환되는 visual token 수
"""
do_resize = do_resize if do_resize is not None else self.do_resize
size = size if size is not None else self.size
size = get_size_dict(size, param_name="size", default_to_square=False)
anyres = anyres if anyres is not None else self.anyres
unpad = unpad if unpad is not None else self.unpad
num_queries_vis_abstractor_image = (
num_queries_vis_abstractor_image
if num_queries_vis_abstractor_image is not None
else self.num_queries_vis_abstractor_image
)
num_queries_vis_abstractor_video_slow = (
num_queries_vis_abstractor_video_slow
if num_queries_vis_abstractor_video_slow is not None
else self.num_queries_vis_abstractor_video_slow
)
num_queries_vis_abstractor_video_fast = (
num_queries_vis_abstractor_video_fast
if num_queries_vis_abstractor_video_fast is not None
else self.num_queries_vis_abstractor_video_fast
)
first_last_frames_slow_video = (
first_last_frames_slow_video
if first_last_frames_slow_video is not None
else self.first_last_frames_slow_video
)
possible_resolutions = possible_resolutions if possible_resolutions is not None else self.possible_resolutions
patch_size = patch_size if patch_size is not None else self.patch_size
pad_to_square = pad_to_square if pad_to_square is not None else self.pad_to_square
resample = resample if resample is not None else self.resample
do_center_crop = do_center_crop if do_center_crop is not None else self.do_center_crop
crop_size = crop_size if crop_size is not None else self.crop_size
crop_size = get_size_dict(crop_size, param_name="crop_size", default_to_square=True)
do_rescale = do_rescale if do_rescale is not None else self.do_rescale
rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
do_normalize = do_normalize if do_normalize is not None else self.do_normalize
image_mean = image_mean if image_mean is not None else self.image_mean
image_std = image_std if image_std is not None else self.image_std
do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
if is_video:
num_queries_vis_abstractor = num_queries_vis_abstractor_video_fast
num_queries_vis_abstractor_slow = num_queries_vis_abstractor_video_slow
unpad = False
else:
num_queries_vis_abstractor = num_queries_vis_abstractor_image
num_queries_vis_abstractor_slow = 0
if return_dummy_image:
images = Image.new("RGB", (224, 224), (0, 0, 0))
images = make_list_of_images(images)
if not valid_images(images):
raise ValueError(
"Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, "
"torch.Tensor, tf.Tensor or jax.ndarray."
)
if do_convert_rgb:
images = [convert_to_rgb(image) for image in images]
# All transformations expect numpy arrays.
images = [to_numpy_array(image) for image in images]
if is_scaled_image(images[0]) and do_rescale:
logger.warning_once(
"It looks like you are trying to rescale already rescaled images. If the input"
" images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
)
if input_data_format is None:
# We assume that all images have the same channel dimension format.
input_data_format = infer_channel_dimension_format(images[0])
new_images = []
image_sizes = [get_image_size(image, channel_dim=input_data_format) for image in images]
vision_query_lengths = []
assert crop_size["height"] == crop_size["width"]
# global image 의 padding 연산은, image original width, height 가 클 때 bottleneck 이 될 수 있음
# 장축의 길이를 size["shortest_edge"] 로 resize 를 먼저 한 뒤에, padding
if anyres:
anyres_global_images = copy.deepcopy(images)
if pad_to_square:
background_color = tuple(int(x * 255) for x in self.image_mean)
anyres_global_images = [
resize_longside(copy.deepcopy(image), size["shortest_edge"], resample, input_data_format)
for image in anyres_global_images
]
anyres_global_images = [
expand2square(image, background_color=background_color, input_data_format=input_data_format)[0]
for image in anyres_global_images
]
else:
anyres_global_images = [
self.resize(
image=image,
size={"height": size["shortest_edge"], "width": size["shortest_edge"]},
resample=resample,
input_data_format=input_data_format,
)
for image in anyres_global_images
]
else:
anyres_global_images = [None for _ in range(len(images))]
if pad_to_square:
background_color = tuple(int(x * 255) for x in self.image_mean)
images = [
resize_longside(image, size["shortest_edge"], resample, input_data_format) for image in images
]
images = [
expand2square(image, background_color=background_color, input_data_format=input_data_format)[0]
for image in images
]
for image, anyres_global_image, image_size in zip(images, anyres_global_images, image_sizes):
if anyres:
# convert image into a list of grids
# we intentially use the same data format as the input data format
image_grids = self.get_image_grids(
image,
possible_resolutions,
grid_size=crop_size["height"],
resample=resample,
data_format=input_data_format,
input_data_format=input_data_format,
)
# video 에 대해서는 global image (thumbnail) 를 사용하지 않음
if not is_video:
image_grids = [anyres_global_image] + image_grids
else:
image_grids = [image]
pixel_values = self._preprocess(
image_grids,
do_resize=do_resize,
size=size,
resample=resample,
do_center_crop=do_center_crop,
crop_size=crop_size,
do_rescale=do_rescale,
rescale_factor=rescale_factor,
do_normalize=do_normalize,
image_mean=image_mean,
image_std=image_std,
data_format=data_format,
input_data_format=input_data_format,
)
pixel_values = np.array(pixel_values)
new_images.append(pixel_values)
vision_query_length = determine_anyres_num_vision_patches(
image_size=image_size,
grid_size=crop_size["height"],
patch_size=patch_size,
possible_resolutions=possible_resolutions,
anyres=anyres,
unpad=unpad,
num_queries_vis_abstractor=num_queries_vis_abstractor,
num_queries_vis_abstractor_slow=num_queries_vis_abstractor_slow,
is_video=is_video,
first_last_frames_slow=first_last_frames_slow,
is_first_or_last_frames=is_first_or_last_frames,
)
vision_query_lengths.append(vision_query_length)
if return_dummy_image:
vision_query_lengths = []
data = {
"pixel_values": [torch.tensor(new_image) for new_image in new_images],
"image_sizes": [{"width": image_size[1], "height": image_size[0]} for image_size in image_sizes],
"vision_query_lengths": vision_query_lengths,
}
return BatchFeature(data=data, tensor_type=return_tensors)
def save_pretrained(
self,
save_directory: Union[str, os.PathLike],
*args,
**kwargs,
):
self.register_for_auto_class()
super().save_pretrained(save_directory, *args, **kwargs)
def determine_anyres_num_vision_patches(
image_size,
grid_size,
patch_size,
possible_resolutions,
anyres=False,
unpad=True,
num_queries_vis_abstractor=0,
num_queries_vis_abstractor_slow=0,
is_video=False,
first_last_frames_slow=False, # sample-wise option
is_first_or_last_frames=False, # grid-wise option
):
"""
Computes the number of visual tokens (patches) based on image resolution, grid configuration, and patch size.
This function supports both fixed-size and any-resolution settings, as well as video-specific configurations
such as handling slow frames and frame position flags.
Args:
num_grids (int): Number of grids per image (e.g., 1 for 1x1, 4 for 2x2, etc.).
image_size (tuple): The original image size as (height, width).
grid_size (int): Size of each grid in pixels (e.g., 336).
patch_size (int): Size of each vision patch (e.g., 14 for ViT models).
possible_resolutions (list): List of possible resolution tuples [(h1, w1), (h2, w2), ...].
anyres (bool, optional): Whether to use any-resolution mode. Defaults to False.
unpad (bool, optional): Whether to unpad the image before computing patches. Defaults to True.
num_queries_vis_abstractor (int, optional): Number of query tokens for vision abstractor (fast path).
num_queries_vis_abstractor_slow (int, optional): Number of query tokens for vision abstractor (slow path).
is_video (bool, optional): Whether the input is a video. Defaults to False.
first_last_frames_slow (bool, optional): Whether to treat first/last video frames as "slow". Defaults to False.
is_first_or_last_frames (bool, optional): Whether current grid corresponds to first/last frame. Defaults to False.
Returns:
int: Total number of visual tokens (patches) after processing.
"""
if not anyres:
return num_queries_vis_abstractor if num_queries_vis_abstractor > 0 else (grid_size // patch_size) ** 2
if num_queries_vis_abstractor > 0:
num_patch_per_grid = int(num_queries_vis_abstractor**0.5)
else:
num_patch_per_grid = grid_size // patch_size
num_global_per_grid = num_patch_per_grid
# In anyres mode, a global image is included, so there are always at least 2 grids.
# However, for video inputs, there is no global image, so it's possible to have only 1 grid.
# Therefore, the assertion below is commented out:
# assert num_grids > 1
# Compute the number of vision patches.
height, width = select_best_resolution(image_size, possible_resolutions)
num_patch_height = (height // grid_size) * num_patch_per_grid
num_patch_width = (width // grid_size) * num_patch_per_grid
# local images
if unpad:
original_height, original_width = image_size
original_aspect_ratio = original_width / original_height
current_aspect_ratio = num_patch_width / num_patch_height
if original_aspect_ratio > current_aspect_ratio:
scale_factor = num_patch_width / original_width
new_height = int(original_height * scale_factor)
padding = (num_patch_height - new_height) // 2
num_patch_height = num_patch_height - padding * 2
else:
scale_factor = num_patch_height / original_height
new_width = int(original_width * scale_factor)
padding = (num_patch_width - new_width) // 2
num_patch_width = num_patch_width - padding * 2
num_patches = num_patch_width * num_patch_height + num_patch_height
else:
num_patches = num_patch_width * num_patch_height
# In the "slow" strategy, when applying to first and last frames only, it is applied exclusively to those two frames.
if num_queries_vis_abstractor_slow > 0:
if first_last_frames_slow:
if is_first_or_last_frames:
num_patches += num_queries_vis_abstractor_slow - num_queries_vis_abstractor
else:
num_patches += num_queries_vis_abstractor_slow - num_queries_vis_abstractor
# The slowfast feature is only applicable when unpad is set to False.
assert unpad is False
# Global image is not included for video inputs.
if not is_video:
num_patches += num_global_per_grid**2
return num_patches
def divide_to_grids(image: np.array, grid_size: int, input_data_format=None) -> List[np.array]:
"""
Divides a local image into grids of size (grid_size x grid_size).
Args:
image (np.array): Input image as a NumPy array.
grid_size (int): The size (in pixels) of each square grid.
input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
Returns:
List[np.array]: A list of image patches, each of size (grid_size x grid_size).
"""
grids = []
height, width = get_image_size(image, channel_dim=input_data_format)
for i in range(0, height, grid_size):
for j in range(0, width, grid_size):
if input_data_format == ChannelDimension.LAST:
grid = image[i : i + grid_size, j : j + grid_size]
else:
grid = image[:, i : i + grid_size, j : j + grid_size]
grids.append(grid)
return grids
def pad(
image: np.array,
target_size: tuple,
background_color=(127, 127, 127),
input_data_format=None,
) -> np.array:
"""
Pads the input image on the sides (top/bottom and left/right) to match the target height and width.
Args:
image (np.array): Input image as a NumPy array.
target_size (tuple): Target size as (target_height, target_width).
background_color (tuple, optional): RGB color value used for padding. Defaults to (127, 127, 127).
input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
Returns:
np.array: The padded image with the specified target size.
"""
target_height, target_width = target_size
height, width = get_image_size(image, channel_dim=input_data_format)
# result = np.ones((target_height, target_width, image.shape[2]), dtype=image.dtype) * background_color
result = np.empty((target_height, target_width, image.shape[2]), dtype=image.dtype)
for i in range(image.shape[2]):
result[..., i].fill(background_color[i])
paste_x = (target_width - width) // 2
paste_y = (target_height - height) // 2
result[paste_y : paste_y + height, paste_x : paste_x + width, :] = image
return result
def expand2square(
image: np.array,
bboxes_dict=None,
background_color=(127, 127, 127),
input_data_format=None,
) -> np.array:
"""
Expands the input image to a square shape by placing it at the center of a new square canvas,
with padding added to the shorter side (either top/bottom or left/right).
The image is always centered on the new canvas, and padding is applied symmetrically.
Args:
image (np.array): Input image as a NumPy array.
bboxes_dict (dict, optional): A dictionary of bounding boxes, where each value is an NDArray of shape (N, 4, 2)
with box coordinates in the format [[xtl, ytl], [xtr, ytr], [xbr, ybr], [xbl, ybl]].
Supports multiple categories (e.g., "ocr", "html") simultaneously.
background_color (tuple, optional): RGB color to fill the padding area. Defaults to (127, 127, 127).
input_data_format (optional): Optional format specifier for image data (e.g., "channels_first" or "channels_last").
Returns:
np.array: A square-shaped image with the original image centered and padded as needed.
Example:
>>> _img = np.ones((80, 100), dtype=np.uint8) * 100
>>> _bboxes_dict = {"words": np.array([[[10, 10], [20, 10], [20, 20], [10, 20]],
... [[30, 30], [40, 30], [40, 40], [30, 40]]])}
>>> _img, _bboxes_dict = expand2square(_img, _bboxes_dict, (255, 255, 255))
>>> _img.shape
(100, 100)
>>> guessed_ocr_bboxes = np.array([[[20, 10], [30, 10], [30, 20], [20, 20]],
... [[40, 30], [50, 30], [50, 40], [40, 40]]])
>>> np.testing.assert_array_almost_equal(_bboxes_dict["words"], guessed_ocr_bboxes) is None
True
"""
height, width = get_image_size(image, channel_dim=input_data_format)
if width == height:
return image, bboxes_dict
elif width > height:
# result = np.ones((width, width, image.shape[2]), dtype=image.dtype) * background_color
result = np.empty((width, width, image.shape[2]), dtype=image.dtype)
for i in range(image.shape[2]):
result[..., i].fill(background_color[i])
result[(width - height) // 2 : (width - height) // 2 + height, :] = image
if bboxes_dict is not None:
for key in bboxes_dict:
bboxes_dict[key][:, :, 1] += (width - height) // 2
return result, bboxes_dict
else:
# result = np.ones((height, height, image.shape[2]), dtype=image.dtype) * background_color
result = np.empty((height, height, image.shape[2]), dtype=image.dtype)
for i in range(image.shape[2]):
result[..., i].fill(background_color[i])
result[:, (height - width) // 2 : (height - width) // 2 + width] = image
if bboxes_dict is not None:
for key in bboxes_dict:
bboxes_dict[key][:, :, 0] += (height - width) // 2
return result, bboxes_dict
def resize_longside(
image: np.array,
size: int,
resample: PILImageResampling = PILImageResampling.BICUBIC, # type: ignore
data_format: Optional[Union[str, ChannelDimension]] = None,
input_data_format: Optional[Union[str, ChannelDimension]] = None,
):
"""
Resizes the image so that its longer side matches the specified size, maintaining the original aspect ratio.
Args:
image (np.array): Input image as a NumPy array.
size (int): Target size for the longer side of the image.
resample (PILImageResampling, optional): Resampling method to use during resizing. Defaults to BICUBIC.
data_format (str or ChannelDimension, optional): Output data format (e.g., "channels_first" or "channels_last").
input_data_format (str or ChannelDimension, optional): Input data format of the image.
Returns:
np.array: The resized image with its aspect ratio preserved.
"""
height, width = get_image_size(image, channel_dim=input_data_format)
if width == height:
target_height, target_width = size, size
elif width > height:
target_width = size
target_height = math.ceil(height / width * size)
else:
target_width = math.ceil(width / height * size)
target_height = size
return resize(
image,
size=(target_height, target_width),
resample=resample,
data_format=data_format,
input_data_format=input_data_format,
)
def _get_local_grids_output_size(image: np.array, target_resolution: tuple, input_data_format=None):
"""
Computes the number of local grids (patches) along the height and width when resizing an image
to the target resolution.
Args:
image (np.array): Input image as a NumPy array.
target_resolution (tuple): Target resolution in the format (target_height, target_width).
input_data_format (optional): Optional format specifier (e.g., "channels_first" or "channels_last").
Returns:
tuple: A tuple (grid_h, grid_w) representing the number of grids along the height and width.
"""
original_height, original_width = get_image_size(image, channel_dim=input_data_format)
target_height, target_width = target_resolution
scale_w = target_width / original_width
scale_h = target_height / original_height
if scale_w < scale_h:
new_width = target_width
new_height = min(math.ceil(original_height * scale_w), target_height)
else:
new_height = target_height
new_width = min(math.ceil(original_width * scale_h), target_width)
return new_height, new_width
def select_best_resolution(original_size: tuple, possible_resolutions: list) -> tuple:
"""
Selects the best-fit resolution from a list of possible resolutions based on the original image size.
This function, adapted from LLaVA-Next
(https://github.com/huggingface/transformers/blob/v4.40.2/src/transformers/models/llava_next/image_processing_llava_next.py),
evaluates each resolution by computing its effective and wasted area compared to the original size.
The optimal resolution is the one that maximizes the effective area while minimizing unused (wasted) space.
Args:
original_size (tuple): The original image size in the format (height, width).
possible_resolutions (list): A list of candidate resolutions in the format [(height1, width1), (height2, width2), ...].
Returns:
tuple: The best-fit resolution in the format (height, width).
"""
original_height, original_width = original_size
best_fit = None
max_effective_resolution = 0
min_wasted_resolution = float("inf")
for height, width in possible_resolutions:
scale = min(width / original_width, height / original_height)
downscaled_width, downscaled_height = int(original_width * scale), int(original_height * scale)
effective_resolution = min(downscaled_width * downscaled_height, original_width * original_height)
wasted_resolution = (width * height) - effective_resolution
if effective_resolution > max_effective_resolution or (
effective_resolution == max_effective_resolution and wasted_resolution < min_wasted_resolution
):
max_effective_resolution = effective_resolution
min_wasted_resolution = wasted_resolution
best_fit = (height, width)
return best_fit

110306
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:340b93b87f93c98b62d2c96ef56e4656d9d68ec8a1cd178fe6812c925f8d8d88
size 4997245472

3
model-00002-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f85a5d24cadbb3d235c670a88f9b0757ff50b226819ba0a3cece51a72a2891e4
size 4920253536

3
model-00003-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a4efe99ddd60fc020ad202df4ed089ab0a280c7b17d5376f011888fde8dd2c44
size 4967583384

829
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,829 @@
{
"metadata": {
"total_size": 14884974080
},
"weight_map": {
"image_newline": "model-00001-of-00003.safetensors",
"language_model.model.embed_tokens.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.input_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.input_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.10.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.10.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.10.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.10.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.10.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.10.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.10.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.10.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.10.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.11.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.12.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.13.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.14.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.15.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.16.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.17.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.18.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.18.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.18.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.18.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.18.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.18.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.18.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.18.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.18.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.19.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.19.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.19.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.19.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.19.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.19.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.19.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.19.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.19.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.2.input_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.2.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.2.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.2.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.2.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.2.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.20.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.20.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.20.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.20.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.20.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.20.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.20.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.20.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.20.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.21.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.22.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.23.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.24.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.25.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.26.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.27.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.28.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.29.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.3.input_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.3.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.3.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.3.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.3.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.3.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.30.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.30.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.30.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.30.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.30.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.30.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.30.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.30.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.30.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.input_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.mlp.down_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.mlp.gate_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.mlp.up_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.post_attention_layernorm.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.self_attn.k_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.self_attn.o_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.self_attn.q_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.31.self_attn.v_proj.weight": "model-00003-of-00003.safetensors",
"language_model.model.layers.4.input_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.4.mlp.down_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.4.mlp.gate_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.4.mlp.up_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.4.post_attention_layernorm.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.4.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.5.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.5.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.5.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.5.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.5.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.5.self_attn.o_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"language_model.model.layers.6.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.6.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.6.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.6.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.6.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.6.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.6.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.6.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.6.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.7.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.8.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.input_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.mlp.down_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.mlp.gate_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.mlp.up_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.post_attention_layernorm.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.self_attn.k_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.self_attn.o_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.self_attn.q_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.layers.9.self_attn.v_proj.weight": "model-00002-of-00003.safetensors",
"language_model.model.norm.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv1.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv1.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv1.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv2.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv2.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv2.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv3.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv3.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.conv3.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.se.fc1.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.se.fc1.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.se.fc2.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b1.se.fc2.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv1.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv1.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv1.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv2.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv2.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv2.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv3.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv3.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.conv3.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.se.fc1.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.se.fc1.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.se.fc2.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b2.se.fc2.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv1.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv1.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv1.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv2.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv2.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv2.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv3.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv3.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.conv3.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.se.fc1.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.se.fc1.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.se.fc2.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.0.b3.se.fc2.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv1.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv1.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv1.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv2.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv2.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv2.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv3.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv3.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.conv3.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.se.fc1.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.se.fc1.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.se.fc2.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b1.se.fc2.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv1.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv1.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv1.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv2.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv2.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv2.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv3.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv3.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.conv3.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.se.fc1.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.se.fc1.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.se.fc2.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b2.se.fc2.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv1.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv1.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv1.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv2.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv2.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv2.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv3.bn.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv3.bn.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.conv3.conv.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.se.fc1.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.se.fc1.weight": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.se.fc2.bias": "model-00003-of-00003.safetensors",
"mm_projector.net.2.b3.se.fc2.weight": "model-00003-of-00003.safetensors",
"mm_projector.pos_emb": "model-00003-of-00003.safetensors",
"mm_projector.readout.0.bias": "model-00003-of-00003.safetensors",
"mm_projector.readout.0.weight": "model-00003-of-00003.safetensors",
"mm_projector.readout.2.bias": "model-00003-of-00003.safetensors",
"mm_projector.readout.2.weight": "model-00003-of-00003.safetensors",
"vision_model.vision_model.embeddings.patch_embedding.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.embeddings.patch_embedding.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.embeddings.position_embedding.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.0.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.1.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.10.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.11.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.12.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.13.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.14.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.15.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.16.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.17.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.18.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.19.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.2.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.20.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.21.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.22.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.23.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.24.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.25.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.26.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.3.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.4.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.5.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.6.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.7.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.8.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.layer_norm1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.layer_norm1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.layer_norm2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.layer_norm2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.self_attn.k_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.self_attn.k_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.self_attn.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.self_attn.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.self_attn.q_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.self_attn.q_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.self_attn.v_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.encoder.layers.9.self_attn.v_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.attention.in_proj_bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.attention.in_proj_weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.attention.out_proj.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.attention.out_proj.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.layernorm.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.layernorm.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.mlp.fc1.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.mlp.fc1.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.mlp.fc2.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.mlp.fc2.weight": "model-00001-of-00003.safetensors",
"vision_model.vision_model.head.probe": "model-00001-of-00003.safetensors",
"vision_model.vision_model.post_layernorm.bias": "model-00001-of-00003.safetensors",
"vision_model.vision_model.post_layernorm.weight": "model-00001-of-00003.safetensors"
}
}

1344
modeling_hyperclovax.py Normal file
View File

File diff suppressed because it is too large Load Diff

135
preprocessor_config.json Normal file
View File

@@ -0,0 +1,135 @@
{
"anyres": true,
"auto_map": {
"AutoImageProcessor": "image_processing_hyperclovax.HCXImageProcessor",
"AutoProcessor": "processing_hyperclovax.HCXProcessor"
},
"crop_size": {
"height": 378,
"width": 378
},
"do_center_crop": true,
"do_convert_rgb": true,
"do_normalize": true,
"do_rescale": true,
"do_resize": true,
"image_mean": [
0.5,
0.5,
0.5
],
"image_processor_class": "AutoImageProcessor",
"image_processor_type": "HCXImageProcessor",
"image_std": [
0.5,
0.5,
0.5
],
"num_queries_vis_abstractor_image": 81,
"num_queries_vis_abstractor_video_slow": 81,
"num_queries_vis_abstractor_video_fast": 9,
"first_last_frames_slow_video": false,
"pad_to_square": true,
"patch_size": 14,
"possible_resolutions": [
[
378,
378
],
[
378,
756
],
[
378,
1134
],
[
378,
1512
],
[
378,
1890
],
[
378,
2268
],
[
378,
2646
],
[
378,
3024
],
[
378,
3402
],
[
756,
378
],
[
756,
756
],
[
756,
1134
],
[
756,
1512
],
[
1134,
378
],
[
1134,
756
],
[
1134,
1134
],
[
1512,
378
],
[
1512,
756
],
[
1890,
378
],
[
2268,
378
],
[
2646,
378
],
[
3024,
378
],
[
3402,
378
]
],
"processor_class": "HCXProcessor",
"resample": 2,
"rescale_factor": 0.00392156862745098,
"size": {
"shortest_edge": 378
},
"unpad": true
}

912
processing_hyperclovax.py Normal file
View File

@@ -0,0 +1,912 @@
import copy
import os
import re
import uuid
from typing import Dict, List, Optional, Union
import numpy as np
import PIL
from PIL import Image
import torch
from transformers.feature_extraction_utils import BatchFeature
from transformers.image_utils import ImageInput, load_image
from transformers.processing_utils import (
AllKwargsForChatTemplate,
ChatTemplateLoadKwargs,
ProcessingKwargs,
ProcessorMixin,
Unpack,
)
from transformers.tokenization_utils_base import AudioInput, TextInput
from transformers.utils import (
is_torch_device,
is_torch_dtype,
logging,
requires_backends,
)
from transformers.utils.chat_template_utils import render_jinja_template
from transformers.video_utils import VideoInput, VideoMetadata, load_video
logger = logging.get_logger(__name__)
class HCXBatchFeature(BatchFeature):
def to(self, *args, **kwargs) -> "BatchFeature":
"""
Send all values to device by calling `v.to(*args, **kwargs)` (PyTorch only). This should support casting in
different `dtypes` and sending the `BatchFeature` to a different `device`.
Args:
args (`Tuple`):
Will be passed to the `to(...)` function of the tensors.
kwargs (`Dict`, *optional*):
Will be passed to the `to(...)` function of the tensors.
To enable asynchronous data transfer, set the `non_blocking` flag in `kwargs` (defaults to `False`).
Returns:
[`BatchFeature`]: The same instance after modification.
"""
requires_backends(self, ["torch"])
import torch # noqa
new_data = {}
device = kwargs.get("device")
non_blocking = kwargs.get("non_blocking", False)
# Check if the args are a device or a dtype
if device is None and len(args) > 0:
# device should be always the first argument
arg = args[0]
if is_torch_dtype(arg):
# The first argument is a dtype
pass
elif isinstance(arg, str) or is_torch_device(arg) or isinstance(arg, int):
device = arg
else:
# it's something else
raise ValueError(f"Attempting to cast a BatchFeature to type {str(arg)}. This is not supported.")
# We cast only floating point tensors to avoid issues with tokenizers casting `LongTensor` to `FloatTensor`
for k, v in self.items():
# check if v is a floating point
if isinstance(v, torch.Tensor) and torch.is_floating_point(v):
# cast and send to device
new_data[k] = v.to(*args, **kwargs)
elif isinstance(v, torch.Tensor) and device is not None:
new_data[k] = v.to(device=device, non_blocking=non_blocking)
elif "pixel_values" in k:
new_pixel_values_batch = []
for _v in v:
pixel_values = [pixel_value.to(device=device, non_blocking=non_blocking) for pixel_value in _v]
new_pixel_values_batch.append(pixel_values)
new_data[k] = new_pixel_values_batch
else:
new_data[k] = v
self.data = new_data
return self
class HCXProcessorKwargs(ProcessingKwargs, total=False):
_defaults = {
"text_kwargs": {
"return_tensors": "pt",
"calc_non_vision_query_lengths": False,
},
"images_kwargs": {},
"audio_kwargs": {},
"videos_kwargs": {
"max_image_cnt": 12,
"max_num_grids": 9,
},
}
class HCXProcessor(ProcessorMixin):
attributes = ["image_processor", "tokenizer"]
valid_kwargs = ["chat_template"]
image_processor_class = "AutoImageProcessor"
tokenizer_class = ("GPT2Tokenizer", "GPT2TokenizerFast")
def __init__(self, image_processor=None, tokenizer=None, chat_template=None, **kwargs):
self.image_token = "<|dummy3|>"
self.video_token = "<|_unuse_missing_100270|>"
self.image_token_pattern = re.compile(r"<\|dummy3\|>")
self.video_token_pattern = re.compile(r"<\|_unuse_missing_100270\|>")
self.image_video_token_pattern = re.compile(r"<\|dummy3\|>|<\|_unuse_missing_100270\|>")
self.image_token_id = (
tokenizer.image_token_id
if getattr(tokenizer, "image_token_id", None)
else tokenizer.convert_tokens_to_ids(self.image_token)
)
self.video_token_id = (
tokenizer.video_token_id
if getattr(tokenizer, "video_token_id", None)
else tokenizer.convert_tokens_to_ids(self.video_token)
)
super().__init__(image_processor, tokenizer, chat_template=chat_template)
def apply_chat_template(
self,
conversation: Union[list[dict[str, str]], list[list[dict[str, str]]]],
chat_template: Optional[str] = None,
**kwargs: Unpack[AllKwargsForChatTemplate],
) -> str:
"""
Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input
conversations to turn them into a single tokenizable string.
The input is expected to be in the following format, where each message content is a list consisting of text and
optionally image or video inputs. One can also provide an image, video, URL or local path which will be used to form
`pixel_values` when `return_dict=True`. If not provided, one will get only the formatted text, optionally tokenized text.
conversation = [
{
"role": "user",
"content": [
{"type": "image", "image": "https://www.ilankelman.org/stopsigns/australia.jpg"},
{"type": "text", "text": "Please describe this image in detail."},
],
},
]
Args:
conversation (`Union[List[Dict, [str, str]], List[List[Dict[str, str]]]]`):
The conversation to format.
chat_template (`Optional[str]`, *optional*):
The Jinja template to use for formatting the conversation. If not provided, the tokenizer's
chat template is used.
"""
if chat_template is None:
if isinstance(self.chat_template, dict) and "default" in self.chat_template:
chat_template = self.chat_template["default"]
elif isinstance(self.chat_template, dict):
raise ValueError(
'The processor has multiple chat templates but none of them are named "default". You need to specify'
" which one to use by passing the `chat_template` argument. Available templates are: "
f"{', '.join(self.chat_template.keys())}"
)
elif self.chat_template is not None:
chat_template = self.chat_template
else:
raise ValueError(
"Cannot use apply_chat_template because this processor does not have a chat template."
)
else:
if isinstance(self.chat_template, dict) and chat_template in self.chat_template:
# It's the name of a template, not a full template string
chat_template = self.chat_template[chat_template]
else:
# It's a template string, render it directly
chat_template = chat_template
if kwargs.get("continue_final_message", False):
if kwargs.get("add_generation_prompt", False):
raise ValueError(
"continue_final_message and add_generation_prompt are not compatible. Use continue_final_message when you want the model to continue the final message, and add_generation_prompt when you want to add a header that will prompt it to start a new assistant message instead."
)
if kwargs.get("return_assistant_tokens_mask", False):
raise ValueError("continue_final_message is not compatible with return_assistant_tokens_mask.")
# Fill sets of kwargs that should be used by different parts of template
processed_kwargs = {
"mm_load_kwargs": {},
"template_kwargs": {},
}
for kwarg_type in processed_kwargs:
for key in AllKwargsForChatTemplate.__annotations__[kwarg_type].__annotations__.keys():
kwarg_type_defaults = AllKwargsForChatTemplate.__annotations__[kwarg_type]
default_value = getattr(kwarg_type_defaults, key, None)
value = kwargs.pop(key, default_value)
if value is not None and not isinstance(value, dict):
processed_kwargs[kwarg_type][key] = value
# Pass unprocessed custom kwargs
processed_kwargs["template_kwargs"].update(kwargs)
if isinstance(conversation, (list, tuple)) and (
isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "content")
):
is_batched = True
conversations = conversation
else:
is_batched = False
conversations = [conversation]
tokenize = processed_kwargs["template_kwargs"].pop("tokenize", False)
return_dict = processed_kwargs["template_kwargs"].pop("return_dict", False)
mm_load_kwargs = processed_kwargs["mm_load_kwargs"]
if tokenize:
batch_images, batch_videos = [], []
batch_audios = []
batch_video_metadata = []
for conversation in conversations:
images, videos = [], []
video_metadata = []
for message in conversation:
visuals = [content for content in message["content"] if content["type"] in ["image", "video"]]
audio_fnames = [
content[key]
for content in message["content"]
for key in ["audio", "url", "path"]
if key in content and content["type"] == "audio"
]
image_fnames = [
vision_info[key]
for vision_info in visuals
for key in ["image", "url", "path", "base64"]
if key in vision_info and vision_info["type"] == "image"
]
video_fnames = [
vision_info[key]
for vision_info in visuals
for key in ["video", "url", "path"]
if key in vision_info and vision_info["type"] == "video"
]
for fname in image_fnames:
images.append(load_image(fname))
# Audio models do not accept nested list of audios (yet!) so we construct a flat input audio list
if not mm_load_kwargs["load_audio_from_video"]:
for fname in audio_fnames:
batch_audios.append(load_audio(fname, sampling_rate=mm_load_kwargs["sampling_rate"]))
else:
for fname in video_fnames:
batch_audios.append(load_audio(fname, sampling_rate=mm_load_kwargs["sampling_rate"]))
for fname in video_fnames:
if isinstance(fname, (list, tuple)) and isinstance(fname[0], str):
video = [np.array(load_image(image_fname)) for image_fname in fname]
# create a 4D video because `load_video` always returns a 4D array
video = np.stack(video)
metadata = None
logger.warning(
"When loading the video from list of images, we cannot infer metadata such as `fps` or `duration`. "
"If your model uses this metadata during processing, please load the whole video and let the model sample frames instead."
)
else:
# TODO: raushan, should be `self.video_processor.load_video_for_model` when API is added
video, metadata = self._load_video_for_model(
fname,
num_frames=mm_load_kwargs.get("num_frames", None),
fps=mm_load_kwargs.get("video_fps", None),
backend=mm_load_kwargs["video_load_backend"],
**kwargs,
)
videos.append(video)
video_metadata.append(metadata)
# Currently all processors can accept nested list of batches, but not flat list of visuals
# So we'll make a batched list of images and let the processor handle it
if images:
batch_images.append(images)
if videos:
batch_videos.append(videos)
batch_video_metadata.append(video_metadata)
# Process conversation with video/image information if needed. Then convert into a prompt using Jinja template
conversations = self._process_messages_for_chat_template(
conversations,
batch_images=batch_images,
batch_videos=batch_videos,
batch_video_metadata=batch_video_metadata,
**processed_kwargs["mm_load_kwargs"],
)
prompt, generation_indices = render_jinja_template(
conversations=conversations,
chat_template=chat_template,
**processed_kwargs["template_kwargs"], # different flags such as `return_assistant_mask`
**self.tokenizer.special_tokens_map, # tokenizer special tokens are used by some templates
)
if not is_batched:
prompt = prompt[0]
if tokenize:
# Tokenizer's `apply_chat_template` never adds special tokens when tokenizing
# But processor's `apply_chat_template` didn't have an option to tokenize, so users had to format the prompt
# and pass it to the processor. Users thus never worried about special tokens relying on processor handling
# everything internally. The below line is to keep BC for that and be able to work with model that have
# special tokens in the template (consistent with tokenizers). We dont want to raise warning, it will flood command line
# without actionable solution for users
single_prompt = prompt[0] if is_batched else prompt
if self.tokenizer.bos_token is not None and single_prompt.startswith(self.tokenizer.bos_token):
kwargs["add_special_tokens"] = False
out = self(
text=prompt,
images=batch_images if batch_images else None,
videos=batch_videos if batch_videos else None,
audio=batch_audios if batch_audios else None,
**kwargs,
)
if return_dict:
if processed_kwargs["template_kwargs"].get("return_assistant_tokens_mask", False):
assistant_masks = []
input_ids = out["input_ids"]
for i in range(len(input_ids)):
current_mask = [0] * len(input_ids[i])
for assistant_start_char, assistant_end_char in generation_indices[i]:
start_token = out.char_to_token(i, assistant_start_char)
end_token = out.char_to_token(i, assistant_end_char - 1)
if start_token is None:
# start_token is out of bounds maybe due to truncation.
break
for token_id in range(start_token, end_token + 1 if end_token else len(input_ids[i])):
current_mask[token_id] = 1
assistant_masks.append(current_mask)
out["assistant_masks"] = assistant_masks
out.convert_to_tensors(tensor_type=kwargs.get("return_tensors", None))
# vllm needs vision_query_lengths, but hf model doesn't need it
del out["vision_query_lengths_images"]
del out["vision_query_lengths_videos"]
return out
else:
return out["input_ids"]
def repeat_dummy_tokens(self, input_ids, target_token_id, vision_query_lengths):
input_ids = input_ids.clone().detach()
batch_indices, target_indices = torch.where(input_ids == target_token_id)
batch_size = input_ids.shape[0]
new_input_ids = [[] for _ in range(batch_size)]
start_indices = [0 for _ in range(batch_size)]
counter = [0 for _ in range(batch_size)]
for batch_idx, target_idx in zip(batch_indices, target_indices):
start_idx = start_indices[batch_idx]
new_input_ids[batch_idx].append(input_ids[batch_idx][start_idx:target_idx])
query_length = vision_query_lengths[batch_idx][counter[batch_idx]]
new_input_ids[batch_idx].append(input_ids[batch_idx][target_idx].repeat(query_length))
start_indices[batch_idx] = target_idx + 1
counter[batch_idx] += 1
for batch_idx in range(batch_size):
start_idx = start_indices[batch_idx]
new_input_ids[batch_idx].append(input_ids[batch_idx][start_idx:]) # append remaining tokens
new_input_ids[batch_idx] = torch.cat(new_input_ids[batch_idx], dim=0)
new_input_ids = torch.stack(new_input_ids)
return new_input_ids
def _load_video_for_model(
self,
video: str,
num_frames: Optional[int] = None,
fps: Optional[int] = None,
backend: str = "opencv",
**kwargs: Unpack[HCXProcessorKwargs],
) -> List[ImageInput]:
"""
Overrided function.
Loads `video` to a List[PIL.Image] (llava style)
Args:
video (`str`):
The video to convert to the numpy array format. Can be a link to video or local path.
num_frames (`int`, *optional*):
Number of frames to sample uniformly. If not passed, the whole video is loaded.
fps (`int`, *optional*):
Number of frames to sample per second. Should be passed only when `num_frames=None`.
If not specified and `num_frames==None`, all frames are sampled.
backend (`str`, *optional*, defaults to `"opencv"`):
The backend to use when loading the video. Can be any of ["decord", "pyav", "opencv", "torchvision"]. Defaults to "opencv".
Returns:
Tuple[`np.array`, Dict]: A tuple containing:
- List[PIL.Image] of frames in RGB.
- Metadata dictionary.
"""
output_kwargs = self._merge_kwargs(
HCXProcessorKwargs,
tokenizer_init_kwargs=self.tokenizer.init_kwargs,
**kwargs,
)
logger.warning_once(f"num_frames control via argument is not supported yet. Ignored num_frames: {num_frames}.")
logger.warning_once(f"fps control via argument is not supported yet. Ignored fps: {fps}.")
logger.warning_once(f"backend control via argument is not supported yet. Ignored backend: {backend}.")
# video_loaded, video_metadata = load_video(
# video, backend="decord", num_frames=32
# )
# frame_interval = int(video_metadata.total_num_frames / 32)
# time_interval = frame_interval / video_metadata.fps
# video_metadata.time_interval = time_interval
def _hcx_sample_indices_fn(metadata: VideoMetadata, num_frames=None, fps=None, **kwargs):
max_num_grids = output_kwargs["videos_kwargs"]["max_num_grids"]
max_image_cnt = output_kwargs["videos_kwargs"]["max_image_cnt"]
frame_indices, time_interval = extract_frame_indices(
metadata.duration,
metadata.total_num_frames,
metadata.fps,
max_num_grids,
max_image_cnt,
default_interval=0.4,
)
metadata.time_interval = time_interval
return np.array(frame_indices)
video_loaded, video_metadata = None, None
for backend in ["decord", "pyav", "opencv", "torchvision"]:
try:
video_loaded, video_metadata = load_video(
video, sample_indices_fn=_hcx_sample_indices_fn, backend=backend
)
break
except Exception as e:
logger.error(f"Error loading video with {backend} backend: {e}")
continue
assert video_loaded is not None, "Failed to load video with any backend"
return video_loaded, video_metadata
def _process_messages_for_chat_template(
self,
conversation: List[List[Dict[str, str]]],
batch_images: List[List[ImageInput]],
batch_videos: List[List[VideoInput]],
batch_video_metadata: List[List[Dict[str, any]]],
**mm_load_kwargs: Unpack[ChatTemplateLoadKwargs],
):
"""
Overrided function.
Used within `apply_chat_template` when a model has a special way to process conversation history. For example,
video models might want to specify in the prompt the duration of video or which frame indices at which timestamps
were sampled. This information cannot be accessed before the video is loaded.
For most models it is a no-op, and must be overridden by model processors which require special processing.
Args:
conversation (`List[Dict, str, str]`):
The conversation to process. Always comes in batched format.
batch_images (`List[List[ImageInput]]`):
Batch of images that were loaded from url/path defined in the conversation. The images
are ordered in the same way as in the conversation. Comes in nested list format, one list of `PIL` images
per batch.
batch_videos (`List[List[ImageInput]]`):
Batch of videos that were loaded from url/path defined in the conversation. The videos
are ordered in the same way as in the conversation. Comes in nested list format, one list of `PIL.Image`
per batch.
batch_video_metadata (`List[List[Dict[[str, any]]]]`):
Batch of metadata returned from loading videos. That includes video fps, duration and total number of framer in original video.
Metadata are ordered in the same way as `batch_videos`. Comes in nested list format, one list of `Dict`
per batch.
"""
is_video_in_conversation = False
for batch_idx, messages in enumerate(conversation):
is_video_in_messages = False
is_image_in_messages = False
for message in messages:
for content in message["content"]:
if content["type"] == "video":
is_video_in_messages = True
elif content["type"] == "image":
is_image_in_messages = True
if not is_video_in_messages:
batch_videos.insert(batch_idx, [])
batch_video_metadata.insert(batch_idx, [])
if not is_image_in_messages:
batch_images.insert(batch_idx, [])
is_video_in_conversation = is_video_in_conversation or is_video_in_messages
if not is_video_in_conversation:
return conversation
# conversation processing
new_conversation = []
for batch_idx, messages in enumerate(conversation):
video_counter = 0
new_messages = []
for message in messages:
new_message = {
"role": message["role"],
"content": [],
}
for content in message["content"]:
if content["type"] == "video":
video = batch_videos[batch_idx][video_counter]
video_meta = batch_video_metadata[batch_idx][video_counter]
time_stamps = calc_timestamp_video_grids(video, video_meta.time_interval, max_grid_shape=(3, 3))
video_counter += 1
if "filename" in content:
filename = content["filename"]
else:
filename = content["video"].split("/")[-1]
if len(filename) > 50:
filename = f"{uuid.uuid4().hex}.mp4"
basename, ext = os.path.splitext(filename)
if ext == "":
ext = ".mp4"
for frame_idx, time_stamp in enumerate(time_stamps):
if frame_idx == len(video) - 1:
# final_grid
new_content = {
"filename": f"{basename}-{frame_idx}{ext}",
"video": content["video"],
"type": "video",
"video_time_stamp": time_stamp,
"lens_keywords": content["lens_keywords"],
"lens_local_keywords": content["lens_local_keywords"],
"speech_to_text": content["speech_to_text"],
"is_final_grid": True,
}
new_message["content"].append(new_content)
else:
new_content = {
"filename": f"{basename}-{frame_idx}{ext}",
"video": content["video"],
"type": "video",
"video_time_stamp": time_stamp,
}
new_message["content"].append(new_content)
else:
new_message["content"].append(copy.deepcopy(content))
new_messages.append(new_message)
new_conversation.append(new_messages)
return new_conversation
def __call__(
self,
text: TextInput = None,
images: List[List[ImageInput]] = None,
videos: List[List[VideoInput]] = None,
audio: AudioInput = None,
**kwargs: Unpack[HCXProcessorKwargs],
):
output_kwargs = self._merge_kwargs(
HCXProcessorKwargs,
tokenizer_init_kwargs=self.tokenizer.init_kwargs,
**kwargs,
)
# prepare model inputs
mm_inputs = {
"pixel_values_images": [],
"image_sizes_images": [],
"vision_query_lengths_images": [],
"pixel_values_videos": [],
# "image_sizes_videos": [],
"vision_query_lengths_videos": [],
}
calc_non_vision_query_lengths = output_kwargs["text_kwargs"].pop("calc_non_vision_query_lengths")
if calc_non_vision_query_lengths:
mm_inputs["non_vision_query_lengths"] = []
# video processing
if videos is not None:
vit_input_size = self.image_processor.crop_size["width"]
video_kwargs = copy.deepcopy(output_kwargs["videos_kwargs"])
for videos_in_single_conversation in videos:
pixel_values_videos = []
vision_query_lengths_videos = []
for video_frames in videos_in_single_conversation:
if len(video_frames) == 0:
mm_inputs["pixel_values_videos"].append([])
mm_inputs["vision_query_lengths_videos"].append([])
continue
video_frames_combined = combine_frames_into_images(
video_frames, max_grid_shape=(3, 3), vit_input_size=vit_input_size
)
video_kwargs["is_video"] = True
video_kwargs["return_tensors"] = None
frames_processed = self.image_processor(images=video_frames_combined, **video_kwargs)
sizes = [(size["width"], size["height"]) for size in frames_processed["image_sizes"]]
pixel_values_videos.extend(frames_processed["pixel_values"])
vision_query_lengths_videos.extend(frames_processed["vision_query_lengths"])
mm_inputs["pixel_values_videos"].append(pixel_values_videos)
mm_inputs["vision_query_lengths_videos"].append(vision_query_lengths_videos)
# image processing
if images is not None:
image_kwargs = copy.deepcopy(output_kwargs["images_kwargs"])
image_kwargs["is_video"] = False
image_kwargs["return_tensors"] = None
for images_in_single_conversation in images:
if isinstance(images_in_single_conversation, PIL.Image.Image): # single item to batch
images_in_single_conversation = [images_in_single_conversation, ]
if len(images_in_single_conversation) == 0:
mm_inputs["pixel_values_images"].append([])
mm_inputs["image_sizes_images"].append([])
mm_inputs["vision_query_lengths_images"].append([])
continue
images_processed = self.image_processor(images=images_in_single_conversation, **image_kwargs)
sizes = [(size["width"], size["height"]) for size in images_processed["image_sizes"]]
mm_inputs["pixel_values_images"].append(images_processed["pixel_values"])
mm_inputs["image_sizes_images"].append(sizes)
mm_inputs["vision_query_lengths_images"].append(images_processed["vision_query_lengths"])
# text processing
def _create_replacer(_target_token, _replacements):
_iterator = iter(_replacements)
def _replacer(match_obj):
# return self.image_token
num_query_tokens = next(_iterator)
return "".join([_target_token for _ in range(num_query_tokens)])
return _replacer
text_inputs = {}
if text is not None:
if not isinstance(text, list):
text = [text]
if images is not None:
new_texts = []
for batch_idx, text_in_single_conversation in enumerate(text):
new_text = self.image_token_pattern.sub(
_create_replacer(self.image_token, mm_inputs["vision_query_lengths_images"][batch_idx]),
text_in_single_conversation,
)
new_texts.append(new_text)
text = new_texts
if videos is not None:
new_texts = []
for batch_idx, text_in_single_conversation in enumerate(text):
new_text = self.video_token_pattern.sub(
_create_replacer(self.video_token, mm_inputs["vision_query_lengths_videos"][batch_idx]),
text_in_single_conversation,
)
new_texts.append(new_text)
text = new_texts
text_inputs = self.tokenizer(text, **output_kwargs["text_kwargs"])
# audio processing
if audio is not None:
raise NotImplementedError("Audio processing is not supported yet.")
return HCXBatchFeature(data={**text_inputs, **mm_inputs})
def decode(self, *args, **kwargs):
"""
This method forwards all its arguments to Siglip2Tokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to
the docstring of this method for more information.
"""
return self.tokenizer.decode(*args, **kwargs)
def batch_decode(self, *args, **kwargs):
"""
This method forwards all its arguments to Siglip2Tokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
refer to the docstring of this method for more information.
"""
return self.tokenizer.batch_decode(*args, **kwargs)
def post_process_image_text_to_text(
self, generated_outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False, **kwargs
):
"""
Post-process the output of the model to decode the text.
Args:
generated_outputs (`torch.Tensor` or `np.ndarray`):
The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
or `(sequence_length,)`.
skip_special_tokens (`bool`, *optional*, defaults to `True`):
Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
Clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method.
**kwargs:
Additional arguments to be passed to the tokenizer's `batch_decode method`.
Returns:
`List[str]`: The decoded text.
"""
return self.tokenizer.batch_decode(
generated_outputs,
skip_special_tokens=skip_special_tokens,
clean_up_tokenization_spaces=clean_up_tokenization_spaces,
**kwargs,
)
@property
def model_input_names(self):
tokenizer_input_names = self.tokenizer.model_input_names
image_processor_input_names = self.image_processor.model_input_names
names_from_processor = list(dict.fromkeys(tokenizer_input_names + image_processor_input_names))
return names_from_processor + []
def extract_frame_indices(play_time, total_frames, fps, max_num_grids, max_image_cnt, default_interval=0.4):
"""
Extracts specific frame indices from a video based on duration, frame count, and sampling strategy.
The function determines which frames to extract given the video duration (`play_time`),
total frame count, and frame rate. It samples frames at regular intervals (default: 0.4s),
but if the number of frames exceeds the limit defined by `max_num_grids * max_image_cnt`,
it performs uniform sampling to stay within that limit.
Args:
play_time (float): Total play time of the video in seconds.
total_frames (int): Total number of frames in the video.
fps (float): Frames per second of the video.
max_num_grids (int): Maximum number of grids to display.
max_image_cnt (int): Maximum number of images per grid.
default_interval (float, optional): Interval in seconds between frame samples. Defaults to 0.4.
Returns:
Tuple:
frame_indices (List[int]): A list of selected frame indices.
time_interval (float): Time interval between selected frames (in seconds).
"""
# Calculate how many frames to extract with the default interval
default_frame_count = int(play_time / default_interval)
# Maximum frames allowed based on max_num_grids and max_image_cnt
max_frames_allowed = max_num_grids * max_image_cnt
# Determine whether we can use the default interval or need uniform sampling
if default_frame_count <= max_frames_allowed:
# Default interval is sufficient, extract frames every 0.4 seconds
frame_interval = int(total_frames / default_frame_count)
else:
# Use uniform sampling to fit within max_frames_allowed
frame_interval = int(total_frames / max_frames_allowed)
# Extract frame indices at the calculated interval
selected_indices = list(range(0, total_frames, frame_interval))
time_interval = frame_interval / fps
# Ensure the number of selected indices does not exceed max_frames_allowed
return selected_indices[:max_frames_allowed], time_interval
def calc_timestamp_video_grids(frames, time_interval, max_grid_shape=(3, 3)):
"""
Calculates the time range labels for each grid in a video.
Args:
frames (List[PIL.Image.Image]): A list of frames extracted from a video.
time_interval (float): Time interval (in seconds) between consecutive frames.
max_grid_shape (Tuple[int, int], optional): The maximum grid shape as (rows, cols). Defaults to (3, 3).
vit_input_size (int, optional): The target size (height and width) for the Vision Transformer input. Defaults to 378.
Returns:
Tuple:
image_time_stamps (List[str]): A list of time span labels for each combined image,
e.g., ["0.00s~1.50s", "1.50s~3.00s", ...].
"""
max_num_grids = max_grid_shape[0] * max_grid_shape[1]
# assert (
# max_grid_shape[1] == 1
# ), f"For video processing, decided to concatenate frames horizontally into a wide image."
# Calculate the number of canvases needed.
num_frames = len(frames)
num_canvases = num_frames // max_num_grids
leftover_frames = num_frames % max_num_grids
time_stamp = 0 # second
image_time_stamps = []
for canvas_idx in range(num_canvases):
# Determine the frames to fill in the current canvas.
start_idx = canvas_idx * max_num_grids
end_idx = min(start_idx + max_num_grids, num_frames)
# Append the current canvas to the result list.
frame_cnt = end_idx - start_idx
image_time_stamps.append(f"{time_stamp:.2f}s~{time_stamp + frame_cnt * time_interval:.2f}s")
time_stamp += frame_cnt * time_interval
if leftover_frames > 0:
# Add the current canvas to the list of combined images.
frame_cnt = leftover_frames
image_time_stamps.append(f"{time_stamp:.2f}s~{time_stamp + frame_cnt * time_interval:.2f}s")
time_stamp += frame_cnt * time_interval
return image_time_stamps
def combine_frames_into_images(frames, max_grid_shape=(3, 3), vit_input_size=378):
"""
Combines a sequence of video frames into grid-based images and generates corresponding time range labels.
Frames are grouped and arranged into a grid (e.g., 3x3) such that each combined image contains up to
`max_grid_shape[0] * max_grid_shape[1]` frames. Each combined image is resized to the given ViT input size.
Args:
frames (NDArray): (num_frames, H, W, C) shape. A list of frames extracted from a video.
time_interval (float): Time interval (in seconds) between consecutive frames.
max_grid_shape (Tuple[int, int], optional): The maximum grid shape as (rows, cols). Defaults to (3, 3).
vit_input_size (int, optional): The target size (height and width) for the Vision Transformer input. Defaults to 378.
Returns:
Tuple:
image_list (List[PIL.Image.Image]): A list of grid-combined images.
"""
max_num_grids = max_grid_shape[0] * max_grid_shape[1]
# assert (
# max_grid_shape[1] == 1
# ), f"For video processing, decided to concatenate frames horizontally into a wide image."
# List to store the resulting combined images.
image_list = []
# Calculate the number of canvases needed.
num_frames = len(frames)
num_canvases = num_frames // max_num_grids
leftover_frames = num_frames % max_num_grids
# change frames (4d numpy tensor) to List[PIL.Image.Image]
frames = [Image.fromarray(frame) for frame in frames]
for canvas_idx in range(num_canvases):
# Initialize the current canvas.
combined_image = Image.new(
"RGB", (vit_input_size * max_grid_shape[0], vit_input_size * max_grid_shape[1]), color=(0, 0, 0)
)
# Determine the frames to fill in the current canvas.
start_idx = canvas_idx * max_num_grids
end_idx = min(start_idx + max_num_grids, num_frames)
for idx in range(start_idx, end_idx):
img = frames[idx]
# Resize each frame to a square shape.
img_resized = img.resize((vit_input_size, vit_input_size))
# Calculate the (row, column) position to place the frame within the grid layout.
local_idx = idx - start_idx
x_offset = (local_idx % max_grid_shape[0]) * vit_input_size
y_offset = (local_idx // max_grid_shape[0]) * vit_input_size
# Calculate the position to place the frame in the grid.
combined_image.paste(img_resized, (x_offset, y_offset))
# Append the current canvas to the result list.
image_list.append(combined_image)
if leftover_frames > 0:
# canvas_idx might be undefined; default to 0 if not previously assigned to avoid "referenced before assignment" error.
canvas_idx = num_canvases
# Add the remaining frames to the final canvas.
# combined_image = Image.new("RGB", (vit_input_size * leftover_frames, vit_input_size * 1), color=(0, 0, 0)) # hsk
combined_image = Image.new(
"RGB", (vit_input_size * max_grid_shape[0], vit_input_size * max_grid_shape[1]), color=(0, 0, 0)
)
for idx in range(leftover_frames):
img = frames[num_canvases * max_num_grids + idx]
# Resize the frame to a square (equal width and height).
img_resized = img.resize((vit_input_size, vit_input_size))
# Calculate the (row, column) position to place the frame within the grid layout.
# x_offset = (idx % leftover_frames) * vit_input_size # hsk
# y_offset = (idx // leftover_frames) * vit_input_size # hsk
x_offset = (idx % max_grid_shape[0]) * vit_input_size
y_offset = (idx // max_grid_shape[0]) * vit_input_size
# Calculate the position to place the frame within the grid layout.
combined_image.paste(img_resized, (x_offset, y_offset))
# Add the current canvas to the list of combined images.
image_list.append(combined_image)
return image_list

6
processor_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"auto_map": {
"AutoProcessor": "processing_hyperclovax.HCXProcessor"
},
"processor_class": "HCXProcessor"
}

86
special_tokens_map.json Normal file
View File

@@ -0,0 +1,86 @@
{
"additional_special_tokens": [
"<|endoftext|>",
"<|fim_prefix|>",
"<|fim_middle|>",
"<|fim_suffix|>",
"<|endofprompt|>",
"<|_unuse_missing_100256|>",
"<|_unuse_missing_100261|>",
"<|_unuse_missing_100262|>",
"<|_unuse_missing_100263|>",
"<|_unuse_missing_100264|>",
"<|_unuse_missing_100265|>",
"<|_unuse_missing_100266|>",
"<|_unuse_missing_100267|>",
"<|_unuse_missing_100268|>",
"<|_unuse_missing_100269|>",
"<|_unuse_missing_100270|>",
"<|dummy3|>",
"<|im_start|>",
"<|im_end|>",
"<|stop|>",
"<|endofturn|>",
"<repo_name>",
"<file_sep>",
"<issue_start>",
"<issue_comment>",
"<issue_closed>",
"<jupyter_start>",
"<jupyter_text>",
"<jupyter_code>",
"<jupyter_output>",
"<jupyter_script>",
"<empty_output>",
"<code_to_intermediate>",
"<intermediate_to_code>",
"<pr>",
"<pr_status>",
"<pr_is_merged>",
"<pr_base>",
"<pr_file>",
"<pr_base_code>",
"<pr_diff>",
"<pr_diff_hunk>",
"<pr_comment>",
"<pr_event_id>",
"<pr_review>",
"<pr_review_state>",
"<pr_review_comment>",
"<pr_in_reply_to_review_id>",
"<pr_in_reply_to_comment_id>",
"<pr_diff_hunk_comment_line>",
"<NAME>",
"<EMAIL>",
"<KEY>",
"<PASSWORD>"
],
"bos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|endofturn|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:609b626b9716c7e16290f368c09efaa66db28cecde98e252f1d129f826ebb340
size 8029664

507
tokenizer_config.json Normal file
View File

@@ -0,0 +1,507 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"100256": {
"content": "<|_unuse_missing_100256|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100257": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100258": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100259": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100260": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100261": {
"content": "<|_unuse_missing_100261|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100262": {
"content": "<|_unuse_missing_100262|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100263": {
"content": "<|_unuse_missing_100263|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100264": {
"content": "<|_unuse_missing_100264|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100265": {
"content": "<|_unuse_missing_100265|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100266": {
"content": "<|_unuse_missing_100266|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100267": {
"content": "<|_unuse_missing_100267|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100268": {
"content": "<|_unuse_missing_100268|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100269": {
"content": "<|_unuse_missing_100269|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100270": {
"content": "<|_unuse_missing_100270|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100271": {
"content": "<|dummy3|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100272": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100273": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100274": {
"content": "<|stop|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100275": {
"content": "<|endofturn|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"100276": {
"content": "<|endofprompt|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110491": {
"content": "<repo_name>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110492": {
"content": "<file_sep>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110493": {
"content": "<issue_start>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110494": {
"content": "<issue_comment>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110495": {
"content": "<issue_closed>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110496": {
"content": "<jupyter_start>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110497": {
"content": "<jupyter_text>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110498": {
"content": "<jupyter_code>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110499": {
"content": "<jupyter_output>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110500": {
"content": "<jupyter_script>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110501": {
"content": "<empty_output>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110502": {
"content": "<code_to_intermediate>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110503": {
"content": "<intermediate_to_code>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110504": {
"content": "<pr>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110505": {
"content": "<pr_status>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110506": {
"content": "<pr_is_merged>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110507": {
"content": "<pr_base>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110508": {
"content": "<pr_file>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110509": {
"content": "<pr_base_code>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110510": {
"content": "<pr_diff>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110511": {
"content": "<pr_diff_hunk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110512": {
"content": "<pr_comment>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110513": {
"content": "<pr_event_id>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110514": {
"content": "<pr_review>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110515": {
"content": "<pr_review_state>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110516": {
"content": "<pr_review_comment>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110517": {
"content": "<pr_in_reply_to_review_id>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110518": {
"content": "<pr_in_reply_to_comment_id>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110519": {
"content": "<pr_diff_hunk_comment_line>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110520": {
"content": "<NAME>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110521": {
"content": "<EMAIL>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110522": {
"content": "<KEY>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"110523": {
"content": "<PASSWORD>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|endoftext|>",
"<|fim_prefix|>",
"<|fim_middle|>",
"<|fim_suffix|>",
"<|endofprompt|>",
"<|_unuse_missing_100256|>",
"<|_unuse_missing_100261|>",
"<|_unuse_missing_100262|>",
"<|_unuse_missing_100263|>",
"<|_unuse_missing_100264|>",
"<|_unuse_missing_100265|>",
"<|_unuse_missing_100266|>",
"<|_unuse_missing_100267|>",
"<|_unuse_missing_100268|>",
"<|_unuse_missing_100269|>",
"<|_unuse_missing_100270|>",
"<|dummy3|>",
"<|im_start|>",
"<|im_end|>",
"<|stop|>",
"<|endofturn|>",
"<repo_name>",
"<file_sep>",
"<issue_start>",
"<issue_comment>",
"<issue_closed>",
"<jupyter_start>",
"<jupyter_text>",
"<jupyter_code>",
"<jupyter_output>",
"<jupyter_script>",
"<empty_output>",
"<code_to_intermediate>",
"<intermediate_to_code>",
"<pr>",
"<pr_status>",
"<pr_is_merged>",
"<pr_base>",
"<pr_file>",
"<pr_base_code>",
"<pr_diff>",
"<pr_diff_hunk>",
"<pr_comment>",
"<pr_event_id>",
"<pr_review>",
"<pr_review_state>",
"<pr_review_comment>",
"<pr_in_reply_to_review_id>",
"<pr_in_reply_to_comment_id>",
"<pr_diff_hunk_comment_line>",
"<NAME>",
"<EMAIL>",
"<KEY>",
"<PASSWORD>"
],
"auto_map": {
"AutoProcessor": "processing_hyperclovax.HCXProcessor"
},
"bos_token": "<|endoftext|>",
"clean_up_tokenization_spaces": true,
"eos_token": "<|endofturn|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<|endoftext|>",
"processor_class": "HCXProcessor",
"tokenizer_class": "GPT2Tokenizer",
"unk_token": "<|endoftext|>"
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long