62 lines
3.9 KiB
Markdown
62 lines
3.9 KiB
Markdown
|
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
|
|||
|
|
|
|||
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|||
|
|
the License. You may obtain a copy of the License at
|
|||
|
|
|
|||
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|||
|
|
|
|||
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|||
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|||
|
|
specific language governing permissions and limitations under the License.
|
|||
|
|
|
|||
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|||
|
|
rendered properly in your Markdown viewer.
|
|||
|
|
|
|||
|
|
-->
|
|||
|
|
*This model was released on 2022-06-06 and added to Hugging Face Transformers on 2023-06-02.*
|
|||
|
|
|
|||
|
|
# MobileViTV2
|
|||
|
|
|
|||
|
|
<div class="flex flex-wrap space-x-1">
|
|||
|
|
<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
|
|||
|
|
</div>
|
|||
|
|
|
|||
|
|
## Overview
|
|||
|
|
|
|||
|
|
The MobileViTV2 model was proposed in [Separable Self-attention for Mobile Vision Transformers](https://huggingface.co/papers/2206.02680) by Sachin Mehta and Mohammad Rastegari.
|
|||
|
|
|
|||
|
|
MobileViTV2 is the second version of MobileViT, constructed by replacing the multi-headed self-attention in MobileViT with separable self-attention.
|
|||
|
|
|
|||
|
|
The abstract from the paper is the following:
|
|||
|
|
|
|||
|
|
*Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires O(k2) time complexity with respect to the number of tokens (or patches) k. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. O(k). A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTV2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTV2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running 3.2× faster on a mobile device.*
|
|||
|
|
|
|||
|
|
This model was contributed by [shehan97](https://huggingface.co/shehan97).
|
|||
|
|
The original code can be found [here](https://github.com/apple/ml-cvnets).
|
|||
|
|
|
|||
|
|
## Usage tips
|
|||
|
|
|
|||
|
|
- MobileViTV2 is more like a CNN than a Transformer model. It does not work on sequence data but on batches of images. Unlike ViT, there are no embeddings. The backbone model outputs a feature map.
|
|||
|
|
- One can use [`MobileViTImageProcessor`] to prepare images for the model. Note that if you do your own preprocessing, the pretrained checkpoints expect images to be in BGR pixel order (not RGB).
|
|||
|
|
- The available image classification checkpoints are pre-trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k) (also referred to as ILSVRC 2012, a collection of 1.3 million images and 1,000 classes).
|
|||
|
|
- The segmentation model uses a [DeepLabV3](https://huggingface.co/papers/1706.05587) head. The available semantic segmentation checkpoints are pre-trained on [PASCAL VOC](http://host.robots.ox.ac.uk/pascal/VOC/).
|
|||
|
|
|
|||
|
|
## MobileViTV2Config
|
|||
|
|
|
|||
|
|
[[autodoc]] MobileViTV2Config
|
|||
|
|
|
|||
|
|
## MobileViTV2Model
|
|||
|
|
|
|||
|
|
[[autodoc]] MobileViTV2Model
|
|||
|
|
- forward
|
|||
|
|
|
|||
|
|
## MobileViTV2ForImageClassification
|
|||
|
|
|
|||
|
|
[[autodoc]] MobileViTV2ForImageClassification
|
|||
|
|
- forward
|
|||
|
|
|
|||
|
|
## MobileViTV2ForSemanticSegmentation
|
|||
|
|
|
|||
|
|
[[autodoc]] MobileViTV2ForSemanticSegmentation
|
|||
|
|
- forward
|