初始化项目，由ModelHub XC社区提供模型

Model: Chat-UniVi/MoH-LLaMA3-8B
Source: Original Platform

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,84 @@
+---
+license: apache-2.0
+pipeline_tag: text-generation
+---
+# MoH: Multi-Head Attention as Mixture-of-Head Attention
+
+**Paper or resources for more information:**
+[[Paper](https://huggingface.co/papers/2410.11842)] [[Code](https://github.com/SkyworkAI/MoH)]
+
+## ⚡ Overview
+We propose Mixture-of-Head attention (MoH), a new architecture that treats attention heads as experts in the Mixture-of-Experts (MoE) mechanism. MoH has two significant advantages:
+* First, MoH enables each token to select the appropriate attention heads, enhancing inference efficiency without compromising accuracy or increasing the number of parameters. 
+* Second, MoH replaces the standard summation in multi-head attention with a weighted summation, introducing flexibility to the attention mechanism and unlocking extra performance potential.
+
+
+
+## 😮 Highlights
+### 💡 General Framework
+We evaluate our proposed MoH across various popular model frameworks, including Vision Transformers (ViT) for image classification, Diffusion models with Transformers (DiT) for class-conditional image generation, and Large Language Models (LLMs) for language tasks.
+
+<div align=center>
+
+|                   Code                    |                                                                                                                         HuggingFace Model                                                                                                                         |  
+|:-----------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+|     **[MoH-ViT](https://github.com/SkyworkAI/MoH/tree/main/MoH-ViT)**      | 🤗 [MoH-ViT-B-75](https://huggingface.co/Chat-UniVi/MoH-ViT-B-75), [MoH-ViT-B-50](https://huggingface.co/Chat-UniVi/MoH-ViT-B-50), [MoH-ViT-S-80](https://huggingface.co/Chat-UniVi/MoH-ViT-S-80), [MoH-ViT-S-75](https://huggingface.co/Chat-UniVi/MoH-ViT-S-75) |
+|     **[MoH-DiT](https://github.com/SkyworkAI/MoH/tree/main/MoH-DiT)**      |                                                                                                 😊 [MoH-DiT-90](https://huggingface.co/Chat-UniVi/MoH-DiT-XL-90)                                                                                                  | 
+| **[MoH-LLaMA3-8B](https://github.com/SkyworkAI/MoH/tree/main/MoH-LLaMA3)** |                                                                                                                        😊 [MoH-LLaMA3-8B](https://huggingface.co/Chat-UniVi/MoH-LLaMA3-8B)                                                                                                                         | 
+
+</div>
+
+### 🔥 High Performance
+Extensive experiments on ViT, DiT, and LLMs demonstrate that MoH outperforms multi-head attention by using only **50%~90%** of the attention heads.
+
+### 🤗 Support Continue-Tuning Starting from the Multi-Head Attention Models
+we demonstrate that pre-trained multi-head attention models, such as LLaMA3-8B, can be further continue-tuned into our MoH models. Notably, MoH-LLaMA3-8B achieves an average accuracy of 64.0% across 14 benchmarks, outperforming LLaMA3-8B by 2.4% by utilizing only 75% of the attention heads.
+
+
+The MoH model quickly recovers to over **95%** of the performance of the original model within a training budget of 10B tokens. Then, the performance gradually improves with the increase of the training tokens.
+
+## 🤖 API for Model Inference
+If you want to load the model from the model hub on Hugging Face or on local, you can use the following code snippets.
+
+### Base Model Inference
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+question = "Hello!"
+
+model = AutoModelForCausalLM.from_pretrained("Chat-UniVi/MoH-LLaMA3-8B", trust_remote_code=True, device_map='auto')
+tokenizer = AutoTokenizer.from_pretrained("Chat-UniVi/MoH-LLaMA3-8B", trust_remote_code=True)
+
+inputs = tokenizer(question, return_tensors='pt').to(model.device)
+response = model.generate(inputs.input_ids, max_length=128)
+print(tokenizer.decode(response.cpu()[0], skip_special_tokens=True))
+```
+
+### Chat Model Inference
+Coming soon...
+
+
+## 🗝️ Training & Validating
+* The training code is built on [Skywork-MoE](https://github.com/SkyworkAI/Skywork-MoE). Unless Skywork-MoE is open source, we can't open source MoH-LLaMA3 alone. We will release the training code after the approval is completed.
+* The evaluation is performed on multiple key benchmarks using the [Eleuther AI Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness).
+
+```python
+# For example, test MoH-LLaMA3-8B on winogrande
+
+CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch \
+--main_process_port 2004 -m lm_eval --model hf \
+--model_args pretrained=Chat-UniVi/MoH-LLaMA3-8B \
+--tasks winogrande \
+--batch_size 1 \
+--output_path Results/winogrande
+```
+
+## ✏️ Citation
+If you find this paper useful, please consider staring 🌟 this repo and citing 📑 our paper:
+```
+@article{jin2024moh,
+  title={MoH: Multi-Head Attention as Mixture-of-Head Attention}, 
+  author={Peng Jin and Bo Zhu and Li Yuan and Shuicheng Yan},
+  year={2024}
+}
+```

config.json

@@ -0,0 +1 @@
+{"architectures": ["LlamaForCausalLM"], "auto_map": {"AutoModelForCausalLM":"modeling_llama.LlamaForCausalLM"},"model_type": "llama", "vocab_size": 160896, "bos_token_id": 1, "eos_token_id": 2, "pad_token_id": 0, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 14336, "max_position_embeddings": 8192, "num_attention_heads": 32, "num_key_value_heads": 8, "num_hidden_layers": 32, "rms_norm_eps": 1e-05, "rotary_percent": 1.0, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "use_cache": true, "transformers_version": "4.33.1", "rope_theta": 500000}

generation_config.json

@@ -0,0 +1 @@
+{"_from_model_config": true, "bos_token_id": 1, "eos_token_id": 2, "pad_token_id": 0, "transformers_version": "4.33.1"}

pytorch_model-00001-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f6c520d24ead67aa24c57640c81472a1527e964e63cef1ecb35a931a6b1a65f
+size 436227233

pytorch_model-00002-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfd00403c118fee4989b6c6363a342b3cd583156455113f6916bf88f4c73d587
+size 436227233

pytorch_model-00003-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ea35cf265e45997ba402d843468529d0bd320825efb1cd1b6ff03809d136170
+size 436227233

pytorch_model-00004-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bc144acb0f9b5bffebd74e86e8eec2157076ae743ef196ce5d74c29aaae0637
+size 436227233

pytorch_model-00005-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:224cc7587af53321c7a1a55c3cd6063082f15ab7b20d8075679b4e4e099205e6
+size 436227233

pytorch_model-00006-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73f11f919712d6f61a175f4fddfe0d01bc218ab56b82e55c5493c48fafce2fb7
+size 436227233

pytorch_model-00007-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc79ebe213b27cda5356ad2bbbe351767ff5b63accd03092a655750d8e6eb95d
+size 436227233

pytorch_model-00008-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:644c3efef1dc986b769f778ece3e6dcd9d6f11b8ba58f2932ed2a57f0e0f721a
+size 436227233

pytorch_model-00009-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3dcad270524a995e5724b8ab72afd29018bae0d4fcc979bb9b8600c852fc6224
+size 436227233

pytorch_model-00010-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05c1262969d4fd631d069249e829e070f087042d3fc7a7e83e3e4a76592d9e15
+size 436227233

pytorch_model-00011-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e08401d0279a24d5f33dc361911ad1f9b04203733d7856f4a07a0a1987b541d
+size 436227233

pytorch_model-00012-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d656d26d54d95c194065a9f3335df04b270f1783a35bbfa26b57353ea95c4f62
+size 436227233

pytorch_model-00013-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7223cd0c3cba7ddead8b5705e608519ac2cb8c0c9e756685f8c1737a93e14d47
+size 436227233

pytorch_model-00014-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96a3ecd0dcd7257db96db7024b3b1da1ecc759ebc9b097adb26088d4d86fc673
+size 436227233

pytorch_model-00015-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fd03376fa4e0dc353db7b2b95cf4238415f7effb4f6fc366dc106f576447c0a
+size 436227233

pytorch_model-00016-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef1b44bbdd5a57c5220d4c4ca8e7085ba00a469c08acab3ae3c4347b25aa32ca
+size 436227233

pytorch_model-00017-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:946b120f4774855b07ac67bd6e8c82f3d5ebc3efc8b0a6f929ed22a7c53ca904
+size 436227233

pytorch_model-00018-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b96a4874f3a5c8810a22fde6b8d3aa7fdb8751c1c88662835b25a22647bd8524
+size 436227233

pytorch_model-00019-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff313fb075c2757dded3cf3c360914e5efb951a7a4670217c4967ab30f8ba3a0
+size 436227233

pytorch_model-00020-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eaa753f91b67175244dac299f4378bda3d8b6f113c4596d09d90a6d7217523af
+size 436227233

pytorch_model-00021-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76853a188210904bc205402cf9f0a33e805c98d041b9078c9f88a7049355e1c5
+size 436227233

pytorch_model-00022-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b49cc526c34d85c98767ed0a28cba5cdd26e245a0e15baa1e6c32dce534bf315
+size 436227233

pytorch_model-00023-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbc742f370749db6b13c721da30684a116dac45a93b8d0e33802be73d25522d6
+size 436227233

pytorch_model-00024-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f8d6ae8387f87fe84dc26e89a44b6d875b1775a9cfb005d955a1ec92efd447a
+size 436227233

pytorch_model-00025-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c161019ed573ed5555afc43ab293d45ebce7486086546b13168b24fddd19933b
+size 436227233

pytorch_model-00026-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d69d2527e9ddfaa8fc51017907653e8c9410fed03b70163f229f8992dab08bb8
+size 436227233

pytorch_model-00027-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0699b69f37e2f62365817e52186ec7b19da2af99a3f639a65d96d7997aa239df
+size 436227233

pytorch_model-00028-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c14ae27463bfa2eac2e59324c53d1532f391b5e23db55c38d6f6fe5694bc1dfb
+size 436227233

pytorch_model-00029-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:287b53bb022823415bc6ce4f0bfb8e83929fba198010ac11cc482443a6eb2f2c
+size 436227233

pytorch_model-00030-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a73b44f28da6eb2651c85e18bbba3160a62b43c4ba46ce362d38e7e022c132ea
+size 436227233

pytorch_model-00031-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dea43ca1eb8a9d01e3b32d13da5fc2ac7d3bb2c0d36b5a2ae8d9147ab67fb5e2
+size 436227233

pytorch_model-00032-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:362e4f07ffaf5ed52ef10e18bc81f0aa2aa77f11809fbbb4e1cf5f8d0d813185
+size 436227233

pytorch_model-00033-of-00033.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:592e2373fcf7823bd6354a0e0fe61f804ee3cf00142431a52ce0f3523111efce
+size 2636129740

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"__type": "AddedToken", "content": "<s>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}, "eos_token": {"__type": "AddedToken", "content": "</s>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}, "unk_token": {"__type": "AddedToken", "content": "<unk>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad64d49d1cfb525fab58584add8b0986c5c55275a2a885794e6a985bdf7d049c
+size 13683070

tokenizer_config.json

@@ -0,0 +1 @@
+{"tokenizer_class": "LlamaTokenizer", "bos_token": {"__type": "AddedToken", "content": "<s>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}, "eos_token": {"__type": "AddedToken", "content": "</s>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}, "unk_token": {"__type": "AddedToken", "content": "<unk>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false}, "pad_token": null, "add_bos_token": false, "add_eos_token": false, "clean_up_tokenization_spaces": false, "legacy": false, "model_max_length": 1000000000000000019884624838656, "sp_model_kwargs": {}}