初始化项目，由ModelHub XC社区提供模型

Model: RedHatAI/SparseLlama-3-8B-pruned_50.2of4
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,85 @@
+---
+base_model: meta-llama/Meta-Llama-3-8B
+inference: true
+model_type: llama
+pipeline_tag: text-generation
+tags:
+- sparse
+---
+
+# SparseLlama-3-8B-pruned_50.2of4
+
+This repo contains model files for a 2:4 (N:M) sparse [Meta-Llama-3-8B](meta-llama/Meta-Llama-3-8B) model pruned in one-shot with [SparseGPT](https://arxiv.org/abs/2301.00774), and then additionally retrained with the [SquareHead](https://arxiv.org/abs/2310.06927) knowledge distillation while maintaining the 2:4 sparsity mask.
+
+**Note:** This is still a work in progress and subject to change. We expect to release new weights with even better accuracy soon.
+
+## Running the model
+
+It can be run naively in transformers for testing purposes:
+```python
+# pip install transformers accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("nm-testing/SparseLlama-3-8B-pruned_50.2of4")
+model = AutoModelForCausalLM.from_pretrained("nm-testing/SparseLlama-3-8B-pruned_50.2of4", device_map="auto")
+
+input_text = "A poem about Machine Learning goes as follows:"
+input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
+
+outputs = model.generate(**input_ids)
+print(tokenizer.decode(outputs[0]))
+```
+
+To take advantage of the 2:4 sparsity present, install [nm-vllm](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: 
+```bash
+pip install nm-vllm[sparse] --extra-index-url https://pypi.neuralmagic.com/simple
+```
+
+```python
+from vllm import LLM, SamplingParams
+
+model = LLM("nm-testing/SparseLlama-3-8B-pruned_50.2of4", sparsity="semi_structured_sparse_w16a16")
+
+prompt = "A poem about Machine Learning goes as follows:"
+sampling_params = SamplingParams(max_tokens=100, temperature=0)
+
+outputs = model.generate(prompt, sampling_params=sampling_params)
+print(outputs[0].outputs[0].text)
+```
+
+## Evaluation Benchmark Results
+
+Model evaluation results obtained via [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) following the configuration of [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard).
+
+| Benchmark                                      | Meta-Llama-3-8B  | SparseLlama-3-8B-pruned_50.2of4<br>(this model) |
+|:----------------------------------------------:|:-----------:|:-----------------------------:|
+| [ARC-c](https://arxiv.org/abs/1911.01547)<br> 25-shot      | 59.47%       | 57.76%                         |
+| [MMLU](https://arxiv.org/abs/2009.03300)<br> 5-shot       | 65.29%       | 60.44%                         |
+| [HellaSwag](https://arxiv.org/abs/1905.07830)<br> 10-shot  |82.14%       | 79.97%                         |
+| [WinoGrande](https://arxiv.org/abs/1907.10641)<br> 5-shot |77.27%       | 77.19%                         |
+| [GSM8K](https://arxiv.org/abs/2110.14168)<br> 5-shot      |  44.81%       | 47.92%                         |
+| [TruthfulQA](https://arxiv.org/abs/2109.07958)<br> 0-shot |  43.96%       | 41.02%                         |
+| **Average<br>Accuracy**  | **62.16%**                    |              **60.72%**                                     |
+| **Recovery**             | **100%**                     |              **97.68%**                                     |
+
+
+Model evaluation results obtained via [Mosaic Eval Gauntlet](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md) following the configuration of [Eval Gauntlet v0.3](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/yamls/eval_gauntlet_v0.3.yaml).
+
+| Benchmark                | Meta-Llama-3-8B  | SparseLlama-3-8B-pruned_50.2of4<br>(this model) |
+|:------------------------:|:----------------:|:----------------------------------------------:|
+| World Knowledge          | 58.08%       | 54.61%                         |
+| Commonsense Reasoning    | 47.66%       | 47.62%                         |
+| Language Understanding   | 71.13%       | 67.58%                         |
+| Symbolic Problem Solving | 38.44%       | 32.15%                         |
+| Reading Comprehension    | 57.48%       | 55.76%                         |
+| **Average Accuracy**     | **54.70%**    |  **51.54%**                   |
+| **Recovery**             | **100%** |  **94.22%**                   |
+
+
+## Help
+
+For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)
+
+## Acknowledgment
+
+This model is built with Meta Llama 3. For more details on its licence please check the model card of [Meta-Llama-3-8B](meta-llama/Meta-Llama-3-8B).

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "eos_token_id": 128001,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 8192,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.39.0.dev0",
+  "use_cache": true,
+  "vocab_size": 128256
+}

configuration.json

@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 128000,
+  "eos_token_id": 128001,
+  "transformers_version": "4.39.0.dev0",
+  "use_cache": true
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3207e1815e3f670425d3d8279d7a1c37811ba9abcf77d377133c935911c81e3f
+size 4976698672

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9d07d15c843256e0adf4d609e98eefebea1b6689e44c4e8ade058865eed3314
+size 4999802720

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1eb35066a615be323cab97d079a88d42e18b52c12168085acfcee3ff6930f7cd
+size 4915916176

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11d0b2e84feffb9dcf681022bfd220c5ef117dcd547762e1e86a81724174c7d9
+size 1168138808

special_tokens_map.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33403d05c86de6d06b07883f26022da68437008344ea498c075b0cbb97e6430d
+size 9884

tokenizer_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b07911533d195b0bd90b6ec87fb22070865f771691eaf21d5c3883831c221227
+size 60122