commit 4701f4518572bf4d28a3d3a402e3159383364ca9 Author: ModelHub XC Date: Fri Jul 3 08:42:12 2026 +0800 初始化项目，由ModelHub XC社区提供模型 Model: RedHatAI/SparseLlama-3-8B-pruned_50.2of4 Source: Original Platform diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..a6344aa --- /dev/null +++ b/.gitattributes @@ -0,0 +1,35 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md new file mode 100644 index 0000000..c33a1f9 --- /dev/null +++ b/README.md @@ -0,0 +1,85 @@ +--- +base_model: meta-llama/Meta-Llama-3-8B +inference: true +model_type: llama +pipeline_tag: text-generation +tags: +- sparse +--- + +# SparseLlama-3-8B-pruned_50.2of4 + +This repo contains model files for a 2:4 (N:M) sparse [Meta-Llama-3-8B](meta-llama/Meta-Llama-3-8B) model pruned in one-shot with [SparseGPT](https://arxiv.org/abs/2301.00774), and then additionally retrained with the [SquareHead](https://arxiv.org/abs/2310.06927) knowledge distillation while maintaining the 2:4 sparsity mask. + +**Note:** This is still a work in progress and subject to change. We expect to release new weights with even better accuracy soon. + +## Running the model + +It can be run naively in transformers for testing purposes: +```python +# pip install transformers accelerate +from transformers import AutoTokenizer, AutoModelForCausalLM + +tokenizer = AutoTokenizer.from_pretrained("nm-testing/SparseLlama-3-8B-pruned_50.2of4") +model = AutoModelForCausalLM.from_pretrained("nm-testing/SparseLlama-3-8B-pruned_50.2of4", device_map="auto") + +input_text = "A poem about Machine Learning goes as follows:" +input_ids = tokenizer(input_text, return_tensors="pt").to("cuda") + +outputs = model.generate(**input_ids) +print(tokenizer.decode(outputs[0])) +``` + +To take advantage of the 2:4 sparsity present, install [nm-vllm](https://github.com/neuralmagic/nm-vllm) for fast inference and low memory-usage: +```bash +pip install nm-vllm[sparse] --extra-index-url https://pypi.neuralmagic.com/simple +``` + +```python +from vllm import LLM, SamplingParams + +model = LLM("nm-testing/SparseLlama-3-8B-pruned_50.2of4", sparsity="semi_structured_sparse_w16a16") + +prompt = "A poem about Machine Learning goes as follows:" +sampling_params = SamplingParams(max_tokens=100, temperature=0) + +outputs = model.generate(prompt, sampling_params=sampling_params) +print(outputs[0].outputs[0].text) +``` + +## Evaluation Benchmark Results + +Model evaluation results obtained via [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) following the configuration of [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard). + +| Benchmark | Meta-Llama-3-8B | SparseLlama-3-8B-pruned_50.2of4
(this model) | +|:----------------------------------------------:|:-----------:|:-----------------------------:| +| [ARC-c](https://arxiv.org/abs/1911.01547)
25-shot | 59.47% | 57.76% | +| [MMLU](https://arxiv.org/abs/2009.03300)
5-shot | 65.29% | 60.44% | +| [HellaSwag](https://arxiv.org/abs/1905.07830)
10-shot |82.14% | 79.97% | +| [WinoGrande](https://arxiv.org/abs/1907.10641)
5-shot |77.27% | 77.19% | +| [GSM8K](https://arxiv.org/abs/2110.14168)
5-shot | 44.81% | 47.92% | +| [TruthfulQA](https://arxiv.org/abs/2109.07958)
0-shot | 43.96% | 41.02% | +| **Average
Accuracy** | **62.16%** | **60.72%** | +| **Recovery** | **100%** | **97.68%** | + + +Model evaluation results obtained via [Mosaic Eval Gauntlet](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/EVAL_GAUNTLET.md) following the configuration of [Eval Gauntlet v0.3](https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/yamls/eval_gauntlet_v0.3.yaml). + +| Benchmark | Meta-Llama-3-8B | SparseLlama-3-8B-pruned_50.2of4
(this model) | +|:------------------------:|:----------------:|:----------------------------------------------:| +| World Knowledge | 58.08% | 54.61% | +| Commonsense Reasoning | 47.66% | 47.62% | +| Language Understanding | 71.13% | 67.58% | +| Symbolic Problem Solving | 38.44% | 32.15% | +| Reading Comprehension | 57.48% | 55.76% | +| **Average Accuracy** | **54.70%** | **51.54%** | +| **Recovery** | **100%** | **94.22%** | + + +## Help + +For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ) + +## Acknowledgment + +This model is built with Meta Llama 3. For more details on its licence please check the model card of [Meta-Llama-3-8B](meta-llama/Meta-Llama-3-8B). \ No newline at end of file diff --git a/config.json b/config.json new file mode 100644 index 0000000..53504d3 --- /dev/null +++ b/config.json @@ -0,0 +1,27 @@ +{ + "architectures": [ + "LlamaForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 128000, + "eos_token_id": 128001, + "hidden_act": "silu", + "hidden_size": 4096, + "initializer_range": 0.02, + "intermediate_size": 14336, + "max_position_embeddings": 8192, + "model_type": "llama", + "num_attention_heads": 32, + "num_hidden_layers": 32, + "num_key_value_heads": 8, + "pretraining_tp": 1, + "rms_norm_eps": 1e-05, + "rope_scaling": null, + "rope_theta": 500000.0, + "tie_word_embeddings": false, + "torch_dtype": "bfloat16", + "transformers_version": "4.39.0.dev0", + "use_cache": true, + "vocab_size": 128256 +} diff --git a/configuration.json b/configuration.json new file mode 100644 index 0000000..bbeeda1 --- /dev/null +++ b/configuration.json @@ -0,0 +1 @@ +{"framework": "pytorch", "task": "text-generation", "allow_remote": true} \ No newline at end of file diff --git a/generation_config.json b/generation_config.json new file mode 100644 index 0000000..e2949df --- /dev/null +++ b/generation_config.json @@ -0,0 +1,7 @@ +{ + "_from_model_config": true, + "bos_token_id": 128000, + "eos_token_id": 128001, + "transformers_version": "4.39.0.dev0", + "use_cache": true +} diff --git a/model-00001-of-00004.safetensors b/model-00001-of-00004.safetensors new file mode 100644 index 0000000..42d384a --- /dev/null +++ b/model-00001-of-00004.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3207e1815e3f670425d3d8279d7a1c37811ba9abcf77d377133c935911c81e3f +size 4976698672 diff --git a/model-00002-of-00004.safetensors b/model-00002-of-00004.safetensors new file mode 100644 index 0000000..a1d096b --- /dev/null +++ b/model-00002-of-00004.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f9d07d15c843256e0adf4d609e98eefebea1b6689e44c4e8ade058865eed3314 +size 4999802720 diff --git a/model-00003-of-00004.safetensors b/model-00003-of-00004.safetensors new file mode 100644 index 0000000..74151ae --- /dev/null +++ b/model-00003-of-00004.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1eb35066a615be323cab97d079a88d42e18b52c12168085acfcee3ff6930f7cd +size 4915916176 diff --git a/model-00004-of-00004.safetensors b/model-00004-of-00004.safetensors new file mode 100644 index 0000000..eaffdd3 --- /dev/null +++ b/model-00004-of-00004.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:11d0b2e84feffb9dcf681022bfd220c5ef117dcd547762e1e86a81724174c7d9 +size 1168138808 diff --git a/model.safetensors.index.json b/model.safetensors.index.json new file mode 100644 index 0000000..a054aad --- /dev/null +++ b/model.safetensors.index.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:146776fce3f6db1103aa6f249e65ee5544c5923ce6f971b092eee79aa6e5d37b +size 23950 diff --git a/special_tokens_map.json b/special_tokens_map.json new file mode 100644 index 0000000..550d879 --- /dev/null +++ b/special_tokens_map.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:33403d05c86de6d06b07883f26022da68437008344ea498c075b0cbb97e6430d +size 9884 diff --git a/tokenizer.json b/tokenizer.json new file mode 100644 index 0000000..5c08aa0 --- /dev/null +++ b/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:52b583ee1699c6ca4468f70885bd5f96dd1f7517292362a67581038afc04f4d8 +size 9085671 diff --git a/tokenizer_config.json b/tokenizer_config.json new file mode 100644 index 0000000..3a1718d --- /dev/null +++ b/tokenizer_config.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b07911533d195b0bd90b6ec87fb22070865f771691eaf21d5c3883831c221227 +size 60122