初始化项目，由ModelHub XC社区提供模型

Model: RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16
Source: Original Platform

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+
+tekken.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,434 @@
+---
+language:
+  - en
+  - fr
+  - de
+  - es
+  - it
+  - pt
+  - zh
+  - ja
+  - ru
+  - ko
+base_model:
+- mistralai/Mistral-Small-24B-Instruct-2501
+pipeline_tag: text-generation
+tags:
+- mistral
+- mistral-small
+- w4a16
+- int4
+- vllm
+- conversational
+- text-generation-inference
+- compressed-tensors
+license: apache-2.0
+license_name: apache-2.0
+name: RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16
+description: This model was obtained by quantizing the weights of Mistral-Small-24B-Instruct-2501 to INT4 data type.
+readme: https://huggingface.co/RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16/main/README.md
+tasks:
+- text-to-text
+provider: Mistral AI
+license_link: https://www.apache.org/licenses/LICENSE-2.0
+validated_on:
+  - RHOAI 2.20
+  - RHAIIS 3.0
+  - RHELAI 1.5
+  - vLLM 0.8.4
+---
+
+<h1 style="display: flex; align-items: center; gap: 10px; margin: 0;">
+  Mistral-Small-24B-Instruct-2501-quantized.w4a16
+  <img src="https://www.redhat.com/rhdc/managed-files/Catalog-Validated_model_0.png" alt="Model Icon" width="40" style="margin: 0; padding: 0;" />
+</h1>
+  
+<a href="https://www.redhat.com/en/products/ai/validated-models" target="_blank" style="margin: 0; padding: 0;">
+<img src="https://www.redhat.com/rhdc/managed-files/Validated_badge-Dark.png" alt="Validated Badge" width="250" style="margin: 0; padding: 0;" />
+</a>
+
+## Model Overview
+- **Model Architecture:** Mistral-Small-24B-Instruct-2501
+  - **Input:** Text
+  - **Output:** Text
+- **Model Optimizations:**
+  - **Weight quantization:** INT4
+  - **Activation quantization:** None
+- **Release Date:** 3/1/2025
+- **Version:** 1.0
+- **Validated on:** RHOAI 2.20, RHAIIS 3.0, RHELAI 1.5
+- **Model Developers:** Neural Magic
+
+Quantized version of [Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501).
+
+### Model Optimizations
+
+This model was obtained by quantizing the weights to INT4 data type, ready for inference with vLLM.
+This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%. Only the weights of the linear operators within transformers blocks are quantized.
+
+## Deployment
+
+### Use with vLLM
+
+1. Initialize vLLM server:
+```
+vllm serve RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16 --tensor_parallel_size 1 --tokenizer_mode mistral
+```
+
+2. Send requests to the server:
+
+```python
+from openai import OpenAI
+
+# Modify OpenAI's API key and API base to use vLLM's API server.
+openai_api_key = "EMPTY"
+openai_api_base = "http://<your-server-host>:8000/v1"
+
+client = OpenAI(
+    api_key=openai_api_key,
+    base_url=openai_api_base,
+)
+
+model = "RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16"
+
+
+messages = [
+    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
+]
+
+outputs = client.chat.completions.create(
+    model=model,
+    messages=messages,
+)
+
+generated_text = outputs.choices[0].message.content
+print(generated_text)
+```
+
+<details>
+  <summary>Deploy on <strong>Red Hat AI Inference Server</strong></summary>
+  
+```bash
+podman run --rm -it --device nvidia.com/gpu=all -p 8000:8000 \
+ --ipc=host \
+--env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
+--env "HF_HUB_OFFLINE=0" -v ~/.cache/vllm:/home/vllm/.cache \
+--name=vllm \
+registry.access.redhat.com/rhaiis/rh-vllm-cuda \
+vllm serve \
+--tensor-parallel-size 8 \
+--max-model-len 32768  \
+--enforce-eager --model RedHatAI/Mistral-Small-24B-Instruct-2501-quantized.w4a16
+```
+​​See [Red Hat AI Inference Server documentation](https://docs.redhat.com/en/documentation/red_hat_ai_inference_server/) for more details.
+</details>
+
+<details>
+  <summary>Deploy on <strong>Red Hat Enterprise Linux AI</strong></summary>
+  
+```bash
+# Download model from Red Hat Registry via docker
+# Note: This downloads the model to ~/.cache/instructlab/models unless --model-dir is specified.
+ilab model download --repository docker://registry.redhat.io/rhelai1/mistral-small-24b-instruct-2501-quantized-w4a16:1.5
+```
+
+```bash
+# Serve model via ilab
+ilab model serve --model-path ~/.cache/instructlab/models/mistral-small-24b-instruct-2501-quantized-w4a16 --gpu 1 -- --trust-remote-code
+  
+# Chat with model
+ilab model chat --model ~/.cache/instructlab/models/mistral-small-24b-instruct-2501-quantized-w4a16
+```
+See [Red Hat Enterprise Linux AI documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_ai/1.4) for more details.
+</details>
+
+<details>
+  <summary>Deploy on <strong>Red Hat Openshift AI</strong></summary>
+  
+```python
+# Setting up vllm server with ServingRuntime
+# Save as: vllm-servingruntime.yaml
+apiVersion: serving.kserve.io/v1alpha1
+kind: ServingRuntime
+metadata:
+ name: vllm-cuda-runtime # OPTIONAL CHANGE: set a unique name
+ annotations:
+   openshift.io/display-name: vLLM NVIDIA GPU ServingRuntime for KServe
+   opendatahub.io/recommended-accelerators: '["nvidia.com/gpu"]'
+ labels:
+   opendatahub.io/dashboard: 'true'
+spec:
+ annotations:
+   prometheus.io/port: '8080'
+   prometheus.io/path: '/metrics'
+ multiModel: false
+ supportedModelFormats:
+   - autoSelect: true
+     name: vLLM
+ containers:
+   - name: kserve-container
+     image: quay.io/modh/vllm:rhoai-2.20-cuda # CHANGE if needed. If AMD: quay.io/modh/vllm:rhoai-2.20-rocm
+     command:
+       - python
+       - -m
+       - vllm.entrypoints.openai.api_server
+     args:
+       - "--port=8080"
+       - "--model=/mnt/models"
+       - "--served-model-name={{.Name}}"
+     env:
+       - name: HF_HOME
+         value: /tmp/hf_home
+     ports:
+       - containerPort: 8080
+         protocol: TCP
+```
+
+```python
+# Attach model to vllm server. This is an NVIDIA template
+# Save as: inferenceservice.yaml
+apiVersion: serving.kserve.io/v1beta1
+kind: InferenceService
+metadata:
+  annotations:
+    openshift.io/display-name: Mistral-Small-24B-Instruct-2501-quantized.w4a16 # OPTIONAL CHANGE
+    serving.kserve.io/deploymentMode: RawDeployment
+  name: Mistral-Small-24B-Instruct-2501-quantized.w4a16         # specify model name. This value will be used to invoke the model in the payload
+  labels:
+    opendatahub.io/dashboard: 'true'
+spec:
+  predictor:
+    maxReplicas: 1
+    minReplicas: 1
+    model:
+      args:
+        - "--trust-remote-code"
+      modelFormat:
+        name: vLLM
+      name: ''
+      resources:
+        limits:
+          cpu: '2'			# this is model specific
+          memory: 8Gi		# this is model specific
+          nvidia.com/gpu: '1'	# this is accelerator specific
+        requests:			# same comment for this block
+          cpu: '1'
+          memory: 4Gi
+          nvidia.com/gpu: '1'
+      runtime: vllm-cuda-runtime	# must match the ServingRuntime name above
+      storageUri: oci://registry.redhat.io/rhelai1/modelcar-mistral-small-24b-instruct-2501-quantized-w4a16:1.5
+    tolerations:
+    - effect: NoSchedule
+      key: nvidia.com/gpu
+      operator: Exists
+```
+
+```bash
+# make sure first to be in the project where you want to deploy the model
+# oc project <project-name>
+# apply both resources to run model
+# Apply the ServingRuntime
+oc apply -f vllm-servingruntime.yaml
+# Apply the InferenceService
+oc apply -f qwen-inferenceservice.yaml
+```
+
+```python
+# Replace <inference-service-name> and <cluster-ingress-domain> below:
+# - Run `oc get inferenceservice` to find your URL if unsure.
+# Call the server using curl:
+curl https://<inference-service-name>-predictor-default.<domain>/v1/chat/completions
+        -H "Content-Type: application/json" \
+        -d '{
+    "model": "Mistral-Small-24B-Instruct-2501-quantized.w4a16",
+    "stream": true,
+    "stream_options": {
+        "include_usage": true
+    },
+    "max_tokens": 1,
+    "messages": [
+        {
+            "role": "user",
+            "content": "How can a bee fly when its wings are so small?"
+        }
+    ]
+}'
+```
+
+See [Red Hat Openshift AI documentation](https://docs.redhat.com/en/documentation/red_hat_openshift_ai/2025) for more details.
+</details>
+
+## Creation
+
+This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below.
+
+```bash
+python quantize.py --model_path mistralai/Mistral-Small-24B-Instruct-2501 --quant_path "output_dir" --calib_size 1024 --dampening_frac 0.05 --observer minmax --actorder false
+```
+
+```python
+from datasets import load_dataset
+from transformers import AutoTokenizer
+from llmcompressor.modifiers.quantization import GPTQModifier
+from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot, apply
+import argparse
+from compressed_tensors.quantization import QuantizationScheme, QuantizationArgs, QuantizationType, QuantizationStrategy
+
+def parse_actorder(value):
+    # Interpret the input value for --actorder
+    if value.lower() == "false":
+        return False
+    elif value.lower() == "group":
+        return "group"
+    elif value.lower() == "weight":
+        return "weight"
+    else:
+        raise argparse.ArgumentTypeError("Invalid value for --actorder. Use 'group' or 'False'.")
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--model_path', type=str)
+parser.add_argument('--quant_path', type=str)
+parser.add_argument('--num_bits', type=int, default=4)
+parser.add_argument('--sequential_update', type=bool, default=True)
+parser.add_argument('--calib_size', type=int, default=256)
+parser.add_argument('--dampening_frac', type=float, default=0.05)
+parser.add_argument('--observer', type=str, default="minmax")
+parser.add_argument(
+    '--actorder',
+    type=parse_actorder,
+    default=False,  # Default value is False
+    help="Specify actorder as 'group' (string) or False (boolean)."
+)
+
+args = parser.parse_args()
+
+model = SparseAutoModelForCausalLM.from_pretrained(
+    args.model_path,
+    device_map="auto",
+    torch_dtype="auto",
+    use_cache=False,
+)
+tokenizer = AutoTokenizer.from_pretrained(args.model_path)
+
+NUM_CALIBRATION_SAMPLES = args.calib_size
+DATASET_ID = "garage-bAInd/Open-Platypus"
+DATASET_SPLIT = "train"
+ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
+ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
+
+def preprocess(example):
+    concat_txt = example["instruction"] + "\n" + example["output"]
+    return {"text": concat_txt}
+
+ds = ds.map(preprocess)
+
+def tokenize(sample):
+    return tokenizer(
+        sample["text"],
+        padding=False,
+        truncation=False,
+        add_special_tokens=True,
+    )
+
+
+ds = ds.map(tokenize, remove_columns=ds.column_names)
+
+quant_scheme = QuantizationScheme(
+    targets=["Linear"],
+    weights=QuantizationArgs(
+        num_bits=args.num_bits,
+        type=QuantizationType.INT,
+        symmetric=True,
+        group_size=128,
+        strategy=QuantizationStrategy.GROUP,
+        observer=args.observer,
+        actorder=args.actorder
+    ),
+    input_activations=None,
+    output_activations=None,
+)
+
+recipe = [
+    GPTQModifier(
+        targets=["Linear"],
+        ignore=["lm_head"],
+        sequential_update=args.sequential_update,
+        dampening_frac=args.dampening_frac,
+        config_groups={"group_0": quant_scheme},
+    )
+]
+oneshot(
+    model=model,
+    dataset=ds,
+    recipe=recipe,
+    num_calibration_samples=args.calib_size,
+)
+
+# Save to disk compressed.
+SAVE_DIR = args.quant_path
+model.save_pretrained(SAVE_DIR, save_compressed=True)
+tokenizer.save_pretrained(SAVE_DIR)
+```
+
+## Evaluation
+
+The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard) and [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/), using the following commands:
+
+OpenLLM Leaderboard V1:
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Mistral-Small-24B-Instruct-2501-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --tasks openllm \
+  --write_out \
+  --batch_size auto \
+  --output_path output_dir \
+  --show_config
+```
+
+OpenLLM Leaderboard V2:
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/Mistral-Small-24B-Instruct-2501-quantized.w4a16",dtype=auto,add_bos_token=False,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --apply_chat_template \
+  --fewshot_as_multiturn \
+  --tasks leaderboard \
+  --write_out \
+  --batch_size auto \
+  --output_path output_dir \
+  --show_config
+
+```
+
+### Accuracy
+
+#### OpenLLM Leaderboard V1 evaluation scores
+
+
+| Metric                                   | mistralai/Mistral-Small-24B-Instruct-2501             | neuralmagic/Mistral-Small-24B-Instruct-2501-quantized.w4a16 |
+|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
+| ARC-Challenge (Acc-Norm, 25-shot)       |  72.18                            |    71.16                                    |
+| GSM8K (Strict-Match, 5-shot)            |  90.14                            |    89.69                                     |
+| HellaSwag (Acc-Norm, 10-shot)           |  85.05                            |    84.43                                    |
+| MMLU (Acc, 5-shot)                      |  80.69                            |    80.00                                    |
+| TruthfulQA (MC2, 0-shot)                |  65.55                            |    63.92                                    |
+| Winogrande (Acc, 5-shot)                |  83.11                            |    82.24                                    |
+| **Average Score**                       | **79.45**                        | **78.57**                                   |
+| **Recovery (%)**                            | **100.00**                       | **98.9**                                   |
+
+#### OpenLLM Leaderboard V2 evaluation scores
+
+| Metric                                                   | mistralai/Mistral-Small-24B-Instruct-2501             | neuralmagic/Mistral-Small-24B-Instruct-2501-quantized.w4a16 |
+|---------------------------------------------------------|:---------------------------------:|:-------------------------------------------:|
+| IFEval (Inst-and-Prompt Level Strict Acc, 0-shot)       |     73.27                        |    74.37                                    |
+| BBH (Acc-Norm, 3-shot)                                  |     45.18                        |    45.15                                       |
+| MMLU-Pro (Acc, 5-shot)                                  |      38.83                       |    36.00                                    |
+| **Average Score**                                       | **52.42**                        | **51.84**                                   |
+| **Recovery (%)**                                            | **100.00**                       | **98.89**                                   |
+| GPQA (Acc-Norm, 0-shot)                                 |      8.29                        |   6.81                                      |
+| MUSR (Acc-Norm, 0-shot)                                 |      7.84                        |   9.46                                     |
+
+Results on GPQA and MUSR are not considred for accuracy recovery calculation because the unquantized model has close to random prediction accuracy (8.29, 7.84) which doesn't provide a reliable baseline for recovery calculation.
+

config.json

@@ -0,0 +1,59 @@
+{
+  "_name_or_path": "mistralai/Mistral-Small-24B-Instruct-2501",
+  "architectures": [
+    "MistralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 32768,
+  "max_position_embeddings": 32768,
+  "model_type": "mistral",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 8,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "input_activations": null,
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": 128,
+          "num_bits": 4,
+          "observer": "minmax",
+          "observer_kwargs": {},
+          "strategy": "group",
+          "symmetric": true,
+          "type": "int"
+        }
+      }
+    },
+    "format": "pack-quantized",
+    "global_compression_ratio": 2.0797400655392253,
+    "ignore": [
+      "lm_head"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed",
+    "sparsity_config": {}
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 100000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.48.2",
+  "use_cache": false,
+  "vocab_size": 131072
+}

configuration.json

@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": 2,
+  "temperature": 0.15,
+  "transformers_version": "4.48.2"
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88c75e7d5c2933ded1ef3452b3770f56ba27d7fdd201346cad3bed08cca72696
+size 4981180656

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80caaaf3de240147958713dd8863eddd82f9c0aa06d642834a1244988ca5c835
+size 4985413200

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a098a6bba64c7389355bb09891d72a23782f639b09d506358096c41d0d7f165
+size 4180946336

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eeef73c38225f74e2eb13556ee452cc6feb16a080c090881bac6bec8cbdc2607
+size 80674

recipe.yaml

@@ -0,0 +1,14 @@
+DEFAULT_stage:
+  DEFAULT_modifiers:
+    GPTQModifier:
+      sequential_update: true
+      dampening_frac: 0.05
+      config_groups:
+        group_0:
+          targets: [Linear]
+          weights: {num_bits: 4, type: int, symmetric: true, group_size: 128, strategy: group,
+            actorder: null, observer: minmax}
+          input_activations: null
+          output_activations: null
+      targets: [Linear]
+      ignore: [lm_head]

special_tokens_map.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bc5d1c106bf86895473568ef5211f3fde8415e6fdee039223d6db8ca52cbe01
+size 21311

tekken.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4b90a968dbc67ef3975129d0b78a2e3cbb6bea340ab9205f22e8a0308b1ffc5
+size 14801223

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b76085f9923309d873994d444989f7eb6ec074b06f25b58f1e8d7b7741070949
+size 17078037

tokenizer_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:350d9c44f99942e4dd52b8f11bca7d0ac1eeee9acf9e8ee13cfc72de753f1cf1
+size 199721