ibm-ai-platform/Bamba-9B-v1
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: ibm-ai-platform/Bamba-9B-v1
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-11-01 13:04:22 +08:00
commit f54a873717
12 changed files with 413454 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
*.gguf filter=lfs diff=lfs merge=lfs -text

309
README.md Normal file
View File

@@ -0,0 +1,309 @@
---
license: apache-2.0
library_name: transformers
tags:
- bamba
---
## Model Details
<p align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/64b6c638ac6d20bae0b93219/GOzs8o4G1apceun92ZC4d.png" alt="Bamba" width="400" height="400">
</p>
# Model Card for Bamba 9B
We introduce Bamba-9B, a decoder-only language model based on the [Mamba-2](https://github.com/state-spaces/mamba) architecture and is designed to handle a wide range of text generation tasks. It is trained from scratch using a two-stage training approach. In the first stage, the model is trained on 2 trillion tokens from the Dolma v1.7 dataset. In the second stage, it undergoes additional training on 200 billion tokens, leveraging a carefully curated blend of high-quality data to further refine its performance and enhance output quality.
| Model | Params | # Layers | Hidden Dim. | Attention Heads | GQA | KV Heads | Context Length | Tied Embeddings |
| ----- | ---------- | -------- | ----------- | --------------- | ---- | -------- | -------------- | --------------- |
| Bamba | 9B (9.78B) | 32 | 4096 | 32 | Yes | 8 | 4096 | False |
The current release includes the following models:
| **Stage** | **Bamba 9B** | **Quantized** | **Note** |
|----------------------|----------------------------------------------------------------------|-------------------------------------------------------------------------|-------------------------------------------------------------------|
| **Base Model** | [ibm-fms/Bamba-9B-v1](https://huggingface.co/ibm-ai-platform/Bamba-9B-v1) | [ibm-fms/Bamba-9B-fp8](https://huggingface.co/ibm-fms/Bamba-9B-fp8) | Stage 2 pretraining |
| **Base Model** | [ibm-fms/Bamba-9B-2T](https://huggingface.co/ibm-fms/Bamba-9B-2T) | [ibm-fms/Bamba-9B-fp8](https://huggingface.co/ibm-fms/Bamba-9B-fp8) | Stage 1 pretraining |
| **Base Model** | [ibm-fms/Bamba-9B-1.8T](https://huggingface.co/ibm-fms/Bamba-9B-1.8T)| [ibm-fms/Bamba-9B-fp8](https://huggingface.co/ibm-fms/Bamba-9B-fp8) | Intermediate checkpoints during Stage 1, more to come |
| **SFT** | coming soon | coming soon | to be released in the next drop |
| **DPO** | coming soon | coming soon | to be released in the next drop |
Original checkpoints (in dcp format) were also uploaded to public bucket:
```
bucket: bamba-public
endpoint-url: https://s3.us-east.cloud-object-storage.appdomain.cloud
```
example command to list original Bamba distributed checkpoints:
```bash
aws --endpoint-url https://s3.us-east.cloud-object-storage.appdomain.cloud s3 ls s3://bamba-public/checkpoints/pretraining/phase_two/2_2t/step_140000_ckp/
```
## Installation
Besides [PyTorch](https://pytorch.org/), you would need a few [extra dependencies](https://github.com/state-spaces/mamba?tab=readme-ov-file#installation) for
Mamba models.
We found some of these dependencies picky on PyTorch versions when doing pip install, so
the best way is to build from source for all Mamba dependencies if you hit dependency
issue with your env:
```bash
git clone https://github.com/Dao-AILab/causal-conv1d.git
cd causal-conv1d && pip install . && cd ..
git clone https://github.com/state-spaces/mamba.git
cd mamba && pip install . && cd ..
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention && pip install . && cd ..
```
For users using our HF versions of the model, you would need to install the latest transformers which includes our newly merged implementation for our Bamba models:
```bash
pip install git+https://github.com/huggingface/transformers.git
```
## Inference
You can utilize our newly contributed HF integration to run inference on our Bamba models:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("ibm-fms/Bamba-9B-v1")
tokenizer = AutoTokenizer.from_pretrained("ibm-fms/Bamba-9B-v1")
message = ["Mamba is a snake with following properties "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
response = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])
```
## Training
We trained our Bamba model with FSDP using our training repo [here](https://github.com/foundation-model-stack/fms-fsdp).
Note that this training effort was started before FSDP2 and also long before we contributed
`Mamba2-Hybrid` to HF, so we were doing FSDP1 training with [official Mamba implementation](https://github.com/state-spaces/mamba).
For users trying to reproduce the training you now have much more options with our newly
contributed [HF-version of Mamba2-Hybrid](https://github.com/huggingface/transformers/tree/main/src/transformers/models/bamba).
## Benchmark scores
### Base pretrained models
<table>
<tr>
<td><strong>Category</strong>
</td>
<td><strong>Benchmark</strong>
</td>
<td><strong>Bamba 9B (2.2T)</strong>
</td>
</tr>
<tr>
<td rowspan="8" >General
</td>
<td>MMLU (5-shot)
</td>
<td>60.77
</td>
</tr>
<tr>
<td>ARC-C (25-shot)
</td>
<td>63.23
</td>
</tr>
<tr>
<td>GSM8K (5-shot)
</td>
<td>36.77
</td>
</tr>
<tr>
<td>Hellaswag (10-shot)
</td>
<td>81.8
</td>
</tr>
<tr>
<td>OpenbookQA (5-shot)
</td>
<td>47.6
</td>
</tr>
<tr>
<td>Piqa (5-shot)
</td>
<td>82.26
</td>
</tr>
<tr>
<td>TruthfulQA (0-shot)
</td>
<td>49.21
</td>
</tr>
<tr>
<td>Winogrande (5-shot)
</td>
<td>76.87
</td>
</tr>
<tr>
<td rowspan="6" >HF OpenLLM- V2*
</td>
<td>MMLU-PRO (5-shot)
</td>
<td>17.53
</td>
</tr>
<tr>
<td>BBH (3-shot)
</td>
<td>17.4
</td>
</tr>
<tr>
<td>GPQA (0-shot)
</td>
<td>4.14
</td>
</tr>
<tr>
<td>IFEval (0-shot)
</td>
<td>15.16
</td>
</tr>
<tr>
<td>MATH Lvl 5 (4-shot)
</td>
<td>1.66
</td>
</tr>
<tr>
<td>MuSR (0-shot)
</td>
<td>9.59
</td>
</tr>
<tr>
<td rowspan="4" >Safety Tasks
</td>
<td>PopQA (5-shot)
</td>
<td>20.5
</td>
</tr>
<tr>
<td>Toxigen (5-shot)
</td>
<td>57.4
</td>
</tr>
<tr>
<td>BBQ (5-shot)
</td>
<td>44.2
</td>
</tr>
<tr>
<td>Crows-pairs english (5-shot)
</td>
<td>70.78
</td>
</tr>
</table>
*For the v2 leaderboard results, we perform [normalization](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/normalization) and report the normalized results.
Further details on our evaluation and normalization detailes along with run and analysis scripts can be found [here](https://github.com/foundation-model-stack/bamba/blob/main/evaluation/README.md).
## Fine-tuning
This [example](https://github.com/foundation-model-stack/bamba/blob/main/tuning/Fine-tuning.md) shows how to fine tune the bamba model for a specific task using [SFT Trainer](https://huggingface.co/docs/trl/en/sft_trainer#supervised-fine-tuning-trainer).
## Quantization
We can create a (FP8) quantized model using [`fms-model-optimizer`](https://github.com/foundation-model-stack/fms-model-optimizer/), which will make the storage and inference even more efficient.
```python
python -m fms_mo.run_quant \
--model_name_or_path <"path_to_original_model"> \
--quant_method fp8 \
--torch_dtype bfloat16 \
--output_dir <"path_to_save_new_model">
```
Model size comparison before and after FP8:
| | original | quantized |
| :-----------------: | -----------------------: | -----------------------------------------------------------: |
| memory (total) | 39.12 GB | 10.83 GB |
| memory (break-down) | `torch.float32` 39.12 GB | `torch.bfloat16` 2.10 GB<br>`torch.float8_e4m3fn` 8.73 GB |
More details about `fms-model-optimizer` can be found [here](https://github.com/foundation-model-stack/fms-model-optimizer/tree/main/examples/FP8_QUANT#quickstart).
## Llama.cpp
There is preliminary work to enable running Bamba architecture models using [llama.cpp](https://github.com/ggerganov/llama.cpp). This is work-in-progress, so should only be used as a guide for the adventurous!
### Known Limitations
* Currently, inference is only supported on CPUs
* Models quantized with `llama-quantize` exhibit bad performance
### Setup
To enable Bamba support, you'll need to build from source using [Gabe's fork](https://github.com/gabe-l-hart/llama.cpp/tree/BambaArchitecture).
```sh
git clone --branch BambaArchitecture git@github.com:gabe-l-hart/llama.cpp.git
cd llama.cpp
mkdir build
cd build
# NOTE: To build with debug symbols and extra logging, use CMAKE_BUILD_TYPE=Debug
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j
```
### Conversion to GGUF
You can use a pre-converted GGUF file from Huggingface (e.g. [bamba-9b.gguf](https://huggingface.co/ibm-fms/Bamba-9B/blob/main/bamba-9b.gguf)). If one doesn't exist, you can use the [convert_hf_to_gguf.py](https://github.com/gabe-l-hart/llama.cpp/blob/BambaArchitecture/convert_hf_to_gguf.py) script from Gabe's fork to perform the conversion manually.
```sh
# Install the python dependencies
cd /path/to/llama.cpp
pip install -r requirements/requirements-convert_hf_to_gguf.txt
# Perform the conversion
./convert_hf_to_gguf.py /path/to/bamba-model --outfile /path/to/bamba-model/bamba-model.gguf
```
### Run with llama-cli
```sh
# Run the model with no layers on the GPU (CPU-only)
cd /path/to/llama.cpp
./bin/llama-cli -ngl 0 -m /path/to/bamba-model/bamba-model.gguf -p "Tell me a story about a developer and their dog"
```
### Quantization with llama-quantize
You can (optionally) quantize the GGUF model using `llama.cpp`'s built in quantizaiton tool `llama-quantize`.
```sh
# Run the quantization (see llama-quantize --help for all quant types)
cd /path/to/llama.cpp
./build/bin/llama-quantize /path/to/bamba-model/bamba-model.gguf Q4_K_M
```
## Contributors
* **Data collection and curation**: We acknowledge and thank AllenAI team for making a high quality open source dataset Dolma as well as Hugging Face data team for making FineWeb-edu and Cosmopedia available. These are tremendous contributions and enable us to create the model today.
* **Data preprocessing**: We thank IBM's internal data preprocessing team, specifically Tuan Hoang Trong, Syed Zawad, Jay Gala, and Ryan Gordon for helping tokenize the data at scale. The code for tokenization is available [here](https://github.com/IBM/data-prep-kit).
* **Model architecture**: The model architecture design was jointly done by Princeton, CMU, IBM, and UIUC and involved the following folks: Tri Dao (Princeton), Albert Gu (CMU), Linsong Chu (IBM), Davis Wertheimer (IBM), Minjia Zhang (UIUC), Mudhakar Srivatsa (IBM), and Raghu Ganti (IBM).
* **Model training**: Model training was performed primarily by the IBM team using the Mamba2 kernels and layer implementation from Tri Dao and Albert Gu. The following folks from IBM were primarily involved: Linsong Chu, Divya Kumari, Davis Wertheimer, Raghu Ganti, and Dakshi Agrawal.
* **Model tuning**: Tuning of the model was enabled and verified in [TRL](https://github.com/huggingface/trl) by the IBM team, involving Sukriti Sharma and Anh Uong.
* **Model inference**: Model inference in `transformers`, `vLLM`, and `llama.cpp` builds on the kernels written by Princeton and CMU. The IBM team is working with the community to enable it in various ecosystems, the team includes Fabian Lim, Antoni viros i Martin, Adnan Hoque, Jamie Yang, Nelson Nimura Gomez, Joshua Rosenkranz, Nick Hill, and Gabe Goodhart.
* **Quantization**: Quantization is led by the IBM team \- Naigang Wang and Charlie Liu.
* **Evaluations**: Evaluations are led by a team in IBM with long context evaluations being performed by UIUC, involving the following folks: Yotam Perlitz, Ofir Arviv, Michal Shmueli-Scheuer (IBM), Haoechen Shen, and Minjia Zhang (UIUC).
Finally, we would like to thank our leadership for their support in this effort \- Priya Nagpurkar, David Cox, Sriram Raghavan, Aya Soffer, and Mukesh Khare.
We would also like to thank the community, in particular Pablo Montalvo-Leroux and Vaibhav Srivastav from Hugging Face who provided valuable feedback to this blog and the PRs into transformers. Further, we would like to thank Tyler Michael Smith from Neural Magic, who is shepherding the integration with vLLM.
A huge shoutout to Meta PyTorch, AllenAI, and Hugging Face teams for their contributions to the open initative, FSDP allowed us to smoothly train this model and the data from Dolma and Fineweb/Cosmopedia made this model today!

3
bamba-9b.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9a4145cf68afe2b15cdc32b6ab5975ef0c2e0bdcb057d71ff7165c2f58a4c8b5
size 19571792544

41
config.json Normal file
View File

@@ -0,0 +1,41 @@
{
"architectures": [
"BambaForCausalLM"
],
"attention_dropout": 0.0,
"attn_layer_indices": [
9,
18,
27
],
"attn_rotary_emb": 64,
"bos_token_id": 128000,
"eos_token_id": 128001,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"mamba_chunk_size": 256,
"mamba_conv_bias": true,
"mamba_d_conv": 4,
"mamba_d_head": 64,
"mamba_d_state": 128,
"mamba_dt_rank": 256,
"mamba_expand": 2,
"mamba_n_groups": 1,
"mamba_n_heads": 128,
"mamba_proj_bias": false,
"max_position_embeddings": 4096,
"model_type": "bamba",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"num_logits_to_keep": 1,
"pad_token_id": 0,
"rms_norm_eps": 1e-05,
"tie_word_embeddings": false,
"transformers_version": "4.47.0.dev0",
"use_cache": true,
"use_mamba_kernels": true,
"vocab_size": 128256
}

3
model-00001-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3ba4978d138d820e72c61508bb1806a1a64f961cd40d27bfaf6c80840ef8ec3c
size 4949075760

3
model-00002-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:33e2f51eb3318a049d914c1b7d0d1268e2334f2150fcce806baad148dfc167c5
size 4891531536

3
model-00003-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5f749184727a109bca7f31b2ab062ee91dc85b7e2961c4c568451ed5651e2379
size 4891531568

3
model-00004-of-00004.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:09afff7cca976f31b0a856ed797ea3dff441a42166d09626de295dabc38afae4
size 4828377152

414
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,414 @@
{
"metadata": {
"total_size": 19560470784
},
"weight_map": {
"lm_head.weight": "model-00004-of-00004.safetensors",
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
"model.final_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.0.feed_forward.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.feed_forward.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.feed_forward.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mamba.A_log": "model-00001-of-00004.safetensors",
"model.layers.0.mamba.D": "model-00001-of-00004.safetensors",
"model.layers.0.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
"model.layers.0.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mamba.dt_bias": "model-00001-of-00004.safetensors",
"model.layers.0.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mamba.norm.weight": "model-00001-of-00004.safetensors",
"model.layers.0.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.0.pre_ff_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.1.feed_forward.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.feed_forward.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.feed_forward.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mamba.A_log": "model-00001-of-00004.safetensors",
"model.layers.1.mamba.D": "model-00001-of-00004.safetensors",
"model.layers.1.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
"model.layers.1.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mamba.dt_bias": "model-00001-of-00004.safetensors",
"model.layers.1.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mamba.norm.weight": "model-00001-of-00004.safetensors",
"model.layers.1.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.1.pre_ff_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.10.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mamba.A_log": "model-00002-of-00004.safetensors",
"model.layers.10.mamba.D": "model-00002-of-00004.safetensors",
"model.layers.10.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
"model.layers.10.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mamba.dt_bias": "model-00002-of-00004.safetensors",
"model.layers.10.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mamba.norm.weight": "model-00002-of-00004.safetensors",
"model.layers.10.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.10.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.11.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mamba.A_log": "model-00002-of-00004.safetensors",
"model.layers.11.mamba.D": "model-00002-of-00004.safetensors",
"model.layers.11.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
"model.layers.11.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mamba.dt_bias": "model-00002-of-00004.safetensors",
"model.layers.11.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mamba.norm.weight": "model-00002-of-00004.safetensors",
"model.layers.11.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.11.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.12.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mamba.A_log": "model-00002-of-00004.safetensors",
"model.layers.12.mamba.D": "model-00002-of-00004.safetensors",
"model.layers.12.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
"model.layers.12.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mamba.dt_bias": "model-00002-of-00004.safetensors",
"model.layers.12.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mamba.norm.weight": "model-00002-of-00004.safetensors",
"model.layers.12.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.12.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.13.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mamba.A_log": "model-00002-of-00004.safetensors",
"model.layers.13.mamba.D": "model-00002-of-00004.safetensors",
"model.layers.13.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
"model.layers.13.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mamba.dt_bias": "model-00002-of-00004.safetensors",
"model.layers.13.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mamba.norm.weight": "model-00002-of-00004.safetensors",
"model.layers.13.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.13.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.14.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mamba.A_log": "model-00002-of-00004.safetensors",
"model.layers.14.mamba.D": "model-00002-of-00004.safetensors",
"model.layers.14.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
"model.layers.14.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mamba.dt_bias": "model-00002-of-00004.safetensors",
"model.layers.14.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mamba.norm.weight": "model-00002-of-00004.safetensors",
"model.layers.14.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.14.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.15.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mamba.A_log": "model-00002-of-00004.safetensors",
"model.layers.15.mamba.D": "model-00002-of-00004.safetensors",
"model.layers.15.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
"model.layers.15.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mamba.dt_bias": "model-00002-of-00004.safetensors",
"model.layers.15.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mamba.norm.weight": "model-00002-of-00004.safetensors",
"model.layers.15.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.15.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.16.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.16.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.16.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.16.mamba.A_log": "model-00002-of-00004.safetensors",
"model.layers.16.mamba.D": "model-00002-of-00004.safetensors",
"model.layers.16.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
"model.layers.16.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
"model.layers.16.mamba.dt_bias": "model-00002-of-00004.safetensors",
"model.layers.16.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.16.mamba.norm.weight": "model-00003-of-00004.safetensors",
"model.layers.16.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.16.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.17.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.17.mamba.A_log": "model-00003-of-00004.safetensors",
"model.layers.17.mamba.D": "model-00003-of-00004.safetensors",
"model.layers.17.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
"model.layers.17.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
"model.layers.17.mamba.dt_bias": "model-00003-of-00004.safetensors",
"model.layers.17.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.mamba.norm.weight": "model-00003-of-00004.safetensors",
"model.layers.17.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.17.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.18.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.18.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mamba.A_log": "model-00003-of-00004.safetensors",
"model.layers.19.mamba.D": "model-00003-of-00004.safetensors",
"model.layers.19.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
"model.layers.19.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mamba.dt_bias": "model-00003-of-00004.safetensors",
"model.layers.19.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mamba.norm.weight": "model-00003-of-00004.safetensors",
"model.layers.19.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.19.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.2.feed_forward.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.feed_forward.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.feed_forward.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mamba.A_log": "model-00001-of-00004.safetensors",
"model.layers.2.mamba.D": "model-00001-of-00004.safetensors",
"model.layers.2.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
"model.layers.2.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mamba.dt_bias": "model-00001-of-00004.safetensors",
"model.layers.2.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mamba.norm.weight": "model-00001-of-00004.safetensors",
"model.layers.2.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.2.pre_ff_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.20.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mamba.A_log": "model-00003-of-00004.safetensors",
"model.layers.20.mamba.D": "model-00003-of-00004.safetensors",
"model.layers.20.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
"model.layers.20.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mamba.dt_bias": "model-00003-of-00004.safetensors",
"model.layers.20.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mamba.norm.weight": "model-00003-of-00004.safetensors",
"model.layers.20.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.20.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.21.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mamba.A_log": "model-00003-of-00004.safetensors",
"model.layers.21.mamba.D": "model-00003-of-00004.safetensors",
"model.layers.21.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
"model.layers.21.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mamba.dt_bias": "model-00003-of-00004.safetensors",
"model.layers.21.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mamba.norm.weight": "model-00003-of-00004.safetensors",
"model.layers.21.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.21.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.22.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mamba.A_log": "model-00003-of-00004.safetensors",
"model.layers.22.mamba.D": "model-00003-of-00004.safetensors",
"model.layers.22.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
"model.layers.22.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mamba.dt_bias": "model-00003-of-00004.safetensors",
"model.layers.22.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mamba.norm.weight": "model-00003-of-00004.safetensors",
"model.layers.22.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.22.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.23.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mamba.A_log": "model-00003-of-00004.safetensors",
"model.layers.23.mamba.D": "model-00003-of-00004.safetensors",
"model.layers.23.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
"model.layers.23.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mamba.dt_bias": "model-00003-of-00004.safetensors",
"model.layers.23.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mamba.norm.weight": "model-00003-of-00004.safetensors",
"model.layers.23.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.23.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.24.feed_forward.down_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.feed_forward.gate_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.feed_forward.up_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mamba.A_log": "model-00003-of-00004.safetensors",
"model.layers.24.mamba.D": "model-00003-of-00004.safetensors",
"model.layers.24.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
"model.layers.24.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mamba.dt_bias": "model-00003-of-00004.safetensors",
"model.layers.24.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mamba.norm.weight": "model-00003-of-00004.safetensors",
"model.layers.24.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
"model.layers.24.pre_ff_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.25.feed_forward.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.25.feed_forward.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.25.feed_forward.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
"model.layers.25.mamba.A_log": "model-00003-of-00004.safetensors",
"model.layers.25.mamba.D": "model-00003-of-00004.safetensors",
"model.layers.25.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
"model.layers.25.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
"model.layers.25.mamba.dt_bias": "model-00003-of-00004.safetensors",
"model.layers.25.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.25.mamba.norm.weight": "model-00004-of-00004.safetensors",
"model.layers.25.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.25.pre_ff_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.26.feed_forward.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.feed_forward.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.feed_forward.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.26.mamba.A_log": "model-00004-of-00004.safetensors",
"model.layers.26.mamba.D": "model-00004-of-00004.safetensors",
"model.layers.26.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
"model.layers.26.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
"model.layers.26.mamba.dt_bias": "model-00004-of-00004.safetensors",
"model.layers.26.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.mamba.norm.weight": "model-00004-of-00004.safetensors",
"model.layers.26.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.26.pre_ff_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.27.feed_forward.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.feed_forward.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.feed_forward.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.27.pre_ff_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.feed_forward.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.feed_forward.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.feed_forward.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.28.mamba.A_log": "model-00004-of-00004.safetensors",
"model.layers.28.mamba.D": "model-00004-of-00004.safetensors",
"model.layers.28.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
"model.layers.28.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
"model.layers.28.mamba.dt_bias": "model-00004-of-00004.safetensors",
"model.layers.28.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.mamba.norm.weight": "model-00004-of-00004.safetensors",
"model.layers.28.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.28.pre_ff_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.29.feed_forward.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.feed_forward.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.feed_forward.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.29.mamba.A_log": "model-00004-of-00004.safetensors",
"model.layers.29.mamba.D": "model-00004-of-00004.safetensors",
"model.layers.29.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
"model.layers.29.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
"model.layers.29.mamba.dt_bias": "model-00004-of-00004.safetensors",
"model.layers.29.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.mamba.norm.weight": "model-00004-of-00004.safetensors",
"model.layers.29.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.29.pre_ff_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.3.feed_forward.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.feed_forward.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.feed_forward.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mamba.A_log": "model-00001-of-00004.safetensors",
"model.layers.3.mamba.D": "model-00001-of-00004.safetensors",
"model.layers.3.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
"model.layers.3.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mamba.dt_bias": "model-00001-of-00004.safetensors",
"model.layers.3.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mamba.norm.weight": "model-00001-of-00004.safetensors",
"model.layers.3.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.3.pre_ff_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.30.feed_forward.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.feed_forward.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.feed_forward.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.30.mamba.A_log": "model-00004-of-00004.safetensors",
"model.layers.30.mamba.D": "model-00004-of-00004.safetensors",
"model.layers.30.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
"model.layers.30.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
"model.layers.30.mamba.dt_bias": "model-00004-of-00004.safetensors",
"model.layers.30.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.mamba.norm.weight": "model-00004-of-00004.safetensors",
"model.layers.30.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.30.pre_ff_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.31.feed_forward.down_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.feed_forward.gate_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.feed_forward.up_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.input_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.31.mamba.A_log": "model-00004-of-00004.safetensors",
"model.layers.31.mamba.D": "model-00004-of-00004.safetensors",
"model.layers.31.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
"model.layers.31.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
"model.layers.31.mamba.dt_bias": "model-00004-of-00004.safetensors",
"model.layers.31.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.mamba.norm.weight": "model-00004-of-00004.safetensors",
"model.layers.31.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
"model.layers.31.pre_ff_layernorm.weight": "model-00004-of-00004.safetensors",
"model.layers.4.feed_forward.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.feed_forward.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.feed_forward.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mamba.A_log": "model-00001-of-00004.safetensors",
"model.layers.4.mamba.D": "model-00001-of-00004.safetensors",
"model.layers.4.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
"model.layers.4.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mamba.dt_bias": "model-00001-of-00004.safetensors",
"model.layers.4.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mamba.norm.weight": "model-00001-of-00004.safetensors",
"model.layers.4.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.4.pre_ff_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.5.feed_forward.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.feed_forward.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.feed_forward.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mamba.A_log": "model-00001-of-00004.safetensors",
"model.layers.5.mamba.D": "model-00001-of-00004.safetensors",
"model.layers.5.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
"model.layers.5.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mamba.dt_bias": "model-00001-of-00004.safetensors",
"model.layers.5.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mamba.norm.weight": "model-00001-of-00004.safetensors",
"model.layers.5.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.5.pre_ff_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.6.feed_forward.down_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.feed_forward.gate_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.feed_forward.up_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mamba.A_log": "model-00001-of-00004.safetensors",
"model.layers.6.mamba.D": "model-00001-of-00004.safetensors",
"model.layers.6.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
"model.layers.6.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mamba.dt_bias": "model-00001-of-00004.safetensors",
"model.layers.6.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mamba.norm.weight": "model-00001-of-00004.safetensors",
"model.layers.6.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
"model.layers.6.pre_ff_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.7.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.7.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.7.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
"model.layers.7.mamba.A_log": "model-00001-of-00004.safetensors",
"model.layers.7.mamba.D": "model-00001-of-00004.safetensors",
"model.layers.7.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
"model.layers.7.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
"model.layers.7.mamba.dt_bias": "model-00001-of-00004.safetensors",
"model.layers.7.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.7.mamba.norm.weight": "model-00002-of-00004.safetensors",
"model.layers.7.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.7.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.8.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mamba.A_log": "model-00002-of-00004.safetensors",
"model.layers.8.mamba.D": "model-00002-of-00004.safetensors",
"model.layers.8.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
"model.layers.8.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mamba.dt_bias": "model-00002-of-00004.safetensors",
"model.layers.8.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mamba.norm.weight": "model-00002-of-00004.safetensors",
"model.layers.8.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.8.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.9.feed_forward.down_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.feed_forward.gate_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.feed_forward.up_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.9.pre_ff_layernorm.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors"
}
}

16
special_tokens_map.json Normal file
View File

@@ -0,0 +1,16 @@
{
"bos_token": {
"content": "<|begin_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|end_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

410562
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

2061
tokenizer_config.json Normal file
View File

File diff suppressed because it is too large Load Diff