初始化项目，由ModelHub XC社区提供模型
Model: sKT-Ai-Labs/SKT-ST-X-0-3B Source: Original Platform
This commit is contained in:
ModelHub XC
37
.gitattributes
vendored
Normal file
37
.gitattributes
vendored
Normal file
@@ -0,0 +1,37 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
Webp/IMG_20260604_001901.png filter=lfs diff=lfs merge=lfs -text
|
||||
Webp/img.png filter=lfs diff=lfs merge=lfs -text
|
||||
43
Index/THIRD_PARTY_NOTICES.MD
Normal file
43
Index/THIRD_PARTY_NOTICES.MD
Normal file
@@ -0,0 +1,43 @@
|
||||
This file lists third-party software contained in SKT-ST-X-0-3B along with their licenses, in compliance with the redistribution clauses of those licenses.
|
||||
|
||||
|
||||
1. TinyLlama
|
||||
|
||||
Some components, tokenizers, or pre-training setups are adapted from the TinyLlama project.
|
||||
|
||||
Source Repository
|
||||
https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||||
|
||||
License Type
|
||||
Apache 2.0
|
||||
|
||||
Copyright Notice
|
||||
Copyright (c) 2023 Zhang Jianguo
|
||||
|
||||
Full License Text
|
||||
MIT License Copyright (c) 2023 Zhang Jianguo Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
|
||||
---
|
||||
|
||||
2. Mixtral (Mistral AI)
|
||||
|
||||
Our model architecture incorporates Mixture-of-Experts (MoE) or specific layers inspired by or adapted from Mixtral.
|
||||
|
||||
Source Repository
|
||||
https://github.com/mistralai/mistral-src
|
||||
|
||||
Files / Directories Used
|
||||
Config.json
|
||||
|
||||
License Type
|
||||
Apache License 2.0
|
||||
|
||||
Copyright Notice
|
||||
Copyright (c) 2023 Mistral AI
|
||||
|
||||
Full License Text
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/6.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
|
||||
41
Index/USED.MD
Normal file
41
Index/USED.MD
Normal file
@@ -0,0 +1,41 @@
|
||||
|
||||
# Open-Source Attribution & Architecture Compliance Notice
|
||||
|
||||
This project utilizes a multi-model architecture incorporating upstream open-source technologies. This document serves as a formal acknowledgment of the respective creators and ensures compliance with the open-source licensing terms governing each component.
|
||||
|
||||
## 1. Model Architecture & Attributions
|
||||
|
||||
### Primary Framework Model
|
||||
* **Model Name:** SKT-ST-X-0-3B
|
||||
* **Developer/Provider:** SKT-Ai-Labs
|
||||
* **Repository:** SKT-Ai-Labs/SKT-ST-X-0-3B
|
||||
* **Role:** Serves as the primary model framework for core processing and structural generation.
|
||||
*
|
||||
### Integrated Knowledge Expert
|
||||
* **Model Name:** TinyLlama-1.1B-Chat-v1.0
|
||||
* **Developer/Provider:** TinyLlama Community (via Hugging Face)
|
||||
* **Repository:** TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||||
* **Role:** Integrated natively as a specialized factual knowledge expert to deliver contextual, accurate, and optimized data processing.
|
||||
*
|
||||
## 2. Apache License 2.0 Compliance Statement
|
||||
The integration of the **TinyLlama-1.1B-Chat-v1.0** model and its community assets complies fully with the **Apache License, Version 2.0 (January 2004)**. Pursuant to Section 4 of the License, the following operational conditions are met:
|
||||
* **Retention of System Notices:** All original copyright, patent, trademark, and attribution notices from the source form of the TinyLlama repository are preserved intact within this distribution.
|
||||
* **Architecture Modification Notice:** This notice explicitly documents the structural configuration wherein TinyLlama is utilized as a secondary expert subsystem alongside SKT-ST-X-0-3B.
|
||||
* **Disclaimer of Warranty:** In alignment with Sections 7 and 8 of the Apache 2.0 License, the integrated components are provided and utilized on an **"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND**, either express or implied.
|
||||
## 3. License Boilerplate Notice
|
||||
|
||||
Copyright 2026 SKT AI lABS
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
```
|
||||
222
README.md
Normal file
222
README.md
Normal file
@@ -0,0 +1,222 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- hi
|
||||
tags:
|
||||
- moe
|
||||
- slm
|
||||
- skt-ai-labs
|
||||
- 3b-model
|
||||
- mixture-of-experts
|
||||
- bilingual
|
||||
library_name: pytorch
|
||||
pipeline_tag: text-generation
|
||||
datasets:
|
||||
- SKT-NRS/SKT-OMNI-CORPUS-2T
|
||||
model-index:
|
||||
- name: SKT-ST-X-0-3B-V1
|
||||
results:
|
||||
- task: {type: Classification}
|
||||
dataset: {type: mteb/mtop_domain, name: MTEB MTOPDomainClassification (en)}
|
||||
metrics: [{type: accuracy, value: 70.95}]
|
||||
- task: {type: Classification}
|
||||
dataset: {type: mteb/amazon_polarity, name: MTEB AmazonPolarityClassification}
|
||||
metrics: [{type: accuracy, value: 46.88}]
|
||||
- task: {type: STS}
|
||||
dataset: {type: mteb/biosses-sts, name: MTEB BIOSSES}
|
||||
metrics: [{type: cos_sim_pearson, value: 47.19}]
|
||||
- task: {type: Reranking}
|
||||
dataset: {type: mteb/scidocs-reranking, name: MTEB SciDocsRR}
|
||||
metrics: [{type: mrr, value: 28.33}]
|
||||
- task: {type: Classification}
|
||||
dataset: {type: mteb/tweet_sentiment, name: Tweet Sentiment}
|
||||
metrics: [{type: f1, value: 26.51}]
|
||||
- task: {type: Clustering}
|
||||
dataset: {type: mteb/stackexchange_clustering, name: StackExchange Clustering}
|
||||
metrics: [{type: v_measure, value: 35.55}]
|
||||
---
|
||||
|
||||
<div style="max-width: 900px; margin: 0 auto; font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;">
|
||||
|
||||
### <span style="color:#B8860B; font-size:24px; font-weight:bold;">SKT</span> <span style="color:#333; font-size:24px; font-weight:bold;">AI LABS </span>
|
||||
|
||||
<div style="background: linear-gradient(90deg, #FFD700 0%, #FFAA00 50%, #B8860B 100%); height: 8px; margin: 10px 0 20px 0; border-radius: 4px;"></div>
|
||||
|
||||
# <span style="color:#000; font-size:48px; font-weight:800; letter-spacing:-1px;">SKT-ST-X-0-3B-V1</span>
|
||||
|
||||
<!-- ==================== EPIC GOLDEN BANNER ==================== -->
|
||||
<div style="background: linear-gradient(135deg, #FFD700 0%, #FFCC00 35%, #B8860B 70%, #FFD700 100%);
|
||||
padding: 40px 20px;
|
||||
border-radius: 20px;
|
||||
text-align: center;
|
||||
margin: 30px 0; border: 4px solid #FFFFFF;
|
||||
box-shadow: 0 15px 30px rgba(184, 134, 11, 0.4);">
|
||||
|
||||
<h2 style="color:#000; font-size:36px; font-weight:800; margin:0 0 15px 0; text-transform: uppercase; letter-spacing: 1px;">
|
||||
COMPACT MOE POWERHOUSE
|
||||
</h2>
|
||||
|
||||
<p style="color:#000; font-size:22px; font-weight:600; margin:15px 0;">
|
||||
3B Total Params • 1.1B Active • English & Hindi
|
||||
</p>
|
||||
|
||||
<p style="color:#222; font-size:18px; max-width:800px; margin:0 auto; line-height:1.6;">
|
||||
A highly efficient Small Language Model (SLM) built on Mixtral MoE architecture for stability.
|
||||
Delivers intelligent responses with a tiny footprint.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<p align="center">
|
||||
<img src="Webp/img.png" alt="SKT AI LABS Model Card" width="100%" style="max-width:600px; border-radius: 16px; border: 4px solid #FFD700; box-shadow: 0 10px 25px rgba(0,0,0,0.2);">
|
||||
</p>
|
||||
|
||||
<!-- Badges -->
|
||||
<p align="center">
|
||||
<a href="https://huggingface.co/sKT-Ai-Labs"><img src="https://img.shields.io/badge/🤗--HuggingFace-yellow?style=for-the-badge&logo=huggingface" alt="HF"></a>
|
||||
<a href="https://www.sktailabs.in"><img src="https://img.shields.io/badge/SKT_AI_LABS-green?style=for-the-badge" alt="Website"></a>
|
||||
<a href="https://opensource.org/licenses/Apache-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge" alt="License"></a>
|
||||
</p>
|
||||
|
||||
<!-- Main Content -->
|
||||
<div style="background: linear-gradient(135deg, #FFF9E6 0%, #FFF0C2 100%); padding: 30px; border-radius: 16px; border-left: 6px solid #B8860B; margin: 25px 0;">
|
||||
<h2 style="color:#B8860B; font-size:32px; margin-bottom:15px;">🏗️ Model Architecture</h2>
|
||||
|
||||
<table style="width:100%; border-collapse: collapse; color:#333;">
|
||||
<tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Total Parameters</strong></td><td style="padding:10px;">~3 Billion</td></tr>
|
||||
<tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Active Parameters</strong></td><td style="padding:10px;">~1.1 Billion (2 Experts/Token)</td></tr>
|
||||
<tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Architecture</strong></td><td style="padding:10px;">Mixture of Experts (MoE)</td></tr>
|
||||
<tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Number of Experts</strong></td><td style="padding:10px;">4</td></tr>
|
||||
<tr style="border-bottom: 1px solid #E6C200;"><td style="padding:10px;"><strong>Context Length</strong></td><td style="padding:10px;">8K Tokens</td></tr>
|
||||
<tr><td style="padding:10px;"><strong>Training Data</strong></td><td style="padding:10px;">40B Tokens (SKT-OMNI-CORPUS-2T)</td></tr>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<div style="background: #fff; padding: 25px; border-radius: 16px; border: 2px solid #FFD700; margin: 20px 0;">
|
||||
<h3 style="color:#B8860B; font-size:24px; margin-top:0;">✨ Key Capabilities</h3>
|
||||
<ul style="color:#333; font-size:16px; line-height:2.0;">
|
||||
<li><strong>Bilingual Mastery:</strong> Fluent in both English and Hindi.</li>
|
||||
<li><strong>Efficient Reasoning:</strong> Logical thinking and problem solving despite small size.</li>
|
||||
<li><strong>Basic Coding:</strong> Python scripts, algorithms, and logic debugging.</li>
|
||||
<li><strong>Creative Writing:</strong> Stories, poems, and roleplay with personality.</li>
|
||||
<li><strong>Knowledge QA:</strong> Accurate general knowledge retrieval.</li> </ul>
|
||||
</div>
|
||||
|
||||
<hr style="border: 0; height: 1px; background: #FFD700; margin: 40px 0;">
|
||||
|
||||
## 🛠️ Quick Start Guide
|
||||
|
||||
### Installation
|
||||
<div style="background: linear-gradient(135deg, #FFFDF0 0%, #FFF8DC 100%); padding: 15px; border-radius: 8px; border: 1px solid #E6C200; margin-bottom: 15px;">
|
||||
|
||||
```bash
|
||||
pip install transformers accelerate torch peft bitsandbytes
|
||||
```
|
||||
</div>
|
||||
|
||||
### Basic Inference
|
||||
<div style="background: linear-gradient(135deg, #FFFDF0 0%, #FFF8DC 100%); padding: 15px; border-radius: 8px; border: 1px solid #E6C200;">
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
model_id = "sKT-Ai-Labs/SKT-ST-X-0-3B"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
device_map="auto",
|
||||
torch_dtype=torch.float16
|
||||
)
|
||||
|
||||
prompt = "What is Quantum Physics?"
|
||||
formatted = f"<|user|>\n{prompt}\n<|assistant|>\n"
|
||||
inputs = tokenizer(formatted, return_tensors="pt").to("cuda")
|
||||
|
||||
outputs = model.generate(**inputs, max_new_tokens=100)
|
||||
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
print(response.split("<|assistant|>")[-1].strip())
|
||||
```
|
||||
</div>
|
||||
|
||||
### ⚡ 4-bit Quantization (Low VRAM)
|
||||
<div style="background: linear-gradient(135deg, #FFFDF0 0%, #FFF8DC 100%); padding: 15px; border-radius: 8px; border: 1px solid #E6C200;">
|
||||
|
||||
```python
|
||||
from transformers import BitsAndBytesConfig
|
||||
|
||||
quant_config = BitsAndBytesConfig(
|
||||
load_in_4bit=True,
|
||||
bnb_4bit_compute_dtype=torch.float16)
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
quantization_config=quant_config,
|
||||
device_map="auto"
|
||||
)
|
||||
```
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## 📊 MTEB Benchmark Highlights
|
||||
|
||||
| Task | Dataset | Metric | Score |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **Classification** | MTOP Domain (en) | Accuracy | **70.95** |
|
||||
| **Classification** | Amazon Polarity | Accuracy | **46.88** |
|
||||
| **STS** | BIOSSES | Cosine Pearson | **47.19** |
|
||||
| **Reranking** | SciDocs RR | MRR | **28.33** |
|
||||
| **Classification** | Tweet Sentiment | F1 | **26.51** |
|
||||
| **Clustering** | StackExchange | V-Measure | **35.55** |
|
||||
|
||||
*Full benchmark results available in the model metadata.*
|
||||
|
||||
---
|
||||
## ❤️ Support Our Mission
|
||||
|
||||
<div style="background: linear-gradient(135deg, #FFF9E6 0%, #FFF0C2 100%); padding: 30px; border-radius: 16px; border: 2px solid #FFD700; text-align: center; margin: 40px 0; box-shadow: 0 5px 15px rgba(0,0,0,0.1);">
|
||||
<h3 style="color:#B8860B; font-size:26px; margin-bottom:15px;">🙏 Drop a Heart ❤ For Our Hard Work!</h3>
|
||||
<p style="color:#333; font-size:18px; margin-bottom:20px;">
|
||||
If you believe in the vision of Sovereign Indian AI, please show your support by dropping a heart below. Your encouragement fuels our journey!
|
||||
</p>
|
||||
|
||||
<div style="display: flex; justify-content: center; gap: 20px; flex-wrap: wrap;">
|
||||
<a href="https://huggingface.co/sKT-Ai-Labs" target="_blank" style="background: linear-gradient(#FFD700, #FFAA00); color: #000; padding: 12px 25px; border-radius: 50px; text-decoration: none; font-weight: bold; font-size: 16px; box-shadow: 0 4px 10px rgba(0,0,0,0.2); transition: transform 0.2s;">
|
||||
❤️ Like This Dataset
|
||||
</a>
|
||||
<a href="https://huggingface.co/sKT-Ai-Labs" target="_blank" style="background: linear-gradient(#B8860B, #8B4513); color: #fff; padding: 12px 25px; border-radius: 50px; text-decoration: none; font-weight: bold; font-size: 16px; box-shadow: 0 4px 10px rgba(0,0,0,0.2); transition: transform 0.2s;">
|
||||
➕ Follow SKT AI LABS </a>
|
||||
</div>
|
||||
|
||||
<p style="color:#666; font-size:15px; margin-top:20px; font-style: italic;">
|
||||
Kindly follow us for more updates and contribute to our open-source journey!
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
## 📜 License & Citation
|
||||
|
||||
This model is released under the **Apache-2.0 License**.
|
||||
|
||||
- [View Full License](Index/USED.MD)
|
||||
- [Third Party Notices](Index/THIRD_PARTY_NOTICES.MD)
|
||||
|
||||
```bibtex
|
||||
@misc{SKT-ST-X-0-3B,
|
||||
author = {SKT AI LABS, India},
|
||||
title = {SKT-ST-X-0-3B: A Compact Mixture of Experts Model},
|
||||
year = {2026},
|
||||
publisher = {Hugging Face},
|
||||
url = {https://huggingface.co/sKT-Ai-Labs/SKT-ST-X-0-3B}
|
||||
}
|
||||
```
|
||||
|
||||
<div style="text-align:center; margin-top:50px; padding: 20px; background: #FFF9E6; border-radius: 10px;">
|
||||
<h3 style="color:#B8860B; margin:0;">Made with ❤️ by SKT AI LABS</h3>
|
||||
<p style="color:#555; margin-top:5px;">Support: support@sktailabs.in</p>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
3
Webp/img.png
Normal file
3
Webp/img.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:00a174d0027ad25195f8102b91a5af5a168a6529e2e1811d6842c5094a049221
|
||||
size 1569488
|
||||
15
chat_template.jinja
Normal file
15
chat_template.jinja
Normal file
@@ -0,0 +1,15 @@
|
||||
{% for message in messages %}
|
||||
{% if message['role'] == 'user' %}
|
||||
{{ '<|user|>
|
||||
' + message['content'] + eos_token }}
|
||||
{% elif message['role'] == 'system' %}
|
||||
{{ '<|system|>
|
||||
' + message['content'] + eos_token }}
|
||||
{% elif message['role'] == 'assistant' %}
|
||||
{{ '<|assistant|>
|
||||
' + message['content'] + eos_token }}
|
||||
{% endif %}
|
||||
{% if loop.last and add_generation_prompt %}
|
||||
{{ '<|assistant|>' }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
38
config.json
Normal file
38
config.json
Normal file
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"architectures": [
|
||||
"MixtralForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"dtype": "float32",
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2048,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 5632,
|
||||
"max_position_embeddings": 2048,
|
||||
"mlp_bias": false,
|
||||
"model_type": "mixtral",
|
||||
"num_attention_heads": 32,
|
||||
"num_experts_per_tok": 2,
|
||||
"num_hidden_layers": 22,
|
||||
"num_key_value_heads": 4,
|
||||
"num_local_experts": 4,
|
||||
"output_router_logits": false,
|
||||
"pad_token_id": null,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 10000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"router_aux_loss_coef": 0.001,
|
||||
"router_jitter_noise": 0.0,
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "5.0.0",
|
||||
"use_cache": true,
|
||||
"vocab_size": 32000
|
||||
}
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"transformers_version": "5.0.0",
|
||||
"use_cache": true
|
||||
}
|
||||
3
model-00001-of-00045.safetensors
Normal file
3
model-00001-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:981107c98e39af62e9853c6520d3026d6bbb268c6f3f59d4a2a37bc5b6db026a
|
||||
size 262144128
|
||||
3
model-00002-of-00045.safetensors
Normal file
3
model-00002-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3d3c56cd8646bf31a87e1c381c9995da47c0d1522db7e3fc2595388ad979140c
|
||||
size 446702320
|
||||
3
model-00003-of-00045.safetensors
Normal file
3
model-00003-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6f981bfee2c19739869c52990158e882811661e80ec6f8f729aa3228e0c143a4
|
||||
size 406898480
|
||||
3
model-00004-of-00045.safetensors
Normal file
3
model-00004-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:9af1e448a1179a9f45f98db265663166ffc2b0223ea6c83309e013ed0b3be5e1
|
||||
size 184549912
|
||||
3
model-00005-of-00045.safetensors
Normal file
3
model-00005-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:40f9de5a65b2af4a7ec9fe98928d2cfaad4812e324146bf042a16dfa9d2a39f1
|
||||
size 406898480
|
||||
3
model-00006-of-00045.safetensors
Normal file
3
model-00006-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:877ab2c5570589c26f92c70ea3f5b141e386b4ee315c40746273c5affb1d4fe2
|
||||
size 184549912
|
||||
3
model-00007-of-00045.safetensors
Normal file
3
model-00007-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:dab021db4c3f4be0809ba84690b396f2b0bc99bbe76fc6afbed2d1d32f7c5ee8
|
||||
size 406898480
|
||||
3
model-00008-of-00045.safetensors
Normal file
3
model-00008-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:032ca0022bc5b9baed0d9714f8b8381fdcfe67241deeca1b961f2f564e0b9368
|
||||
size 184549912
|
||||
3
model-00009-of-00045.safetensors
Normal file
3
model-00009-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:40ca90dc17af92dd84f066242d0f6ff737fdbb8318e44f6e1e8999ac5e0c34af
|
||||
size 406898480
|
||||
3
model-00010-of-00045.safetensors
Normal file
3
model-00010-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:10a6aae7356056566afd780df6780d404ecfb608b8a16f2173f10c7a63511ab0
|
||||
size 184549912
|
||||
3
model-00011-of-00045.safetensors
Normal file
3
model-00011-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6c9cf406c8c104fc3eb32710f281c982a4428fd1144317b3a9580356e5d27c7b
|
||||
size 406898480
|
||||
3
model-00012-of-00045.safetensors
Normal file
3
model-00012-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:68b4824637bc4084ee7387d499777c18a8d2f9353b438e2670048f8f2dea933e
|
||||
size 184549912
|
||||
3
model-00013-of-00045.safetensors
Normal file
3
model-00013-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3ceab62a3a4e34f95133aa8bdc8aca28692cf16323b8a757fb508bde36129ec2
|
||||
size 406898480
|
||||
3
model-00014-of-00045.safetensors
Normal file
3
model-00014-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a651ab0ae0a6f66ef738ff38ae94fd876cdbcb28e68b6089076ab3ec1d43bbec
|
||||
size 184549912
|
||||
3
model-00015-of-00045.safetensors
Normal file
3
model-00015-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:5a8347a3d1f42a325b9047647426d2f9b6ed1bdacb23fe024a45a785d09e9fa6
|
||||
size 406898480
|
||||
3
model-00016-of-00045.safetensors
Normal file
3
model-00016-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:41dd27624f3595249ab9370df9b9bc2927d81846a5523dcff5edf5daa5409f6d
|
||||
size 184549912
|
||||
3
model-00017-of-00045.safetensors
Normal file
3
model-00017-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fc35fd3f6b7e3f0b440ca4a217ba6b4584c94b91bbaaf2630260809a10b79c13
|
||||
size 406898480
|
||||
3
model-00018-of-00045.safetensors
Normal file
3
model-00018-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:afbb197b68121817c0f19aa9b2d136447b11cfd93d61783a635fd4a4223c0839
|
||||
size 184549912
|
||||
3
model-00019-of-00045.safetensors
Normal file
3
model-00019-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ae233e7131f173acf31ca073e404d8bc57c26b2ec2453ea220d8d913de95f413
|
||||
size 406898480
|
||||
3
model-00020-of-00045.safetensors
Normal file
3
model-00020-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7b829e690e61a9c7f83c730a2c043409ec46fbe06cc7a7ea55095fed13c6aebb
|
||||
size 184549912
|
||||
3
model-00021-of-00045.safetensors
Normal file
3
model-00021-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:bd40826993382a3148372a92c39387f8c29bc48983fab35195176d86b15ae1a2
|
||||
size 406898472
|
||||
3
model-00022-of-00045.safetensors
Normal file
3
model-00022-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b0dd980d2563f80294a14c96dfbe8ed410d22dc8d03aa89648f38862d4e9580c
|
||||
size 184549920
|
||||
3
model-00023-of-00045.safetensors
Normal file
3
model-00023-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1d821c39fa5df7f7af26491025074a494b8b41ab36c2edaaacead1916aeb98a0
|
||||
size 406898496
|
||||
3
model-00024-of-00045.safetensors
Normal file
3
model-00024-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3487aca9cbaa6cc585696bdf10852bdff97e00d29e59b523cce082c27b95044b
|
||||
size 184549920
|
||||
3
model-00025-of-00045.safetensors
Normal file
3
model-00025-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8eeabbb8e76479054f963ab9718646e39619f3c8260de096311deccd50c5cfb6
|
||||
size 406898496
|
||||
3
model-00026-of-00045.safetensors
Normal file
3
model-00026-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:270285c28ed2d7b12fb9bee818ea17cc7249343ebb55bd08411a32dbaa12a026
|
||||
size 184549920
|
||||
3
model-00027-of-00045.safetensors
Normal file
3
model-00027-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b69c19dc2000171a18aa75a7b1fc11b0f5ae055120c9a47bd5292824bbaf8824
|
||||
size 406898496
|
||||
3
model-00028-of-00045.safetensors
Normal file
3
model-00028-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4c0a2365f203f872375ed19d756282c33b93ff1e6267896dc1c9cf67eb6dae1a
|
||||
size 184549920
|
||||
3
model-00029-of-00045.safetensors
Normal file
3
model-00029-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:67ea43df5fe77ce574b23d677867327efd8a5a1f83a600408841648f93aab7c0
|
||||
size 406898496
|
||||
3
model-00030-of-00045.safetensors
Normal file
3
model-00030-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:84032a0f77a7ad81b243a8b037f5b181445536a3b8ca7da08d7ba1e0d53fc910
|
||||
size 184549920
|
||||
3
model-00031-of-00045.safetensors
Normal file
3
model-00031-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8080f1c4f90e96049ecdc8d8f1cda2018ce36f17fca3ef1dfe4d4a7bb642eaf9
|
||||
size 406898496
|
||||
3
model-00032-of-00045.safetensors
Normal file
3
model-00032-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1f5c57a93efa015d888ff494339fcac208edc1ce29370d302a1d8293ed08c8ee
|
||||
size 184549920
|
||||
3
model-00033-of-00045.safetensors
Normal file
3
model-00033-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1708e87accc64e0682b8a0dc950bd7fdba65ad40e36e68ad7b8be5c221d34f66
|
||||
size 406898496
|
||||
3
model-00034-of-00045.safetensors
Normal file
3
model-00034-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2fdb034242611d62bd0d19a3907f6962be4e0d8172bffedddeae0748fd27a197
|
||||
size 184549920
|
||||
3
model-00035-of-00045.safetensors
Normal file
3
model-00035-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:82bf7ba392d0ab3c4c79964eaef9304e9b4631ccf38f1f9de9bedbdaf9e83cc7
|
||||
size 406898496
|
||||
3
model-00036-of-00045.safetensors
Normal file
3
model-00036-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7585b34d5c058cafec37d0a5d8021a488a1542b78b50aa49e38be0185b324833
|
||||
size 184549920
|
||||
3
model-00037-of-00045.safetensors
Normal file
3
model-00037-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:aacbfec152164f6487ca8d85c8857d984de1b5dbf954fd1ca590d6cf38da9da2
|
||||
size 406898496
|
||||
3
model-00038-of-00045.safetensors
Normal file
3
model-00038-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4dba9f83991dacbc11c2b18d967402661981e7d0f2e6679b6fd6fb09b1d6b9f2
|
||||
size 184549920
|
||||
3
model-00039-of-00045.safetensors
Normal file
3
model-00039-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:fc763223d3f5f9b0c71897e1f5823d7845453cbb7b2f5ae5cbea17f6671e1bd5
|
||||
size 406898496
|
||||
3
model-00040-of-00045.safetensors
Normal file
3
model-00040-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d5bf4e4ca9397ff9c469d8fcf035971f8851a1a93652e68a76a14aceca289368
|
||||
size 184549920
|
||||
3
model-00041-of-00045.safetensors
Normal file
3
model-00041-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:50d890ce1de3fc0e172baddf821e7f94f98baf8f59aaa2651a14f21455ae4c5e
|
||||
size 406898496
|
||||
3
model-00042-of-00045.safetensors
Normal file
3
model-00042-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3cdf0dcff69cffe82d56c6f4fc601e95ca4b5b91f0b255d5855e8935e0fa9a8b
|
||||
size 184549920
|
||||
3
model-00043-of-00045.safetensors
Normal file
3
model-00043-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:057fb4503ba074a878d8cb5aa9ae5cdab3595a76353632a4d9e2cf5a6e98ec3c
|
||||
size 406898496
|
||||
3
model-00044-of-00045.safetensors
Normal file
3
model-00044-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c18ca8a9c082884ba676292453dec4446828a8b154e046f662e7f05f9501cd98
|
||||
size 184549920
|
||||
3
model-00045-of-00045.safetensors
Normal file
3
model-00045-of-00045.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6b5e14506b721acb90605062d60b301b8d1251796db3281126a88aec621c30b6
|
||||
size 406898480
|
||||
429
model.safetensors.index.json
Normal file
429
model.safetensors.index.json
Normal file
@@ -0,0 +1,429 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_parameters": 3384027136,
|
||||
"total_size": 13536108544
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "model-00001-of-00045.safetensors",
|
||||
"model.embed_tokens.weight": "model-00002-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.block_sparse_moe.gate.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00002-of-00045.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.3.w3.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.block_sparse_moe.gate.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00003-of-00045.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.0.w1.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.0.w2.weight": "model-00022-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.0.w3.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.1.w1.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.1.w2.weight": "model-00022-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.1.w3.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.2.w1.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.2.w2.weight": "model-00022-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.2.w3.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.3.w1.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.3.w2.weight": "model-00022-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.3.w3.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.block_sparse_moe.gate.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.0.w1.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.0.w2.weight": "model-00024-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.0.w3.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.1.w1.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.1.w2.weight": "model-00024-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.1.w3.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.2.w1.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.2.w2.weight": "model-00024-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.2.w3.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.3.w1.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.3.w2.weight": "model-00024-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.3.w3.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.block_sparse_moe.gate.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00023-of-00045.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.0.w1.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.0.w2.weight": "model-00026-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.0.w3.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.1.w1.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.1.w2.weight": "model-00026-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.1.w3.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.2.w1.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.2.w2.weight": "model-00026-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.2.w3.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.3.w1.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.3.w2.weight": "model-00026-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.3.w3.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.block_sparse_moe.gate.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00025-of-00045.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.0.w1.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.0.w2.weight": "model-00028-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.0.w3.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.1.w1.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.1.w2.weight": "model-00028-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.1.w3.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.2.w1.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.2.w2.weight": "model-00028-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.2.w3.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.3.w1.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.3.w2.weight": "model-00028-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.3.w3.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.block_sparse_moe.gate.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00027-of-00045.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.0.w1.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.0.w2.weight": "model-00030-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.0.w3.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.1.w1.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.1.w2.weight": "model-00030-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.1.w3.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.2.w1.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.2.w2.weight": "model-00030-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.2.w3.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.3.w1.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.3.w2.weight": "model-00030-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.3.w3.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.block_sparse_moe.gate.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00029-of-00045.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.0.w1.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.0.w2.weight": "model-00032-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.0.w3.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.1.w1.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.1.w2.weight": "model-00032-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.1.w3.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.2.w1.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.2.w2.weight": "model-00032-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.2.w3.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.3.w1.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.3.w2.weight": "model-00032-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.3.w3.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.block_sparse_moe.gate.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00031-of-00045.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.0.w1.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.0.w2.weight": "model-00034-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.0.w3.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.1.w1.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.1.w2.weight": "model-00034-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.1.w3.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.2.w1.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.2.w2.weight": "model-00034-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.2.w3.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.3.w1.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.3.w2.weight": "model-00034-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.3.w3.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.block_sparse_moe.gate.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00033-of-00045.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.0.w1.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.0.w2.weight": "model-00036-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.0.w3.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.1.w1.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.1.w2.weight": "model-00036-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.1.w3.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.2.w1.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.2.w2.weight": "model-00036-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.2.w3.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.3.w1.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.3.w2.weight": "model-00036-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.3.w3.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.block_sparse_moe.gate.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00035-of-00045.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.0.w1.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.0.w2.weight": "model-00038-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.0.w3.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.1.w1.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.1.w2.weight": "model-00038-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.1.w3.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.2.w1.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.2.w2.weight": "model-00038-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.2.w3.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.3.w1.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.3.w2.weight": "model-00038-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.3.w3.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.block_sparse_moe.gate.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00037-of-00045.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.0.w1.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.0.w2.weight": "model-00040-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.0.w3.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.1.w1.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.1.w2.weight": "model-00040-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.1.w3.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.2.w1.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.2.w2.weight": "model-00040-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.2.w3.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.3.w1.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.3.w2.weight": "model-00040-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.3.w3.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.block_sparse_moe.gate.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00039-of-00045.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.0.w1.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.0.w3.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.1.w1.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.1.w2.weight": "model-00006-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.1.w3.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.2.w1.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.2.w2.weight": "model-00006-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.2.w3.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.3.w1.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.3.w3.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.block_sparse_moe.gate.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00005-of-00045.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.0.w1.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.0.w2.weight": "model-00042-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.0.w3.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.1.w1.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.1.w2.weight": "model-00042-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.1.w3.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.2.w1.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.2.w2.weight": "model-00042-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.2.w3.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.3.w1.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.3.w2.weight": "model-00042-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.3.w3.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.block_sparse_moe.gate.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00041-of-00045.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.0.w1.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.0.w2.weight": "model-00044-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.0.w3.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.1.w1.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.1.w2.weight": "model-00044-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.1.w3.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.2.w1.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.2.w2.weight": "model-00044-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.2.w3.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.3.w1.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.3.w2.weight": "model-00044-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.3.w3.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.block_sparse_moe.gate.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00043-of-00045.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00045-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.0.w1.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.0.w2.weight": "model-00008-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.0.w3.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.1.w1.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.1.w2.weight": "model-00008-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.1.w3.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.2.w1.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.2.w2.weight": "model-00008-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.2.w3.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.3.w1.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.3.w2.weight": "model-00008-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.3.w3.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.block_sparse_moe.gate.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00007-of-00045.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.0.w1.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.0.w2.weight": "model-00010-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.0.w3.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.1.w1.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.1.w2.weight": "model-00010-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.1.w3.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.2.w1.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.2.w2.weight": "model-00010-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.2.w3.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.3.w1.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.3.w2.weight": "model-00010-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.3.w3.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.block_sparse_moe.gate.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00009-of-00045.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.0.w1.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.0.w2.weight": "model-00012-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.0.w3.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.1.w1.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.1.w2.weight": "model-00012-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.1.w3.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.2.w1.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.2.w2.weight": "model-00012-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.2.w3.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.3.w1.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.3.w2.weight": "model-00012-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.3.w3.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.block_sparse_moe.gate.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00011-of-00045.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.0.w1.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.0.w2.weight": "model-00014-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.0.w3.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.1.w1.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.1.w2.weight": "model-00014-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.1.w3.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.2.w1.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.2.w2.weight": "model-00014-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.2.w3.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.3.w1.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.3.w2.weight": "model-00014-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.3.w3.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.block_sparse_moe.gate.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00013-of-00045.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.0.w1.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.0.w2.weight": "model-00016-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.0.w3.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.1.w1.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.1.w2.weight": "model-00016-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.1.w3.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.2.w1.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.2.w2.weight": "model-00016-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.2.w3.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.3.w1.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.3.w2.weight": "model-00016-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.3.w3.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.block_sparse_moe.gate.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00015-of-00045.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.0.w1.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.0.w2.weight": "model-00018-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.0.w3.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.1.w1.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.1.w2.weight": "model-00018-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.1.w3.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.2.w1.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.2.w2.weight": "model-00018-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.2.w3.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.3.w1.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.3.w2.weight": "model-00018-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.3.w3.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.block_sparse_moe.gate.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00017-of-00045.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.0.w1.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.0.w2.weight": "model-00020-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.0.w3.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.1.w1.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.1.w2.weight": "model-00020-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.1.w3.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.2.w1.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.2.w2.weight": "model-00020-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.2.w3.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.3.w1.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.3.w2.weight": "model-00020-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.3.w3.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.block_sparse_moe.gate.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00019-of-00045.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00021-of-00045.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00021-of-00045.safetensors",
|
||||
"model.norm.weight": "model-00045-of-00045.safetensors"
|
||||
}
|
||||
}
|
||||
277136
tokenizer.json
Normal file
277136
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
19
tokenizer_config.json
Normal file
19
tokenizer_config.json
Normal file
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"add_prefix_space": null,
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"is_local": false,
|
||||
"local_files_only": false,
|
||||
"max_length": null,
|
||||
"model_max_length": 2048,
|
||||
"pad_to_multiple_of": null,
|
||||
"pad_token": "<s>",
|
||||
"pad_token_type_id": 0,
|
||||
"padding_side": "left",
|
||||
"sp_model_kwargs": {},
|
||||
"tokenizer_class": "TokenizersBackend",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user