初始化项目，由ModelHub XC社区提供模型
Model: mkd-hossain/keural-dpo-5500 Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
289
README.md
Normal file
289
README.md
Normal file
@@ -0,0 +1,289 @@
|
||||
---
|
||||
language:
|
||||
- ko
|
||||
- en
|
||||
license: apache-2.0
|
||||
library_name: transformers
|
||||
tags:
|
||||
- mixtral
|
||||
- moe
|
||||
- korean
|
||||
- bilingual
|
||||
- causal-lm
|
||||
- dpo
|
||||
- rlhf
|
||||
- instruction-tuned
|
||||
- chat
|
||||
base_model: mkd-hossain/keural-sft-18k
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# Keural-DPO-14.83B (checkpoint 5500)
|
||||
|
||||
Keural is a bilingual Korean–English Mixture-of-Experts language model trained **entirely from scratch** — no base model was used.
|
||||
This is the **DPO (Direct Preference Optimization) checkpoint** at step 5,500 (~79% of 1 epoch), aligned from the Keural SFT-18k base using human preference data.
|
||||
|
||||
> This checkpoint is more mature than the 3500-step release. At step 5500 the model has seen ~80% of the full preference dataset, producing noticeably better instruction-following and more consistent language matching compared to the SFT base.
|
||||
|
||||
## Model Details
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Architecture | Mixtral-style MoE (8 experts, top-2 routing) |
|
||||
| Parameters | **14.83B total** / ~7.42B active per token |
|
||||
| Layers | 24 |
|
||||
| Hidden size | 4096 |
|
||||
| Attention heads | 32 (GQA — 8 KV heads) |
|
||||
| KV heads | 8 |
|
||||
| Head dim | 128 |
|
||||
| Expert intermediate size | 5,632 |
|
||||
| Experts | 8 total, top-2 per token |
|
||||
| Context length | 4,096 tokens |
|
||||
| Vocabulary | 131,074 (131,072 SPM + `<|im_start|>` + `<|im_end|>`) |
|
||||
| RoPE theta | 500,000 |
|
||||
| Sliding window | 512 (alternating every other layer) |
|
||||
| Norm | RMSNorm (eps=1e-5) |
|
||||
| Activation | SiLU |
|
||||
| Dtype | bfloat16 |
|
||||
| Languages | Korean (primary), English |
|
||||
|
||||
## Full Training Pipeline
|
||||
|
||||
| Stage | Steps | Tokens | Data | Hardware |
|
||||
|---|---|---|---|---|
|
||||
| Pretraining Stage 1 | 100,000 | ~50B | Korean + English web corpus | 2× H200 SXM |
|
||||
| Pretraining Stage 2 | 120,000 | ~13B | Korean + English web corpus (continued) | 2× H200 SXM |
|
||||
| SFT | 18,000 | 710M | mkd-chanwoo/keural-SFT (1.14M ChatML samples) | 2× H200 SXM |
|
||||
| **DPO (this checkpoint)** | **5,500 / 6,927** | — | keural-dpo-raw (440K preference pairs) | 2× H200 SXM |
|
||||
|
||||
### DPO Training Details
|
||||
|
||||
| Hyperparameter | Value |
|
||||
|---|---|
|
||||
| Algorithm | Direct Preference Optimization (DPO) |
|
||||
| Learning rate | 2e-6 → 2e-7 cosine decay |
|
||||
| Min learning rate | 2e-7 |
|
||||
| LR at step 5500 | ~3.87e-7 |
|
||||
| Warmup steps | 100 |
|
||||
| Beta (KL penalty) | 0.1 |
|
||||
| Batch size per GPU | 2 |
|
||||
| Gradient accumulation | 16 steps |
|
||||
| Effective batch size | 64 (2 × 16 × 2 GPUs) |
|
||||
| Max sequence length | 1,024 tokens |
|
||||
| Optimizer | AdamW (β1=0.9, β2=0.95, ε=1e-8) |
|
||||
| Weight decay | 0.1 |
|
||||
| Gradient clipping | 1.0 |
|
||||
| Total steps (1 epoch) | 6,927 |
|
||||
| Dataset size | 440,627 preference pairs |
|
||||
| Parallelism | FSDP FULL_SHARD (ZeRO-3 equivalent) |
|
||||
| Precision | bfloat16 + gradient checkpointing |
|
||||
| Hardware | 2× NVIDIA H200 SXM (139 GiB each) |
|
||||
| Speed | ~40 seconds/step |
|
||||
|
||||
**DPO loss at step 5500:** ~0.6924 (stable)
|
||||
**Margin at step 5500:** +0.0009 to +0.0018 (consistently positive — model reliably prefers chosen responses)
|
||||
**GradNorm:** 0.20–0.31 (clean, no explosion)
|
||||
|
||||
### SFT Hyperparameters (base checkpoint)
|
||||
|
||||
| Hyperparameter | Value |
|
||||
|---|---|
|
||||
| Learning rate | 1e-5 → 1e-6 cosine decay |
|
||||
| Effective batch size | 64 (4 per GPU × 8 grad accum × 2 GPUs) |
|
||||
| Max sequence length | 4,096 tokens |
|
||||
| Weight decay | 0.05 |
|
||||
| Steps | 18,000 |
|
||||
| Dataset | mkd-chanwoo/keural-SFT (1.14M samples) |
|
||||
|
||||
## Chat Format (ChatML)
|
||||
|
||||
This model uses **ChatML** format. You **must** use this exact format for good results.
|
||||
|
||||
```
|
||||
<|im_start|>system
|
||||
You are a helpful bilingual Korean-English assistant. Always respond in the same language as the user.<|im_end|>
|
||||
<|im_start|>user
|
||||
안녕하세요! 오늘 날씨가 어때요?<|im_end|>
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
The model generates until it produces `<|im_end|>` (token ID 131073).
|
||||
|
||||
> **Important:** Always include a system prompt. Without it, the model may default to Korean regardless of input language.
|
||||
|
||||
## How to Use
|
||||
|
||||
### With `transformers`
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
import torch
|
||||
|
||||
model_id = "mkd-hossain/keural-dpo-5500"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": (
|
||||
"You are a helpful bilingual Korean-English assistant. "
|
||||
"Always respond in the same language as the user's message. "
|
||||
"If the user writes in English, respond in English. "
|
||||
"If the user writes in Korean, respond in Korean."
|
||||
)
|
||||
},
|
||||
{"role": "user", "content": "파이썬에서 리스트를 정렬하는 방법을 알려주세요."},
|
||||
]
|
||||
|
||||
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
output = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=512,
|
||||
temperature=0.7,
|
||||
top_p=0.9,
|
||||
top_k=50,
|
||||
repetition_penalty=1.1,
|
||||
no_repeat_ngram_size=8,
|
||||
do_sample=True,
|
||||
eos_token_id=131073, # <|im_end|>
|
||||
)
|
||||
|
||||
response = tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=False)
|
||||
response = response.split("<|im_end|>")[0].strip()
|
||||
print(response)
|
||||
```
|
||||
|
||||
### With vLLM (recommended for serving)
|
||||
|
||||
```bash
|
||||
pip install vllm
|
||||
|
||||
python -m vllm.entrypoints.openai.api_server \
|
||||
--model mkd-hossain/keural-dpo-5500 \
|
||||
--tokenizer mkd-hossain/keural-dpo-5500 \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 4096 \
|
||||
--tensor-parallel-size 1
|
||||
```
|
||||
|
||||
Call the OpenAI-compatible endpoint:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="mkd-hossain/keural-dpo-5500",
|
||||
messages=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are a helpful bilingual assistant. Respond in the same language as the user."
|
||||
},
|
||||
{"role": "user", "content": "What is the capital of South Korea?"},
|
||||
],
|
||||
max_tokens=512,
|
||||
temperature=0.7,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Multi-GPU serving (2× GPU)
|
||||
|
||||
```bash
|
||||
python -m vllm.entrypoints.openai.api_server \
|
||||
--model mkd-hossain/keural-dpo-5500 \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 4096 \
|
||||
--tensor-parallel-size 2
|
||||
```
|
||||
|
||||
### Manual ChatML prompt (without `apply_chat_template`)
|
||||
|
||||
```python
|
||||
prompt = (
|
||||
"<|im_start|>system\n"
|
||||
"You are a helpful bilingual Korean-English assistant. "
|
||||
"Always respond in the same language as the user.\n"
|
||||
"<|im_end|>\n"
|
||||
"<|im_start|>user\n"
|
||||
"Tell me about Seoul.<|im_end|>\n"
|
||||
"<|im_start|>assistant\n"
|
||||
)
|
||||
```
|
||||
|
||||
## Special Tokens
|
||||
|
||||
| Token | ID | Purpose |
|
||||
|---|---|---|
|
||||
| `<|im_start|>` | 131072 | Marks the start of each conversation turn |
|
||||
| `<|im_end|>` | 131073 | Marks the end of each turn / generation stop token |
|
||||
| `<bos>` | 1 | Beginning of sequence |
|
||||
| `<eos>` | 2 | End of sequence |
|
||||
| `<pad>` | 0 | Padding token |
|
||||
|
||||
> **Critical:** Always set `eos_token_id=131073` (`<|im_end|>`) when generating. Using `eos_token_id=2` will cause generation to not stop correctly.
|
||||
|
||||
## Recommended Generation Settings
|
||||
|
||||
```python
|
||||
# For conversational / creative tasks
|
||||
generation_config = {
|
||||
"max_new_tokens": 512,
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.9,
|
||||
"top_k": 50,
|
||||
"repetition_penalty": 1.1,
|
||||
"no_repeat_ngram_size": 8,
|
||||
"do_sample": True,
|
||||
"eos_token_id": 131073,
|
||||
}
|
||||
|
||||
# For factual / deterministic tasks
|
||||
generation_config = {
|
||||
"max_new_tokens": 512,
|
||||
"temperature": 0.1,
|
||||
"repetition_penalty": 1.1,
|
||||
"no_repeat_ngram_size": 8,
|
||||
"do_sample": False,
|
||||
"eos_token_id": 131073,
|
||||
}
|
||||
```
|
||||
|
||||
## DPO Dataset
|
||||
|
||||
Training used the `keural-dpo-raw` dataset — 440,627 chosen/rejected preference pairs in ChatML format, covering:
|
||||
- General conversation (Korean and English)
|
||||
- Question answering
|
||||
- Instruction following
|
||||
- Knowledge and reasoning tasks
|
||||
|
||||
## Comparison to Previous Checkpoints
|
||||
|
||||
| Checkpoint | Stage | Key Difference |
|
||||
|---|---|---|
|
||||
| mkd-hossain/keural-pretrained | Pretraining (120k steps) | Raw base model, no instruction tuning |
|
||||
| mkd-hossain/keural-sft-18k | SFT (18k steps) | Instruction following, ChatML format |
|
||||
| mkd-hossain/keural-dpo-3500 | DPO 50% | Early alignment, margins emerging |
|
||||
| **mkd-hossain/keural-dpo-5500** | **DPO 79%** | **Stronger alignment, consistent margins** |
|
||||
|
||||
## Limitations
|
||||
|
||||
- This is a **late-training checkpoint** (step 5,500 of 6,927 — 79% of 1 epoch). A full-epoch checkpoint will be released when training completes.
|
||||
- Maximum context is 4,096 tokens. Inputs longer than this will be truncated.
|
||||
- The pretraining corpus is Korean-dominant. Always include a system prompt for correct bilingual behavior.
|
||||
- Not safety-aligned — do not deploy in production without additional safety fine-tuning.
|
||||
- DPO margins are small (0.001–0.002) due to the large model size and low LR — this is normal for 14B+ models.
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
32
config.json
Normal file
32
config.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"architectures": [
|
||||
"KeuralMoECausalLM"
|
||||
],
|
||||
"model_type": "keural",
|
||||
"vocab_size": 131074,
|
||||
"hidden_size": 4096,
|
||||
"intermediate_size": 5632,
|
||||
"num_hidden_layers": 24,
|
||||
"num_attention_heads": 32,
|
||||
"num_key_value_heads": 8,
|
||||
"head_dim": 128,
|
||||
"num_local_experts": 8,
|
||||
"num_experts_per_tok": 2,
|
||||
"max_position_embeddings": 4096,
|
||||
"rope_theta": 500000.0,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"sliding_window": 512,
|
||||
"hidden_act": "silu",
|
||||
"initializer_range": 0.02,
|
||||
"use_cache": true,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.40.0",
|
||||
"output_router_logits": false,
|
||||
"router_aux_loss_coef": 0.001,
|
||||
"keural_step": 5500,
|
||||
"auto_map": {
|
||||
"AutoConfig": "configuration_keural.KeuralConfig",
|
||||
"AutoModelForCausalLM": "modeling_keural.KeuralMoECausalLM"
|
||||
}
|
||||
}
|
||||
13
configuration_keural.py
Normal file
13
configuration_keural.py
Normal file
@@ -0,0 +1,13 @@
|
||||
"""
|
||||
KeuralConfig — custom configuration class for Keural MoE.
|
||||
Registered as model_type "keural" so HuggingFace AutoConfig can resolve it.
|
||||
"""
|
||||
|
||||
from transformers import MixtralConfig
|
||||
|
||||
|
||||
class KeuralConfig(MixtralConfig):
|
||||
model_type = "keural"
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"pad_token_id": 0,
|
||||
"max_new_tokens": 512,
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.9,
|
||||
"top_k": 50,
|
||||
"repetition_penalty": 1.1,
|
||||
"do_sample": true,
|
||||
"transformers_version": "4.40.0"
|
||||
}
|
||||
3
model-00001-of-00007.safetensors
Normal file
3
model-00001-of-00007.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3104e300cda5d354f48b7d32a8200f7056895ad3c92747233983440f18493faf
|
||||
size 4823806664
|
||||
3
model-00002-of-00007.safetensors
Normal file
3
model-00002-of-00007.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:f6be94acc7bd003cb9aa02bccf41d35e8c6d67a8bf8145eb772e1f81b061b3bd
|
||||
size 4811210208
|
||||
3
model-00003-of-00007.safetensors
Normal file
3
model-00003-of-00007.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:16bd6e372b41ee489cf4e01b0e5d0e7408f32a1955ee2c2bed203dfe03721a6d
|
||||
size 4811210248
|
||||
3
model-00004-of-00007.safetensors
Normal file
3
model-00004-of-00007.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:0b01c279ab30b9c425f2ffb9c487147ff05cc4abb82c6dbedafa26045da46b4e
|
||||
size 4811210336
|
||||
3
model-00005-of-00007.safetensors
Normal file
3
model-00005-of-00007.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1d5d4721eece97213f5a836e6bc8fd55c8d4e2ef7a2e3eccb4537bc7e2809574
|
||||
size 4811210336
|
||||
3
model-00006-of-00007.safetensors
Normal file
3
model-00006-of-00007.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7e07978a3a9ec4395e2fe3b7ab41426d8918bbb144115ead1c6a3c511e2a0b47
|
||||
size 4811210336
|
||||
3
model-00007-of-00007.safetensors
Normal file
3
model-00007-of-00007.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:0d0117b94f5b5d9f857e899b75743a0ee2bf5486640421b5d3f1d7909654cc33
|
||||
size 1858103664
|
||||
754
model.safetensors.index.json
Normal file
754
model.safetensors.index.json
Normal file
@@ -0,0 +1,754 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 30737866752
|
||||
},
|
||||
"weight_map": {
|
||||
"model.embed_tokens.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.gate.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.0.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.gate.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.1.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.gate.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.0.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.1.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.1.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.1.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.2.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.2.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.2.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.3.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.3.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.3.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.4.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.4.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.4.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.5.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.5.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.5.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.6.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.6.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.6.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.7.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.7.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.2.block_sparse_moe.experts.7.w2.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.gate.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.0.w1.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.0.w3.weight": "model-00001-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.3.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.gate.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.4.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.gate.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.5.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.gate.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.1.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.1.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.1.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.2.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.2.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.2.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.3.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.3.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.3.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.4.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.4.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.4.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.5.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.5.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.5.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.6.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.6.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.6.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.7.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.7.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.6.block_sparse_moe.experts.7.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.gate.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.0.w1.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.0.w3.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.0.w2.weight": "model-00002-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.7.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.gate.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.8.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.gate.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.9.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.gate.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.1.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.1.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.2.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.2.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.2.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.3.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.3.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.3.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.4.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.4.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.4.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.5.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.5.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.5.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.6.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.6.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.6.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.7.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.7.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.10.block_sparse_moe.experts.7.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.gate.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.0.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.0.w3.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.0.w2.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.1.w1.weight": "model-00003-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.2.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.2.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.3.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.5.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.6.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.6.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.6.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.7.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.7.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.11.block_sparse_moe.experts.7.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.gate.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.2.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.2.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.3.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.5.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.6.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.6.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.6.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.7.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.7.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.12.block_sparse_moe.experts.7.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.gate.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.2.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.2.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.3.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.5.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.6.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.6.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.6.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.7.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.7.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.13.block_sparse_moe.experts.7.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.gate.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.1.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.2.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.2.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.2.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.3.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.3.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.3.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.4.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.4.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.4.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.5.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.5.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.5.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.6.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.6.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.6.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.7.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.7.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.14.block_sparse_moe.experts.7.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.gate.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.0.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.0.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.0.w2.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.1.w1.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.1.w3.weight": "model-00004-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.3.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.3.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.4.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.4.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.4.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.5.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.5.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.15.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.gate.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.3.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.3.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.4.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.4.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.4.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.5.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.5.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.16.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.gate.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.3.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.3.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.4.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.4.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.4.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.5.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.5.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.17.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.gate.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.2.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.2.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.2.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.3.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.3.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.3.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.4.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.4.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.4.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.5.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.5.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.5.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.6.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.6.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.6.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.7.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.7.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.18.block_sparse_moe.experts.7.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.gate.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.0.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.0.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.0.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.1.w1.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.1.w3.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.1.w2.weight": "model-00005-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.2.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.2.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.2.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.3.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.7.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.19.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.gate.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.0.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.1.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.1.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.1.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.2.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.2.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.2.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.3.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.7.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.20.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.gate.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.0.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.1.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.1.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.1.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.2.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.2.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.2.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.3.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.7.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.21.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.gate.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.0.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.1.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.1.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.1.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.2.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.2.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.2.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.3.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.3.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.3.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.4.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.4.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.4.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.5.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.5.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.5.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.6.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.6.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.6.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.7.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.7.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.22.block_sparse_moe.experts.7.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.gate.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.0.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.0.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.0.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.1.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.1.w3.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.1.w2.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.2.w1.weight": "model-00006-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.2.w3.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.2.w2.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.3.w1.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.3.w3.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.3.w2.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.4.w1.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.4.w3.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.4.w2.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.5.w1.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.5.w3.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.5.w2.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.6.w1.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.6.w3.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.6.w2.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.7.w1.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.7.w3.weight": "model-00007-of-00007.safetensors",
|
||||
"model.layers.23.block_sparse_moe.experts.7.w2.weight": "model-00007-of-00007.safetensors",
|
||||
"model.norm.weight": "model-00007-of-00007.safetensors",
|
||||
"lm_head.weight": "model-00007-of-00007.safetensors"
|
||||
}
|
||||
}
|
||||
43
modeling_keural.py
Normal file
43
modeling_keural.py
Normal file
@@ -0,0 +1,43 @@
|
||||
"""
|
||||
KeuralMoECausalLM — Keural Mixture-of-Experts Causal Language Model.
|
||||
|
||||
Bilingual Korean-English MoE LLM trained entirely from scratch.
|
||||
Developed by MKD Corp AI Research, Republic of Korea.
|
||||
|
||||
Architecture:
|
||||
- 14.83B total parameters (~7.42B active per token)
|
||||
- 24 layers, hidden=4096, GQA 32/8 heads
|
||||
- 8 experts per layer, top-2 routing
|
||||
- Sliding window attention (512, alternating layers)
|
||||
- RoPE theta=500,000, context length=4096
|
||||
- Vocabulary: 131,074 tokens (131,072 SPM + <|im_start|> + <|im_end|>)
|
||||
|
||||
Load with trust_remote_code=True:
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
"mkd-hossain/keural-sft3-final",
|
||||
torch_dtype="bfloat16",
|
||||
device_map="auto",
|
||||
trust_remote_code=True,
|
||||
)
|
||||
"""
|
||||
|
||||
from transformers import MixtralForCausalLM
|
||||
from transformers.utils import logging
|
||||
|
||||
try:
|
||||
from configuration_keural import KeuralConfig
|
||||
except ImportError:
|
||||
from .configuration_keural import KeuralConfig
|
||||
|
||||
logger = logging.get_logger(__name__)
|
||||
|
||||
|
||||
class KeuralMoECausalLM(MixtralForCausalLM):
|
||||
"""
|
||||
Keural MoE Causal Language Model.
|
||||
Bilingual Korean-English 14.83B MoE LLM trained from scratch by MKD Corp AI Research.
|
||||
"""
|
||||
|
||||
config_class = KeuralConfig
|
||||
_keys_to_ignore_on_load_missing = ["lm_head.weight"]
|
||||
30
special_tokens_map.json
Normal file
30
special_tokens_map.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<bos>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|im_end|>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"single_word": false,
|
||||
"lstrip": false,
|
||||
"rstrip": false,
|
||||
"normalized": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.model
Normal file
3
tokenizer.model
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b982818ea2f2057ba791e2006d17683799f1d8ceb9c91322018a638c4ec4b170
|
||||
size 2657284
|
||||
16
tokenizer_config.json
Normal file
16
tokenizer_config.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"bos_token": "<bos>",
|
||||
"eos_token": "<|im_end|>",
|
||||
"unk_token": "<unk>",
|
||||
"pad_token": "<pad>",
|
||||
"model_max_length": 4096,
|
||||
"tokenizer_class": "LlamaTokenizer",
|
||||
"sp_model_kwargs": {},
|
||||
"legacy": false,
|
||||
"chat_template": "{% for message in messages %}<|im_start|>{{ message['role'] }}\n{{ message['content'] }}<|im_end|>\n{% endfor %}<|im_start|>assistant\n",
|
||||
"added_tokens_decoder": {
|
||||
"131072": {"content": "<|im_start|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
|
||||
"131073": {"content": "<|im_end|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false}
|
||||
},
|
||||
"additional_special_tokens": ["<|im_start|>", "<|im_end|>"]
|
||||
}
|
||||
Reference in New Issue
Block a user