hkust-nlp/drkernel-14b
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: hkust-nlp/drkernel-14b
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-04 05:07:18 +08:00
commit 60a8cfe820
19 changed files with 152776 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

316
README.md Normal file
View File

@@ -0,0 +1,316 @@
---
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen3-14B-Base
tags:
- qwen3
- triton
- kernel-generation
- reinforcement-learning
- code
datasets:
- hkust-nlp/drkernel-coldstart-8k
- hkust-nlp/drkernel-rl-data
- hkust-nlp/drkernel-validation-data
---
# DR.Kernel-14B
[![Model](https://img.shields.io/badge/🤗%20Model-hkust--nlp/drkernel--14b-yellow)](https://huggingface.co/hkust-nlp/drkernel-14b)
[![Paper](https://img.shields.io/badge/arXiv-2602.05885-b31b1b)](https://arxiv.org/abs/2602.05885)
`hkust-nlp/drkernel-14b` is a Qwen3-14B-based model specialized for GPU kernel generation and optimization (especially Triton) in the DR.Kernel framework.
It is trained for iterative optimization with execution feedback from KernelGYM, rather than single-shot code generation only.
## Model Summary
- Model type: `Qwen3ForCausalLM`
- Parameter count: `14,768,307,200` (from `model.safetensors.index.json`)
- Weight dtype: BF16
- Base model family: Qwen3-14B
- Main capability: generate and iteratively refine optimized `ModelNew` kernel implementations from PyTorch reference tasks
## Model Specs
From `config.json`:
- `num_hidden_layers`: 40
- `hidden_size`: 5120
- `intermediate_size`: 17408
- `num_attention_heads`: 40
- `num_key_value_heads`: 8
- `max_position_embeddings`: 32768
- `vocab_size`: 151936
- `transformers_version`: 4.56.0
From `model.safetensors.index.json`:
- `total_parameters`: 14,768,307,200
- `total_size`: 29,536,614,400 bytes
- Expected checkpoint shards: `model-00001-of-00007.safetensors` ... `model-00007-of-00007.safetensors`
## Training Recipe (DR.Kernel)
The 14B model follows the same two-stage DR.Kernel pipeline:
1. Cold-start SFT
- Dataset: `hkust-nlp/drkernel-coldstart-8k`
- Multi-turn trajectory warm-up for kernel generation/refinement
2. Multi-turn RL
- Train dataset: `hkust-nlp/drkernel-rl-data`
- Validation dataset: `hkust-nlp/drkernel-validation-data` (KernelBench Level 2 validation split)
- Core methods: TRLOO + MRS + PR + PRS
- Execution environment: KernelGYM with compilation/correctness/performance/profiling feedback
Related training scripts in this repo:
- `drkernel/kernel/scripts/sft/14b-coldstart.sh`
- `drkernel/kernel/scripts/rl/14b_trloo_mrs_pr_prs.sh`
## Intended Use
- Kernel generation research and benchmarking
- Triton kernel optimization with iterative feedback
- Multi-turn agentic code refinement under execution-based reward
## Not Intended Use
- Safety-critical production deployment without additional verification
- General-purpose coding assistant use where kernel-evaluation feedback is unavailable
## Prompting Format
The model is trained with kernel-optimization prompts that:
- Provide a PyTorch reference architecture (`Model`, `get_inputs`, `get_init_inputs`)
- Require returning an optimized `ModelNew`
- In multi-turn settings, append server feedback and request iterative improvement
For best behavior, keep the same task style as DR.Kernel datasets and use chat-format messages.
## Quick Start (Transformers)
Use the same fixed 1-shot first-turn prompt template as DR.Kernel data (recommended):
````python
import textwrap
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "hkust-nlp/drkernel-14b"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
ref_code = textwrap.dedent(
"""
import torch
import torch.nn as nn
class Model(nn.Module):
def __init__(self):
super().__init__()
def forward(self, x):
x = torch.abs(x)
x = x - 1.0
return x
def get_inputs():
return [torch.randn(64, 128)]
def get_init_inputs():
return []
"""
).strip()
example_ref_code = textwrap.dedent(
"""
import torch
import torch.nn as nn
import torch.nn.functional as F
class Model(nn.Module):
def __init__(self) -> None:
super().__init__()
def forward(self, a, b):
return a + b
def get_inputs():
# randomly generate input tensors based on the model architecture
a = torch.randn(1, 128).cuda()
b = torch.randn(1, 128).cuda()
return [a, b]
def get_init_inputs():
# randomly generate tensors required for initialization based on the model architecture
return []
"""
).strip()
example_kernel_code = textwrap.dedent(
'''
import torch
import torch.nn as nn
import torch.nn.functional as F
import triton
import triton.language as tl
@triton.jit
def add_kernel(
x_ptr, # Pointer to first input
y_ptr, # Pointer to second input
out_ptr, # Pointer to output
n_elements, # Total number of elements in input/output
BLOCK_SIZE: tl.constexpr,
):
# Each program handles a contiguous block of data of size BLOCK_SIZE
block_start = tl.program_id(0) * BLOCK_SIZE
# Create a range of offsets [0..BLOCK_SIZE-1]
offsets = block_start + tl.arange(0, BLOCK_SIZE)
# Mask to ensure we don't go out of bounds
mask = offsets < n_elements
# Load input values
x = tl.load(x_ptr + offsets, mask=mask, other=0.0)
y = tl.load(y_ptr + offsets, mask=mask, other=0.0)
# Perform the elementwise addition
out = x + y
# Store the result
tl.store(out_ptr + offsets, out, mask=mask)
def triton_add(x: torch.Tensor, y: torch.Tensor):
"""
This function wraps the Triton kernel call. It:
1. Ensures the inputs are contiguous on GPU.
2. Calculates the grid (blocks) needed.
3. Launches the Triton kernel.
"""
assert x.is_cuda and y.is_cuda, "Tensors must be on CUDA."
x = x.contiguous()
y = y.contiguous()
# Prepare output tensor
out = torch.empty_like(x)
# Number of elements in the tensor
n_elements = x.numel()
BLOCK_SIZE = 128 # Tunable parameter for block size
# Determine the number of blocks needed
grid = lambda meta: ((n_elements + meta["BLOCK_SIZE"] - 1) // meta["BLOCK_SIZE"],)
# Launch the Triton kernel
add_kernel[grid](x, y, out, n_elements, BLOCK_SIZE=BLOCK_SIZE)
return out
class ModelNew(nn.Module):
def __init__(self) -> None:
super().__init__()
def forward(self, a, b):
# Instead of "return a + b", call our Triton-based addition
return triton_add(a, b)
'''
).strip()
prompt_template = textwrap.dedent(
"""\
You write custom Triton kernels to replace the pytorch operators in the given architecture to get speedups.
You have complete freedom to choose the set of operators you want to replace. You may make the decision to replace some operators with custom Triton kernels and leave others unchanged. You may replace multiple operators with custom implementations, consider operator fusion opportunities (combining multiple operators into a single kernel, for example, combining matmul+relu), or algorithmic changes (such as online softmax). You are only limited by your imagination.
Here's an example to show you the syntax of inline embedding custom Triton kernels in torch: The example given architecture is:
```python
{example_ref_code}
```
The example new arch with custom Triton kernels looks like this:
```python
{example_kernel_code}
```
You are given the following architecture:
```python
{ref_code}
```
Optimize the architecture named Model with custom Triton operators! Name your optimized output architecture ModelNew. Output the new code in codeblocks. Please generate real code, NOT pseudocode, make sure the code compiles and is fully functional. Let's think step by step.
"""
).strip()
prompt = prompt_template.format(
example_ref_code=example_ref_code,
example_kernel_code=example_kernel_code,
ref_code=ref_code,
)
messages = [{"role": "user", "content": prompt}]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
with torch.no_grad():
outputs = model.generate(
inputs,
max_new_tokens=2048,
do_sample=True,
temperature=1.0,
top_p=1.0,
)
# Only print newly generated tokens
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=False))
````
## Evaluation
Use KernelGYM-based evaluation scripts in this repo:
- `drkernel/kernel/scripts/eval/drkernel-14b-maxturns3.sh`
- `drkernel/kernel/scripts/eval/grading_common.sh` for custom evaluation runs
Validation data:
- `hkust-nlp/drkernel-validation-data` (KernelBench Level 2 validation tasks)
## Data and Attribution
- Query/task source includes:
- [ByteDance-Seed/cudaLLM-data](https://huggingface.co/datasets/ByteDance-Seed/cudaLLM-data)
- SFT cold-start trajectories:
- [hkust-nlp/drkernel-coldstart-8k](https://huggingface.co/datasets/hkust-nlp/drkernel-coldstart-8k)
- RL train data:
- [hkust-nlp/drkernel-rl-data](https://huggingface.co/datasets/hkust-nlp/drkernel-rl-data)
- Validation/eval data:
- [hkust-nlp/drkernel-validation-data](https://huggingface.co/datasets/hkust-nlp/drkernel-validation-data)
- Benchmark source:
- [KernelBench](https://github.com/ScalingIntelligence/KernelBench)
Please acknowledge original dataset/benchmark authors when using this model.
## Related Resources
- Paper: [Dr.Kernel: Reinforcement Learning Done Right for Triton Kernel Generations](https://arxiv.org/abs/2602.05885)
- Codebase: [KernelGYM](https://github.com/hkust-nlp/KernelGYM)
- Training docs: `drkernel/README.md`
## Citation
```bibtex
@article{liuetal2026,
title={Dr.Kernel: Reinforcement Learning Done Right for Triton Kernel Generations},
author={Wei Liu, Jiawei Xu, Yingru Li, Longtao Zheng, Tianjian Li, Qian Liu, Junxian He},
journal={arXiv:2602.05885},
year={2026}
}
```

28
added_tokens.json Normal file
View File

@@ -0,0 +1,28 @@
{
"</think>": 151668,
"</tool_call>": 151658,
"</tool_response>": 151666,
"<think>": 151667,
"<tool_call>": 151657,
"<tool_response>": 151665,
"<|box_end|>": 151649,
"<|box_start|>": 151648,
"<|endoftext|>": 151643,
"<|file_sep|>": 151664,
"<|fim_middle|>": 151660,
"<|fim_pad|>": 151662,
"<|fim_prefix|>": 151659,
"<|fim_suffix|>": 151661,
"<|im_end|>": 151645,
"<|im_start|>": 151644,
"<|image_pad|>": 151655,
"<|object_ref_end|>": 151647,
"<|object_ref_start|>": 151646,
"<|quad_end|>": 151651,
"<|quad_start|>": 151650,
"<|repo_name|>": 151663,
"<|video_pad|>": 151656,
"<|vision_end|>": 151653,
"<|vision_pad|>": 151654,
"<|vision_start|>": 151652
}

184
chat_template.jinja Normal file
View File

@@ -0,0 +1,184 @@
{% macro render_extra_keys(json_dict, handled_keys) %}
{%- if json_dict is mapping %}
{%- for json_key in json_dict if json_key not in handled_keys %}
{%- if json_dict[json_key] is mapping %}
{{- '
<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '</' ~ json_key ~ '>' }}
{%- else %}
{{-'
<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}
{%- endif %}
{%- endfor %}
{%- endif %}
{% endmacro %}
{%- if messages[0]["role"] == "system" %}
{%- set system_message = messages[0]["content"] %}
{%- set loop_messages = messages[1:] %}
{%- else %}
{%- set loop_messages = messages %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = [] %}
{%- endif %}
{%- if system_message is defined %}
{{- "<|im_start|>system
" + system_message }}
{%- else %}
{%- if tools is iterable and tools | length > 0 %}
{{- "<|im_start|>system
You are Qwen, a helpful AI assistant that can interact with a computer to solve tasks." }}
{%- endif %}
{%- endif %}
{%- if tools is iterable and tools | length > 0 %}
{{- "
You have access to the following functions:
" }}
{{- "<tools>" }}
{%- for tool in tools %}
{%- if tool.function is defined %}
{%- set tool = tool.function %}
{%- endif %}
{{- "
<function>
<name>" ~ tool.name ~ "</name>" }}
{%- if tool.description is defined %}
{{- '
<description>' ~ (tool.description | trim) ~ '</description>' }}
{%- endif %}
{{- '
<parameters>' }}
{%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
{%- for param_name, param_fields in tool.parameters.properties|items %}
{{- '
<parameter>' }}
{{- '
<name>' ~ param_name ~ '</name>' }}
{%- if param_fields.type is defined %}
{{- '
<type>' ~ (param_fields.type | string) ~ '</type>' }}
{%- endif %}
{%- if param_fields.description is defined %}
{{- '
<description>' ~ (param_fields.description | trim) ~ '</description>' }}
{%- endif %}
{%- set handled_keys = ['name', 'type', 'description'] %}
{{- render_extra_keys(param_fields, handled_keys) }}
{{- '
</parameter>' }}
{%- endfor %}
{%- endif %}
{% set handled_keys = ['type', 'properties'] %}
{{- render_extra_keys(tool.parameters, handled_keys) }}
{{- '
</parameters>' }}
{%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
{{- render_extra_keys(tool, handled_keys) }}
{{- '
</function>' }}
{%- endfor %}
{{- "
</tools>" }}
{{- '
If you choose to call a function ONLY reply in the following format with NO suffix:
<tool_call>
<function=example_function_name>
<parameter=example_parameter_1>
value_1
</parameter>
<parameter=example_parameter_2>
This is the value for the second parameter
that can span
multiple lines
</parameter>
</function>
</tool_call>
<IMPORTANT>
Reminder:
- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags
- Required parameters MUST be specified
- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after
- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls
</IMPORTANT>' }}
{%- endif %}
{%- if system_message is defined %}
{{- '<|im_end|>
' }}
{%- else %}
{%- if tools is iterable and tools | length > 0 %}
{{- '<|im_end|>
' }}
{%- endif %}
{%- endif %}
{%- for message in loop_messages %}
{%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
{{- '<|im_start|>' + message.role }}
{%- if message.content is defined and message.content is string and message.content | trim | length > 0 %}
{{- '
' + message.content | trim + '
' }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '
<tool_call>
<function=' + tool_call.name + '>
' }}
{%- if tool_call.arguments is defined %}
{%- for args_name, args_value in tool_call.arguments|items %}
{{- '<parameter=' + args_name + '>
' }}
{%- set args_value = args_value | tojson | safe if args_value is mapping else args_value | string %}
{{- args_value }}
{{- '
</parameter>
' }}
{%- endfor %}
{%- endif %}
{{- '</function>
</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>
' }}
{%- elif message.role == "user" or message.role == "system" or message.role == "assistant" %}
{{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>' + '
' }}
{%- elif message.role == "tool" %}
{%- if loop.previtem and loop.previtem.role != "tool" %}
{{- '<|im_start|>user
' }}
{%- endif %}
{{- '<tool_response>
' }}
{{- message.content }}
{{- '
</tool_response>
' }}
{%- if not loop.last and loop.nextitem.role != "tool" %}
{{- '<|im_end|>
' }}
{%- elif loop.last %}
{{- '<|im_end|>
' }}
{%- endif %}
{%- else %}
{{- '<|im_start|>' + message.role + '
' + message.content + '<|im_end|>
' }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant
' }}
{%- endif %}

72
config.json Normal file
View File

@@ -0,0 +1,72 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"dtype": "bfloat16",
"eos_token_id": 151643,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 17408,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 32768,
"max_window_layers": 40,
"model_type": "qwen3",
"num_attention_heads": 40,
"num_hidden_layers": 40,
"num_key_value_heads": 8,
"pad_token_id": 151643,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": false,
"transformers_version": "4.56.0",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"bos_token_id": 151643,
"eos_token_id": 151643,
"max_new_tokens": 2048,
"transformers_version": "4.56.0"
}

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f1fd3c82c8b94bd5dd549d50ed7b13bf9004ffcdfedd86e75f9418f64750831e
size 4823627440

3
model-00002-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aa70c49bec7a3fdd19178fdba067937ef55f6fa8086503b2fac265624dad5229
size 4834008136

3
model-00003-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f4f31bb3f5186b71c23158003c2fbde8f6d046456569115ca9cc2a8c4473ec29
size 4897059448

3
model-00004-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2f688dc67dae74b3ab9439e935eeec945b445b4b9cba6dc3f9d57acb0f962ee2
size 4869405032

3
model-00005-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:38ea9c255f7644c2cd255175931644082a439e8f6cd85e3cadb2a7a2e05e8e44
size 3921846224

3
model-00006-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:54927feff827bbf71d10d1e37256bf706b47b6c748b5d466620e8f7da01dcffa
size 4932391472

3
model-00007-of-00007.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b10866c01bd803cb5a46e3db6dd1cc6ab5ed11bb32a7cc3348b665807cae53fd
size 1258327816

451
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,451 @@
{
"metadata": {
"total_parameters": 14768307200,
"total_size": 29536614400
},
"weight_map": {
"lm_head.weight": "model-00006-of-00007.safetensors",
"model.embed_tokens.weight": "model-00004-of-00007.safetensors",
"model.layers.0.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.0.self_attn.k_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.1.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.1.self_attn.k_norm.weight": "model-00006-of-00007.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.1.self_attn.q_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.10.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.10.self_attn.k_norm.weight": "model-00004-of-00007.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.10.self_attn.q_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.11.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.11.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.11.self_attn.q_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.12.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.12.self_attn.k_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.13.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.13.self_attn.q_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.14.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.14.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.14.self_attn.q_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.15.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.15.self_attn.k_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.16.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.16.self_attn.q_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.17.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.17.self_attn.k_norm.weight": "model-00004-of-00007.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.17.self_attn.q_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.18.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.18.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.18.self_attn.q_norm.weight": "model-00006-of-00007.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.19.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.self_attn.q_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.2.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.2.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.20.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.20.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.20.self_attn.q_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.21.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.21.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.21.self_attn.q_norm.weight": "model-00004-of-00007.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.22.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.22.self_attn.k_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.22.self_attn.q_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.23.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00007-of-00007.safetensors",
"model.layers.23.self_attn.k_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.23.self_attn.q_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.24.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.24.self_attn.k_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.24.self_attn.q_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.25.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.25.self_attn.k_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.25.self_attn.q_norm.weight": "model-00005-of-00007.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.26.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.26.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.26.self_attn.q_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.27.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.27.self_attn.k_norm.weight": "model-00004-of-00007.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.27.self_attn.q_norm.weight": "model-00006-of-00007.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.28.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.28.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.28.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.28.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.28.post_attention_layernorm.weight": "model-00007-of-00007.safetensors",
"model.layers.28.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.28.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.28.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.28.self_attn.q_norm.weight": "model-00006-of-00007.safetensors",
"model.layers.28.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.28.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.29.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.29.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.29.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.29.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.29.self_attn.k_norm.weight": "model-00006-of-00007.safetensors",
"model.layers.29.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.29.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.29.self_attn.q_norm.weight": "model-00004-of-00007.safetensors",
"model.layers.29.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.29.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.3.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00007-of-00007.safetensors",
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.3.self_attn.q_norm.weight": "model-00004-of-00007.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.30.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.30.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.30.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.30.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.30.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.30.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.30.self_attn.q_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.30.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.30.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.31.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.31.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.31.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.31.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.31.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.31.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.31.self_attn.k_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.31.self_attn.o_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.31.self_attn.q_norm.weight": "model-00005-of-00007.safetensors",
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.31.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.32.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.32.mlp.down_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.32.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.32.mlp.up_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.32.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.32.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.32.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.32.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.32.self_attn.q_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.32.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.32.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.33.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.33.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.33.mlp.gate_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.33.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.33.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.33.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.33.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.33.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.33.self_attn.q_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.33.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.33.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.34.input_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.34.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.34.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.34.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.34.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.34.self_attn.k_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.34.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.34.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.34.self_attn.q_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.34.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.34.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.35.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.35.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.35.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.35.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.35.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.35.self_attn.k_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.35.self_attn.k_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.35.self_attn.o_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.35.self_attn.q_norm.weight": "model-00006-of-00007.safetensors",
"model.layers.35.self_attn.q_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.35.self_attn.v_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.36.input_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.36.mlp.down_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.36.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.36.mlp.up_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.36.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.36.self_attn.k_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.36.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.36.self_attn.o_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.36.self_attn.q_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.36.self_attn.q_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.36.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.37.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.37.mlp.down_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.37.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.37.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.37.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.37.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.37.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.37.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.37.self_attn.q_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.37.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.37.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.38.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.38.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.38.mlp.gate_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.38.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.38.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
"model.layers.38.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.38.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.38.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.38.self_attn.q_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.38.self_attn.q_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.38.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.39.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.39.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.39.mlp.gate_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.39.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.39.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.39.self_attn.k_norm.weight": "model-00004-of-00007.safetensors",
"model.layers.39.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.39.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.39.self_attn.q_norm.weight": "model-00005-of-00007.safetensors",
"model.layers.39.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.39.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.4.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.4.self_attn.k_norm.weight": "model-00007-of-00007.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.4.self_attn.q_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.5.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00007.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.5.self_attn.q_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.6.input_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.6.self_attn.k_norm.weight": "model-00005-of-00007.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.6.self_attn.q_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.7.input_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
"model.layers.7.self_attn.k_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.7.self_attn.q_norm.weight": "model-00003-of-00007.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.8.input_layernorm.weight": "model-00002-of-00007.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00005-of-00007.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
"model.layers.8.self_attn.k_norm.weight": "model-00002-of-00007.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00003-of-00007.safetensors",
"model.layers.8.self_attn.q_norm.weight": "model-00005-of-00007.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00004-of-00007.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00002-of-00007.safetensors",
"model.layers.9.input_layernorm.weight": "model-00003-of-00007.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00007.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
"model.layers.9.self_attn.k_norm.weight": "model-00006-of-00007.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00007-of-00007.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.9.self_attn.q_norm.weight": "model-00004-of-00007.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00006-of-00007.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00003-of-00007.safetensors",
"model.norm.weight": "model-00006-of-00007.safetensors"
}
}

31
special_tokens_map.json Normal file
View File

@@ -0,0 +1,31 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

BIN
tokenizer.json (Stored with Git LFS) Normal file
View File

Binary file not shown.

239
tokenizer_config.json Normal file
View File

@@ -0,0 +1,239 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151665": {
"content": "<tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151666": {
"content": "</tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151667": {
"content": "<think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151668": {
"content": "</think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|endoftext|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long