初始化项目，由ModelHub XC社区提供模型
Model: EPFLiGHT/Apertus-8B-MeditronFO Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
133
README.md
Normal file
133
README.md
Normal file
@@ -0,0 +1,133 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
library_name: transformers
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- medical
|
||||
- clinical
|
||||
- healthcare
|
||||
- meditron
|
||||
- fully-open
|
||||
- medical-llm
|
||||
base_model: swiss-ai/Apertus-8B-Instruct-2509
|
||||
base_model_relation: finetune
|
||||
datasets:
|
||||
- EPFLiGHT/fully-open-meditron
|
||||
---
|
||||
|
||||
# Apertus-8B-MeditronFO
|
||||
|
||||
<div align="center">
|
||||
<img src=https://raw.githubusercontent.com/EPFLiGHT/MeditronFO/3caab22b7efaf9e56e2894bac65dab6d3a31aaa0/static/images/MeditronFO-logo-cube.png width="15%"/>
|
||||
</div>
|
||||
<p align="center">
|
||||
👋 Join our <a href="https://www.light-laboratory.org/" target="_blank">LiGHT</a> community.
|
||||
<br>
|
||||
📖 Check out the MeditronFO <a href="https://epflight.github.io/MeditronFO/" target="_blank">blog</a> and MeditronFO <a href="https://arxiv.org/abs/2605.16215" target="_blank">preprint</a>.
|
||||
<br>
|
||||
🔜 If you are a clinician join the MOOVE initiative <a href="https://jointhemoove.org/">here</a>.
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
[<a href="https://huggingface.co/collections/EPFLiGHT/meditronfo" target="_blank">Hugging Face</a>]
|
||||
[<a href="https://arxiv.org/abs/2605.16215" target="_blank">Preprint</a>]
|
||||
[<a href="https://github.com/EPFLiGHT/FullyOpenMeditron" target="_blank">GitHub</a>]
|
||||
[<a href="https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron" target="_blank">Dataset</a>]
|
||||
<br>
|
||||
<b>License</b>: <a href="https://www.apache.org/licenses/LICENSE-2.0.txt" target="_blank">Apache 2.0</a> | <b>Authors</b>: <a href="https://www.light-laboratory.org/" target="_blank">LiGHT</a>
|
||||
</p>
|
||||
|
||||
> [!Note]
|
||||
> We're introducing Apertus-8B-MeditronFO, our small fully open medical specialist LLM, medical specialization of [Apertus-8B-Instruct](https://huggingface.co/swiss-ai/Apertus-70B-Instruct-2509) on the [Fully Open Meditron Corpus](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron). This model is part of the **Fully Open Meditron** family — the first end-to-end auditable pipeline for clinical LLMs, with open weights, open data, open training recipe, and clinician-vetted corpus construction.
|
||||
|
||||
|
||||
- **Part of the Fully Open Meditron family:** End to end fully open clinical LLMs
|
||||
- **Establishes a new state of the art:** Best medical accuracy on medical benchmarks among small fully open medical LLMs
|
||||
- **Is prefered over Apertus-70B in 86.9% of comparisons:** Comparison on the clinician validated LM judge evaluation AutoMOOVE.
|
||||
- **Apertus-8B-MeditronFO improves +13.35 points over its base on aggregate medical benchmarks:** the largest gain in the MeditronFO family.
|
||||
|
||||
<div align="center">
|
||||
<img src=https://raw.githubusercontent.com/EPFLiGHT/MeditronFO/3caab22b7efaf9e56e2894bac65dab6d3a31aaa0/static/images/plot_fig1_healthbench_timeline_easy_color.png width="100%"/>
|
||||
</div>
|
||||
|
||||
## Benchmark
|
||||
|
||||
Accuracy (%) on standard medical benchmarks. See the paper for full evaluation details, confidence intervals, and open-ended Auto-MOOVE results.
|
||||
|
||||
| Benchmark | Apertus-8B-Instruct | **Apertus-8B-MeditronFO** | Δ |
|
||||
|---|---:|---:|---:|
|
||||
| MedMCQA | 45.80 | **48.74** | +2.94 |
|
||||
| MedQA | 51.14 | **58.44** | +7.30 |
|
||||
| PubMedQA | 37.60 | **75.60** | +38.00 |
|
||||
| MedXpertQA | 11.71 | **13.67** | +1.96 |
|
||||
| HealthBench Hard | 21.55 | **38.11** | +16.56 |
|
||||
| **Average** | 33.56 | **46.91** | +13.35 |
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
model_id = "EPFLiGHT/Apertus-8B-MeditronFO"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": "A 62-year-old woman presents with a three-day history of dyspnea on exertion and a productive cough. What is the differential diagnosis?"},
|
||||
]
|
||||
inputs = tokenizer.apply_chat_template(
|
||||
messages,
|
||||
add_generation_prompt=True,
|
||||
tokenize=True,
|
||||
return_dict=True,
|
||||
return_tensors="pt",
|
||||
).to(model.device)
|
||||
|
||||
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
|
||||
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Training
|
||||
|
||||
- **Base model:** [Apertus-8B-Instruct](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509)
|
||||
- **Corpus:** [Fully Open Meditron](https://huggingface.co/datasets/EPFLiGHT/fully-open-meditron) 601k examples (~150M tokens), aggregating eight public medical QA datasets with three clinician-vetted synthetic components: exam-style QA, guideline-grounded QA from 46,469 clinical practice guidelines, and open-ended clinical vignettes
|
||||
- **Hardware:** 8 NVIDIA GH200 nodes
|
||||
- **Framework:** Axolotl with FSDP v2 / DeepSpeed ZeRO-3, Flash Attention 2, bf16 mixed precision
|
||||
- **Decontamination:** System-wide two-stage n-gram and token-alignment decontamination against all evaluation benchmarks
|
||||
|
||||
Full hyperparameters are in Appendix I of the paper.
|
||||
|
||||
## Compute & footprint
|
||||
The training was done on 8 nodes of 4 NVIDIA GH200 GPUs for approximately 6 hours on the CSCS Swiss National Supercomputing Centre.
|
||||
Our trainings have a carbon neutral footprint as the CSCS data center is carbon neutral ([CSCS energy efficiency](https://ethz.ch/en/news-and-events/eth-news/news/2022/11/at-cscs-energy-efficiency-is-a-key-priority-even-at-high-performance.html)).
|
||||
|
||||
## Limitations & intended use
|
||||
|
||||
MeditronFO can produce text on a variety of topics, but the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data. MeditronFO has been trained to be specialised for Medicine and is intended to be used for Medicine related tasks evaluation. These models should be used as assistive tools rather than definitive sources of information. Users should always verify important information and critically evaluate any generated content.
|
||||
|
||||
## Citation
|
||||
|
||||
If you find MeditronFO useful in your research, please cite our preprint:
|
||||
|
||||
```bibtex
|
||||
@misc{theimerlienhard2026fullyopenmeditronauditable,
|
||||
title = {Fully Open Meditron: An Auditable Pipeline for Clinical LLMs},
|
||||
author = {Xavier Theimer-Lienhard and Mushtaha El-Amin and Fay Elhassan and Sahaj Vaidya and Victor Cartier-Negadi and David Sasu and Lars Klein and Mary-Anne Hartley},
|
||||
year = {2026},
|
||||
eprint = {2605.16215},
|
||||
archivePrefix = {arXiv},
|
||||
primaryClass = {cs.AI},
|
||||
url = {https://arxiv.org/abs/2605.16215}
|
||||
}
|
||||
```
|
||||
|
||||
## Contact
|
||||
Please use the community tab for any discussions or issue related to this model.
|
||||
Questions related to the project can be sent to xavier.theimer-lienhard@epfl.ch or mary-anne.hartley@epfl.ch.
|
||||
337
chat_template.jinja
Normal file
337
chat_template.jinja
Normal file
@@ -0,0 +1,337 @@
|
||||
{%- macro render_typescript_type(param_spec, required_params, is_nullable=false) -%}
|
||||
{%- if param_spec.type == "array" -%}
|
||||
{%- if param_spec['items'] -%}
|
||||
{%- if param_spec['items']['type'] == "string" -%}
|
||||
{{- "string[]" }}
|
||||
{%- elif param_spec['items']['type'] == "number" -%}
|
||||
{{- "number[]" }}
|
||||
{%- elif param_spec['items']['type'] == "integer" -%}
|
||||
{{- "number[]" }}
|
||||
{%- elif param_spec['items']['type'] == "boolean" -%}
|
||||
{{- "boolean[]" }}
|
||||
{%- else -%}
|
||||
{%- set inner_type = render_typescript_type(param_spec['items'], required_params) -%}
|
||||
{%- if inner_type == "object | object" or inner_type|length > 50 -%}
|
||||
{{- "any[]" }}
|
||||
{%- else -%}
|
||||
{{- inner_type + "[]" }}
|
||||
{%- endif -%}
|
||||
{%- endif -%}
|
||||
{%- if param_spec.nullable -%}
|
||||
{{- " | null" }}
|
||||
{%- endif -%}
|
||||
{%- else -%}
|
||||
{{- "any[]" }}
|
||||
{%- if param_spec.nullable -%}
|
||||
{{- " | null" }}
|
||||
{%- endif -%}
|
||||
{%- endif -%}
|
||||
{%- elif param_spec.type is defined and param_spec.type is iterable and param_spec.type is not string and param_spec.type is not mapping and param_spec.type[0] is defined -%}
|
||||
{#- Handle array of types like ["object", "object"] from Union[dict, list] #}
|
||||
{%- if param_spec.type | length > 1 -%}
|
||||
{{- param_spec.type | join(" | ") }}
|
||||
{%- else -%}
|
||||
{{- param_spec.type[0] }}
|
||||
{%- endif -%}
|
||||
{%- elif param_spec.oneOf -%}
|
||||
{#- Handle oneOf schemas - check for complex unions and fallback to any #}
|
||||
{%- set has_object_variants = false -%}
|
||||
{%- for variant in param_spec.oneOf -%}
|
||||
{%- if variant.type == "object" -%}
|
||||
{%- set has_object_variants = true -%}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{%- if has_object_variants and param_spec.oneOf|length > 1 -%}
|
||||
{{- "any" }}
|
||||
{%- else -%}
|
||||
{%- for variant in param_spec.oneOf -%}
|
||||
{{- render_typescript_type(variant, required_params) -}}
|
||||
{%- if variant.description %}
|
||||
{{- "// " + variant.description }}
|
||||
{%- endif -%}
|
||||
{%- if variant.default is defined %}
|
||||
{{ "// default: " + variant.default|tojson }}
|
||||
{%- endif -%}
|
||||
{%- if not loop.last %}
|
||||
{{- " | " }}
|
||||
{% endif -%}
|
||||
{%- endfor -%}
|
||||
{%- endif -%}
|
||||
{%- elif param_spec.type == "string" -%}
|
||||
{%- if param_spec.enum -%}
|
||||
{{- '"' + param_spec.enum|join('" | "') + '"' -}}
|
||||
{%- else -%}
|
||||
{{- "string" }}
|
||||
{%- if param_spec.nullable %}
|
||||
{{- " | null" }}
|
||||
{%- endif -%}
|
||||
{%- endif -%}
|
||||
{%- elif param_spec.type == "number" -%}
|
||||
{{- "number" }}
|
||||
{%- elif param_spec.type == "integer" -%}
|
||||
{{- "number" }}
|
||||
{%- elif param_spec.type == "boolean" -%}
|
||||
{{- "boolean" }}
|
||||
{%- elif param_spec.type == "object" -%}
|
||||
{%- if param_spec.properties -%}
|
||||
{{- "{\n" }}
|
||||
{%- for prop_name, prop_spec in param_spec.properties.items() -%}
|
||||
{{- prop_name -}}
|
||||
{%- if prop_name not in (param_spec.required or []) -%}
|
||||
{{- "?" }}
|
||||
{%- endif -%}
|
||||
{{- ": " }}
|
||||
{{ render_typescript_type(prop_spec, param_spec.required or []) }}
|
||||
{%- if not loop.last -%}
|
||||
{{-", " }}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{{- "}" }}
|
||||
{%- else -%}
|
||||
{{- "object" }}
|
||||
{%- endif -%}
|
||||
{%- else -%}
|
||||
{{- "any" }}
|
||||
{%- endif -%}
|
||||
{%- endmacro -%}
|
||||
|
||||
{%- macro render_tools(tools) -%}
|
||||
{%- for tool in tools %}
|
||||
{{- "// " + tool.description + "\n" }}
|
||||
{{- "type "+ tool.name + " = " }}
|
||||
{%- if tool.parameters and tool.parameters.properties %}
|
||||
{{- "(_: {\n" }}
|
||||
{%- for param_name, param_spec in tool.parameters.properties.items() %}
|
||||
{%- if param_spec.description %}
|
||||
{{- "// " + param_spec.description + "\n" }}
|
||||
{%- endif %}
|
||||
{{- param_name }}
|
||||
{%- if param_name not in (tool.parameters.required or []) -%}
|
||||
{{- "?" }}
|
||||
{%- endif -%}
|
||||
{{- ": " }}
|
||||
{{- render_typescript_type(param_spec, tool.parameters.required or []) }}
|
||||
{%- if param_spec.default is defined -%}
|
||||
{%- if param_spec.enum %}
|
||||
{{- ", // default: " + param_spec.default }}
|
||||
{%- elif param_spec.oneOf %}
|
||||
{{- "// default: " + param_spec.default }}
|
||||
{%- else %}
|
||||
{{- ", // default: " + param_spec.default|tojson }}
|
||||
{%- endif -%}
|
||||
{%- endif -%}
|
||||
{%- if not loop.last %}
|
||||
{{- ",\n" }}
|
||||
{%- else %}
|
||||
{{- "\n" }}
|
||||
{%- endif -%}
|
||||
{%- endfor %}
|
||||
{{- "}) => any;" }}
|
||||
{%- else -%}
|
||||
{{- "() => any;" }}
|
||||
{%- endif -%}
|
||||
{%- if not loop.last -%}
|
||||
{{- "\n" }}
|
||||
{%- endif -%}
|
||||
{%- endfor %}
|
||||
{%- endmacro -%}
|
||||
|
||||
{{ bos_token }}
|
||||
|
||||
{%- set system_token = '<|system_start|>' -%}
|
||||
{%- set end_system_token = '<|system_end|>' -%}
|
||||
{%- set developer_token = '<|developer_start|>' -%}
|
||||
{%- set end_developer_token = '<|developer_end|>' -%}
|
||||
{%- set user_token = '<|user_start|>' -%}
|
||||
{%- set end_user_token = '<|user_end|>' -%}
|
||||
{%- set assistant_token = '<|assistant_start|>' -%}
|
||||
{%- set end_assistant_token = '<|assistant_end|>' -%}
|
||||
{%- set inner_token = '<|inner_prefix|>' -%}
|
||||
{%- set outer_token = '<|inner_suffix|>' -%}
|
||||
{%- set tool_calls_token = '<|tools_prefix|>' -%}
|
||||
{%- set end_tool_calls_token = '<|tools_suffix|>' -%}
|
||||
{%- set image_token = '<|image|>' -%}
|
||||
|
||||
{%- set ns = namespace(in_assistant=false, in_tool=false, in_inner=false, waiting_for_tool_outputs=false, assistant_format=none) -%}
|
||||
|
||||
{%- if messages and messages[0].role == 'system' -%}
|
||||
{%- if "content" in messages[0] -%}
|
||||
{%- if messages[0].content is string -%}
|
||||
{{ system_token + messages[0].content + end_system_token }}
|
||||
{%- elif messages[0].content is mapping and "text" in messages[0].content -%}
|
||||
{{ system_token + messages[0].content.text + end_system_token }}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid system message") -}}
|
||||
{%- endif -%}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid system message") -}}
|
||||
{%- endif -%}
|
||||
{%- set loop_messages = messages[1:] -%}
|
||||
{%- else -%}
|
||||
{{ system_token + 'You are Apertus, a helpful assistant created by the SwissAI initiative.\nKnowledge cutoff: 2024-04\nCurrent date: ' + strftime_now('%Y-%m-%d') + end_system_token }}
|
||||
{%- set loop_messages = messages -%}
|
||||
{%- endif -%}
|
||||
|
||||
{{ developer_token + 'Deliberation: ' }}
|
||||
{%- if enable_thinking is defined and enable_thinking -%}
|
||||
{{ 'enabled\n' }}
|
||||
{%- else -%}
|
||||
{{ 'disabled\n' }}
|
||||
{%- endif -%}
|
||||
{%- if tools is defined and tools -%}
|
||||
{{ 'Tool Capabilities:\n' + render_tools(tools) }}
|
||||
{%- else -%}
|
||||
{{ 'Tool Capabilities: disabled' }}
|
||||
{%- endif -%}
|
||||
{{ end_developer_token }}
|
||||
|
||||
{%- for message in loop_messages -%}
|
||||
{%- if message.role == 'user' -%}
|
||||
{%- set ns.in_inner = false -%}
|
||||
{%- if ns.in_tool -%}
|
||||
{{ ']' }}
|
||||
{%- set ns.in_tool = false -%}
|
||||
{%- endif -%}
|
||||
{%- if ns.in_assistant -%}
|
||||
{{ end_assistant_token }}
|
||||
{%- set ns.in_assistant = false -%}
|
||||
{%- endif -%}
|
||||
{%- if "content" in message -%}
|
||||
{{ user_token }}
|
||||
{%- if message.content is string -%}
|
||||
{{ message.content }}
|
||||
{%- elif message.content is mapping and "parts" in message.content -%}
|
||||
{%- set parts = message.content.parts -%}
|
||||
{%- for part in parts -%}
|
||||
{%- if part.type == "text" -%}
|
||||
{{ part.text }}
|
||||
{%- elif part.type == "image" -%}
|
||||
{{ image_token }}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid user part: " + part.type) -}}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid user message: " + message.role) -}}
|
||||
{%- endif -%}
|
||||
{{ end_user_token }}
|
||||
{%- endif -%}
|
||||
{%- elif message.role == 'assistant' -%}
|
||||
{%- if not ns.in_assistant -%}
|
||||
{{ assistant_token }}
|
||||
{%- set ns.in_assistant = true -%}
|
||||
{%- endif -%}
|
||||
{%- if "content" in message -%}
|
||||
{%- if message.content is string and (ns.assistant_format is none or ns.assistant_format == "string") -%}
|
||||
{%- if ns.in_tool -%}
|
||||
{{ ']' }}
|
||||
{%- set ns.in_tool = false -%}
|
||||
{%- endif -%}
|
||||
{%- set ns.assistant_format = "string" -%}
|
||||
{{ message.content }}
|
||||
{%- elif message.content is mapping and "blocks" in message.content and (ns.assistant_format is none or ns.assistant_format == "mapping") -%}
|
||||
{%- set ns.assistant_format = "mapping" -%}
|
||||
{%- set blocks = message.content.blocks -%}
|
||||
{%- for block in blocks -%}
|
||||
{%- if block.type == 'thoughts' -%}
|
||||
{%- if ns.in_tool -%}
|
||||
{{ ']' }}
|
||||
{%- set ns.in_tool = false -%}
|
||||
{%- endif -%}
|
||||
{%- if not ns.in_inner -%}
|
||||
{%- set ns.in_inner = true -%}
|
||||
{{ inner_token }}
|
||||
{%- endif -%}
|
||||
{{ block.text }}
|
||||
{%- elif block.type == 'tool_calls' -%}
|
||||
{%- if ns.in_tool -%}
|
||||
{{ ']' }}
|
||||
{%- set ns.in_tool = false -%}
|
||||
{%- endif -%}
|
||||
{%- if ns.in_inner and not loop.first and block.calls|length == 1 and block.calls[0].name == 'display_answers' -%}
|
||||
{%- set ns.in_inner = false -%}
|
||||
{{ outer_token }}
|
||||
{%- endif -%}
|
||||
{{ tool_calls_token + '[' }}
|
||||
{%- for tool_call in block.calls -%}
|
||||
{{- '{"' + tool_call.name + '": ' + tool_call.arguments + '}' }}
|
||||
{%- if not loop.last -%}
|
||||
{{- ", " }}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{{ ']' + end_tool_calls_token }}
|
||||
{%- set ns.waiting_for_tool_outputs = true -%}
|
||||
{%- elif block.type == 'tool_outputs' -%}
|
||||
{%- if ns.in_tool -%}
|
||||
{{- raise_exception("Cannot have both tool outputs as separate messages and tool outputs as blocks") -}}
|
||||
{%- endif -%}
|
||||
{{ '[' }}
|
||||
{%- for tool_output in block.outputs -%}
|
||||
{{- tool_output.output }}
|
||||
{%- if not loop.last -%}
|
||||
{{- ", " }}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{{- ']' }}
|
||||
{%- set ns.waiting_for_tool_outputs = false -%}
|
||||
{%- elif block.type == 'response' -%}
|
||||
{%- if ns.in_tool -%}
|
||||
{{ ']' }}
|
||||
{%- set ns.in_tool = false -%}
|
||||
{%- endif -%}
|
||||
{%- if (not loop.first and ns.in_inner) or (ns.in_assistant and ns.in_inner) -%}
|
||||
{%- set ns.in_inner = false -%}
|
||||
{{ outer_token }}
|
||||
{%- endif -%}
|
||||
{{ block.text }}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid assistant block type: " + block.type) -}}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid assistant content") -}}
|
||||
{%- endif -%}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid assistant message") -}}
|
||||
{%- endif -%}
|
||||
{%- if "tool_calls" in message and message.tool_calls -%}
|
||||
{{ tool_calls_token + '[' }}
|
||||
{%- for tool_call in message.tool_calls -%}
|
||||
{%- if tool_call.type == 'function' -%}
|
||||
{%- set function = tool_call.function -%}
|
||||
{{- '{"' + function.name + '": ' + function.arguments + '}' }}
|
||||
{%- if not loop.last -%}
|
||||
{{- ", " }}
|
||||
{%- endif -%}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid tool call type: " + tool_call.type) -}}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{{ ']' + end_tool_calls_token }}
|
||||
{%- set ns.waiting_for_tool_outputs = true -%}
|
||||
{%- endif -%}
|
||||
{%- elif message.role == 'tool' -%}
|
||||
{%- if not ns.in_assistant -%}
|
||||
{{- raise_exception("Tool message outside of assistant") -}}
|
||||
{%- endif -%}
|
||||
{%- if not ns.in_tool -%}
|
||||
{{ '[' }}
|
||||
{%- set ns.in_tool = true -%}
|
||||
{%- else -%}
|
||||
{{ ", "}}
|
||||
{%- endif -%}
|
||||
{{ message.content }}
|
||||
{%- set ns.waiting_for_tool_outputs = false -%}
|
||||
{%- else -%}
|
||||
{{- raise_exception("Invalid message role") -}}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{%- if ns.in_tool -%}
|
||||
{{ ']' }}
|
||||
{%- endif -%}
|
||||
{%- if ns.in_assistant and not (continue_assistant_message is defined and continue_assistant_message) and not ns.waiting_for_tool_outputs -%}
|
||||
{{ end_assistant_token }}
|
||||
{%- endif -%}
|
||||
{%- if add_generation_prompt -%}
|
||||
{{ assistant_token }}
|
||||
{%- endif -%}
|
||||
38
config.json
Normal file
38
config.json
Normal file
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"architectures": [
|
||||
"ApertusForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 68,
|
||||
"hidden_act": "xielu",
|
||||
"hidden_dropout": 0.0,
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 21504,
|
||||
"max_position_embeddings": 65536,
|
||||
"mlp_bias": false,
|
||||
"model_type": "apertus",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 8,
|
||||
"pad_token_id": 3,
|
||||
"post_norm": false,
|
||||
"qk_norm": true,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": {
|
||||
"factor": 8.0,
|
||||
"high_freq_factor": 4.0,
|
||||
"low_freq_factor": 1.0,
|
||||
"original_max_position_embeddings": 8192,
|
||||
"rope_type": "llama3",
|
||||
"type": "llama3"
|
||||
},
|
||||
"rope_theta": 12000000,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "4.57.1",
|
||||
"use_cache": false,
|
||||
"vocab_size": 131072
|
||||
}
|
||||
3
model-00001-of-00004.safetensors
Normal file
3
model-00001-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:9942849a1efe88f66146806480cebfba761ceaf3ebe1a9d9e137664cd99b9b1d
|
||||
size 4999774828
|
||||
3
model-00002-of-00004.safetensors
Normal file
3
model-00002-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7e391657b83378c244529abf92599247466f6dba51b3621d69fe6be710d40552
|
||||
size 4882371964
|
||||
3
model-00003-of-00004.safetensors
Normal file
3
model-00003-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:66811d4c32cf10a256d439bd22351c9e3c65fa41c64512c5012c19c6a94eb484
|
||||
size 4974645580
|
||||
3
model-00004-of-00004.safetensors
Normal file
3
model-00004-of-00004.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:596eaa7c79284bd5a3690d236c6e45dac034190838b04ef50d683777705c0573
|
||||
size 1249934492
|
||||
458
model.safetensors.index.json
Normal file
458
model.safetensors.index.json
Normal file
@@ -0,0 +1,458 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_size": 16106676480
|
||||
},
|
||||
"weight_map": {
|
||||
"lm_head.weight": "model-00004-of-00004.safetensors",
|
||||
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.0.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.1.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.10.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.10.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.11.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.12.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.13.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.14.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.15.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.16.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.17.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.18.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.19.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.2.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.2.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.20.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.20.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.21.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.21.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.22.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.23.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.28.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.29.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.3.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.3.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.30.attention_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.feedforward_layernorm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.act_fn.alpha_n": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.act_fn.alpha_p": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.30.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.attention_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.feedforward_layernorm.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.mlp.act_fn.alpha_n": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.mlp.act_fn.alpha_p": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
|
||||
"model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
||||
"model.layers.4.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.4.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.5.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.6.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.7.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.attention_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.feedforward_layernorm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.act_fn.alpha_n": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.act_fn.alpha_p": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.8.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
||||
"model.layers.9.attention_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.feedforward_layernorm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.act_fn.alpha_n": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.act_fn.alpha_p": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.act_fn.beta": "model-00004-of-00004.safetensors",
|
||||
"model.layers.9.mlp.act_fn.eps": "model-00004-of-00004.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
||||
"model.norm.weight": "model-00004-of-00004.safetensors"
|
||||
}
|
||||
}
|
||||
30
special_tokens_map.json
Normal file
30
special_tokens_map.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|assistant_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:010095abf1dc6f52d4867584e7b3f0d4eece854593ae310220ec7782dd1b0a66
|
||||
size 17078474
|
||||
8020
tokenizer_config.json
Normal file
8020
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user