fenyo/ministral-8b-instruct-FAQ-MES-WEB
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: fenyo/ministral-8b-instruct-FAQ-MES-WEB
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-24 12:32:18 +08:00
commit 401771d269
17 changed files with 1176 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

601
README.md Normal file
View File

@@ -0,0 +1,601 @@
---
datasets:
- fenyo/FAQ-MES-WEB
base_model:
- mistralai/Ministral-8B-Instruct-2410
library_name: transformers
---
# Model Card for ministral-8b-instruct-FAQ-MES-WEB
**Ce modèle est comparé à d'autres modèles ici : https://huggingface.co/fenyo/MonEspaceSante-FAQ-Mistral-Small-24B-GGUF/blob/main/REPORT.md**
This model is a fine-tuned version of [mistralai/Ministral-8B-Instruct-2410](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) on the
[fenyo/FAQ-MES-WEB](https://huggingface.co/datasets/fenyo/FAQ-MES-WEB) dataset.
It has been trained using [TRL](https://github.com/huggingface/trl) on aPC running Windows 11, WSL (Ubuntu 24.04) and one Nvidia RTX-5090 GPU.
## Quick start
```python
from transformers import pipeline
question = "Qu'est-ce que Mon espace santé ?"
generator = pipeline("text-generation", model="fenyo/ministral-8b-instruct-merged", device="cuda")
output = generator([{"role": "system", "content": "You are a helpful chatbot assistant for the Mon Espace Santé website. You answer only based on the information present in the FAQ. If the information is not available, you must respond with the predefined refusal message and nothing else."}, {"role": "user", "content": question}], max_new_tokens=4096, return_full_text=False)[0]
print(output["generated_text"])
```
## Training procedure
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/alexandre-fenyo-fenyonet/fenyo-FAQ-MES-WEB-ministral/runs/eq7aqysw)
<img src="https://fenyo.net/hf/ministral-FAQ-MES-WEB.png" alt="Weight & Biases"/>
This model was trained with SFT.
## Training script and parameters
Parameters file:
```
wandb_notebook_name="ministral-8b-instruct-FAQ-MES"
model_name="Ministral-8B-Instruct-2410"
tokenizer="Ministral-8B-Instruct-2410"
# Note: ce system_prompt n'est pas injecte par ft-small.py.
# Il doit deja etre present dans la colonne messages du dataset pour qu'il soit appris.
system_prompt="You are a helpful chatbot assistant for the Mon Espace Santé website. You answer only based on the information present in the FAQ. If the information is not available, you must respond with the predefined refusal message and nothing else."
var_dataset_name="fenyo/FAQ-MES-WEB"
var_wandb_project="fenyo-FAQ-MES-WEB-ministral"
var_wandb_run="ministral-8b-instruct"
attn_implementation="eager"
ft_bf16=True
ft_assistant_only_loss=True
lora_r=16
lora_alpha=32
lora_dropout=0.05
lora_bias="none"
lora_target_modules="q_proj" "k_proj" "v_proj" "o_proj" "gate_proj" "up_proj" "down_proj"
ft_learning_rate=5e-5
ft_gradient_checkpointing=True
ft_num_train_epochs=3
ft_logging_steps=1
ft_per_device_train_batch_size=4
ft_gradient_accumulation_steps=8
ft_max_length=1024
ft_warmup_steps=25
ft_lr_scheduler_type="cosine"
ft_output_dir="ministral-8b-instruct-lora"
ft_push_to_hub=False
ft_report_to="wandb"
ft_eval_strategy="steps"
ft_eval_steps=125
```
Script de finetuning :
```python
import argparse
import os
import shlex
from huggingface_hub import login
from datasets import load_dataset
import transformers
from transformers import AutoTokenizer
import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, TaskType
from trl.trainer.sft_config import SFTConfig
from trl.trainer.sft_trainer import SFTTrainer
def _coerce_value(raw):
lowered = raw.lower()
if lowered == "true":
return True
if lowered == "false":
return False
try:
return int(raw)
except ValueError:
pass
try:
return float(raw)
except ValueError:
pass
return raw
def _load_params(path):
params = {}
with open(path, "r", encoding="utf-8") as f:
for line in f:
stripped = line.strip()
if not stripped or stripped.startswith("#"):
continue
tokens = shlex.split(stripped)
if not tokens or "=" not in tokens[0]:
continue
key, first_value = tokens[0].split("=", 1)
values = [first_value] + tokens[1:]
if len(values) == 1:
params[key] = _coerce_value(values[0])
else:
params[key] = [_coerce_value(v) for v in values]
return params
def _get_param(params, key, alias=None, default=None, required=True):
if key in params:
return params[key]
if alias and alias in params:
return params[alias]
if required:
raise ValueError(f"Missing required parameter: {key}")
return default
def _wants_wandb(report_to):
if report_to is None:
return False
if isinstance(report_to, str):
values = [report_to]
else:
values = report_to
normalized = {str(value).strip().lower() for value in values}
return "wandb" in normalized
def _require_env(name, help_text):
value = os.environ.get(name)
if value:
return value
raise RuntimeError(f"Missing environment variable {name!r}. {help_text}")
def _get_env(name):
value = os.environ.get(name)
if value:
return value
return None
def _parse_args():
parser = argparse.ArgumentParser()
parser.add_argument(
"--max-examples",
type=int,
default=None,
help="Limit train and validation datasets to the first N examples for quick tests.",
)
parser.add_argument(
"--push-to-hub",
action="store_true",
help="Enable Hub uploads for the current run. By default, uploads are disabled.",
)
return parser.parse_args()
def _limit_dataset(dataset, max_examples):
if max_examples is None:
return dataset
if max_examples <= 0:
raise ValueError("--max-examples must be a positive integer")
return dataset.select(range(min(max_examples, len(dataset))))
def _ensure_empty_output_dir(output_dir):
if not os.path.exists(output_dir):
return
if not os.path.isdir(output_dir):
raise RuntimeError(
f"Output path exists and is not a directory: {os.path.abspath(output_dir)}"
)
if os.listdir(output_dir):
raise RuntimeError(
"Output directory is not empty. "
f"Refusing to run: {os.path.abspath(output_dir)}"
)
def _ensure_assistant_generation_template(tokenizer, assistant_only_loss):
if not assistant_only_loss:
return
chat_template = getattr(tokenizer, "chat_template", None)
if not chat_template:
raise RuntimeError(
"assistant_only_loss=True requires a tokenizer chat template, but none is configured."
)
if "{% generation %}" in chat_template:
return
assistant_block = '{{- message["content"] + eos_token}}'
if assistant_block not in chat_template:
raise RuntimeError(
"assistant_only_loss=True requires assistant generation markers, and the chat template "
"does not contain the expected assistant block to patch automatically."
)
tokenizer.chat_template = chat_template.replace(
assistant_block,
'{% generation %}{{- message["content"] + eos_token}}{% endgeneration %}',
1,
)
print(
"Patched tokenizer chat template with generation markers so assistant_only_loss can be enabled."
)
def _visible_text(text):
return (
text.replace("\r", "<CR>")
.replace("\t", "<TAB>")
.replace("\n", "<EOL>\n")
)
def _masked_visible_text(text, char_mask):
masked_chars = []
for ch, keep in zip(text, char_mask):
if keep:
masked_chars.append(ch)
elif ch == "\n":
masked_chars.append("\n")
elif ch == "\t":
masked_chars.append("\t")
else:
masked_chars.append(".")
return _visible_text("".join(masked_chars))
def _preview_training_example(dataset, tokenizer, assistant_only_loss):
if len(dataset) == 0:
raise RuntimeError("Training dataset is empty.")
example = dataset[0]
messages = example["messages"]
rendered = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=False,
)
tokenized = tokenizer.apply_chat_template(
messages,
tokenize=True,
return_dict=True,
add_generation_prompt=False,
return_assistant_tokens_mask=assistant_only_loss,
)
input_ids = tokenized["input_ids"]
assistant_mask = tokenized.get("assistant_masks")
if assistant_mask is None:
assistant_mask = [1] * len(input_ids)
tokenized_with_offsets = tokenizer(
rendered,
add_special_tokens=False,
return_offsets_mapping=True,
)
offset_ids = tokenized_with_offsets["input_ids"]
offsets = tokenized_with_offsets["offset_mapping"]
if input_ids != offset_ids:
raise RuntimeError(
"Preview tokenization mismatch between chat template rendering and tokenizer offsets."
)
char_mask = [False] * len(rendered)
for keep, (start, end) in zip(assistant_mask, offsets):
if keep:
for idx in range(start, end):
char_mask[idx] = True
trainable_text = "".join(
ch for ch, keep in zip(rendered, char_mask) if keep
)
print("\n=== First Training Example Preview ===")
print(f"Messages in example: {len(messages)}")
print(f"Rendered characters: {len(rendered)}")
print(f"Rendered tokens: {len(input_ids)}")
print(f"Trainable tokens: {sum(assistant_mask)}")
print(
"\nView 1: Full model input."
" This is the exact text produced by the chat template and sent to the model."
" It should include control tokens such as <s>, [INST], [/INST], the user prompt,"
" and the assistant answer."
)
print("\nRendered chat template:")
print(_visible_text(rendered))
print(
"\nView 2: Same model input after applying the loss mask."
" Visible characters are the ones whose tokens contribute to the training loss."
" Dots mark text that is still provided to the model as context but should not be optimized,"
" such as [INST], the prompt, and other prompt-side control tokens."
)
print("\nRendered chat template after loss mask:")
print(_masked_visible_text(rendered, char_mask))
print(
"\nView 3: Trainable text only."
" This is the assistant-side text extracted from the masked view."
" It should contain only what we want the fine-tuning objective to learn to predict."
)
print("\nTrainable text only:")
print(_visible_text(trainable_text))
print("=== End Preview ===\n")
def _load_tokenizer(tokenizer_name):
attempts = [
{
"kwargs": {"use_fast": True, "fix_mistral_regex": True},
"reason": "preferred tokenizer load with mistral regex fix",
},
{
"kwargs": {"use_fast": True, "fix_mistral_regex": False},
"reason": "fallback when the local tokenizers API is incompatible with the mistral regex patch",
},
{
"kwargs": {"use_fast": False},
"reason": "last-resort tokenizer load without fast-tokenizer features",
},
]
errors = []
for attempt in attempts:
try:
if errors:
print(
f"Retrying tokenizer load for {tokenizer_name!r}: {attempt['reason']}."
)
return AutoTokenizer.from_pretrained(tokenizer_name, **attempt["kwargs"])
except Exception as exc:
errors.append((attempt["kwargs"], exc))
message = str(exc)
can_retry = False
if (
attempt["kwargs"].get("fix_mistral_regex") is True
and "backend_tokenizer" in message
):
can_retry = True
elif "fix_mistral_regex" in message:
can_retry = True
if not can_retry:
raise
attempted = ", ".join(str(kwargs) for kwargs, _ in errors)
raise RuntimeError(
f"Unable to load tokenizer {tokenizer_name!r} after trying: {attempted}"
) from errors[-1][1]
def _fold_system_into_user(example):
messages = example.get("messages", [])
if len(messages) < 2:
return example
first_message = messages[0]
second_message = messages[1]
if first_message.get("role") != "system" or second_message.get("role") != "user":
return example
system_text = (first_message.get("content") or "").strip()
user_text = (second_message.get("content") or "").strip()
merged_user = dict(second_message)
merged_user["content"] = (
"Instructions:\n"
f"{system_text}\n\n"
"Question utilisateur:\n"
f"{user_text}"
)
example["messages"] = [merged_user] + messages[2:]
return example
args = _parse_args()
params_path = os.environ.get("PARAMS_CFG", "params-small.cfg")
if not os.path.isabs(params_path):
params_path = os.path.join(os.path.dirname(__file__), params_path)
params = _load_params(params_path)
known_keys = {
"var_dataset_name",
"var_wandb_project",
"var_wandb_run",
"wandb_notebook_name",
"model_name",
"tokenizer",
"system_prompt",
"attn_implementation",
"ft_bf16",
"ft_assistant_only_loss",
"lora_r",
"lora_alpha",
"lora_dropout",
"lora_bias",
"lora_target_modules",
"ft_learning_rate",
"ft_gradient_checkpointing",
"ft_num_train_epochs",
"ft_logging_steps",
"ft_per_device_train_batch_size",
"ft_gradient_accumulation_steps",
"ft_max_length",
"ft_warmup_steps",
"ft_lr_scheduler_type",
"ft_output_dir",
"ft_push_to_hub",
"ft_report_to",
"ft_eval_strategy",
"ft_eval_steps",
}
unknown_keys = sorted(k for k in params.keys() if k not in known_keys)
if unknown_keys:
print(f"Unknown keys in {params_path}: {', '.join(unknown_keys)}")
target_modules = _get_param(params, "lora_target_modules")
if isinstance(target_modules, str):
target_modules = [target_modules]
resolved_params = {
"var_dataset_name": _get_param(params, "var_dataset_name"),
"var_wandb_project": _get_param(params, "var_wandb_project"),
"var_wandb_run": _get_param(params, "var_wandb_run"),
"wandb_notebook_name": _get_param(params, "wandb_notebook_name"),
"model_name": _get_param(params, "model_name", default=None, required=False),
"tokenizer": _get_param(params, "tokenizer"),
"attn_implementation": _get_param(params, "attn_implementation", default="eager", required=False),
"ft_bf16": _get_param(params, "ft_bf16", default=True, required=False),
"ft_assistant_only_loss": _get_param(params, "ft_assistant_only_loss", default=True, required=False),
"lora_r": _get_param(params, "lora_r"),
"lora_alpha": _get_param(params, "lora_alpha"),
"lora_dropout": _get_param(params, "lora_dropout"),
"lora_bias": _get_param(params, "lora_bias"),
"lora_target_modules": target_modules,
"ft_learning_rate": _get_param(params, "ft_learning_rate"),
"ft_gradient_checkpointing": _get_param(params, "ft_gradient_checkpointing"),
"ft_num_train_epochs": _get_param(params, "ft_num_train_epochs"),
"ft_logging_steps": _get_param(params, "ft_logging_steps"),
"ft_per_device_train_batch_size": _get_param(params, "ft_per_device_train_batch_size"),
"ft_gradient_accumulation_steps": _get_param(params, "ft_gradient_accumulation_steps"),
"ft_max_length": _get_param(params, "ft_max_length"),
"ft_warmup_steps": _get_param(params, "ft_warmup_steps"),
"ft_lr_scheduler_type": _get_param(params, "ft_lr_scheduler_type"),
"ft_output_dir": _get_param(params, "ft_output_dir"),
"ft_push_to_hub": _get_param(params, "ft_push_to_hub"),
"ft_report_to": _get_param(params, "ft_report_to"),
"ft_eval_strategy": _get_param(params, "ft_eval_strategy"),
"ft_eval_steps": _get_param(params, "ft_eval_steps"),
}
resolved_params["ft_push_to_hub"] = args.push_to_hub
_ensure_empty_output_dir(resolved_params["ft_output_dir"])
print(f"Config values from {params_path}:")
for key in sorted(resolved_params.keys()):
print(f" {key}={resolved_params[key]}")
var_dataset_name = resolved_params["var_dataset_name"]
var_wandb_project = resolved_params["var_wandb_project"]
var_wandb_run = resolved_params["var_wandb_run"]
wandb_notebook_name = resolved_params["wandb_notebook_name"]
tokenizer_name = resolved_params["tokenizer"]
model_name = resolved_params["model_name"] or tokenizer_name
model_kwargs = dict(
attn_implementation=resolved_params["attn_implementation"],
dtype=torch.bfloat16,
use_cache=False,
)
os.environ['WANDB_NOTEBOOK_NAME'] = wandb_notebook_name
hf_token = _get_env("hfkey")
if hf_token:
login(token=hf_token)
else:
print("hfkey is not set; relying on local Hugging Face access and cached credentials.")
if _wants_wandb(resolved_params["ft_report_to"]):
try:
import wandb
except ModuleNotFoundError as exc:
raise ModuleNotFoundError(
"Weights & Biases reporting is enabled but the 'wandb' package is not installed. "
"Install it with '.venv/bin/python -m pip install wandb' or disable W&B by setting "
"ft_report_to=\"none\" in the params file."
) from exc
wandb.login(key=_require_env("wandbkey", "Export your Weights & Biases token or disable W&B with ft_report_to=\"none\"."))
wandb.init(project=var_wandb_project, entity="alexandre-fenyo-fenyonet", name=var_wandb_run)
# Chargement du tokenizer
tokenizer = _load_tokenizer(tokenizer_name)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
_ensure_assistant_generation_template(
tokenizer, resolved_params["ft_assistant_only_loss"]
)
# Chargement du modèle
model = AutoModelForCausalLM.from_pretrained(model_name, **model_kwargs)
model.config.pad_token_id = tokenizer.pad_token_id
# Chargement des jeux de données
train_dataset = load_dataset(var_dataset_name, split="train")
# Conserve uniquement la colonne "messages" (ou adapte selon ton schéma)
train_dataset = train_dataset.remove_columns([c for c in train_dataset.column_names if c != "messages"])
train_dataset = train_dataset.map(_fold_system_into_user)
train_dataset = _limit_dataset(train_dataset, args.max_examples)
eval_dataset = load_dataset(var_dataset_name, split="validation")
eval_dataset = eval_dataset.remove_columns([c for c in eval_dataset.column_names if c != "messages"])
eval_dataset = eval_dataset.map(_fold_system_into_user)
eval_dataset = _limit_dataset(eval_dataset, args.max_examples)
print(f"Dataset sizes: train={len(train_dataset)} validation={len(eval_dataset)}")
_preview_training_example(
train_dataset, tokenizer, resolved_params["ft_assistant_only_loss"]
)
peft_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=resolved_params["lora_r"],
lora_alpha=resolved_params["lora_alpha"],
lora_dropout=resolved_params["lora_dropout"],
bias=resolved_params["lora_bias"],
target_modules=target_modules,
)
training_args = SFTConfig(
learning_rate=resolved_params["ft_learning_rate"],
bf16=resolved_params["ft_bf16"],
gradient_checkpointing=resolved_params["ft_gradient_checkpointing"],
num_train_epochs=resolved_params["ft_num_train_epochs"],
logging_steps=resolved_params["ft_logging_steps"],
per_device_train_batch_size=resolved_params["ft_per_device_train_batch_size"],
gradient_accumulation_steps=resolved_params["ft_gradient_accumulation_steps"],
max_length=resolved_params["ft_max_length"],
warmup_steps=resolved_params["ft_warmup_steps"],
lr_scheduler_type=resolved_params["ft_lr_scheduler_type"],
output_dir=resolved_params["ft_output_dir"],
push_to_hub=resolved_params["ft_push_to_hub"],
report_to=resolved_params["ft_report_to"],
eval_strategy=resolved_params["ft_eval_strategy"],
eval_steps=resolved_params["ft_eval_steps"],
assistant_only_loss=resolved_params["ft_assistant_only_loss"],
save_strategy="epoch",
save_total_limit=100,
)
trainer = SFTTrainer(
model=model,
peft_config=peft_config,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
processing_class=tokenizer,
)
trainer.train()
trainer.save_model(training_args.output_dir)
# trainer.push_to_hub(dataset_name=var_dataset_name)
```
### Framework versions
```
accelerate==1.13.0
datasets==4.8.4
huggingface_hub==1.8.0
nbclient==0.10.4
nbformat==5.10.4
peft==0.18.1
tokenizers==0.22.2
torch==2.11.0
transformers==5.4.0
trl==0.29.1
wandb==0.25.1
```

87
chat_template.jinja Normal file
View File

@@ -0,0 +1,87 @@
{%- if messages[0]["role"] == "system" %}
{%- set system_message = messages[0]["content"] %}
{%- set loop_messages = messages[1:] %}
{%- else %}
{%- set loop_messages = messages %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = none %}
{%- endif %}
{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
{%- set ns = namespace() %}
{%- set ns.index = 0 %}
{%- for message in loop_messages %}
{%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
{%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
{{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
{%- endif %}
{%- set ns.index = ns.index + 1 %}
{%- endif %}
{%- endfor %}
{{- bos_token }}
{%- for message in loop_messages %}
{%- if message["role"] == "user" %}
{%- if tools is not none and (message == user_messages[-1]) %}
{{- "[AVAILABLE_TOOLS][" }}
{%- for tool in tools %}
{%- set tool = tool.function %}
{{- '{"type": "function", "function": {' }}
{%- for key, val in tool.items() if key != "return" %}
{%- if val is string %}
{{- '"' + key + '": "' + val + '"' }}
{%- else %}
{{- '"' + key + '": ' + val|tojson }}
{%- endif %}
{%- if not loop.last %}
{{- ", " }}
{%- endif %}
{%- endfor %}
{{- "}}" }}
{%- if not loop.last %}
{{- ", " }}
{%- else %}
{{- "]" }}
{%- endif %}
{%- endfor %}
{{- "[/AVAILABLE_TOOLS]" }}
{%- endif %}
{%- if loop.last and system_message is defined %}
{{- "[INST]" + system_message + "\n\n" + message["content"] + "[/INST]" }}
{%- else %}
{{- "[INST]" + message["content"] + "[/INST]" }}
{%- endif %}
{%- elif (message.tool_calls is defined and message.tool_calls is not none) %}
{{- "[TOOL_CALLS][" }}
{%- for tool_call in message.tool_calls %}
{%- set out = tool_call.function|tojson %}
{{- out[:-1] }}
{%- if not tool_call.id is defined or tool_call.id|length != 9 %}
{{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
{%- endif %}
{{- ', "id": "' + tool_call.id + '"}' }}
{%- if not loop.last %}
{{- ", " }}
{%- else %}
{{- "]" + eos_token }}
{%- endif %}
{%- endfor %}
{%- elif message["role"] == "assistant" %}
{% generation %}{{- message["content"] + eos_token}}{% endgeneration %}
{%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
{%- if message.content is defined and message.content.content is defined %}
{%- set content = message.content.content %}
{%- else %}
{%- set content = message.content %}
{%- endif %}
{{- '[TOOL_RESULTS]{"content": ' + content|string + ", " }}
{%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
{{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
{%- endif %}
{{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
{%- else %}
{{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
{%- endif %}
{%- endfor %}

68
config.json Normal file
View File

@@ -0,0 +1,68 @@
{
"architectures": [
"MistralForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 1,
"dtype": "bfloat16",
"eos_token_id": 2,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 12288,
"layer_types": [
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention",
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention",
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention",
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention",
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention",
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention",
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention",
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention",
"full_attention",
"sliding_attention",
"sliding_attention",
"sliding_attention"
],
"max_position_embeddings": 32768,
"model_type": "ministral",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"pad_token_id": null,
"rms_norm_eps": 1e-05,
"rope_parameters": {
"rope_theta": 100000000.0,
"rope_type": "default"
},
"sliding_window": 32768,
"tie_word_embeddings": false,
"transformers_version": "5.4.0",
"use_cache": true,
"vocab_size": 131072
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"_from_model_config": true,
"bos_token_id": 1,
"eos_token_id": 2,
"transformers_version": "5.4.0"
}

3
model-00001-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fc9a76c2486fc7bccaf27cca8dbdfd1668e8330a3755c3203a5202fecfa932ff
size 1929415304

3
model-00002-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b9e091d56b94d69f9982ea8b8a142fa403de53a1eed67407ed201952b751dbf6
size 1929466952

3
model-00003-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b1d95fb816b87478a8e9e819fcecc4c245413068192b146952fb01994331bb09
size 1929466968

3
model-00004-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b3905b8ffb91f9b6f062442149bc1e351e417e7d0ec421813915e936589629b0
size 1929467000

3
model-00005-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d99c98569df42f8a9ecb1f6e070a9699ecde87b18e746ed6cd328c2ab713f4e0
size 1929467000

3
model-00006-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1c833fe1691299929f0a87910471b4f3239fd31466162faea47dbd53dad51563
size 1929467000

3
model-00007-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e3f2b52916541782a73bc5b7a1e8bdfd5202883d21c44417c8573384811267bb
size 1929467000

3
model-00008-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f73c4f73727a000a92ab8593795337042692f30e3b42602ac3be12fe83a46b18
size 1459695328

3
model-00009-of-00009.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e75c6183282198d40135032c9e757fb70c2455f8c0f4ded7b0895b49b72bbd5f
size 1073741952

334
model.safetensors.index.json Normal file
View File

@@ -0,0 +1,334 @@
{
"metadata": {
"total_size": 16039616512
},
"weight_map": {
"model.embed_tokens.weight": "model-00001-of-00009.safetensors",
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.0.input_layernorm.weight": "model-00001-of-00009.safetensors",
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00009.safetensors",
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.1.input_layernorm.weight": "model-00001-of-00009.safetensors",
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00009.safetensors",
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00009.safetensors",
"model.layers.2.mlp.gate_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.2.mlp.up_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.2.mlp.down_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.2.input_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.2.post_attention_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.3.self_attn.q_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.3.self_attn.k_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.3.self_attn.v_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.3.self_attn.o_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.3.mlp.gate_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.3.mlp.up_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.3.mlp.down_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.3.input_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.3.post_attention_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.4.self_attn.q_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.4.self_attn.k_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.4.self_attn.v_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.4.self_attn.o_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.4.mlp.gate_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.4.mlp.up_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.4.mlp.down_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.4.input_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.4.post_attention_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.5.self_attn.q_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.5.self_attn.k_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.5.self_attn.v_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.5.self_attn.o_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.5.mlp.gate_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.5.mlp.up_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.5.mlp.down_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.5.input_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.5.post_attention_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.6.self_attn.q_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.6.self_attn.k_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.6.self_attn.v_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.6.self_attn.o_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.6.mlp.gate_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.6.mlp.up_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.6.mlp.down_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.6.input_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.6.post_attention_layernorm.weight": "model-00002-of-00009.safetensors",
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.7.self_attn.k_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.7.self_attn.v_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.7.self_attn.o_proj.weight": "model-00002-of-00009.safetensors",
"model.layers.7.mlp.gate_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.7.mlp.up_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.7.mlp.down_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.7.input_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.7.post_attention_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.8.self_attn.q_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.8.self_attn.k_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.8.self_attn.v_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.8.self_attn.o_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.8.mlp.gate_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.8.mlp.up_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.8.mlp.down_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.8.input_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.8.post_attention_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.9.self_attn.q_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.9.self_attn.k_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.9.self_attn.v_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.9.self_attn.o_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.9.mlp.gate_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.9.mlp.up_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.9.mlp.down_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.9.input_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.9.post_attention_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.10.self_attn.q_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.10.self_attn.k_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.10.self_attn.v_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.10.self_attn.o_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.10.mlp.gate_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.10.mlp.up_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.10.mlp.down_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.10.input_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.10.post_attention_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.11.self_attn.q_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.11.self_attn.k_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.11.self_attn.v_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.11.self_attn.o_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.11.mlp.gate_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.11.mlp.up_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.11.mlp.down_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.11.input_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.11.post_attention_layernorm.weight": "model-00003-of-00009.safetensors",
"model.layers.12.self_attn.q_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.12.self_attn.k_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.12.self_attn.v_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.12.self_attn.o_proj.weight": "model-00003-of-00009.safetensors",
"model.layers.12.mlp.gate_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.12.mlp.up_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.12.mlp.down_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.12.input_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.12.post_attention_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.13.self_attn.q_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.13.self_attn.k_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.13.self_attn.v_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.13.self_attn.o_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.13.mlp.gate_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.13.mlp.up_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.13.mlp.down_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.13.input_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.13.post_attention_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.14.self_attn.q_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.14.self_attn.k_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.14.self_attn.v_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.14.self_attn.o_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.14.mlp.gate_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.14.mlp.up_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.14.mlp.down_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.14.input_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.14.post_attention_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.15.self_attn.q_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.15.self_attn.k_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.15.self_attn.v_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.15.self_attn.o_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.15.mlp.gate_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.15.mlp.up_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.15.mlp.down_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.15.input_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.15.post_attention_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.16.self_attn.q_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.16.self_attn.k_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.16.self_attn.v_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.16.self_attn.o_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.16.mlp.gate_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.16.mlp.up_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.16.mlp.down_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.16.input_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.16.post_attention_layernorm.weight": "model-00004-of-00009.safetensors",
"model.layers.17.self_attn.q_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.17.self_attn.k_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.17.self_attn.v_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.17.self_attn.o_proj.weight": "model-00004-of-00009.safetensors",
"model.layers.17.mlp.gate_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.17.mlp.up_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.17.mlp.down_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.17.input_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.17.post_attention_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.18.self_attn.q_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.18.self_attn.k_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.18.self_attn.v_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.18.self_attn.o_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.18.mlp.gate_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.18.mlp.up_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.18.mlp.down_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.18.input_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.18.post_attention_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.19.self_attn.q_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.19.self_attn.k_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.19.self_attn.v_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.19.self_attn.o_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.19.mlp.gate_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.19.mlp.up_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.19.mlp.down_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.19.input_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.19.post_attention_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.20.self_attn.q_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.20.self_attn.k_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.20.self_attn.v_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.20.self_attn.o_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.20.mlp.gate_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.20.mlp.up_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.20.mlp.down_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.20.input_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.20.post_attention_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.21.self_attn.q_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.21.self_attn.k_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.21.self_attn.v_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.21.self_attn.o_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.21.mlp.gate_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.21.mlp.up_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.21.mlp.down_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.21.input_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.21.post_attention_layernorm.weight": "model-00005-of-00009.safetensors",
"model.layers.22.self_attn.q_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.22.self_attn.k_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.22.self_attn.v_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.22.self_attn.o_proj.weight": "model-00005-of-00009.safetensors",
"model.layers.22.mlp.gate_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.22.mlp.up_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.22.mlp.down_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.22.input_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.22.post_attention_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.23.self_attn.q_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.23.self_attn.k_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.23.self_attn.v_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.23.self_attn.o_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.23.mlp.gate_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.23.mlp.up_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.23.mlp.down_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.23.input_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.23.post_attention_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.24.self_attn.q_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.24.self_attn.k_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.24.self_attn.v_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.24.self_attn.o_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.24.mlp.gate_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.24.mlp.up_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.24.mlp.down_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.24.input_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.24.post_attention_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.25.self_attn.q_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.25.self_attn.k_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.25.self_attn.v_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.25.self_attn.o_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.25.mlp.gate_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.25.mlp.up_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.25.mlp.down_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.25.input_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.25.post_attention_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.26.self_attn.q_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.26.self_attn.k_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.26.self_attn.v_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.26.self_attn.o_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.26.mlp.gate_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.26.mlp.up_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.26.mlp.down_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.26.input_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.26.post_attention_layernorm.weight": "model-00006-of-00009.safetensors",
"model.layers.27.self_attn.q_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.27.self_attn.k_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.27.self_attn.v_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.27.self_attn.o_proj.weight": "model-00006-of-00009.safetensors",
"model.layers.27.mlp.gate_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.27.mlp.up_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.27.mlp.down_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.27.input_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.27.post_attention_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.28.self_attn.q_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.28.self_attn.k_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.28.self_attn.v_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.28.self_attn.o_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.28.mlp.gate_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.28.mlp.up_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.28.mlp.down_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.28.input_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.28.post_attention_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.29.self_attn.q_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.29.self_attn.k_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.29.self_attn.v_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.29.self_attn.o_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.29.mlp.gate_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.29.mlp.up_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.29.mlp.down_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.29.input_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.29.post_attention_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.30.self_attn.q_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.30.self_attn.k_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.30.self_attn.v_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.30.self_attn.o_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.30.mlp.gate_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.30.mlp.up_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.30.mlp.down_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.30.input_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.30.post_attention_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.31.self_attn.q_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.31.self_attn.k_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.31.self_attn.v_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.31.self_attn.o_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.31.mlp.gate_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.31.mlp.up_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.31.mlp.down_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.31.input_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.31.post_attention_layernorm.weight": "model-00007-of-00009.safetensors",
"model.layers.32.self_attn.q_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.32.self_attn.k_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.32.self_attn.v_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.32.self_attn.o_proj.weight": "model-00007-of-00009.safetensors",
"model.layers.32.mlp.gate_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.32.mlp.up_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.32.mlp.down_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.32.input_layernorm.weight": "model-00008-of-00009.safetensors",
"model.layers.32.post_attention_layernorm.weight": "model-00008-of-00009.safetensors",
"model.layers.33.self_attn.q_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.33.self_attn.k_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.33.self_attn.v_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.33.self_attn.o_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.33.mlp.gate_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.33.mlp.up_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.33.mlp.down_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.33.input_layernorm.weight": "model-00008-of-00009.safetensors",
"model.layers.33.post_attention_layernorm.weight": "model-00008-of-00009.safetensors",
"model.layers.34.self_attn.q_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.34.self_attn.k_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.34.self_attn.v_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.34.self_attn.o_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.34.mlp.gate_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.34.mlp.up_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.34.mlp.down_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.34.input_layernorm.weight": "model-00008-of-00009.safetensors",
"model.layers.34.post_attention_layernorm.weight": "model-00008-of-00009.safetensors",
"model.layers.35.self_attn.q_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.35.self_attn.k_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.35.self_attn.v_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.35.self_attn.o_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.35.mlp.gate_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.35.mlp.up_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.35.mlp.down_proj.weight": "model-00008-of-00009.safetensors",
"model.layers.35.input_layernorm.weight": "model-00008-of-00009.safetensors",
"model.layers.35.post_attention_layernorm.weight": "model-00008-of-00009.safetensors",
"model.norm.weight": "model-00008-of-00009.safetensors",
"lm_head.weight": "model-00009-of-00009.safetensors"
}
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d7edbeaf20dd7f571b5dd1c54d9ace4f9b6299127cc7ba2afb14a6d51a4a79a4
size 17078136

14
tokenizer_config.json Normal file
View File

@@ -0,0 +1,14 @@
{
"add_prefix_space": null,
"backend": "tokenizers",
"bos_token": "<s>",
"clean_up_tokenization_spaces": false,
"eos_token": "</s>",
"fix_mistral_regex": false,
"is_local": true,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "</s>",
"tokenizer_class": "TokenizersBackend",
"unk_token": "<unk>",
"use_default_system_prompt": false
}