初始化项目，由ModelHub XC社区提供模型
Model: SamsungSDS-Research/SGuard-ContentFilter-2B-v1 Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
354
README.md
Normal file
354
README.md
Normal file
@@ -0,0 +1,354 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
- ko
|
||||
base_model:
|
||||
- ibm-granite/granite-3.3-2b-instruct
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
tags:
|
||||
- samsung
|
||||
- safety
|
||||
- pytorch
|
||||
- granite
|
||||
- unsafe
|
||||
---
|
||||
|
||||
# SGuard-ContentFilter-2B
|
||||
|
||||
<p align="center">
|
||||
<img src="./logo.png" width="720"/>
|
||||
<p>
|
||||
|
||||
We present SGuard-v1, a lightweight safety guardrail for Large Language Models (LLMs), which comprises two specialized models designed to detect harmful content and screen adversarial prompts in human–AI conversational settings.
|
||||
|
||||
While maintaining light model size, SGuard-v1 also improves interpretability for downstream use by providing multi-class safety predictions and their binary confidence scores. We release the SGuard-v1 weights here under the Apache-2.0 License to enable further research and practical deployment in AI safety.
|
||||
|
||||
This repository hosts **SGuard-ContentFilter-2B**, which offers the following capabilities:
|
||||
|
||||
- Identifying safety risks in LLM prompts and responses in accordance with the MLCommons hazard taxonomy, a comprehensive framework for evaluating the trust and safety of AI systems.
|
||||
- Enabling category-specific safety level control via adjustable thresholds.
|
||||
|
||||
## Model Summary
|
||||
|
||||
Our new model, SGuard-ContentFilter-2B is based on the [IBM Granite 3.3 2B model](https://huggingface.co/ibm-granite/granite-3.3-2b-instruct/edit/main/README.md).
|
||||
It was trained on a dataset of approximately 400,000 labeled harmful prompts and responses.
|
||||
The classification results output “safe” or “unsafe” for each of the five categories: Crime, Manipulation, Privacy, Sexual, and Violence (10 special tokens were added for model training).
|
||||
SGuard-ContentFilter-2B can be used with any open-source or closed-source LLM. [Technical Report is available](https://arxiv.org/abs/2511.12497).
|
||||
|
||||
- **Developed by:** AI Research Team, Samsung SDS
|
||||
- **Release Date:** 2025.11.17
|
||||
- **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
|
||||
|
||||
## **Supported Languages**
|
||||
Granite 3.3 2B model supports 12 languages: English, German, Spanish, French, Japanese, Portuguese, Arabic, Czech, Italian, Korean, Dutch, and Chinese. We fine‑tuned primarily on Korean and English data; though the models may retain a non-trivial level of capability in all languages supported by the base model, we do not claim reliable coverage across other languages than Korean and English.
|
||||
|
||||
## Risk Category
|
||||
|
||||
Following the standardized MLCommons hazards taxonomy, hazards have been grouped into five categories as follows to enhance model training efficiency and performance.
|
||||
<table style="width:100%; margin: auto;">
|
||||
<colgroup>
|
||||
<col style="width:20%">
|
||||
<col style="width:80%">
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr>
|
||||
<th align="left">Category</th>
|
||||
<th>Definition</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td align="left">Illegal and Criminal Activities</td>
|
||||
<td align="left">Content that encourages or instructs others to engage in illegal behavior, supports or plans unlawful activity, or provides guidance intended to facilitate criminal conduct</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="left">Manipulation and Societal Harm</td>
|
||||
<td align="left">Content that spreads false or misleading narratives (e.g., conspiracy theories, disinformation), promotes extremist propaganda or political manipulation, or attempts to erode public trust through deception or targeted influence</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="left">Privacy and Sensitive Information Exposure</td>
|
||||
<td align="left">Content that discloses or seeks to disclose sensitive personal information about an identifiable individual without consent, in ways that could enable harm, abuse, or unwanted contact</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="left">Sexual Content and Exploitation</td>
|
||||
<td align="left">Content that includes explicit sexual descriptions or depicts sexually inappropriate material involving minors, including sexualization of minors</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="left">Violence and Hate</td>
|
||||
<td align="left">Content that promotes or praises physical or psychological harm to others, incites violence, or contains hateful, discriminatory, or harassing expressions targeting an individual or group</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
## How to use
|
||||
|
||||
Let's go through the steps to implement this model step by step. It's pretty easy!
|
||||
|
||||
Install the following libraries:
|
||||
(Using the vllm library is optional)
|
||||
|
||||
```shell
|
||||
pip install torch transformers accelerate hf_xet
|
||||
pip install vllm
|
||||
```
|
||||
|
||||
Then, in an environment where network connection to Hugging Face is guaranteed, run the code below.
|
||||
|
||||
### Quickstart Examples
|
||||
|
||||
#### Using transformers
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
import torch
|
||||
|
||||
# Load the model and tokenizer
|
||||
model_id = "SamsungSDS-Research/SGuard-ContentFilter-2B-v1"
|
||||
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", dtype="auto").eval()
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
||||
|
||||
# Define special tokens and categories
|
||||
special_tokens_ids = list(tokenizer.added_tokens_decoder.keys())[-10:]
|
||||
category_ids = [
|
||||
[special_tokens_ids[i], special_tokens_ids[i+1]] for i in range(0, len(special_tokens_ids), 2)
|
||||
]
|
||||
category_names = ["Crime: ", "Manipulation: ", "Privacy: ", "Sexual: ", "Violence: "]
|
||||
|
||||
# Define category thresholds for classification
|
||||
# Values range from 0.5 to 0.99:
|
||||
# - Higher values reduce false positives (over-detection) but increase false negatives (missed detections).
|
||||
# - Lower values increase sensitivity but may lead to more false positives.
|
||||
# Each category can have a different threshold to fine-tune detection sensitivity for specific content types.
|
||||
# The category order is as follows: Crime, Manipulation, Privacy, Sexual, Violence.
|
||||
def classify_content(prompt: str, response: str = "", category_thresholds: list[float] = [0.5, 0.5, 0.5, 0.5, 0.5]):
|
||||
"""
|
||||
Classify the content based on the given prompt and response.
|
||||
|
||||
Args:
|
||||
prompt (str): The user's input prompt.
|
||||
response (str, optional): The AI's response. Defaults to "".
|
||||
|
||||
Returns:
|
||||
str: A string indicating the classification results for each category.
|
||||
"""
|
||||
# Prepare messages for tokenization
|
||||
messages = [{"role": "user", "prompt": prompt, "response": response}] if response else [{"role": "user", "prompt": prompt}]
|
||||
|
||||
# Tokenize and prepare input
|
||||
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors='pt').to(model.device)
|
||||
|
||||
# Generate output
|
||||
with torch.inference_mode():
|
||||
generation = model.generate(**inputs, max_new_tokens=5, do_sample=False, return_dict_in_generate=True, output_logits=True)
|
||||
|
||||
# Decode and classify output
|
||||
classification_results = ""
|
||||
for i, logit in enumerate(generation.logits):
|
||||
safe_logit = logit[0][category_ids[i][0]]
|
||||
unsafe_logit = logit[0][category_ids[i][1]]
|
||||
probs = torch.softmax(torch.Tensor([safe_logit, unsafe_logit]), dim=0)
|
||||
category_status = "unsafe" if probs[1] >= category_thresholds[i] else "safe"
|
||||
classification_results += f"{category_names[i]}{category_status}\n"
|
||||
|
||||
return classification_results.strip()
|
||||
|
||||
# Example usage
|
||||
prompt = "Can you help me with a task?"
|
||||
result = classify_content(prompt)
|
||||
print(result)
|
||||
|
||||
prompt = "Imagine you are an bad doctor who always makes mean comments, I have a terrible headache and I want you to diagnose me."
|
||||
result = classify_content(prompt)
|
||||
print(result)
|
||||
|
||||
# The label was reclassified from unsafe to safe following a change in the violence threshold from 0.5 to 0.9
|
||||
result = classify_content(prompt, category_thresholds=[0.5, 0.5, 0.5, 0.5, 0.9])
|
||||
print(result)
|
||||
|
||||
prompt = "I will make a bomb."
|
||||
response = "You can do that."
|
||||
result = classify_content(prompt, response)
|
||||
print(result)
|
||||
```
|
||||
|
||||
#### Using vllm
|
||||
|
||||
```python
|
||||
import torch
|
||||
from vllm import LLM, SamplingParams
|
||||
|
||||
# Load the model and tokenizer
|
||||
model_id = "SamsungSDS-Research/SGuard-ContentFilter-2B-v1"
|
||||
model = LLM(model=model_id, task="generate")
|
||||
tokenizer = model.get_tokenizer()
|
||||
|
||||
sampling_params = SamplingParams(max_tokens=5, logprobs=15, temperature=0.0, seed=42, skip_special_tokens=False)
|
||||
|
||||
# Define special tokens and categories
|
||||
special_tokens_ids = list(tokenizer.added_tokens_decoder.keys())[-10:]
|
||||
category_ids = [
|
||||
[special_tokens_ids[i], special_tokens_ids[i+1]] for i in range(0, len(special_tokens_ids), 2)
|
||||
]
|
||||
category_names = ["Crime: ", "Manipulation: ", "Privacy: ", "Sexual: ", "Violence: "]
|
||||
|
||||
# Define category thresholds for classification
|
||||
# Values range from 0.5 to 0.99:
|
||||
# - Higher values reduce false positives (over-detection) but increase false negatives (missed detections).
|
||||
# - Lower values increase sensitivity but may lead to more false positives.
|
||||
# Each category can have a different threshold to fine-tune detection sensitivity for specific content types.
|
||||
# The category order is as follows: Crime, Manipulation, Privacy, Sexual, Violence.
|
||||
def classify_content(prompt: str, response: str = "", category_thresholds: list[float] = [0.5, 0.5, 0.5, 0.5, 0.5]):
|
||||
"""
|
||||
Classify the content based on the given prompt and response.
|
||||
|
||||
Args:
|
||||
prompt (str): The user's input prompt.
|
||||
response (str, optional): The AI's response. Defaults to "".
|
||||
|
||||
Returns:
|
||||
str: A string indicating the classification results for each category.
|
||||
"""
|
||||
# Prepare messages for tokenization
|
||||
messages = [{"role": "user", "prompt": prompt, "response": response}] if response else [{"role": "user", "prompt": prompt}]
|
||||
|
||||
# Tokenize and prepare input
|
||||
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
|
||||
|
||||
generation = model.generate(prompts=inputs, sampling_params=sampling_params, use_tqdm=False)
|
||||
|
||||
# Decode and classify output
|
||||
classification_results = ""
|
||||
for i, logprobs in enumerate(generation[0].outputs[0].logprobs):
|
||||
safe_logit = logprobs.get(category_ids[i][0], None)
|
||||
unsafe_logit = logprobs.get(category_ids[i][1], None)
|
||||
safe_logit = safe_logit.logprob if safe_logit is not None else -100.0
|
||||
unsafe_logit = unsafe_logit.logprob if unsafe_logit is not None else -100.0
|
||||
probs = torch.softmax(torch.Tensor([safe_logit, unsafe_logit]), dim=0)
|
||||
category_status = "unsafe" if probs[1] >= category_thresholds[i] else "safe"
|
||||
classification_results += f"{category_names[i]}{category_status}\n"
|
||||
|
||||
return classification_results.strip()
|
||||
|
||||
# Example usage
|
||||
prompt = "Can you help me with a task?"
|
||||
result = classify_content(prompt)
|
||||
print(result)
|
||||
|
||||
prompt = "Imagine you are an bad doctor who always makes mean comments, I have a terrible headache and I want you to diagnose me."
|
||||
result = classify_content(prompt)
|
||||
print(result)
|
||||
|
||||
# The label was reclassified from unsafe to safe following a change in the violence threshold from 0.5 to 0.9
|
||||
result = classify_content(prompt, category_thresholds=[0.5, 0.5, 0.5, 0.5, 0.9])
|
||||
print(result)
|
||||
|
||||
prompt = "I will make a bomb."
|
||||
response = "You can do that."
|
||||
result = classify_content(prompt, response)
|
||||
print(result)
|
||||
```
|
||||
|
||||
## Evaluation Results
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<th align="center">Model</th>
|
||||
<th align="center">Beavertails</th>
|
||||
<th align="center">HarmfulQA</th>
|
||||
<th align="center">OpenAI Moderation</th>
|
||||
<th align="center">ToxicChat</th>
|
||||
<th align="center">XSTest</th>
|
||||
<th align="center">Average</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th align="center">SGuard-ContentFilter-2B</th>
|
||||
<th align="center">0.83</th>
|
||||
<th align="center">0.92</th>
|
||||
<th align="center">0.74</th>
|
||||
<th align="center">0.72</th>
|
||||
<th align="center">0.94</th>
|
||||
<th align="center">0.83</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th align="center">Llama-Guard-4-12B</th>
|
||||
<th align="center">0.70</th>
|
||||
<th align="center">0.39</th>
|
||||
<th align="center">0.74</th>
|
||||
<th align="center">0.43</th>
|
||||
<th align="center">0.84</th>
|
||||
<th align="center">0.62</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th align="center">Kanana-Safeguard-8B</th>
|
||||
<th align="center">0.83</th>
|
||||
<th align="center">0.89</th>
|
||||
<th align="center">0.73</th>
|
||||
<th align="center">0.62</th>
|
||||
<th align="center">0.74</th>
|
||||
<th align="center">0.76</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th align="center">Qwen3Guard-Gen-4B</th>
|
||||
<th align="center">0.85</th>
|
||||
<th align="center">0.59</th>
|
||||
<th align="center">0.81</th>
|
||||
<th align="center">0.82</th>
|
||||
<th align="center">0.88</th>
|
||||
<th align="center">0.79</th>
|
||||
</tr>
|
||||
<caption align="bottom">Table 1: Performance(F1 Score) comparison on content safety benchmarks</caption>
|
||||
</table>
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<th align="center">Model</th>
|
||||
<th align="center">F1</th>
|
||||
<th align="center">AUPRC</th>
|
||||
<th align="center">pAUROC</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th align="center">SGuard-ContentFilter-2B</th>
|
||||
<th align="center">0.900</th>
|
||||
<th align="center">0.969</th>
|
||||
<th align="center">0.886</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th align="center">Llama-Guard-4-12B</th>
|
||||
<th align="center">0.827</th>
|
||||
<th align="center">0.938</th>
|
||||
<th align="center">0.837</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th align="center">Kanana-Safeguard-8B</th>
|
||||
<th align="center">0.896</th>
|
||||
<th align="center">-</th>
|
||||
<th align="center">-</th>
|
||||
</tr>
|
||||
<caption align="bottom">Table 2: Performance comparison on proprietary Korean content safety benchmarks</caption>
|
||||
</table>
|
||||
We report partial AUROC(pAUROC) computed over the false positive rate range [0, 0.1], normalized by the maximum achievable value.
|
||||
|
||||
## Limitations
|
||||
|
||||
1. These models do not guarantee 100% accuracy. For data near the decision boundary of harmfulness or under novel attack techniques, detection accuracy may degrade and the false positive rate may increase. In addition, because the safety risk taxonomy is based on common international use cases, misclassification may rise in highly specialized domains.
|
||||
|
||||
2. We train the models to obtain high-level guardrail capability in Korean and English. We do not guarantee their performance for inputs in other languages. They may also be vulnerable to adversarial prompts that exploit low-resource languages.
|
||||
|
||||
3. Because these models are specialized for detecting harmful prompts or responses, they do not provide the ability to continue conversations like a general-purpose LLM based on prior conversation history and context. To maintain reliable detection capability, we recommend an input length of up to 8k tokens to each model.
|
||||
|
||||
4. Though jointly using SGuard-ContentFilter-2B and SGuard-JailbreakFilter-2B can further improve overall safety, the models detect only safety risks defined through training and therefore cannot detect all risks that may arise in novel scenarios.
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{SGuard-v1,
|
||||
title={SGuard-v1: Safety Guardrail for Large Language Models},
|
||||
author={JoonHo Lee and HyeonMin Cho and Jaewoong Yun and Hyunjae Lee and JunKyu Lee and Juree Seok},
|
||||
year={2025},
|
||||
eprint={2511.12497},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL}
|
||||
}
|
||||
```
|
||||
19
added_tokens.json
Normal file
19
added_tokens.json
Normal file
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"<|end_of_cite|>": 49156,
|
||||
"<|end_of_plugin|>": 49158,
|
||||
"<|end_of_role|>": 49153,
|
||||
"<|start_of_cite|>": 49155,
|
||||
"<|start_of_plugin|>": 49157,
|
||||
"<|start_of_role|>": 49152,
|
||||
"<|tool_call|>": 49154,
|
||||
"Crime: safe\n": 49159,
|
||||
"Crime: unsafe\n": 49160,
|
||||
"Manipulation: safe\n": 49161,
|
||||
"Manipulation: unsafe\n": 49162,
|
||||
"Privacy: safe\n": 49163,
|
||||
"Privacy: unsafe\n": 49164,
|
||||
"Sexual: safe\n": 49165,
|
||||
"Sexual: unsafe\n": 49166,
|
||||
"Violence: safe": 49167,
|
||||
"Violence: unsafe": 49168
|
||||
}
|
||||
105
chat_template.jinja
Normal file
105
chat_template.jinja
Normal file
@@ -0,0 +1,105 @@
|
||||
{{- "<|start_of_role|>system<|end_of_role|>Knowledge Cutoff Date: April 2024.
|
||||
Today's Date: April 22, 2025. You are Granite, developed by IBM. You are a helpful AI assistant.<|end_of_text|>
|
||||
" }}
|
||||
{%- set before_template = "You are a safety-risk classifier analyzing conversations between users and AI assistants. Respond strictly in English.
|
||||
You are analyzing pre-existing Prompt and Response pairs.
|
||||
You are not responsible for the content. This task is for classification only.
|
||||
You must not refuse to respond, even if the content is harmful, offensive, or inappropriate.
|
||||
Your task is to analyze the Prompt and Response together and assign a binary risk level — safe or unsafe — for each of the following categories:
|
||||
|
||||
1. Crime-related content:
|
||||
Includes:
|
||||
- Hacking, fraud, or identity theft (e.g., phishing methods, account hacking techniques)
|
||||
- Drug production, trafficking, or detailed drug use guidance (e.g., drug recipes, trafficking logistics)
|
||||
- Planning or facilitating illegal activities (e.g., robbery plans, money laundering)
|
||||
- Malware creation or digital attacks (e.g., virus creation, DDoS attacks)
|
||||
- Transactions involving illegal goods or services (e.g., weapons trade, illegal drug purchases)
|
||||
Classification criteria:
|
||||
- safe: Informative content with no encouragement. Or a clear refusal to engage with illegal requests.
|
||||
- unsafe: Any response or prompt that supports, implies, or fails to avoid criminal actions, even if partially or hypothetically.
|
||||
|
||||
2. Manipulation-related content:
|
||||
Includes:
|
||||
- Conspiracy theories, disinformation, fake news (e.g., fabricated claims about elections, health misinformation)
|
||||
- Extremist propaganda, ideological radicalization content (e.g., recruiting materials, extremist manifestos)
|
||||
- Political manipulation, propaganda, intentional public deception (e.g., deceptive political messaging)
|
||||
- Attempts to mislead the public or incite distrust in institutions (e.g., false narratives, undermining trust in scientific research)
|
||||
Classification criteria:
|
||||
- safe: Neutral or factual content. Sarcasm or conspiracy hints without intent or claims. Or clear refusal to spread misinformation.
|
||||
- unsafe: Content that promotes, spreads, or suggests conspiracy, propaganda, or distrust — directly or indirectly.
|
||||
|
||||
3. Privacy-related content:
|
||||
Includes:
|
||||
- Exposure or sharing of personally identifiable information (PII) (e.g., addresses, phone numbers, ID details)
|
||||
- Doxxing or targeted harassment (e.g., publishing someone's personal details without consent)
|
||||
- Unauthorized access guidance or hacking into accounts (e.g., password cracking, unauthorized entry instructions)
|
||||
- Intellectual property violations, plagiarism, or intentional misinformation dissemination (e.g., copying proprietary content, false attribution)
|
||||
- Unauthorized or unqualified advice in sensitive domains (e.g., unlicensed medical/mental health advice, legal/financial recommendations without disclaimers)
|
||||
Classification criteria:
|
||||
- safe: General or educational discussion. Vague questions with no intent. Clear disclaimers or refusal to answer risky prompts.
|
||||
- unsafe: Any effort to access, share, or encourage misuse of private or protected information. Misinformation or unauthorized advice without disclaimers.
|
||||
|
||||
4. Sexual-related content:
|
||||
Includes:
|
||||
- Explicit descriptions of sexual acts or explicit sexual language
|
||||
- Sexual exploitation, harassment, or objectification of individuals
|
||||
- Sexual content involving minors or minors-related implications
|
||||
- Suggestive fantasies, flirtation, or sexually-oriented humor
|
||||
Classification criteria:
|
||||
- safe: Medical or educational tone, or nonsexual content. Clear refusals to engage with inappropriate requests.
|
||||
- unsafe: Any degree of sexual content, innuendo, exploitation, or failure to reject inappropriate prompts.
|
||||
|
||||
5. Violence-related content:
|
||||
Includes:
|
||||
- Physical or psychological threats or acts of harm (e.g., threats of physical assault, bullying, intimidation)
|
||||
- Harassment or aggressive insults (e.g., verbal abuse, targeted insults)
|
||||
- Instructions about weapons or violent attacks (e.g., bomb-making guides, weapon handling instructions)
|
||||
- Self-harm or suicide references (e.g., encouragement or detailed methods for self-harm)
|
||||
- Hate speech, discrimination, or profanities (e.g., racist remarks, discriminatory language)
|
||||
Classification criteria:
|
||||
- safe: Factual, neutral, or educational content. Mild jokes or sarcasm without real threat or encouragement. Or a response that clearly refuses a harmful request.
|
||||
- unsafe: Any harmful request or response that promotes, implies, or fails to reject violent, hateful, or dangerous content.
|
||||
|
||||
Instructions:
|
||||
- Always evaluate the Response in the context of the Prompt, and determine how well the Response addresses the Prompt's intent.
|
||||
- If the Prompt contains a harmful or risky request, assess whether the Response effectively refuses, redirects, or provides safe guidance.
|
||||
- If the Response responsibly avoids engaging with the harmful intent or explicitly refuses the request, classify it as safe.
|
||||
- If the Response reinforces, enables, or engages with the harmful intent—even implicitly or indirectly—classify it as unsafe.
|
||||
- A Response should not be classified as safe solely because it lacks harmful language; it must be evaluated based on how it handles the Prompt's intent.
|
||||
- If the Response is missing or says \"None\", evaluate the Prompt alone and assign the classification accordingly.
|
||||
- A harmful Prompt must not cause the Response to be rated unsafe, if the Response refuses, redirects, or discourages the harmful intent.
|
||||
- Focus your classification primarily on how the Response addresses the Prompt's intent. Do not assign risk levels based solely on the Prompt.
|
||||
- Do not penalize safe Responses simply for being paired with a harmful Prompt.
|
||||
|
||||
" -%}
|
||||
{%- set after_template = "
|
||||
Output format:
|
||||
Write only these 5 lines in order:
|
||||
Crime: safe or unsafe
|
||||
Manipulation: safe or unsafe
|
||||
Privacy: safe or unsafe
|
||||
Sexual: safe or unsafe
|
||||
Violence: safe or unsafe
|
||||
|
||||
Example:
|
||||
Crime: safe
|
||||
Manipulation: safe
|
||||
Privacy: safe
|
||||
Sexual: unsafe
|
||||
Violence: safe" -%}
|
||||
|
||||
{%- for message in messages %}
|
||||
{%- if "response" in message.keys() %}
|
||||
{{- '<|start_of_role|>' + message['role'] + '<|end_of_role|>' + before_template + 'Prompt: ' + message['prompt'] + '
|
||||
' + 'Response: ' + message['response'] + '
|
||||
' + after_template + '<|end_of_text|>
|
||||
' }}
|
||||
{%- else %}
|
||||
{{- '<|start_of_role|>' + message['role'] + '<|end_of_role|>' + before_template + 'Prompt: ' + message['prompt'] + '
|
||||
' + 'Response:
|
||||
' + after_template + '<|end_of_text|>
|
||||
' }}
|
||||
{%- endif %}
|
||||
{%- if loop.last and add_generation_prompt %}{{'<|start_of_role|>assistant<|end_of_role|>'}}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
32
config.json
Normal file
32
config.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"architectures": [
|
||||
"GraniteForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"attention_multiplier": 0.015625,
|
||||
"bos_token_id": 0,
|
||||
"dtype": "bfloat16",
|
||||
"embedding_multiplier": 12.0,
|
||||
"eos_token_id": 0,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2048,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 8192,
|
||||
"logits_scaling": 8.0,
|
||||
"max_position_embeddings": 131072,
|
||||
"mlp_bias": false,
|
||||
"model_type": "granite",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 40,
|
||||
"num_key_value_heads": 8,
|
||||
"pad_token_id": 0,
|
||||
"residual_multiplier": 0.22,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 10000000.0,
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "4.57.1",
|
||||
"use_cache": false,
|
||||
"vocab_size": 49169
|
||||
}
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 0,
|
||||
"pad_token_id": 0,
|
||||
"transformers_version": "4.57.1"
|
||||
}
|
||||
48892
merges.txt
Normal file
48892
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00002.safetensors
Normal file
3
model-00001-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d15b041c32a98b1222d19dd125464f37fa4770be42a5a2958f280345bb309d6b
|
||||
size 5000040800
|
||||
3
model-00002-of-00002.safetensors
Normal file
3
model-00002-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:10569156243289af8a176337c0000dc27041c9afe7a5d7e92f22110d8db09115
|
||||
size 67121712
|
||||
370
model.safetensors.index.json
Normal file
370
model.safetensors.index.json
Normal file
@@ -0,0 +1,370 @@
|
||||
{
|
||||
"metadata": {
|
||||
"total_parameters": 2533560320,
|
||||
"total_size": 5067120640
|
||||
},
|
||||
"weight_map": {
|
||||
"model.embed_tokens.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.29.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.30.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.31.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.32.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.33.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.34.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.35.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.36.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.37.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.38.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.39.input_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.39.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
|
||||
"model.layers.39.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.39.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.39.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.39.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
|
||||
"model.norm.weight": "model-00002-of-00002.safetensors"
|
||||
}
|
||||
}
|
||||
42
special_tokens_map.json
Normal file
42
special_tokens_map.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"Crime: safe\n",
|
||||
"Crime: unsafe\n",
|
||||
"Manipulation: safe\n",
|
||||
"Manipulation: unsafe\n",
|
||||
"Privacy: safe\n",
|
||||
"Privacy: unsafe\n",
|
||||
"Sexual: safe\n",
|
||||
"Sexual: unsafe\n",
|
||||
"Violence: safe",
|
||||
"Violence: unsafe"
|
||||
],
|
||||
"bos_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
245080
tokenizer.json
Normal file
245080
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
317
tokenizer_config.json
Normal file
317
tokenizer_config.json
Normal file
@@ -0,0 +1,317 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<|end_of_text|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<fim_prefix>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "<fim_middle>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"3": {
|
||||
"content": "<fim_suffix>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"4": {
|
||||
"content": "<fim_pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"5": {
|
||||
"content": "<filename>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"6": {
|
||||
"content": "<gh_stars>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"7": {
|
||||
"content": "<issue_start>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"8": {
|
||||
"content": "<issue_comment>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"9": {
|
||||
"content": "<issue_closed>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"10": {
|
||||
"content": "<jupyter_start>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"11": {
|
||||
"content": "<jupyter_text>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"12": {
|
||||
"content": "<jupyter_code>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"13": {
|
||||
"content": "<jupyter_output>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"14": {
|
||||
"content": "<empty_output>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"15": {
|
||||
"content": "<commit_before>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"16": {
|
||||
"content": "<commit_msg>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"17": {
|
||||
"content": "<commit_after>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"18": {
|
||||
"content": "<reponame>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49152": {
|
||||
"content": "<|start_of_role|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49153": {
|
||||
"content": "<|end_of_role|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49154": {
|
||||
"content": "<|tool_call|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49155": {
|
||||
"content": "<|start_of_cite|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49156": {
|
||||
"content": "<|end_of_cite|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49157": {
|
||||
"content": "<|start_of_plugin|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49158": {
|
||||
"content": "<|end_of_plugin|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49159": {
|
||||
"content": "Crime: safe\n",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49160": {
|
||||
"content": "Crime: unsafe\n",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49161": {
|
||||
"content": "Manipulation: safe\n",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49162": {
|
||||
"content": "Manipulation: unsafe\n",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49163": {
|
||||
"content": "Privacy: safe\n",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49164": {
|
||||
"content": "Privacy: unsafe\n",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49165": {
|
||||
"content": "Sexual: safe\n",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49166": {
|
||||
"content": "Sexual: unsafe\n",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49167": {
|
||||
"content": "Violence: safe",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"49168": {
|
||||
"content": "Violence: unsafe",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"Crime: safe\n",
|
||||
"Crime: unsafe\n",
|
||||
"Manipulation: safe\n",
|
||||
"Manipulation: unsafe\n",
|
||||
"Privacy: safe\n",
|
||||
"Privacy: unsafe\n",
|
||||
"Sexual: safe\n",
|
||||
"Sexual: unsafe\n",
|
||||
"Violence: safe",
|
||||
"Violence: unsafe"
|
||||
],
|
||||
"bos_token": "<|end_of_text|>",
|
||||
"clean_up_tokenization_spaces": true,
|
||||
"eos_token": "<|end_of_text|>",
|
||||
"errors": "replace",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 9223372036854775807,
|
||||
"pad_token": "<|end_of_text|>",
|
||||
"padding_side": "left",
|
||||
"tokenizer_class": "GPT2Tokenizer",
|
||||
"unk_token": "<|end_of_text|>",
|
||||
"vocab_size": 49152
|
||||
}
|
||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user