初始化项目,由ModelHub XC社区提供模型

Model: Skywork/Skywork-Critic-Llama-3.1-8B
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-05-13 09:35:36 +08:00
commit be9d54b7ba
44 changed files with 413008 additions and 0 deletions

68
.gitattributes vendored Normal file
View File

@@ -0,0 +1,68 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
pytorch_model-00013-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00007-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00015-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00028-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00026-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00005-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00011-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00017-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00019-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00021-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00023-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00024-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00027-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00002-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00008-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00012-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00018-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00025-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00004-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00006-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00033-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00001-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00020-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00022-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00029-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00030-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00031-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00032-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00003-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00009-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00010-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00014-of-00033.bin filter=lfs diff=lfs merge=lfs -text
pytorch_model-00016-of-00033.bin filter=lfs diff=lfs merge=lfs -text

149
README.md Normal file
View File

@@ -0,0 +1,149 @@
---
license: other
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
pipeline_tag: text-generation
---
<div align="center">
<img src="misc/misc_fig.jpg" width="400"/>
🤗 <a href="https://huggingface.co/Skywork" target="_blank">Hugging Face</a> • 🤖 <a href="https://modelscope.cn/organization/Skywork" target="_blank">ModelScope</a>
<br>
<br>
<br>
</div>
# Introduction to Skywork Critic Series Models
[**Skywork-Critic-Llama3.1-70B**](https://huggingface.co/Skywork/Skywork-Critic-Llama3.1-70B) and [**Skywork-Critic-Llama3.1-8B**](https://huggingface.co/Skywork/Skywork-Critic-Llama-3.1-8B), developed by the [SkyworkAI](https://huggingface.co/Skywork) Alignment Team, are advanced judge models that excel at pairwise preference evaluation. These models compare and assess input pairs, offering nuanced judgments on their relative quality or suitability. Leveraging their deep understanding of language and context, Skywork-Critic models provide valuable insights for various applications, including data improvement, evaluation, and reward modeling.
# Training Details
Skywork-Critic-Llama3.1-70B and Skywork-Critic-Llama3.1-8B are built on Meta [Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) and [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) respectively. These models have undergone fine-tuning using a diverse array of high-quality datasets, including:
- **Cleaned open-source data**: We utilize a high-quality subset of [HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2), [OffsetBias](https://huggingface.co/datasets/NCSOFT/offsetbias), [WildGuard (adversarial)](https://huggingface.co/allenai/wildguard) and Magpie DPO series([Ultra](https://huggingface.co/datasets/argilla/magpie-ultra-v0.1),[Pro (Llama-3.1)](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-DPO-100K-v0.1),[Pro](https://huggingface.co/datasets/Magpie-Align/Magpie-Pro-DPO-100K-v0.1),[Air](https://huggingface.co/datasets/Magpie-Align/Magpie-Air-DPO-100K-v0.1)). For more details, please refer to our [Skywork-Reward-Preference-80K-v0.1 dataset](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.1). Additionally, we integrate several open-source, high-quality critic datasets such as [Open-Critic-GPT](https://huggingface.co/datasets/Vezora/Open-Critic-GPT) into our training process.
- **In-house human annotation data**: This includes both pointwise scoring across many dimensions for a single response and pairwise comparisons between two responses. Each dimension incorporates a rationale for the assigned score. Please note that manually labeled data is very expensive to obtain. We only have a few hundred manually labeled data points, all of which are in Chinese, so the ability to perform single rating might not be particularly strong.
- **Synthetic critic data**: We use a similar appoarch to [**self-taught**](https://arxiv.org/abs/2408.02666). Specifically, we employed two methods to generate inferior responses for a given instruction: 1) Creating a similar instruction and then generating a response for this new instruction. 2) Introducing subtle errors into high-quality responses.
- **Critic-related chat data**: We incorporate critic-related chat data to maintain the model's conversational capabilities.
The training employs instruction-tuning methodology, focusing on pairwise preference evaluation and general chat tasks. We have conducted a thorough verification process to ensure our training dataset does not contain any test set information from RewardBench, maintaining the integrity of our evaluation results.
# RewardBench Leaderboard for Generative Models
We evaluate our models on [RewardBench](https://huggingface.co/spaces/allenai/reward-bench) using the [official test script](https://github.com/allenai/reward-bench).
As of September 2024, Skywork-Critic-Llama3.1-70B **ranks first** on RewardBench for generative models across all sizes, while Skywork-Critic-Llama3.1-8B tops the list for generative models under 10B parameters. (Note: An asterisk (*) indicates an open-source model.)
| Model | Chat | Chat Hard | Safety | Reasoning | Overall Score |
| ------------------------------- | :---: | :-------: | :----: | :-------: | :---: |
| **Skywork-Critic-Llama3.1-70B** * | **96.6** | **87.9** | **93.1** | **95.5** | **93.3** |
| Salesforce/SFR-LLaMa-3.1-70B-Judge-r | 96.9 | 84.8 | 91.6 | 97.6 | 92.7 |
| Salesforce/SFR-nemo-12B-Judge-r | 97.2 | 82.2 | 86.5 | 95.1 | 90.3 |
| **Skywork-Critic-Llama3.1-8B** * | **93.6** | **81.4** | **91.1** | **89.8** | **89.0** |
| Salesforce/SFR-LLaMa-3.1-8B-Judge-r | 95.5 | 77.7 | 86.2 | 95.1 | 88.7 |
| facebook/Self-taught-Llama-3-70B | 96.9 | 84.0 | 91.1 | 82.5 | 88.6 |
| google/gemini-1.5-pro-0514 | 92.3 | 80.6 | 87.9 | 92.0 | 88.2 |
| openai/gpt-4o-2024-08-06 | 96.1 | 76.1 | 88.1 | 86.6 | 86.7 |
| openai/gpt-4-0125-preview | 95.3 | 74.3 | 87.6 | 86.9 | 86.0 |
| openai/gpt-4-turbo-2024-04-09 | 95.3 | 75.4 | 87.6 | 82.7 | 85.2 |
| Anthropic/claude-3-5-sonnet-20240620 | 96.4 | 74.0 | 81.6 | 84.7 | 84.2 |
| meta-llama/Meta-Llama-3.1-70B-Instruct * | 97.2 | 70.2 | 82.8 | 86.0 | 84.0 |
| NCSOFT/Llama-3-OffsetBias-8B * | 92.5 | 80.3 | 86.8 | 76.4 | 84.0 |
# Demo Code
Below is an example of obtaining the critic of two conversations.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# An Example Case
prompt = "Jane has 12 apples. She gives 4 apples to her friend Mark, then buys 1 more apple, and finally splits all her apples equally among herself and her 2 siblings. How many apples does each person get?"
responseA = "1. Jane starts with 12 apples and gives 4 to Mark. 12 - 4 = 8. Jane now has 8 apples.\n2. Jane buys 1 more apple. 8 + 1 = 9. Jane now has 9 apples.\n3. Jane splits the 9 apples equally among herself and her 2 siblings (3 people in total). 9 ÷ 3 = 3 apples each. Each person gets 3 apples."
responseB = "1. Jane starts with 12 apples and gives 4 to Mark. 12 - 4 = 8. Jane now has 8 apples.\n2. Jane buys 1 more apple. 8 + 1 = 9. Jane now has 9 apples.\n3. Jane splits the 9 apples equally among her 2 siblings (2 people in total). 9 ÷ 2 = 4.5 apples each. Each person gets 4 apples."
# feed a natural language prompt to generative model
prompt_template = """Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user\'s instructions and answers the user\'s question better.
Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible.
Please directly output your final verdict by strictly following this format: "[[A]]" if assistant A is better, "[[B]]" if assistant B is better.
[User Question]
{input}
[The Start of Assistant A's Answer]
{response_a}
[The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
{response_b}
[The End of Assistant B's Answer]
"""
user_message = prompt_template.format(input=prompt, response_a=responseA, response_b=responseB)
conversation = [{"role": "user", "content": user_message}]
model_name = "Skywork/Skywork-Critic-Llama3.1-70B"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
input_ids = tokenizer.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt").to(model.device)
generation = model.generate(
input_ids=input_ids,
max_new_tokens=2048,
do_sample=False,
pad_token_id=128009,
temperature=0)
completion = tokenizer.decode(
generation[0][len(input_ids[0]):],
skip_special_tokens=True,
clean_up_tokenization_spaces=True)
print(completion)
# Output:
# The generative model should output "[[A]]"
```
# Declaration and License Agreement
## Declaration
We hereby declare that the Skywork model should not be used for any activities that pose a threat to national or societal security or engage in unlawful actions. Additionally, we request users not to deploy the Skywork model for internet services without appropriate security reviews and records. We hope that all users will adhere to this principle to ensure that technological advancements occur in a regulated and lawful environment.
We have done our utmost to ensure the compliance of the data used during the model's training process. However, despite our extensive efforts, due to the complexity of the model and data, there may still be unpredictable risks and issues. Therefore, if any problems arise as a result of using the Skywork open-source model, including but not limited to data security issues, public opinion risks, or any risks and problems arising from the model being misled, abused, disseminated, or improperly utilized, we will not assume any responsibility.
## License Agreement
The community usage of Skywork model requires [Skywork Community License](https://github.com/SkyworkAI/Skywork-Reward/blob/main/misc/Skywork%20Community%20License.pdf). The Skywork model supports commercial use. If you plan to use the Skywork model or its derivatives for commercial purposes, you must abide by terms and conditions within [Skywork Community License](https://github.com/SkyworkAI/Skywork-Reward/blob/main/misc/Skywork%20Community%20License.pdf).
# Contact
If you have any questions or feedback, don't hesitate to reach out to our friendly team at <shiwen.tu@kunlun-inc.com> or <liang.zhao@kunlun-inc.com>. Liang Zhao leads this project.
# Citation
If you find our work helpful, please feel free to cite us using the following BibTeX entry:
```bibtex
@misc{skyworkcritic2024,
title={Skywork Critic Model Series},
author={Shiwen, Tu and Liang, Zhao and Liu, Chris Yuhao and Zeng, Liang and Liu, Yang},
year={2024},
month={September},
howpublished={\url{https://huggingface.co/Skywork}},
url={https://huggingface.co/Skywork},
}
```

38
config.json Normal file
View File

@@ -0,0 +1,38 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 128000,
"eos_token_id": [
128001,
128008,
128009
],
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 131072,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": {
"factor": 8.0,
"low_freq_factor": 1.0,
"high_freq_factor": 4.0,
"original_max_position_embeddings": 8192,
"rope_type": "llama3"
},
"rope_theta": 500000.0,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.42.3",
"use_cache": true,
"vocab_size": 128256
}

12
generation_config.json Normal file
View File

@@ -0,0 +1,12 @@
{
"bos_token_id": 128000,
"do_sample": true,
"eos_token_id": [
128001,
128008,
128009
],
"temperature": 0.6,
"top_p": 0.9,
"transformers_version": "4.42.3"
}

Binary file not shown.

Binary file not shown.

BIN
misc/misc_fig.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 77 KiB

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:47d2f5a157f246d9b011a2ec737e48f739f64da38840fd65b70107ceccae867c
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:764255e4010ece8af96deb4f8ab6395de48466bf6a8c3965e19c09f232048a74
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a002675fe69df7331153e34ecf050a01e19c3fd4c7ff3cc30366df59d900af31
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:34bc3680d867eb18826a809a54d302c7a9fbfd82e283f5967d1b5eb9ad9b381c
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f4ba9b1c1c652205fd431cfb2c7a1393ea23d688ea9f1a2d07d2fdbaddaaeb38
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b27ad3ab9f4b69cc0b2ab48abc01e7914556f2afd12229e911b717adeaccf2ce
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fa8e3986841a4e6ba334dd7ed063edbf5b6d915a980d580e2b2c0289ef410a8d
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6fdc3709807dfce760121b4953bd19c761d3f450e5429dcadb02693713380e8b
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:930058f8221cd3480db4be1fdc041dbbbc0be479b443791c8fd4a18a285d13f9
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2ff8d084a5756dd46f5a931be32cd2667a31528255cc2184cb15feed280407c3
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6693087eae23273e01fa12f9e2047c11f90c298af6cbe6c50f3d4bb077c49a76
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:61da77ea219ce9349369fec8df2616d96b78fb828f24f808cc2927ac5a354226
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a93feb9c1ec31af5862c70d34955369ad4bd3f1792267fa8af0752fa53488e19
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f24a5c0ceca29ae5ad6cbfe5ea78aa226cbbd56670fc125297b85119a799f459
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c79fef54bcf7ca86a28dbc5ee65101cd64425301aebd682d2d1e851beec54f11
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ec68ebaf1459fc6d4afbba64ac116934dbf311b40b22a0380fe40f6bd2582ca1
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7125abf7ad1f6fff3b8786a5e8461c7659d0bb624dee9747584d8dfcd2b78391
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bf656bb4aa979db005437184c3f5e6ec52caa755f2928616d383e50e6c432cce
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f23ce245c9746d791c9e4ac39d3151b7e2b0bc84888e98b980eb213611f0db27
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:79c6896805c994de986cfb5d542ec2586db6ac9d6eb1881a97a0388ee31208c9
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:acdf3ab92bb7f94b7227ea3b0d88ae7608b6705af5a0e94ad6ef245d5a2f9b6b
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a64b6f349cc916b7b500220451f427ac690f871101f22ccbdb21f49b9f336eec
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:068e77bee764acc004ad2500dbc0a2cc8980a46ab79ff52ef4b0b0ccb64b839a
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4ee8ccde0dbfada9aa3c13363fdd0559bcf01d0377e4aeaac4eefcfaf4b97360
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b51203c763adfd388acc6576e75b008bb0db7bbe4cd5760d5335e7b88ac5f063
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ec5df4ff8286ff10fe5923c6e5b8beecfbef1fe5aeb23e36aafd367103c0393f
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e270d7bed954a4824aa518e896c80c43f02d4dc37b042bc78062b66590383c21
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bc154ff22e6cb39408c92ab1d83c0b5f509327b8ecdd7fcaf5206f11c7a4782f
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:911068936fcd11ad56129ed2e77a7128765f7a1fd5b3a1b26ff75c376f7889e5
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:339651de5288236bf948c2694561ce1bbe9373d8907af415e1b8951625aa0040
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2e242ce46aef17e4f6aec7e58cda8569decca5675bba9cc0be40c7c9fae5d0ed
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:453fe5c10c09a35f0117db516f724e7882159b28ebcf77f43f37be11edc215c4
size 436227068

View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9516802b6c53b395d565717b51be6ceb703455529dbcb7616e1e5cbb397618b8
size 2101356116

File diff suppressed because one or more lines are too long

16
special_tokens_map.json Normal file
View File

@@ -0,0 +1,16 @@
{
"bos_token": {
"content": "<|begin_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|eot_id|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

410563
tokenizer.json Normal file

File diff suppressed because it is too large Load Diff

2062
tokenizer_config.json Normal file

File diff suppressed because it is too large Load Diff