初始化项目，由ModelHub XC社区提供模型
Model: flowaicom/Flow-Judge-v0.1-AWQ Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
818
README.md
Normal file
818
README.md
Normal file
@@ -0,0 +1,818 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
tags:
|
||||
- lm-judge
|
||||
- phi3
|
||||
- evaluation
|
||||
- nlp
|
||||
- conversational
|
||||
- autoawq
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
metrics:
|
||||
- accuracy
|
||||
- f1
|
||||
- precision
|
||||
- recall
|
||||
- pearsonr
|
||||
- spearmanr
|
||||
- kendall-tau
|
||||
model_name: Flow-Judge-v0.1-AWQ
|
||||
base_model: microsoft/Phi-3.5-mini-instruct
|
||||
inference: false
|
||||
model_creator: Flow AI
|
||||
model_type: phi3.5
|
||||
quantized_by: Flow AI
|
||||
---
|
||||
# Flow-Judge-v0.1-AWQ
|
||||
- Original model: [Flow-Judge-v0.1](https://huggingface.co/flowaicom/Flow-Judge-v0.1)
|
||||
- Model collection: [Flow-Judge-v0.1 models](https://huggingface.co/collections/flowaicom/flow-judge-v01-66e6af5fc3b3a128bde07dec)
|
||||
- Technical report: [Flow Judge: An Open Small Language Model for LLM System Evaluations](https://huggingface.co/flowaicom/Flow-Judge-v0.1)
|
||||
- Model website: [flow-ai.com/judge](https://www.flow-ai.com/blog/flow-judge)
|
||||
- About us: [Flow AI](https://www.flow-ai.com/about)
|
||||
|
||||
<!-- description start -->
|
||||
## Description
|
||||
|
||||
This repo contains AWQ safetensors quant for [Flow-Judge-v0.1](https://huggingface.co/flowaicom/Flow-Judge-v0.1).
|
||||
|
||||
## Quantization config
|
||||
```python
|
||||
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
|
||||
model = AutoAWQForCausalLM.from_pretrained(merged_path, **{"low_cpu_mem_usage": True, "use_cache": False},
|
||||
attn_implementation="flash_attention_2", torch_dtype="auto", device_map="auto")
|
||||
tokenizer = AutoTokenizer.from_pretrained(lora_path, trust_remote_code=False)
|
||||
model.quantize(tokenizer, quant_config=quant_config)
|
||||
model.save_quantized(quant_path)
|
||||
tokenizer.save_pretrained(quant_path)
|
||||
```
|
||||
|
||||
## Library versions used for quantization
|
||||
|
||||
```raw
|
||||
autoawq 0.2.6
|
||||
autoawq_kernels 0.0.7
|
||||
```
|
||||
|
||||
## Running the AWQ model
|
||||
|
||||
First install the flow judge library
|
||||
```shell
|
||||
git clone https://github.com/flowaicom/flow-judge
|
||||
cd flow-judge
|
||||
pip install -e ".[vllm]"
|
||||
```
|
||||
|
||||
Quickstart with Python:
|
||||
```python
|
||||
from flow_judge import Vllm, Llamafile, Hf, EvalInput, FlowJudge
|
||||
from flow_judge.metrics import RESPONSE_FAITHFULNESS_5POINT
|
||||
from IPython.display import Markdown, display
|
||||
|
||||
# If you are running on an Ampere GPU or newer, create a model using VLLM
|
||||
model = Vllm(quantization=True)
|
||||
|
||||
# If you have other applications open taking up VRAM, you can use less VRAM by setting gpu_memory_utilization to a lower value.
|
||||
# model = Vllm(gpu_memory_utilization=0.70)
|
||||
|
||||
# Or create a model using Llamafile if not running an Nvidia GPU & running a Silicon MacOS for example
|
||||
# model = Llamafile()
|
||||
|
||||
# Initialize the judge
|
||||
faithfulness_judge = FlowJudge(
|
||||
metric=RESPONSE_FAITHFULNESS_5POINT,
|
||||
model=model
|
||||
)
|
||||
|
||||
# Sample to evaluate
|
||||
query = ...
|
||||
context = ...
|
||||
response = ...
|
||||
|
||||
# Create an EvalInput
|
||||
# We want to evaluate the response to the customer issue based on the context and the user instructions
|
||||
eval_input = EvalInput(
|
||||
inputs=[
|
||||
{"query": query},
|
||||
{"context": context},
|
||||
],
|
||||
output={"response": response},
|
||||
)
|
||||
|
||||
# Run the evaluation
|
||||
result = faithfulness_judge.evaluate(eval_input, save_results=False)
|
||||
|
||||
# Display the result
|
||||
display(Markdown(f"__Feedback:__\n{result.feedback}\n\n__Score:__\n{result.score}"))
|
||||
```
|
||||
|
||||
Discover more at our repository [https://github.com/flowaicom/flow-judge](https://github.com/flowaicom/flow-judge)
|
||||
|
||||
# Original model card: Flow-Judge-v0.1
|
||||
|
||||
<p align="center">
|
||||
<img src="https://cdn-uploads.huggingface.co/production/uploads/63368577d184e6b53c50e6d0/6kSJKgPh2pDh4tA-Ky0xW.png" alt="Centered image">
|
||||
</p>
|
||||
<p align="center">🚀 <a href="https://www.flow-ai.com/judge">Flow Judge</a> | 📄 <a href="https://www.flow-ai.com/blog/flow-judge">Technical report</a> | 💻 <a href="https://github.com/flowaicom/flow-judge">flow-judge</a></p>
|
||||
|
||||
## Model Summary
|
||||
|
||||
Flow-Judge-v0.1 is a compact yet powerful 3.8B model that offers customizable LLM system evaluations across various fields. The model inherits it's architecture from Phi-3.5-mini instruct model which enables Flow-Judge to deliver high-quality results while maintaining a small footprint. Despite its smaller size, it achieves performance comparable to larger models in both held-out and out-of-domain benchmarks. Flow-Judge-v0.1 supports multiple scoring scales, provides qualitative feedback, and generates structured evaluation outputs. Trained on a smaller synthetic dataset, it represents an efficient approach to AI development. Released under the Apache 2.0 license, Flow Judge is an open and accessible model suitable for developers and companies seeking cost-effective and rapid evaluations using custom rubrics.
|
||||
|
||||
__Quantized weights__
|
||||
- [flowaicom/Flow-Judge-v0.1-AWQ](https://huggingface.co/flowaicom/Flow-Judge-v0.1-AWQ)
|
||||
- [flowaicom/Flow-Judge-v0.1-GGUF](https://huggingface.co/flowaicom/Flow-Judge-v0.1-GGUF)
|
||||
|
||||
__Quickstart__
|
||||
- [Quickstart](https://github.com/flowaicom/flow-judge/examples/1_quickstart.ipynb)
|
||||
|
||||
## Intended Use Case
|
||||
Flow Judge is intended to be used on custom LLM system evaluation tasks.
|
||||
|
||||
- Customizable evaluations: Users can define their own evaluation criteria and rubrics, tailoring Flow Judge to their specific needs and requirements. This flexibility allows for the creation of highly targeted assessments that accurately measure performance of their LLM system
|
||||
|
||||
- Flow Judge supports three different scoring scales:
|
||||
- Pass/fail: Suitable for binary assessments, such as determining whether a piece of text meets a specific standard or contains errors.
|
||||
- 3-Likert: Allows for more granular evaluations, with scores ranging from negative to neutral to positive. Useful for assessing the overall quality or sentiment of a piece of text.
|
||||
- 5-Likert: Provides an even more nuanced assessment, with scores ranging from strongly negative to strongly positive, enabling users to capture subtle differences in quality or sentiment.
|
||||
|
||||
- Easy to interpret results:
|
||||
- Flow Judge produces structured evaluations with `<feedback>` and `<score>` tags.
|
||||
- Qualitative feedback: Flow Judge detects errors and grades outputs and provides qualitative feedback that explains its reasoning for assigning a particular score from the rubric while highlighting problematic parts of the responses.
|
||||
- Score: Based on a grading rubric Flow Judge will return a numerical score on binary, likert-3 or likert-5 scale.
|
||||
|
||||
## Training
|
||||
|
||||
### Model
|
||||
|
||||
Flow Judge is based on the Phi-3.5-mini architecture, and the base model checkpoint used is specifically its instruct version. The model uses the same tokenizer, supports MQA and Flash Attention 2, and has weights in bfloat16 precision. However, post-finetuning, the model's support for languages and long context lengths has not been fully tested. Due to specialized Supervised Fine-Tuning (SFT), Flow Judge might show different benchmark results and support a maximum context length of 8192, shorter than the base model's.
|
||||
|
||||
|
||||
### Training Datasets
|
||||
|
||||
Flow-Judge-v0.1 has been trained on synthetically generated datasets. The construction of training datasets for Flow Judge involves a multi-step process:
|
||||
|
||||
1. Manually curating seed rubrics to serve as a foundation
|
||||
2. Synthetically generating domain-adapted metrics and rubrics for various domains
|
||||
3. Synthetically generating training instances with multiple inputs, such as user queries and contextual information
|
||||
4. Employing a dual-evaluation strategy with consensus to ensure quality and consistency
|
||||
|
||||
This process creates a comprehensive and diverse set of training instances that enable accurate, domain-specific evaluations of LLM systems in generative AI products while minimizing human intervention.
|
||||
|
||||
Read more about the dataset construction from [here](https://www.flow-ai.com/blog/flow-judge#dataset-construction)
|
||||
|
||||
|
||||
### Fine-tuning
|
||||
|
||||
For fine-tuning we used Axolotl's preprocessing to ensure input training data is consistent. We then conducted supervised fine-tuning based on microsoft/Phi-3.5-mini-instruct using RSLoRa. More detailed information about the fine-tuning process is provided in our [technical report](https://www.flow-ai.com/blog/flow-judge#fine-tuning).
|
||||
|
||||
## Usage
|
||||
|
||||
### Prompt format
|
||||
|
||||
#### Prompt template with inputs
|
||||
```text
|
||||
# GOAL
|
||||
Your job is to evaluate a task carried out by an AI system powered by a large language model.
|
||||
You will be provided with the inputs and output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.
|
||||
|
||||
# INPUT
|
||||
Below are the inputs required for performing the task:
|
||||
<inputs>
|
||||
{INPUTS}
|
||||
</inputs>
|
||||
|
||||
# OUTPUT
|
||||
Below is the output of the task:
|
||||
<output>
|
||||
{OUTPUT}
|
||||
</output>
|
||||
|
||||
# EVALUATION CRITERIA AND SCORING RUBRIC
|
||||
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
|
||||
<evaluation_criteria>
|
||||
{EVALUATION_CRITERIA}
|
||||
</evaluation_criteria>
|
||||
|
||||
<scoring_rubric>
|
||||
{RUBRIC}
|
||||
</scoring_rubric>
|
||||
|
||||
# INSTRUCTIONS FOR THE EVALUATION
|
||||
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
|
||||
2. Review the inputs and output: Look at the inputs provided for the task. Examine the output generated from completing the task.
|
||||
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
|
||||
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
|
||||
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
|
||||
6. Assign a final score based on the scoring rubric.
|
||||
|
||||
## FORMAT FOR THE EVALUATION
|
||||
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
|
||||
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.
|
||||
|
||||
Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.
|
||||
```
|
||||
|
||||
#### Prompt template without inputs
|
||||
```text
|
||||
# GOAL
|
||||
Your job is to evaluate a task carried out by an AI system powered by a large language model.
|
||||
|
||||
You will be provided the output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.
|
||||
|
||||
# OUTPUT
|
||||
Below is the output of the task:
|
||||
<output>
|
||||
{OUTPUT}
|
||||
</output>
|
||||
|
||||
# EVALUATION CRITERIA AND SCORING RUBRIC
|
||||
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
|
||||
<evaluation_criteria>
|
||||
{EVALUATION_CRITERIA}
|
||||
</evaluation_criteria>
|
||||
|
||||
<scoring_rubric>
|
||||
{RUBRIC}
|
||||
</scoring_rubric>
|
||||
|
||||
# INSTRUCTIONS FOR THE EVALUATION
|
||||
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
|
||||
2. Review the output: Examine the output generated from completing the task.
|
||||
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
|
||||
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
|
||||
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
|
||||
6. Assign a final score based on the scoring rubric.
|
||||
|
||||
## FORMAT FOR THE EVALUATION
|
||||
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
|
||||
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.
|
||||
|
||||
Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.
|
||||
```
|
||||
|
||||
#### Formatted prompt example with inputs
|
||||
```text
|
||||
# GOAL
|
||||
Your job is to evaluate a task carried out by an AI system powered by a large language model.
|
||||
You will be provided with the inputs and output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.
|
||||
# INPUT/s
|
||||
Below are the inputs required for performing the task:
|
||||
<inputs>
|
||||
<customer_query>
|
||||
Subject: Issues with recent order #12345
|
||||
Hello,
|
||||
I recently received my order #12345, but I'm having a few problems:
|
||||
1. The blue t-shirt I ordered is actually green. Can I exchange it?
|
||||
2. The phone case doesn't fit my iPhone 13 Pro Max. The website said it was compatible.
|
||||
3. I was charged for express shipping, but it took over a week to arrive. Can I get a refund on the shipping cost?
|
||||
Please let me know how we can resolve these issues. I've been a loyal customer for years and this is the first time I've had problems like this.
|
||||
Thank you,
|
||||
Jane Smith
|
||||
</customer_query>
|
||||
</inputs>
|
||||
|
||||
# OUTPUT
|
||||
Below is the output of the task:
|
||||
<output>
|
||||
Dear Jane Smith,
|
||||
Thank you for reaching out about the issues with your recent order #12345. We sincerely apologize for the inconvenience you've experienced. Let me address each of your concerns:
|
||||
1. T-shirt color discrepancy: We're sorry about the mix-up with the t-shirt color. We'd be happy to exchange it for the correct blue t-shirt. I'll send you a prepaid return label for the green shirt, and we'll ship out the blue one as soon as we receive the return.
|
||||
2. Phone case compatibility: We apologize for the confusion regarding the phone case compatibility. Our website information should have been more accurate. We'll gladly accept a return of the incompatible case and issue a full refund for it. Additionally, I can recommend a case that will fit your iPhone 13 Pro Max if you're interested.
|
||||
3. Shipping charge: You're absolutely right about the shipping charge. Since the delivery didn't meet our express shipping standards, we will refund the entire shipping cost to your original payment method. This refund should process within 3-5 business days.
|
||||
To initiate these resolutions, please reply to this email confirming that you'd like to proceed with the t-shirt exchange and phone case return. Once confirmed, I'll send you the necessary return labels and process the shipping refund immediately.
|
||||
We truly value your loyalty as a long-time customer and sincerely regret that you've encountered these issues. Rest assured, we're taking steps to prevent similar problems in the future. As a gesture of goodwill, we'd like to offer you a 20% discount on your next order.
|
||||
If you have any further questions or concerns, please don't hesitate to reach out. We're here to ensure your complete satisfaction.
|
||||
Best regards,
|
||||
Alex Johnson
|
||||
Customer Service Representative
|
||||
</output>
|
||||
|
||||
# EVALUATION CRITERIA AND SCORING RUBRIC
|
||||
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
|
||||
<evaluation_criteria>
|
||||
How well the response addresses the specific issues raised in the customer's query?
|
||||
</evaluation_criteria>
|
||||
<scoring_rubric>
|
||||
- Score 1: The response completely fails to address the customer's needs and ignores the specific issues raised.
|
||||
- Score 2: The response barely addresses the customer's query and misses most of the specific issues raised.
|
||||
- Score 3: The response partially addresses the customer's query, touching on some of the specific issues but leaving others unaddressed.
|
||||
- Score 4: The response adequately addresses most aspects of the customer's query and the specific issues raised.
|
||||
- Score 5: The response fully and comprehensively addresses all aspects of the customer's query and all specific issues raised in a highly satisfactory manner.
|
||||
</scoring_rubric>
|
||||
|
||||
# INSTRUCTIONS FOR THE EVALUATION
|
||||
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
|
||||
2. Review the inputs and output: Look at the inputs provided for the task. Examine the output generated from completing the task.
|
||||
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
|
||||
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
|
||||
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
|
||||
6. Assign a final score based on the scoring rubric.
|
||||
|
||||
## FORMAT FOR THE EVALUATION
|
||||
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
|
||||
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.
|
||||
Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.
|
||||
```
|
||||
>Note that inputs and output are formatted with XML tags. See [flow-judge](https://github.com/flowaicom/flow-judge) repository formatting functions for more details.
|
||||
|
||||
### Inference
|
||||
|
||||
Evaluations can easily be run using our [flow-judge](https://github.com/flowaicom/flow-judge) library. It currently supports both Transformers and vllm engine.
|
||||
|
||||
To run Flow Judge efficiently, ensure your hardware meets the following requirements:
|
||||
|
||||
- Modern GPU with at least 4 GB VRAM (e.g., NVIDIA RTX series)
|
||||
- Minimum of 8 GB of system memory
|
||||
- At least 10GB of free storage for model files and dependencies.
|
||||
|
||||
## Evaluation
|
||||
### Held-out test sets
|
||||
|
||||
<table border="1" cellpadding="10" cellspacing="0" style="border-collapse: collapse; width: auto;">
|
||||
<thead>
|
||||
<tr>
|
||||
<th rowspan="2" style="text-align: left;">Evaluator</th>
|
||||
<th colspan="3" style="text-align: center;">Pass / Fail Held-out Test set</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th style="text-align: center;">Precision</th>
|
||||
<th style="text-align: center;">Recall</th>
|
||||
<th style="text-align: center;">F1</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align: left;">microsoft/Phi-3.5-mini-instruct</td>
|
||||
<td style="text-align: center;">0.685</td>
|
||||
<td style="text-align: center;"><strong>1.000</strong></td>
|
||||
<td style="text-align: center;">0.813</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">meta-llama/Meta-Llama-3.1-8B-Instruct</td>
|
||||
<td style="text-align: center;"><u>0.870</u></td>
|
||||
<td style="text-align: center;">0.982</td>
|
||||
<td style="text-align: center;"><u>0.923</u></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">mistralai/Mistral-Nemo-Instruct-2407</td>
|
||||
<td style="text-align: center;">0.709</td>
|
||||
<td style="text-align: center;"><u>0.994</u></td>
|
||||
<td style="text-align: center;">0.827</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">gpt-4o-mini</td>
|
||||
<td style="text-align: center;">0.834</td>
|
||||
<td style="text-align: center;">1.000</td>
|
||||
<td style="text-align: center;">0.910</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">flowaicom/Flow-Judge-v0.1</td>
|
||||
<td style="text-align: center;"><strong>0.940</strong></td>
|
||||
<td style="text-align: center;">0.972</td>
|
||||
<td style="text-align: center;"><strong>0.955</strong></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<table border="1" cellpadding="10" cellspacing="0" style="border-collapse: collapse; width: auto;">
|
||||
<thead>
|
||||
<tr>
|
||||
<th rowspan="2" style="text-align: left;">Evaluator</th>
|
||||
<th colspan="3" style="text-align: center;">3-Likert Held-out Test set</th>
|
||||
<th colspan="3" style="text-align: center;">5-Likert Held-out Test set</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th style="text-align: center;">pearsonr</th>
|
||||
<th style="text-align: center;">spearmanr</th>
|
||||
<th style="text-align: center;">kendall-tau</th>
|
||||
<th style="text-align: center;">pearsonr</th>
|
||||
<th style="text-align: center;">spearmanr</th>
|
||||
<th style="text-align: center;">kendall-tau</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align: left;">microsoft/Phi-3.5-mini-instruct</td>
|
||||
<td style="text-align: center;">0.756</td>
|
||||
<td style="text-align: center;">0.749</td>
|
||||
<td style="text-align: center;">0.695</td>
|
||||
<td style="text-align: center;">0.808</td>
|
||||
<td style="text-align: center;">0.819</td>
|
||||
<td style="text-align: center;">0.739</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">prometheus-eval/prometheus-7b-v2.0*</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;"><u>0.910</u></td>
|
||||
<td style="text-align: center;"><u>0.908</u></td>
|
||||
<td style="text-align: center;"><u>0.838</u></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">meta-llama/Meta-Llama-3.1-8B-Instruct</td>
|
||||
<td style="text-align: center;"><u>0.836</u></td>
|
||||
<td style="text-align: center;"><u>0.833</u></td>
|
||||
<td style="text-align: center;"><u>0.789</u></td>
|
||||
<td style="text-align: center;">0.854</td>
|
||||
<td style="text-align: center;">0.868</td>
|
||||
<td style="text-align: center;">0.791</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">mistralai/Mistral-Nemo-Instruct-2407</td>
|
||||
<td style="text-align: center;">0.813</td>
|
||||
<td style="text-align: center;">0.807</td>
|
||||
<td style="text-align: center;">0.758</td>
|
||||
<td style="text-align: center;">0.870</td>
|
||||
<td style="text-align: center;">0.867</td>
|
||||
<td style="text-align: center;">0.789</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">gpt-4o-mini</td>
|
||||
<td style="text-align: center;">0.890</td>
|
||||
<td style="text-align: center;">0.888</td>
|
||||
<td style="text-align: center;">0.851</td>
|
||||
<td style="text-align: center;">0.923</td>
|
||||
<td style="text-align: center;">0.923</td>
|
||||
<td style="text-align: center;">0.864</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">flowaicom/Flow-Judge-v0.1</td>
|
||||
<td style="text-align: center;"><strong>0.888</strong></td>
|
||||
<td style="text-align: center;"><strong>0.888</strong></td>
|
||||
<td style="text-align: center;"><strong>0.852</strong></td>
|
||||
<td style="text-align: center;"><strong>0.919</strong></td>
|
||||
<td style="text-align: center;"><strong>0.919</strong></td>
|
||||
<td style="text-align: center;"><strong>0.856</strong></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
\* _Reported in model paper_
|
||||
|
||||
|
||||
### RAGTruth
|
||||
<table border="1" cellpadding="10" cellspacing="0" style="border-collapse: collapse; width: auto;">
|
||||
<tr>
|
||||
<th rowspan="2" style="text-align: left;">Evaluator</th>
|
||||
<th colspan="3" style="text-align:center;">RAGTruth QA</th>
|
||||
<th colspan="3" style="text-align:center;">RAGTruth Data-to-Text</th>
|
||||
<th colspan="3" style="text-align:center;">RAGTruth Summarization</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th style="text-align:center;">Precision</th>
|
||||
<th style="text-align:center;">Recall</th>
|
||||
<th style="text-align:center;">F1</th>
|
||||
<th style="text-align:center;">Precision</th>
|
||||
<th style="text-align:center;">Recall</th>
|
||||
<th style="text-align:center;">F1</th>
|
||||
<th style="text-align:center;">Precision</th>
|
||||
<th style="text-align:center;">Recall</th>
|
||||
<th style="text-align:center;">F1</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>microsoft/Phi-3.5-mini-instruct</td>
|
||||
<td style="text-align:center;">0.817</td>
|
||||
<td style="text-align:center;">0.963</td>
|
||||
<td style="text-align:center;">0.884</td>
|
||||
<td style="text-align:center;">0.356</td>
|
||||
<td style="text-align:center;"><strong>1.000</strong></td>
|
||||
<td style="text-align:center;">0.525</td>
|
||||
<td style="text-align:center;">0.776</td>
|
||||
<td style="text-align:center;"><strong>1.000</strong></td>
|
||||
<td style="text-align:center;"><strong>0.874</strong></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>meta-llama/Meta-Llama-3.1-8B-Instruct</td>
|
||||
<td style="text-align:center;"><strong>0.844</strong></td>
|
||||
<td style="text-align:center;"><u>0.986</u></td>
|
||||
<td style="text-align:center;"><strong>0.910</strong></td>
|
||||
<td style="text-align:center;">0.382</td>
|
||||
<td style="text-align:center;">0.537</td>
|
||||
<td style="text-align:center;">0.447</td>
|
||||
<td style="text-align:center;"><u>0.797</u></td>
|
||||
<td style="text-align:center;"><u>0.940</u></td>
|
||||
<td style="text-align:center;">0.863</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>mistralai/Mistral-Nemo-Instruct-2407</td>
|
||||
<td style="text-align:center;">0.821</td>
|
||||
<td style="text-align:center;"><strong>0.995</strong></td>
|
||||
<td style="text-align:center;"><u>0.900</u></td>
|
||||
<td style="text-align:center;">0.357</td>
|
||||
<td style="text-align:center;"><strong>1.000</strong></td>
|
||||
<td style="text-align:center;">0.526</td>
|
||||
<td style="text-align:center;">0.775</td>
|
||||
<td style="text-align:center;"><strong>1.000</strong></td>
|
||||
<td style="text-align:center;"><u>0.873</u></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>gpt-4o-mini</td>
|
||||
<td style="text-align:center;">0.830</td>
|
||||
<td style="text-align:center;">0.966</td>
|
||||
<td style="text-align:center;">0.893</td>
|
||||
<td style="text-align:center;">0.398</td>
|
||||
<td style="text-align:center;">0.994</td>
|
||||
<td style="text-align:center;">0.569</td>
|
||||
<td style="text-align:center;">0.786</td>
|
||||
<td style="text-align:center;">0.997</td>
|
||||
<td style="text-align:center;">0.879</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Luna*</td>
|
||||
<td style="text-align:center;">0.378</td>
|
||||
<td style="text-align:center;">0.800</td>
|
||||
<td style="text-align:center;">0.513</td>
|
||||
<td style="text-align:center;">0.649</td>
|
||||
<td style="text-align:center;">0.912</td>
|
||||
<td style="text-align:center;"><u>0.759</u></td>
|
||||
<td style="text-align:center;">0.400</td>
|
||||
<td style="text-align:center;">0.765</td>
|
||||
<td style="text-align:center;">0.525</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>RAGAS Faithfuless*</td>
|
||||
<td style="text-align:center;">0.312</td>
|
||||
<td style="text-align:center;">0.419</td>
|
||||
<td style="text-align:center;">0.357</td>
|
||||
<td style="text-align:center;"><strong>0.792</strong></td>
|
||||
<td style="text-align:center;">0.508</td>
|
||||
<td style="text-align:center;">0.619</td>
|
||||
<td style="text-align:center;">0.642</td>
|
||||
<td style="text-align:center;">0.299</td>
|
||||
<td style="text-align:center;">0.408</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Trulens Groundedness*</td>
|
||||
<td style="text-align:center;">0.228</td>
|
||||
<td style="text-align:center;">0.925</td>
|
||||
<td style="text-align:center;">0.366</td>
|
||||
<td style="text-align:center;"><u>0.669</u></td>
|
||||
<td style="text-align:center;"><u>0.965</u></td>
|
||||
<td style="text-align:center;"><strong>0.790</strong></td>
|
||||
<td style="text-align:center;">0.402</td>
|
||||
<td style="text-align:center;">0.500</td>
|
||||
<td style="text-align:center;">0.445</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>flowaicom/Flow-Judge-v0.1</td>
|
||||
<td style="text-align:center;"><u>0.835</u></td>
|
||||
<td style="text-align:center;">0.961</td>
|
||||
<td style="text-align:center;">0.894</td>
|
||||
<td style="text-align:center;">0.541</td>
|
||||
<td style="text-align:center;">0.249</td>
|
||||
<td style="text-align:center;">0.341</td>
|
||||
<td style="text-align:center;"><strong>0.834</strong></td>
|
||||
<td style="text-align:center;">0.836</td>
|
||||
<td style="text-align:center;">0.835</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
\* _reported in model paper_
|
||||
|
||||
|
||||
### HaluEval, Covid-QA, PubMedQA
|
||||
<table border="1" cellpadding="10" cellspacing="0" style="border-collapse: collapse; width: auto;">
|
||||
<thead>
|
||||
<tr>
|
||||
<th rowspan="2" style="text-align: left;">Evaluator</th>
|
||||
<th colspan="4" style="text-align: center;">HaluEval</th>
|
||||
<th colspan="4" style="text-align: center;">Covid-QA</th>
|
||||
<th colspan="4" style="text-align: center;">PubMedQA</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th style="text-align: center;">Precision</th>
|
||||
<th style="text-align: center;">Recall</th>
|
||||
<th style="text-align: center;">F1</th>
|
||||
<th style="text-align: center;">Accuracy</th>
|
||||
<th style="text-align: center;">Precision</th>
|
||||
<th style="text-align: center;">Recall</th>
|
||||
<th style="text-align: center;">F1</th>
|
||||
<th style="text-align: center;">Accuracy</th>
|
||||
<th style="text-align: center;">Precision</th>
|
||||
<th style="text-align: center;">Recall</th>
|
||||
<th style="text-align: center;">F1</th>
|
||||
<th style="text-align: center;">Accuracy</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align: left;">microsoft/Phi-3.5-mini-instruct</td>
|
||||
<td style="text-align: center;">0.730</td>
|
||||
<td style="text-align: center;"><u>0.914</u></td>
|
||||
<td style="text-align: center;">0.812</td>
|
||||
<td style="text-align: center;">0.788</td>
|
||||
<td style="text-align: center;">0.617</td>
|
||||
<td style="text-align: center;">0.964</td>
|
||||
<td style="text-align: center;">0.752</td>
|
||||
<td style="text-align: center;">0.681</td>
|
||||
<td style="text-align: center;">0.623</td>
|
||||
<td style="text-align: center;"><u>0.986</u></td>
|
||||
<td style="text-align: center;">0.764</td>
|
||||
<td style="text-align: center;">0.696</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">meta-llama/Meta-Llama-3.1-8B-Instruct</td>
|
||||
<td style="text-align: center;"><strong>0.864</strong></td>
|
||||
<td style="text-align: center;">0.891</td>
|
||||
<td style="text-align: center;"><strong>0.878</strong></td>
|
||||
<td style="text-align: center;"><u>0.874</u></td>
|
||||
<td style="text-align: center;"><u>0.663</u></td>
|
||||
<td style="text-align: center;"><u>0.976</u></td>
|
||||
<td style="text-align: center;"><u>0.790</u></td>
|
||||
<td style="text-align: center;">0.734</td>
|
||||
<td style="text-align: center;"><u>0.681</u></td>
|
||||
<td style="text-align: center;">0.962</td>
|
||||
<td style="text-align: center;"><strong>0.797</strong></td>
|
||||
<td style="text-align: center;">0.750</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">mistralai/Mistral-Nemo-Instruct-2407</td>
|
||||
<td style="text-align: center;">0.655</td>
|
||||
<td style="text-align: center;"><strong>0.993</strong></td>
|
||||
<td style="text-align: center;">0.789</td>
|
||||
<td style="text-align: center;">0.735</td>
|
||||
<td style="text-align: center;">0.651</td>
|
||||
<td style="text-align: center;"><strong>0.982</strong></td>
|
||||
<td style="text-align: center;">0.783</td>
|
||||
<td style="text-align: center;">0.728</td>
|
||||
<td style="text-align: center;">0.602</td>
|
||||
<td style="text-align: center;"><strong>0.994</strong></td>
|
||||
<td style="text-align: center;"><u>0.750</u></td>
|
||||
<td style="text-align: center;">0.669</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">gpt-4o-mini</td>
|
||||
<td style="text-align: center;">0.846</td>
|
||||
<td style="text-align: center;">0.940</td>
|
||||
<td style="text-align: center;">0.891</td>
|
||||
<td style="text-align: center;">0.885</td>
|
||||
<td style="text-align: center;">0.795</td>
|
||||
<td style="text-align: center;">0.964</td>
|
||||
<td style="text-align: center;">0.872</td>
|
||||
<td style="text-align: center;">0.858</td>
|
||||
<td style="text-align: center;">0.791</td>
|
||||
<td style="text-align: center;">0.904</td>
|
||||
<td style="text-align: center;">0.843</td>
|
||||
<td style="text-align: center;">0.832</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">flowaicom/Flow-Judge-v0.1</td>
|
||||
<td style="text-align: center;"><u>0.826</u></td>
|
||||
<td style="text-align: center;">0.895</td>
|
||||
<td style="text-align: center;"><u>0.859</u></td>
|
||||
<td style="text-align: center;">0.854</td>
|
||||
<td style="text-align: center;"><strong>0.767</strong></td>
|
||||
<td style="text-align: center;">0.877</td>
|
||||
<td style="text-align: center;"><strong>0.818</strong></td>
|
||||
<td style="text-align: center;">0.807</td>
|
||||
<td style="text-align: center;"><strong>0.874</strong></td>
|
||||
<td style="text-align: center;">0.624</td>
|
||||
<td style="text-align: center;">0.728</td>
|
||||
<td style="text-align: center;">0.767</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">gpt-4o*</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.879</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.821</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.821</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">Claude 3 Sonnet*</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.845</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.829</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.829</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">RAGAS Faithfulness*</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.706</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.750</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.669</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">Lynx 8B*</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">0.857</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;"><u>0.963</u></td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;"><u>0.852</u></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align: left;">Lynx 70B*</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;"><strong>0.884</strong></td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;"><strong>0.975</strong></td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;">-</td>
|
||||
<td style="text-align: center;"><strong>0.904</strong></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
\* _reported in model paper_
|
||||
### Feedback Bench
|
||||
|
||||
<table border="1" cellpadding="10" cellspacing="0" style="border-collapse: collapse; width: auto;">
|
||||
<tr>
|
||||
<th rowspan="2">Evaluator</th>
|
||||
<th colspan="3" style="text-align:center;">Feedback bench</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th style="text-align:center;">pearsonr</th>
|
||||
<th style="text-align:center;">spearmanr</th>
|
||||
<th style="text-align:center;">kendall-tau</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>microsoft/Phi-3.5-mini-instruct</td>
|
||||
<td style="text-align:center;">0.710</td>
|
||||
<td style="text-align:center;">0.721</td>
|
||||
<td style="text-align:center;">0.622</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>prometheus-eval/prometheus-7b-v2.0*</td>
|
||||
<td style="text-align:center;"><strong>0.878</strong></td>
|
||||
<td style="text-align:center;"><strong>0.909</strong></td>
|
||||
<td style="text-align:center;"><strong>0.773</strong></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>meta-llama/Meta-Llama-3.1-8B-Instruct</td>
|
||||
<td style="text-align:center;">0.742</td>
|
||||
<td style="text-align:center;">0.749</td>
|
||||
<td style="text-align:center;">0.654</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>mistralai/Mistral-Nemo-Instruct-2407</td>
|
||||
<td style="text-align:center;">0.720</td>
|
||||
<td style="text-align:center;">0.724</td>
|
||||
<td style="text-align:center;">0.632</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>gpt-4o-mini</td>
|
||||
<td style="text-align:center;">0.797</td>
|
||||
<td style="text-align:center;">0.795</td>
|
||||
<td style="text-align:center;">0.701</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>flowaicom/Flow-Judge-v0.1</td>
|
||||
<td style="text-align:center;"><u>0.787</u></td>
|
||||
<td style="text-align:center;"><u>0.789</u></td>
|
||||
<td style="text-align:center;"><u>0.688</u></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
\* _reported in model paper using reference answers_
|
||||
|
||||
## License
|
||||
We opted for the Apache 2.0 license for Flow Judge to provide the community with an open, small yet powerful LM evaluator. Our goal is to support the wider adoption of rigorous evaluation techniques in LLM system development, making them more accessible to practitioners and researchers.
|
||||
|
||||
## Limitations and future work
|
||||
Multilingual evaluation: Flow Judge has been fine-tuned exclusively on English data. While the foundation model (Phi-3.5-mini-instruct [17]) may possess multilingual capabilities, we have not systematically evaluated Flow Judge performance in non-English contexts. We plan to explore multi-lingual LM evaluators in the future.
|
||||
|
||||
Long context and structured Inputs: Our training dataset encompasses a wide range of custom metrics relevant to evaluating LLM systems. However, it does not include examples with long context inputs or structured data formats such as JSON, since these are harder to synthetically generate. This limitation may impact Flow Judge's performance when evaluating responses that require processing extensive context or parsing structured input. Extending our model’s capabilities to handle these input types represents an important area for future research.
|
||||
|
||||
Math and coding: The current version has not been trained on specific task domains such as arithmetic problems or code evaluation. As a result, its performance in these specialized areas may be limited. Future iterations of the model should address these gaps.
|
||||
|
||||
Domain-specific knowledge and complex multi-step evaluations: Flow Judge may struggle with highly specialized domain knowledge or proprietary data outside the training scope of its foundation model. Additionally, evaluation tasks requiring multi-step reasoning or complex logical processes may challenge the model's capabilities. We strongly recommend conducting meta-evaluations of the model performance before deploying it in specialized or highly complex evaluation scenarios.
|
||||
13
added_tokens.json
Normal file
13
added_tokens.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"<|assistant|>": 32001,
|
||||
"<|endoftext|>": 32000,
|
||||
"<|end|>": 32007,
|
||||
"<|placeholder1|>": 32002,
|
||||
"<|placeholder2|>": 32003,
|
||||
"<|placeholder3|>": 32004,
|
||||
"<|placeholder4|>": 32005,
|
||||
"<|placeholder5|>": 32008,
|
||||
"<|placeholder6|>": 32009,
|
||||
"<|system|>": 32006,
|
||||
"<|user|>": 32010
|
||||
}
|
||||
146
config.json
Normal file
146
config.json
Normal file
@@ -0,0 +1,146 @@
|
||||
{
|
||||
"_name_or_path": "/home/ks/repos/axolotl/flow-judge-ckpt-231-merged",
|
||||
"architectures": [
|
||||
"Phi3ForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"auto_map": {
|
||||
"AutoConfig": "configuration_phi3.Phi3Config",
|
||||
"AutoModelForCausalLM": "modeling_phi3.Phi3ForCausalLM"
|
||||
},
|
||||
"bos_token_id": 1,
|
||||
"embd_pdrop": 0.0,
|
||||
"eos_token_id": 32000,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 3072,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 8192,
|
||||
"max_position_embeddings": 131072,
|
||||
"model_type": "phi3",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 32,
|
||||
"original_max_position_embeddings": 4096,
|
||||
"pad_token_id": 32000,
|
||||
"quantization_config": {
|
||||
"bits": 4,
|
||||
"group_size": 128,
|
||||
"modules_to_not_convert": null,
|
||||
"quant_method": "awq",
|
||||
"version": "gemm",
|
||||
"zero_point": true
|
||||
},
|
||||
"resid_pdrop": 0.0,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": {
|
||||
"long_factor": [
|
||||
1.0800000429153442,
|
||||
1.1100000143051147,
|
||||
1.1399999856948853,
|
||||
1.340000033378601,
|
||||
1.5899999141693115,
|
||||
1.600000023841858,
|
||||
1.6200000047683716,
|
||||
2.620000123977661,
|
||||
3.2300000190734863,
|
||||
3.2300000190734863,
|
||||
4.789999961853027,
|
||||
7.400000095367432,
|
||||
7.700000286102295,
|
||||
9.09000015258789,
|
||||
12.199999809265137,
|
||||
17.670000076293945,
|
||||
24.46000099182129,
|
||||
28.57000160217285,
|
||||
30.420001983642578,
|
||||
30.840002059936523,
|
||||
32.590003967285156,
|
||||
32.93000411987305,
|
||||
42.320003509521484,
|
||||
44.96000289916992,
|
||||
50.340003967285156,
|
||||
50.45000457763672,
|
||||
57.55000305175781,
|
||||
57.93000411987305,
|
||||
58.21000289916992,
|
||||
60.1400032043457,
|
||||
62.61000442504883,
|
||||
62.62000274658203,
|
||||
62.71000289916992,
|
||||
63.1400032043457,
|
||||
63.1400032043457,
|
||||
63.77000427246094,
|
||||
63.93000411987305,
|
||||
63.96000289916992,
|
||||
63.970001220703125,
|
||||
64.02999877929688,
|
||||
64.06999969482422,
|
||||
64.08000183105469,
|
||||
64.12000274658203,
|
||||
64.41000366210938,
|
||||
64.4800033569336,
|
||||
64.51000213623047,
|
||||
64.52999877929688,
|
||||
64.83999633789062
|
||||
],
|
||||
"short_factor": [
|
||||
1.0,
|
||||
1.0199999809265137,
|
||||
1.0299999713897705,
|
||||
1.0299999713897705,
|
||||
1.0499999523162842,
|
||||
1.0499999523162842,
|
||||
1.0499999523162842,
|
||||
1.0499999523162842,
|
||||
1.0499999523162842,
|
||||
1.0699999332427979,
|
||||
1.0999999046325684,
|
||||
1.1099998950958252,
|
||||
1.1599998474121094,
|
||||
1.1599998474121094,
|
||||
1.1699998378753662,
|
||||
1.2899998426437378,
|
||||
1.339999794960022,
|
||||
1.679999828338623,
|
||||
1.7899998426437378,
|
||||
1.8199998140335083,
|
||||
1.8499997854232788,
|
||||
1.8799997568130493,
|
||||
1.9099997282028198,
|
||||
1.9399996995925903,
|
||||
1.9899996519088745,
|
||||
2.0199997425079346,
|
||||
2.0199997425079346,
|
||||
2.0199997425079346,
|
||||
2.0199997425079346,
|
||||
2.0199997425079346,
|
||||
2.0199997425079346,
|
||||
2.0299997329711914,
|
||||
2.0299997329711914,
|
||||
2.0299997329711914,
|
||||
2.0299997329711914,
|
||||
2.0299997329711914,
|
||||
2.0299997329711914,
|
||||
2.0299997329711914,
|
||||
2.0299997329711914,
|
||||
2.0299997329711914,
|
||||
2.0799996852874756,
|
||||
2.0899996757507324,
|
||||
2.189999580383301,
|
||||
2.2199995517730713,
|
||||
2.5899994373321533,
|
||||
2.729999542236328,
|
||||
2.749999523162842,
|
||||
2.8399994373321533
|
||||
],
|
||||
"type": "longrope"
|
||||
},
|
||||
"rope_theta": 10000.0,
|
||||
"sliding_window": 262144,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.44.2",
|
||||
"use_cache": false,
|
||||
"vocab_size": 32064
|
||||
}
|
||||
227
configuration_phi3.py
Normal file
227
configuration_phi3.py
Normal file
@@ -0,0 +1,227 @@
|
||||
# coding=utf-8
|
||||
# Copyright 2024 Microsoft and the HuggingFace Inc. team. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
""" Phi-3 model configuration"""
|
||||
|
||||
|
||||
from transformers.configuration_utils import PretrainedConfig
|
||||
from transformers.utils import logging
|
||||
|
||||
|
||||
logger = logging.get_logger(__name__)
|
||||
|
||||
PHI3_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
||||
"microsoft/Phi-3-mini-4k-instruct": "https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/resolve/main/config.json",
|
||||
"microsoft/Phi-3-mini-128k-instruct": "https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/resolve/main/config.json",
|
||||
}
|
||||
|
||||
|
||||
class Phi3Config(PretrainedConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a [`Phi3Model`]. It is used to instantiate a Phi-3
|
||||
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
|
||||
defaults will yield a similar configuration to that of the
|
||||
[microsoft/Phi-3-mini-4k-instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct).
|
||||
|
||||
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
|
||||
documentation from [`PretrainedConfig`] for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (`int`, *optional*, defaults to 32064):
|
||||
Vocabulary size of the Phi-3 model. Defines the number of different tokens that can be represented by the
|
||||
`inputs_ids` passed when calling [`Phi3Model`].
|
||||
hidden_size (`int`, *optional*, defaults to 3072):
|
||||
Dimension of the hidden representations.
|
||||
intermediate_size (`int`, *optional*, defaults to 8192):
|
||||
Dimension of the MLP representations.
|
||||
num_hidden_layers (`int`, *optional*, defaults to 32):
|
||||
Number of hidden layers in the Transformer decoder.
|
||||
num_attention_heads (`int`, *optional*, defaults to 32):
|
||||
Number of attention heads for each attention layer in the Transformer decoder.
|
||||
num_key_value_heads (`int`, *optional*):
|
||||
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
|
||||
`num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
|
||||
`num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
|
||||
converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
|
||||
by meanpooling all the original heads within that group. For more details checkout [this
|
||||
paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
|
||||
`num_attention_heads`.
|
||||
resid_pdrop (`float`, *optional*, defaults to 0.0):
|
||||
Dropout probability for mlp outputs.
|
||||
embd_pdrop (`int`, *optional*, defaults to 0.0):
|
||||
The dropout ratio for the embeddings.
|
||||
attention_dropout (`float`, *optional*, defaults to 0.0):
|
||||
The dropout ratio after computing the attention scores.
|
||||
hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
|
||||
The non-linear activation function (function or string) in the decoder.
|
||||
max_position_embeddings (`int`, *optional*, defaults to 4096):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
original_max_position_embeddings (`int`, *optional*, defaults to 4096):
|
||||
The maximum sequence length that this model was trained with. This is used to determine the size of the
|
||||
original RoPE embeddings when using long scaling.
|
||||
initializer_range (`float`, *optional*, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
rms_norm_eps (`float`, *optional*, defaults to 1e-05):
|
||||
The epsilon value used for the RMSNorm.
|
||||
use_cache (`bool`, *optional*, defaults to `True`):
|
||||
Whether or not the model should return the last key/values attentions (not used by all models). Only
|
||||
relevant if `config.is_decoder=True`. Whether to tie weight embeddings or not.
|
||||
tie_word_embeddings (`bool`, *optional*, defaults to `False`):
|
||||
Whether to tie weight embeddings
|
||||
rope_theta (`float`, *optional*, defaults to 10000.0):
|
||||
The base period of the RoPE embeddings.
|
||||
rope_scaling (`dict`, *optional*):
|
||||
The scaling strategy for the RoPE embeddings. If `None`, no scaling is applied. If a dictionary, it must
|
||||
contain the following keys: `type`, `short_factor` and `long_factor`. The `type` must be `longrope` and
|
||||
the `short_factor` and `long_factor` must be lists of numbers with the same length as the hidden size
|
||||
divided by the number of attention heads divided by 2.
|
||||
bos_token_id (`int`, *optional*, defaults to 1):
|
||||
The id of the "beginning-of-sequence" token.
|
||||
eos_token_id (`int`, *optional*, defaults to 32000):
|
||||
The id of the "end-of-sequence" token.
|
||||
pad_token_id (`int`, *optional*, defaults to 32000):
|
||||
The id of the padding token.
|
||||
sliding_window (`int`, *optional*):
|
||||
Sliding window attention window size. If `None`, no sliding window is applied.
|
||||
|
||||
Example:
|
||||
|
||||
```python
|
||||
>>> from transformers import Phi3Model, Phi3Config
|
||||
|
||||
>>> # Initializing a Phi-3 style configuration
|
||||
>>> configuration = Phi3Config.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
|
||||
|
||||
>>> # Initializing a model from the configuration
|
||||
>>> model = Phi3Model(configuration)
|
||||
|
||||
>>> # Accessing the model configuration
|
||||
>>> configuration = model.config
|
||||
```"""
|
||||
|
||||
model_type = "phi3"
|
||||
keys_to_ignore_at_inference = ["past_key_values"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vocab_size=32064,
|
||||
hidden_size=3072,
|
||||
intermediate_size=8192,
|
||||
num_hidden_layers=32,
|
||||
num_attention_heads=32,
|
||||
num_key_value_heads=None,
|
||||
resid_pdrop=0.0,
|
||||
embd_pdrop=0.0,
|
||||
attention_dropout=0.0,
|
||||
hidden_act="silu",
|
||||
max_position_embeddings=4096,
|
||||
original_max_position_embeddings=4096,
|
||||
initializer_range=0.02,
|
||||
rms_norm_eps=1e-5,
|
||||
use_cache=True,
|
||||
tie_word_embeddings=False,
|
||||
rope_theta=10000.0,
|
||||
rope_scaling=None,
|
||||
bos_token_id=1,
|
||||
eos_token_id=32000,
|
||||
pad_token_id=32000,
|
||||
sliding_window=None,
|
||||
**kwargs,
|
||||
):
|
||||
self.vocab_size = vocab_size
|
||||
self.hidden_size = hidden_size
|
||||
self.intermediate_size = intermediate_size
|
||||
self.num_hidden_layers = num_hidden_layers
|
||||
self.num_attention_heads = num_attention_heads
|
||||
|
||||
if num_key_value_heads is None:
|
||||
num_key_value_heads = num_attention_heads
|
||||
|
||||
self.num_key_value_heads = num_key_value_heads
|
||||
self.resid_pdrop = resid_pdrop
|
||||
self.embd_pdrop = embd_pdrop
|
||||
self.attention_dropout = attention_dropout
|
||||
self.hidden_act = hidden_act
|
||||
self.max_position_embeddings = max_position_embeddings
|
||||
self.original_max_position_embeddings = original_max_position_embeddings
|
||||
self.initializer_range = initializer_range
|
||||
self.rms_norm_eps = rms_norm_eps
|
||||
self.use_cache = use_cache
|
||||
self.rope_theta = rope_theta
|
||||
self.rope_scaling = rope_scaling
|
||||
self._rope_scaling_adjustment()
|
||||
self._rope_scaling_validation()
|
||||
self.sliding_window = sliding_window
|
||||
|
||||
super().__init__(
|
||||
bos_token_id=bos_token_id,
|
||||
eos_token_id=eos_token_id,
|
||||
pad_token_id=pad_token_id,
|
||||
tie_word_embeddings=tie_word_embeddings,
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
def _rope_scaling_adjustment(self):
|
||||
"""
|
||||
Adjust the `type` of the `rope_scaling` configuration for backward compatibility.
|
||||
"""
|
||||
if self.rope_scaling is None:
|
||||
return
|
||||
|
||||
rope_scaling_type = self.rope_scaling.get("type", None)
|
||||
|
||||
# For backward compatibility if previous version used "su" or "yarn"
|
||||
if rope_scaling_type is not None and rope_scaling_type in ["su", "yarn"]:
|
||||
self.rope_scaling["type"] = "longrope"
|
||||
|
||||
def _rope_scaling_validation(self):
|
||||
"""
|
||||
Validate the `rope_scaling` configuration.
|
||||
"""
|
||||
if self.rope_scaling is None:
|
||||
return
|
||||
|
||||
if not isinstance(self.rope_scaling, dict) or len(self.rope_scaling) != 3:
|
||||
raise ValueError(
|
||||
"`rope_scaling` must be a dictionary with three fields, `type`, `short_factor` and `long_factor`, "
|
||||
f"got {self.rope_scaling}"
|
||||
)
|
||||
rope_scaling_type = self.rope_scaling.get("type", None)
|
||||
rope_scaling_short_factor = self.rope_scaling.get("short_factor", None)
|
||||
rope_scaling_long_factor = self.rope_scaling.get("long_factor", None)
|
||||
if rope_scaling_type is None or rope_scaling_type not in ["longrope"]:
|
||||
raise ValueError(f"`rope_scaling`'s type field must be one of ['longrope'], got {rope_scaling_type}")
|
||||
if not (
|
||||
isinstance(rope_scaling_short_factor, list)
|
||||
and all(isinstance(x, (int, float)) for x in rope_scaling_short_factor)
|
||||
):
|
||||
raise ValueError(
|
||||
f"`rope_scaling`'s short_factor field must be a list of numbers, got {rope_scaling_short_factor}"
|
||||
)
|
||||
if not len(rope_scaling_short_factor) == self.hidden_size // self.num_attention_heads // 2:
|
||||
raise ValueError(
|
||||
f"`rope_scaling`'s short_factor field must have length {self.hidden_size // self.num_attention_heads // 2}, got {len(rope_scaling_short_factor)}"
|
||||
)
|
||||
if not (
|
||||
isinstance(rope_scaling_long_factor, list)
|
||||
and all(isinstance(x, (int, float)) for x in rope_scaling_long_factor)
|
||||
):
|
||||
raise ValueError(
|
||||
f"`rope_scaling`'s long_factor field must be a list of numbers, got {rope_scaling_long_factor}"
|
||||
)
|
||||
if not len(rope_scaling_long_factor) == self.hidden_size // self.num_attention_heads // 2:
|
||||
raise ValueError(
|
||||
f"`rope_scaling`'s long_factor field must have length {self.hidden_size // self.num_attention_heads // 2}, got {len(rope_scaling_long_factor)}"
|
||||
)
|
||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"do_sample": true,
|
||||
"eos_token_id": [
|
||||
32007,
|
||||
32001,
|
||||
32000
|
||||
],
|
||||
"pad_token_id": 32000,
|
||||
"transformers_version": "4.44.2"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b5efe582972470ebc9f4ba4154194a8c35d0a37ce934057065477536c37a1441
|
||||
size 2277171688
|
||||
1570
modeling_phi3.py
Normal file
1570
modeling_phi3.py
Normal file
File diff suppressed because it is too large
Load Diff
30
special_tokens_map.json
Normal file
30
special_tokens_map.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
93463
tokenizer.json
Normal file
93463
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
Binary file not shown.
131
tokenizer_config.json
Normal file
131
tokenizer_config.json
Normal file
@@ -0,0 +1,131 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_eos_token": false,
|
||||
"add_prefix_space": null,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"32000": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32001": {
|
||||
"content": "<|assistant|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32002": {
|
||||
"content": "<|placeholder1|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32003": {
|
||||
"content": "<|placeholder2|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32004": {
|
||||
"content": "<|placeholder3|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32005": {
|
||||
"content": "<|placeholder4|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32006": {
|
||||
"content": "<|system|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32007": {
|
||||
"content": "<|end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32008": {
|
||||
"content": "<|placeholder5|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32009": {
|
||||
"content": "<|placeholder6|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32010": {
|
||||
"content": "<|user|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": true,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"bos_token": "<s>",
|
||||
"chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'system') %}{{'<|system|>' + '\n' + message['content'] + '<|endoftext|>' + '\n'}}{% elif (message['role'] == 'user') %}{{'<|user|>' + '\n' + message['content'] + '<|endoftext|>' + '\n' + '<|assistant|>' + '\n'}}{% elif message['role'] == 'assistant' %}{{message['content'] + '<|endoftext|>' + '\n'}}{% endif %}{% endfor %}",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|endoftext|>",
|
||||
"legacy": false,
|
||||
"model_max_length": 131072,
|
||||
"pad_token": "<|endoftext|>",
|
||||
"padding_side": "left",
|
||||
"sp_model_kwargs": {},
|
||||
"tokenizer_class": "LlamaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user