初始化项目，由ModelHub XC社区提供模型
Model: adgomant/adele-judge-qwen3-14B-cre Source: Original Platform
This commit is contained in:
ModelHub XC
151
README.md
Normal file
151
README.md
Normal file
@@ -0,0 +1,151 @@
|
||||
---
|
||||
library_name: transformers
|
||||
tags:
|
||||
- text-generation
|
||||
- peft
|
||||
- adele
|
||||
- judge
|
||||
base_model: Qwen/Qwen3-14B
|
||||
datasets:
|
||||
- CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0
|
||||
---
|
||||
|
||||
# ADeLe Distilled Judge
|
||||
|
||||
This repository contains an ADeLe-suite-specific distilled judge. It scores a model response against a question and reference answer with an ordinal score from 1 to 5, then derives binary correctness with the ADeLe threshold.
|
||||
|
||||
The repository root contains a merged Transformers model for standard loading. The original LoRA adapter is also included under `adapter/` for provenance and reuse.
|
||||
|
||||
## Intended Use
|
||||
|
||||
Use this model to score ADeLe-style examples where a question, reference answer, and model response are available. It is intended for out-of-model evaluation within the ADeLe benchmark suite, not as a general-purpose evaluator.
|
||||
|
||||
## Input Format
|
||||
|
||||
The recommended helper accepts:
|
||||
|
||||
- `question`
|
||||
- `reference_answer` or `ground_truth`
|
||||
- `model_response`
|
||||
|
||||
## Score Rubric
|
||||
|
||||
Allowed scores: 1, 2, 3, 4, 5
|
||||
|
||||
- 1: surely incorrect
|
||||
- 2: likely incorrect
|
||||
- 3: minimally correct or sufficient
|
||||
- 4: likely correct
|
||||
- 5: surely correct
|
||||
|
||||
Binary label: scores greater than or equal to 3 are `CORRECT`; lower scores are `INCORRECT`.
|
||||
|
||||
## Training And Validation Data
|
||||
|
||||
| Split | Examples | Models |
|
||||
| --- | --- | --- |
|
||||
| train | 239,420 | 16 |
|
||||
| validation | 45,738 | 3 |
|
||||
|
||||
- `train` models: `DK-R1-Dist-Qwen-1.5B`, `DK-R1-Dist-Qwen-32B`, `DK-R1-Dist-Qwen-7B`, `gemini-2.5-flash`, `gemini-3.1-pro`, `gpt-35-turbo`, `gpt-5.2`, `gpt4o`, `llama3d1-405b`, `llama3d2-11b`, `llama3d2-1b`, `llama3d2-90b`, `llama4-17B-128E`, `o1-mini`, `o1_re=low`, `o3-mini`
|
||||
- `validation` models: `DK-R1-Dist-Qwen-14B`, `gemini-3-flash`, `llama3d2-3b`
|
||||
|
||||
## Data Quality And Label Construction
|
||||
|
||||
Training labels are distilled from two proprietary judge scores used by the ADeLe evaluation pipeline to derive the official correctness signal. The configured source columns are `score_gpt4o` and `score_sonnet`.
|
||||
|
||||
- Ordinal target: `floor(mean(score_gpt4o, score_sonnet))`.
|
||||
- Binary target: `CORRECT` when the ordinal target is >= `3`.
|
||||
- Judge-agreement filter: keep examples with `abs(score_gpt4o - score_sonnet) <= 1`.
|
||||
- Response-length filter: keep responses with at most `4096` base-tokenizer tokens before prompt formatting.
|
||||
- Sequence-length filter: keep full chat-formatted examples within `max_seq_length=8192`.
|
||||
|
||||
## Validation Results
|
||||
|
||||
Source artifact: `validation_trainer_metrics.json`.
|
||||
|
||||
| Metric | Value |
|
||||
| --- | --- |
|
||||
| Epoch | 1.0000 |
|
||||
| Binary accuracy | 0.9894 |
|
||||
| Binary macro F1 | 0.9880 |
|
||||
| Precision, CORRECT | 0.9932 |
|
||||
| Recall, CORRECT | 0.9909 |
|
||||
| Precision, INCORRECT | 0.9817 |
|
||||
| Recall, INCORRECT | 0.9863 |
|
||||
| False negative rate, CORRECT | 0.0091 |
|
||||
| False positive rate, CORRECT | 0.0137 |
|
||||
| Ordinal accuracy | 0.9639 |
|
||||
| Ordinal macro F1 | 0.7351 |
|
||||
| Mean confidence | 0.9604 |
|
||||
|
||||
## Recommended Inference
|
||||
|
||||
Do not use free-form generation as the primary prediction method. The recommended path scores the restricted continuations `"1"`, `"2"`, `"3"`, `"4"`, and `"5"`.
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
judge = pipeline(
|
||||
"adele-judge",
|
||||
model="adgomant/adele-judge-qwen3-14-cre",
|
||||
trust_remote_code=True,
|
||||
device_map="auto",
|
||||
)
|
||||
result = judge(
|
||||
{"question": "...", "reference_answer": "...", "model_response": "..."}
|
||||
)
|
||||
print(result)
|
||||
|
||||
results = judge([
|
||||
{"question": "...", "reference_answer": "...", "model_response": "..."},
|
||||
{"question": "...", "ground_truth": "...", "model_response": "..."},
|
||||
], batch_size=8)
|
||||
```
|
||||
|
||||
The result has this shape:
|
||||
|
||||
```python
|
||||
{
|
||||
"score": 4,
|
||||
"label": "CORRECT",
|
||||
"probs": {"1": 0.01, "2": 0.02, "3": 0.08, "4": 0.70, "5": 0.19},
|
||||
"logprobs": {"1": -5.0, "2": -4.2, "3": -2.9, "4": -0.8, "5": -2.1},
|
||||
"confidence": 0.70,
|
||||
"margin": 1.3,
|
||||
"entropy": 0.82,
|
||||
}
|
||||
```
|
||||
|
||||
## Standard Transformers Loading
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)
|
||||
```
|
||||
|
||||
`generation_config.json` uses safe one-token defaults for debugging, but `generate()` is not the recommended scoring method.
|
||||
|
||||
## Metadata
|
||||
|
||||
Training, filtering, split, tokenization, and metric artifacts available at packaging time are stored in `adele_judge_metadata.json`.
|
||||
|
||||
The model is trained on distilled judge targets. These targets are useful for reproducing the ADeLe paper-style correctness signal at lower inference cost, but they should not be interpreted as independent human annotations.
|
||||
|
||||
## References
|
||||
|
||||
- ADeLe project page: [ADeLe v1.0](https://kinds-of-intelligence-cfi.github.io/ADELE/).
|
||||
- ADeLe paper and official correctness definition: [General scales unlock AI evaluation with explanatory and predictive power](https://www.nature.com/articles/s41586-026-10303-2).
|
||||
- Official ADeLe dataset: [CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0](https://huggingface.co/datasets/CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0).
|
||||
- Official instance-level model-response data used for distillation: [https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data](https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data).
|
||||
- Training and Hub packaging implementation: [https://github.com/adgomant/adele-judge](https://github.com/adgomant/adele-judge).
|
||||
|
||||
## Limitations
|
||||
|
||||
- ADeLe-specific judge; not a general-purpose evaluator.
|
||||
- Distilled from proprietary judge labels and inherits their noise, calibration, and biases.
|
||||
- Intended for scoring responses against a reference answer.
|
||||
- It should not produce explanations; the expected output is a single score.
|
||||
- Validation is out-of-model within the ADeLe suite, so transfer outside that suite should be measured before relying on it.
|
||||
Reference in New Issue
Block a user