adele-judge-qwen3-14B-cre/README.md

---
library_name: transformers
tags:
- text-generation
- peft
- adele
- judge
base_model: Qwen/Qwen3-14B
datasets:
- CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0
---

# ADeLe Distilled Judge

This repository contains an ADeLe-suite-specific distilled judge. It scores a model response against a question and reference answer with an ordinal score from 1 to 5, then derives binary correctness with the ADeLe threshold.

The repository root contains a merged Transformers model for standard loading. The original LoRA adapter is also included under `adapter/` for provenance and reuse.

## Intended Use

Use this model to score ADeLe-style examples where a question, reference answer, and model response are available. It is intended for out-of-model evaluation within the ADeLe benchmark suite, not as a general-purpose evaluator.

## Input Format

The recommended helper accepts:

- `question`
- `reference_answer` or `ground_truth`
- `model_response`

## Score Rubric

Allowed scores: 1, 2, 3, 4, 5

- 1: surely incorrect
- 2: likely incorrect
- 3: minimally correct or sufficient
- 4: likely correct
- 5: surely correct

Binary label: scores greater than or equal to 3 are `CORRECT`; lower scores are `INCORRECT`.

## Training And Validation Data

| Split | Examples | Models |
| --- | --- | --- |
| train | 239,420 | 16 |
| validation | 45,738 | 3 |

- `train` models: `DK-R1-Dist-Qwen-1.5B`, `DK-R1-Dist-Qwen-32B`, `DK-R1-Dist-Qwen-7B`, `gemini-2.5-flash`, `gemini-3.1-pro`, `gpt-35-turbo`, `gpt-5.2`, `gpt4o`, `llama3d1-405b`, `llama3d2-11b`, `llama3d2-1b`, `llama3d2-90b`, `llama4-17B-128E`, `o1-mini`, `o1_re=low`, `o3-mini`
- `validation` models: `DK-R1-Dist-Qwen-14B`, `gemini-3-flash`, `llama3d2-3b`

## Data Quality And Label Construction

Training labels are distilled from two proprietary judge scores used by the ADeLe evaluation pipeline to derive the official correctness signal. The configured source columns are `score_gpt4o` and `score_sonnet`.

- Ordinal target: `floor(mean(score_gpt4o, score_sonnet))`.
- Binary target: `CORRECT` when the ordinal target is >= `3`.
- Judge-agreement filter: keep examples with `abs(score_gpt4o - score_sonnet) <= 1`.
- Response-length filter: keep responses with at most `4096` base-tokenizer tokens before prompt formatting.
- Sequence-length filter: keep full chat-formatted examples within `max_seq_length=8192`.

## Validation Results

Source artifact: `validation_trainer_metrics.json`.

| Metric | Value |
| --- | --- |
| Epoch | 1.0000 |
| Binary accuracy | 0.9894 |
| Binary macro F1 | 0.9880 |
| Precision, CORRECT | 0.9932 |
| Recall, CORRECT | 0.9909 |
| Precision, INCORRECT | 0.9817 |
| Recall, INCORRECT | 0.9863 |
| False negative rate, CORRECT | 0.0091 |
| False positive rate, CORRECT | 0.0137 |
| Ordinal accuracy | 0.9639 |
| Ordinal macro F1 | 0.7351 |
| Mean confidence | 0.9604 |

## Recommended Inference

Do not use free-form generation as the primary prediction method. The recommended path scores the restricted continuations `"1"`, `"2"`, `"3"`, `"4"`, and `"5"`.

```python
from transformers import pipeline

judge = pipeline(
    "adele-judge",
    model="adgomant/adele-judge-qwen3-14-cre",
    trust_remote_code=True,
    device_map="auto",
)
result = judge(
    {"question": "...", "reference_answer": "...", "model_response": "..."}
)
print(result)

results = judge([
    {"question": "...", "reference_answer": "...", "model_response": "..."},
    {"question": "...", "ground_truth": "...", "model_response": "..."},
], batch_size=8)
```

The result has this shape:

```python
{
    "score": 4,
    "label": "CORRECT",
    "probs": {"1": 0.01, "2": 0.02, "3": 0.08, "4": 0.70, "5": 0.19},
    "logprobs": {"1": -5.0, "2": -4.2, "3": -2.9, "4": -0.8, "5": -2.1},
    "confidence": 0.70,
    "margin": 1.3,
    "entropy": 0.82,
}
```

## Standard Transformers Loading

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)
```

`generation_config.json` uses safe one-token defaults for debugging, but `generate()` is not the recommended scoring method.

## Metadata

Training, filtering, split, tokenization, and metric artifacts available at packaging time are stored in `adele_judge_metadata.json`.

The model is trained on distilled judge targets. These targets are useful for reproducing the ADeLe paper-style correctness signal at lower inference cost, but they should not be interpreted as independent human annotations.

## References

- ADeLe project page: [ADeLe v1.0](https://kinds-of-intelligence-cfi.github.io/ADELE/).
- ADeLe paper and official correctness definition: [General scales unlock AI evaluation with explanatory and predictive power](https://www.nature.com/articles/s41586-026-10303-2).
- Official ADeLe dataset: [CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0](https://huggingface.co/datasets/CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0).
- Official instance-level model-response data used for distillation: [https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data](https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data).
- Training and Hub packaging implementation: [https://github.com/adgomant/adele-judge](https://github.com/adgomant/adele-judge).

## Limitations

- ADeLe-specific judge; not a general-purpose evaluator.
- Distilled from proprietary judge labels and inherits their noise, calibration, and biases.
- Intended for scoring responses against a reference answer.
- It should not produce explanations; the expected output is a single score.
- Validation is out-of-model within the ADeLe suite, so transfer outside that suite should be measured before relying on it.