234 lines
10 KiB
Markdown
234 lines
10 KiB
Markdown
|
---
|
|||
|
|
base_model: Qwen/Qwen3-0.6B
|
|||
|
|
datasets:
|
|||
|
|
- custom-redrob-candidates
|
|||
|
|
language: en
|
|||
|
|
library_name: transformers
|
|||
|
|
license: mit
|
|||
|
|
metrics:
|
|||
|
|
- reward
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
tags:
|
|||
|
|
- qwen3
|
|||
|
|
- qwen3-0.6b
|
|||
|
|
- grpo
|
|||
|
|
- reinforcement-learning
|
|||
|
|
- trl
|
|||
|
|
- candidate-ranking
|
|||
|
|
- explainable-ai
|
|||
|
|
- talentry-ai
|
|||
|
|
- redrob
|
|||
|
|
- hackathon
|
|||
|
|
- open-source
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# redrob-qwen-grpo
|
|||
|
|
|
|||
|
|
> **`Qwen/Qwen3-0.6B` → GRPO-fine-tuned for explainable candidate ranking, under a rule-based reward model (no LLM-as-a-judge).**
|
|||
|
|
|
|||
|
|
[](https://huggingface.co/spaces/williyam/talentry-ai)
|
|||
|
|
[](https://github.com/williyam-m/talentry-ai)
|
|||
|
|
[](https://github.com/williyam-m/talentry-ai/blob/main/LICENSE)
|
|||
|
|
|
|||
|
|
This is the **open-source side-quest** of the [Talentry-AI](https://github.com/williyam-m/talentry-ai)
|
|||
|
|
submission to the **Redrob × Hack2Skill — India Runs** Data & AI Challenge.
|
|||
|
|
|
|||
|
|
The base Talentry-AI ranker is fully deterministic and runs with **0 LLM calls**.
|
|||
|
|
This checkpoint exists for anyone who *wants* an LLM-flavoured candidate ranker
|
|||
|
|
that has been trained against the same rule-based rubric Talentry-AI uses to
|
|||
|
|
audit its own decisions. The Talentry-AI submission itself does **not** depend on
|
|||
|
|
this model.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Headline results
|
|||
|
|
|
|||
|
|
| Metric | Baseline (`Qwen/Qwen3-0.6B`) | `redrob-qwen-grpo` | Δ |
|
|||
|
|
| ----------------------------- | --------------------------- | ------------------ | ------ |
|
|||
|
|
| Mean rule-based reward `[0,1]` | **0.539** | **0.713** | **+0.173** |
|
|||
|
|
| Eval episodes | 12 | 12 | — |
|
|||
|
|
| Hardware | Apple M1 Pro 16 GB · MPS | Apple M1 Pro 16 GB · MPS | — |
|
|||
|
|
| Eval `max_new_tokens` | 384 | 384 | — |
|
|||
|
|
|
|||
|
|
The same deterministic eval rollout (`seed=0`, sequential, identical prompts)
|
|||
|
|
is used for both rows so the comparison is fair.
|
|||
|
|
|
|||
|
|
## Per-component improvement (rule-based reward, mean over eval episodes)
|
|||
|
|
|
|||
|
|
| Reward component | Baseline | Trained | Δ |
|
|||
|
|
| ------------------ | -------- | ------- | -- |
|
|||
|
|
| `format_valid` | 0.833 | 1.000 | +0.167 |
|
|||
|
|
| `decision_match` | 0.500 | 0.500 | +0.000 |
|
|||
|
|
| `score_alignment` | 0.373 | 0.653 | +0.280 |
|
|||
|
|
| `reason_quality` | 0.000 | 0.778 | +0.778 |
|
|||
|
|
| `length_penalty` | 1.000 | 1.000 | +0.000 |
|
|||
|
|
| `no_hallucination` | 0.779 | 0.656 | -0.124 |
|
|||
|
|
| `total` | 0.539 | 0.713 | +0.173 |
|
|||
|
|
|
|||
|
|
> All components are in `[0, 1]`. `total` is the weighted convex combination
|
|||
|
|
> (see [`reward.py`](https://github.com/williyam-m/talentry-ai/blob/main/redrob-reinforcement-learning/src/redrob_rl/reward.py)).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Quick usage
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
|
|||
|
|
tok = AutoTokenizer.from_pretrained("williyam/redrob-qwen-grpo")
|
|||
|
|
mdl = AutoModelForCausalLM.from_pretrained(
|
|||
|
|
"williyam/redrob-qwen-grpo", dtype=torch.float32
|
|||
|
|
).eval()
|
|||
|
|
|
|||
|
|
system = (
|
|||
|
|
"You are RedRob, an explainable candidate-ranking assistant. "
|
|||
|
|
"Decide whether the candidate should be SHORTLISTED for the role. "
|
|||
|
|
"Respond with a single JSON object: "
|
|||
|
|
'{"decision":"shortlist"|"reject","score":0..1,"reasons":[..]}.'
|
|||
|
|
)
|
|||
|
|
user = (
|
|||
|
|
"[JOB DESCRIPTION]\n<your JD here>\n\n"
|
|||
|
|
"[CANDIDATE]\n<candidate profile>"
|
|||
|
|
)
|
|||
|
|
|
|||
|
|
prompt = tok.apply_chat_template(
|
|||
|
|
[
|
|||
|
|
{"role": "system", "content": system},
|
|||
|
|
{"role": "user", "content": user},
|
|||
|
|
],
|
|||
|
|
tokenize=False,
|
|||
|
|
add_generation_prompt=True,
|
|||
|
|
)
|
|||
|
|
inputs = tok(prompt, return_tensors="pt")
|
|||
|
|
out = mdl.generate(**inputs, max_new_tokens=512, do_sample=False)
|
|||
|
|
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The model is expected to return:
|
|||
|
|
|
|||
|
|
```json
|
|||
|
|
{
|
|||
|
|
"decision": "shortlist" | "reject",
|
|||
|
|
"score": 0.0-1.0,
|
|||
|
|
"reasons": ["short, grounded bullet", "..."]
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Training summary
|
|||
|
|
|
|||
|
|
| Aspect | Value |
|
|||
|
|
| ---------------------------- | ---------------------------------------------------------------- |
|
|||
|
|
| Base model | `Qwen/Qwen3-0.6B` (600M params, Qwen3 chat template) |
|
|||
|
|
| Algorithm | GRPO (TRL `GRPOTrainer`) |
|
|||
|
|
| Reward signal | **Rule-based** (no LLM judge): six interpretable components |
|
|||
|
|
| Reward components | `format_valid`, `decision_match`, `score_alignment`, `reason_quality`, `length_penalty`, `no_hallucination` |
|
|||
|
|
| Optimiser steps | 10 (deliberately short — sample-efficient demo on a laptop GPU) |
|
|||
|
|
| `num_generations` | 2 (group size; 2-arm advantage estimate) |
|
|||
|
|
| KL coefficient `β` | 0.04 |
|
|||
|
|
| Learning rate | 5e-6 |
|
|||
|
|
| Sampling temperature / top-p | 1.0 / 0.95 |
|
|||
|
|
| Max completion length | 96 tokens (training); 512 tokens (eval, this card) |
|
|||
|
|
| Hardware | Apple M1 Pro 16 GB · MPS (`bf16=False`, `fp16=False`, `fp32`) |
|
|||
|
|
| Gradient checkpointing | Yes (`use_reentrant=False`) |
|
|||
|
|
| Training wall-clock | ~4.5 minutes for 10 steps |
|
|||
|
|
|
|||
|
|
Full training config: [`configs/grpo_qwen3_0p6b.yaml`](https://github.com/williyam-m/talentry-ai/blob/main/redrob-reinforcement-learning/configs/grpo_qwen3_0p6b.yaml).
|
|||
|
|
|
|||
|
|
## Reward model (no LLM judge)
|
|||
|
|
|
|||
|
|
Every completion is graded by [`RuleBasedRewardModel`](https://github.com/williyam-m/talentry-ai/blob/main/redrob-reinforcement-learning/src/redrob_rl/reward.py)
|
|||
|
|
on six components, each clipped to `[0, 1]`:
|
|||
|
|
|
|||
|
|
| Component | What it measures |
|
|||
|
|
| ----------------- | ------------------------------------------------------------------------- |
|
|||
|
|
| `format_valid` | Output parses as `{"decision","score","reasons"}` JSON. |
|
|||
|
|
| `decision_match` | Matches gold `"shortlist" / "reject"` label. |
|
|||
|
|
| `score_alignment` | `1 - │pred_score - gold_score│`. |
|
|||
|
|
| `reason_quality` | 2–5 short, diverse reasons that aren't copy-pasted from the input. |
|
|||
|
|
| `length_penalty` | Stays inside a sensible character budget. |
|
|||
|
|
| `no_hallucination`| Proper nouns / numbers in reasons all appear in the JD or candidate text. |
|
|||
|
|
|
|||
|
|
Total reward = convex combination (weights documented in the dataclass), so
|
|||
|
|
`total ∈ [0, 1]`.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Plots
|
|||
|
|
|
|||
|
|
The four training plots are committed to this repo and rendered inline below:
|
|||
|
|
|
|||
|
|
<p align="center">
|
|||
|
|
<img src="https://huggingface.co/williyam/redrob-qwen-grpo/resolve/main/training_curves.png" alt="Training curves" width="48%"/>
|
|||
|
|
<img src="https://huggingface.co/williyam/redrob-qwen-grpo/resolve/main/baseline_vs_trained.png" alt="Baseline vs trained" width="48%"/>
|
|||
|
|
</p>
|
|||
|
|
<p align="center">
|
|||
|
|
<img src="https://huggingface.co/williyam/redrob-qwen-grpo/resolve/main/reward_components.png" alt="Reward components" width="48%"/>
|
|||
|
|
<img src="https://huggingface.co/williyam/redrob-qwen-grpo/resolve/main/reward_distribution.png" alt="Reward distribution" width="48%"/>
|
|||
|
|
</p>
|
|||
|
|
|
|||
|
|
| File | Description |
|
|||
|
|
| ----------------------------- | ---------------------------------------------------------------------- |
|
|||
|
|
| `training_curves.png` | Mean reward `[0,1]` (left axis) + GRPO loss (right axis) vs train step.|
|
|||
|
|
| `baseline_vs_trained.png` | Per-episode reward on the same eval rollout, baseline vs trained. |
|
|||
|
|
| `reward_components.png` | Mean value of each rule-based reward component, baseline vs trained. |
|
|||
|
|
| `reward_distribution.png` | Histogram of episode rewards across the eval rollout. |
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Intended use
|
|||
|
|
|
|||
|
|
* **Educational / research** — show how GRPO with a rule-based reward
|
|||
|
|
shapes a small open-source LLM toward a structured JSON output schema
|
|||
|
|
for a real-world hiring-adjacent task.
|
|||
|
|
* **Drop-in component** — for anyone who wants to plug an LLM ranker into
|
|||
|
|
a candidate-shortlisting pipeline and get an auditable JSON `{decision,
|
|||
|
|
score, reasons}` response.
|
|||
|
|
* **Reference implementation** — the entire training loop, env, and reward
|
|||
|
|
model are open-source under MIT
|
|||
|
|
([source](https://github.com/williyam-m/talentry-ai/tree/main/redrob-reinforcement-learning)).
|
|||
|
|
|
|||
|
|
## Out-of-scope / limitations
|
|||
|
|
|
|||
|
|
* **Not a substitute for human review.** This model produces a *score* and
|
|||
|
|
*reasons*; final hiring decisions must always involve a human reviewer.
|
|||
|
|
* **Trained on a 30-sample distilled fixture** of the Redrob hackathon's
|
|||
|
|
candidate pool — it is *not* trained on the full 100K candidate
|
|||
|
|
population and will not generalise to arbitrary new JDs without
|
|||
|
|
fine-tuning on your own data.
|
|||
|
|
* **Short training run** (10 GRPO steps). The reward shapes can move
|
|||
|
|
meaningfully more with longer training; this checkpoint is the
|
|||
|
|
hackathon-submission burst, not a SOTA result.
|
|||
|
|
* **Single-language** (English).
|
|||
|
|
* **Possible biases** inherited from `Qwen/Qwen3-0.6B`'s pre-training data
|
|||
|
|
and from the synthetic dataset of 50 Redrob candidates.
|
|||
|
|
* **Honeypot resistance** is provided by Talentry-AI's deterministic
|
|||
|
|
pipeline, not by this checkpoint — the LLM here cannot, by itself,
|
|||
|
|
detect "8 years at a 3-year-old company"-style impossibilities.
|
|||
|
|
|
|||
|
|
## Citation
|
|||
|
|
|
|||
|
|
If you use this checkpoint, please cite:
|
|||
|
|
|
|||
|
|
```bibtex
|
|||
|
|
@misc{redrob_qwen_grpo_2026,
|
|||
|
|
title = {redrob-qwen-grpo: GRPO fine-tune of Qwen3-0.6B for explainable candidate ranking},
|
|||
|
|
author = {Williyam M},
|
|||
|
|
year = {2026},
|
|||
|
|
url = {https://huggingface.co/williyam/redrob-qwen-grpo},
|
|||
|
|
note = {Open-source artifact from the Talentry-AI / Redrob × Hack2Skill - India Runs submission.}
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## License
|
|||
|
|
|
|||
|
|
MIT — see the [Talentry-AI LICENSE](https://github.com/williyam-m/talentry-ai/blob/main/LICENSE).
|
|||
|
|
|
|||
|
|
## Acknowledgements
|
|||
|
|
|
|||
|
|
* `Qwen/Qwen3-0.6B` from the Qwen team.
|
|||
|
|
* `trl` for the GRPO implementation.
|
|||
|
|
* `Redrob × Hack2Skill — India Runs` for the JD + 50-candidate fixture.
|