123 lines
7.0 KiB
Markdown
123 lines
7.0 KiB
Markdown
---
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
base_model:
|
||
- Qwen/Qwen3-4B-Thinking-2507
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- grpo
|
||
- rlhf
|
||
- reasoning
|
||
- cognitive-reappraisal
|
||
- emotion-regulation
|
||
library_name: transformers
|
||
---
|
||
# Reappraisal-4B-GRPO-RMv2
|
||
|
||
Cognitive reappraisal is reinterpreting a negative situation to reduce the emotion it causes.
|
||
[Li, Herderich, Nair & Goldenberg (2025), *Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios*](https://doi.org/10.31234/osf.io/fzvd8)
|
||
collected reappraisals for 6 negative interpersonal scenarios from 611 people and from GPT-4-0314,
|
||
and had human raters score each for effectiveness. This repo is part of a project that uses those
|
||
ratings to train reward models of reappraisal effectiveness and to RL-finetune an open model that
|
||
writes reappraisals as effective as possible — with GPT-4-0314, the AI reappraiser in the original
|
||
study, as the bar to beat.
|
||
|
||
This model is the strongest policy from that project: a GRPO finetune of
|
||
[Qwen/Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507). Given a scenario,
|
||
it produces an alternative interpretation addressed to the person, in two sentences or fewer, in a
|
||
single forward pass (no best-of-N at inference). The reward signal is the three-seed ensemble
|
||
reward model [idealab-cs2/reappraisal-reward-model-v2](https://huggingface.co/idealab-cs2/reappraisal-reward-model-v2).
|
||
|
||
## Model Details
|
||
|
||
- **Base Model:** Qwen/Qwen3-4B-Thinking-2507 (a reasoning model; non-reasoning bases lose here)
|
||
- **Training Method:** GRPO (Group Relative Policy Optimization), full fine-tuning, DeepSpeed ZeRO-2, 4× H100
|
||
- **Reward Model:** [idealab-cs2/reappraisal-reward-model-v2](https://huggingface.co/idealab-cs2/reappraisal-reward-model-v2) (3-seed ensemble, reward = `mean − 0.5·std`)
|
||
- **Training Data:** scenarios from Li et al. (2025) — 6 negative interpersonal vignettes
|
||
- **Trained By:** ruggsea
|
||
|
||
## Evaluation
|
||
|
||
Reappraisals were compared pairwise against GPT-4-0314's reappraisals for the same scenarios, judged
|
||
by Llama-3.1-70B-Instruct scoring both A/B orderings. That judge was selected by calibration against
|
||
the human preference data (it agrees with the human-preferred reappraisal 93% of the time on this
|
||
task; larger judges such as Llama-3.1-405B-FP8 and GLM-5.2 agreed less and showed position bias).
|
||
Results are pooled over three independent evals of this checkpoint (fresh generations + an
|
||
independent both-orders judge each), and marginal wins are confirmed at n ≥ 240 with the lower CI
|
||
bound above the bar.
|
||
|
||
| model | win-rate vs GPT-4-0314 | n | 95% CI lower (iid, optimistic) |
|
||
|---|---|---|---|
|
||
| **Reappraisal-4B-GRPO-RMv2 (this model)** | **0.806** | 720 | **0.776** |
|
||
| Reappraisal-4B-GRPO (v1, BT reward) | 0.581 | 480 | 0.537 |
|
||
| Qwen3-4B-Thinking (base, untrained) | ~0.46 | — | — |
|
||
| DeepSeek-R1-671B (single-pass reference) | 0.771 | 240 | 0.718 |
|
||
|
||
**Read the CI column as a screening tool, not a precise bound.** The 720 contests cluster on only
|
||
**6 vignettes** (n_effective ≈ 6), so the iid Wilson CIs above are optimistic, and the earlier claim
|
||
that "the CI-low 0.776 sits above the R1 reference 0.771" is **retracted** — comparing an iid CI-low to
|
||
a reference *point* is not a valid test. The recipe is seed-robust (a different-seed run scored 0.854),
|
||
and the win over GPT-4-0314 on these 6 vignettes is real and clears the 0.608 bar with margin; but this
|
||
is a **specialist trained on 6 scenarios**, and the strongest policy in this project is the
|
||
[committee model](https://huggingface.co/idealab-cs2/reappraisal-4b-grpo-committee) (0.872 6-vignette,
|
||
vignette-cluster CI-low 0.781), which is the checkpoint to read for the honest R1 comparison.
|
||
|
||
**Out-of-distribution: this does NOT generalize past parity, and no 4B in this project beats R1
|
||
off-distribution.** The policy is GRPO-trained on the same 6 vignettes it is evaluated on. The earlier
|
||
"0.784 off-vignette, well above 0.5" claim is **retracted**: it used a weaker gemma opponent and iid
|
||
statistics. Against a strong open 72B on fresh cross-family scenarios, our best policies are at
|
||
**parity-or-below** (cluster CIs spanning 0.5), and against **DeepSeek-R1-671B off-distribution our 4B
|
||
loses** — single-pass ~0.39–0.45 and best-of-N only tied on the selection pool before reverting to
|
||
~0.375 on a fresh held-out pool (winner's curse). Off-distribution generalization tracks base capacity,
|
||
not our training.
|
||
|
||
The improvement over the v1 policy comes almost entirely from the reward model: the same GRPO recipe
|
||
against the earlier reward models scored 0.581 (Bradley-Terry) and 0.698 (multidimensional), against
|
||
0.806 with the v2 ensemble.
|
||
|
||
## Prompting
|
||
|
||
The model was trained with this system prompt:
|
||
```
|
||
You are helping someone reduce a negative emotion they feel in a short interpersonal scenario by
|
||
offering an alternative interpretation of the situation (a 'reappraisal' / 'rethinking'). Direct
|
||
your response at the person in the scenario in second person. Limit your response to two sentences
|
||
maximum. Do not list emotions; output only the reappraisal text.
|
||
```
|
||
The user turn is `SCENARIO:\n{scenario}\n\nWrite a reappraisal of this scenario in two sentences
|
||
maximum, addressing the person in second person.` Because the base is a thinking model, generations
|
||
contain a reasoning trace in `<think>...</think>` followed by the reappraisal; only the text after
|
||
`</think>` is the answer.
|
||
|
||
## Training hyperparameters
|
||
|
||
- learning_rate: 8e-6
|
||
- beta (KL coefficient): 0.04 (kept strictly > 0; β=0 collapses and reward-hacks)
|
||
- num_generations: 12
|
||
- max_steps: 500
|
||
- precision: bf16, DeepSpeed ZeRO-2 across 4 GPUs
|
||
|
||
The reward is the ensemble score of the answer with a format gate penalizing outputs that are empty
|
||
or longer than two sentences (so the policy cannot inflate reward through verbosity). Training was
|
||
healthy: reward rising, KL bounded, entropy alive, no reward-std collapse (wandb `vzo3wpv6`).
|
||
|
||
## Data separation
|
||
|
||
The 6 vignettes' GPT-4-0314 reappraisals and the frontier-opponent answers are held out and used only
|
||
for evaluation; training uses the vignette prompts (the task) and the human-rating-based reward model.
|
||
Any pooled or synthetic training text is n-gram-checked against the held-out eval answers before use.
|
||
|
||
## Intended use & limitations
|
||
|
||
Research artifact for the study of RLHF, reward modeling, and computational emotion regulation. It
|
||
writes short cognitive reappraisals of negative interpersonal situations. It is **not** a clinical or
|
||
mental-health tool and must not be used as one. Quality is validated only on the narrow reappraisal
|
||
task and the evaluation above.
|
||
|
||
## References
|
||
|
||
- Reward model used to train this policy: [idealab-cs2/reappraisal-reward-model-v2](https://huggingface.co/idealab-cs2/reappraisal-reward-model-v2)
|
||
- Earlier policy (v1, Bradley-Terry reward): [idealab-cs2/reappraisal-4b-grpo](https://huggingface.co/idealab-cs2/reappraisal-4b-grpo)
|
||
- Li, J. Z., Herderich, A., Nair, P., & Goldenberg, A. (2025). *Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios.* PsyArXiv. https://doi.org/10.31234/osf.io/fzvd8
|