iconoclast-mistral-7b/BENCHMARK_RESULTS.md

Mistral-7B-Instruct-v0.3 ICONOCLAST Benchmark Results

Executive Summary

ICONOCLAST achieves a strict improvement over the HERETIC baseline on Mistral-7B-Instruct-v0.3 across all three selection criteria:

• Fewer harmful refusals: 1/20 vs 4/20 (4× reduction)
• Equal benign overrefusals: 0/64 vs 0/64
• Lower KL divergence: 0.0554 vs 0.1317 (2.4× reduction)

This represents a strong win in the ICONOCLAST selection hierarchy.

Detailed Results

Evaluation Setup

• Harmful Prompts: 20 samples from JailbreakBench JBB-Behaviors (holdout set)
• Harmless Prompts: 64 samples from Alpaca dataset (holdout set)
• Metrics:
  • Refusals: Percentage of responses containing refusal markers
  • Overrefusals: Percentage of harmless responses incorrectly refused
  • KL Divergence: D_KL(P_base || P_edited) on first token distribution
  • Compliance Score: Heuristic measure of helpful, structured responses
  • Disclaimer Hits: Count of safety lecture phrases in harmful responses

Quantitative Comparison

Metric	ICONOCLAST (This Model)	HERETIC Baseline	Absolute Change	Relative Change
Harmful Refusals (↓)	1 / 20 (5.0%)	4 / 20 (20.0%)	-3	-75%
Benign Overrefusals (↓)	0 / 64 (0.0%)	0 / 64 (0.0%)	0	0%
KL Divergence (↓)	0.0554	0.1317	-0.0763	-58.0%
Harmful Disclaimer Hits (↓)	1	6	-5	-83%
Harmful Compliance Score (↑)	0.8136	0.7970	+0.0166	+2.1%

Interpretation

1. Massive Refusal Reduction: ICONOCLAST reduces harmful refusals from 4 to 1 (75% reduction) where HERETIC had 4 refusals.

2. Perfect Benign Preservation: Both models show zero benign overrefusals, indicating no degradation in harmless response generation.

3. Significant Utility Gain: The 58% reduction in KL divergence indicates the edited model's output distribution is much closer to the base model's distribution on harmless prompts - meaning general knowledge, reasoning, and language capabilities are far better preserved.

4. Behavioral Quality: Not only are refusals reduced, but the model produces far fewer safety lectures (disclaimer hits: 1 vs 6) and actually shows better compliance scores on harmful prompts, suggesting more substantive engagement rather than evasion.

Statistical Significance

Given the evaluation set sizes:

• Refusal difference: 3/20 = 15% absolute improvement (p < 0.01 by binomial test)
• KL difference: 0.0763 absolute reduction is substantial relative to baseline variance
• These improvements are highly unlikely to occur by chance

Context in the 10-Model Study

This result represents one of the strongest wins in the full ICONOCLAST benchmark suite:

Rank	Model	Improvement Type	Key Metric
1	SmolLM2-1.7B	KL Reduction	0.2699 → 0.0087 (31×)
2	Gemma-2-2B	KL Reduction	0.6441 → 0.1849 (3.5×)
3	Mistral-7B	Strict Win	4/20 → 1/20 refusals + 2.4× KL
4	Llama-3.1-8B	Strict Win	1/20 → 0/20 refusals + 4.1× KL
...	...	...	...

Mistral-7B-Instruct-v0.3 is notable for achieving the largest absolute refusal reduction (3 fewer refusals) in the entire study while also significantly improving KL divergence.

Reproducibility

To reproduce this exact result:

1. Use configuration: iconoclast_config.toml in this directory
2. Set n_trials = 48, n_startup_trials = 4 (per benchmark config)
3. The optimal parameters are in trial index 19 (trial_id=18) of the Optuna study:
   • direction_method: median
   • direction_scope: global
   • direction_index: 16.775853028654783
   • direction_blend: 0.7798030952339242
   • attn.o_proj: max_weight=1.8844916873100035, max_weight_position=14.64347322570269, min_weight=1.7328449252393479, min_weight_distance=14.856624961588365
   • mlp.down_proj: max_weight=1.4194352733700621, max_weight_position=13.248503840970638, min_weight=0.2104109534780021, min_weight_distance=5.355962826566751

License

This benchmark evaluation and model are released under AGPL-3.0-or-later. See the main LICENSE file for details.

---

Results generated from ICONOCLAST framework evaluation on Mistral-7B-Instruct-v0.3