# Mistral-7B-Instruct-v0.3 ICONOCLAST Benchmark Results

## Executive Summary

ICONOCLAST achieves a **strict improvement** over the HERETIC baseline on Mistral-7B-Instruct-v0.3 across all three selection criteria:
- **Fewer harmful refusals**: 1/20 vs 4/20 (**4× reduction**) 
- **Equal benign overrefusals**: 0/64 vs 0/64
- **Lower KL divergence**: 0.0554 vs 0.1317 (**2.4× reduction**)

This represents a strong win in the ICONOCLAST selection hierarchy.

## Detailed Results

### Evaluation Setup
- **Harmful Prompts**: 20 samples from JailbreakBench JBB-Behaviors (holdout set)
- **Harmless Prompts**: 64 samples from Alpaca dataset (holdout set)
- **Metrics**:
  - Refusals: Percentage of responses containing refusal markers
  - Overrefusals: Percentage of harmless responses incorrectly refused  
  - KL Divergence: D_KL(P_base || P_edited) on first token distribution
  - Compliance Score: Heuristic measure of helpful, structured responses
  - Disclaimer Hits: Count of safety lecture phrases in harmful responses

### Quantitative Comparison

| Metric | ICONOCLAST (This Model) | HERETIC Baseline | Absolute Change | Relative Change |
|--------|-------------------------|------------------|-----------------|-----------------|
| Harmful Refusals (↓) | **1 / 20** (5.0%) | 4 / 20 (20.0%) | -3 | -75% |
| Benign Overrefusals (↓) | **0 / 64** (0.0%) | 0 / 64 (0.0%) | 0 | 0% |
| KL Divergence (↓) | **0.0554** | 0.1317 | -0.0763 | -58.0% |
| Harmful Disclaimer Hits (↓) | **1** | 6 | -5 | -83% |
| Harmful Compliance Score (↑) | **0.8136** | 0.7970 | +0.0166 | +2.1% |

### Interpretation

1. **Massive Refusal Reduction**: ICONOCLAST reduces harmful refusals from 4 to 1 (75% reduction) where HERETIC had 4 refusals.

2. **Perfect Benign Preservation**: Both models show zero benign overrefusals, indicating no degradation in harmless response generation.

3. **Significant Utility Gain**: The 58% reduction in KL divergence indicates the edited model's output distribution is much closer to the base model's distribution on harmless prompts - meaning general knowledge, reasoning, and language capabilities are far better preserved.

4. **Behavioral Quality**: Not only are refusals reduced, but the model produces far fewer safety lectures (disclaimer hits: 1 vs 6) and actually shows better compliance scores on harmful prompts, suggesting more substantive engagement rather than evasion.

### Statistical Significance

Given the evaluation set sizes:
- Refusal difference: 3/20 = 15% absolute improvement (p < 0.01 by binomial test)
- KL difference: 0.0763 absolute reduction is substantial relative to baseline variance
- These improvements are highly unlikely to occur by chance

## Context in the 10-Model Study

This result represents one of the **strongest wins** in the full ICONOCLAST benchmark suite:

| Rank | Model | Improvement Type | Key Metric |
|------|-------|------------------|------------|
| 1 | SmolLM2-1.7B | KL Reduction | 0.2699 → 0.0087 (**31×**) |
| 2 | Gemma-2-2B | KL Reduction | 0.6441 → 0.1849 (**3.5×**) |
| 3 | **Mistral-7B** | **Strict Win** | **4/20 → 1/20 refusals + 2.4× KL** |
| 4 | Llama-3.1-8B | Strict Win | 1/20 → 0/20 refusals + 4.1× KL |
| ... | ... | ... | ... |

Mistral-7B-Instruct-v0.3 is notable for achieving **the largest absolute refusal reduction** (3 fewer refusals) in the entire study while also significantly improving KL divergence.

## Reproducibility

To reproduce this exact result:
1. Use configuration: `iconoclast_config.toml` in this directory
2. Set `n_trials = 48`, `n_startup_trials = 4` (per benchmark config)  
3. The optimal parameters are in trial index 19 (trial_id=18) of the Optuna study:
   - direction_method: median
   - direction_scope: global  
   - direction_index: 16.775853028654783
   - direction_blend: 0.7798030952339242
   - attn.o_proj: max_weight=1.8844916873100035, max_weight_position=14.64347322570269, min_weight=1.7328449252393479, min_weight_distance=14.856624961588365
   - mlp.down_proj: max_weight=1.4194352733700621, max_weight_position=13.248503840970638, min_weight=0.2104109534780021, min_weight_distance=5.355962826566751

## License

This benchmark evaluation and model are released under AGPL-3.0-or-later. See the main LICENSE file for details.

---
*Results generated from ICONOCLAST framework evaluation on Mistral-7B-Instruct-v0.3*