aux-2015/README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - frontier-to-pit
  - divergence-decoding
  - point-in-time
  - look-ahead-bias
  - auxiliary-model
---

# Aux 2015

A 3B point-in-time model trained from scratch with a **knowledge cutoff of December 31, 2015**,
released as part of [**Frontier to Point-in-Time**](https://frontiertopit.com/) — a toolkit for
reducing **look-ahead bias** in forecasting with LLMs, without sacrificing what makes them
useful in the first place. The toolkit adapts open-source frontier models to substantially
reduce look-ahead bias, applies the methods to Qwen 3.5 27B to convert it into a 2015
point-in-time model, and ships production-ready inference code.

This model plays two roles in that release:

- **Auxiliary model for Divergence Decoding** — the 2015 half of the temporal pair below, used
  to unlearn post-cutoff knowledge from the frontier model at inference time, with no
  retraining of the large model.
- **From-scratch point-in-time model** — with a **128K context window** and best-in-class
  instruction following (the instruction-tuned point-in-time models benchmarked on the project
  page have 1.7K–4K context windows). It appears as **"Aux 2015"** in the project's examples
  and benchmarks.

The temporal pair:

- **[Aux 2015](https://huggingface.co/fin-ai-lab/aux-2015)** — knowledge cutoff December 2015
- **[Aux 2024](https://huggingface.co/fin-ai-lab/aux-2024)** — trained through the end of 2024
  (knowledge cutoff January 2025)

The two models are era-matched: same architecture, tokenizer, and training recipe, differing
only in the time span of their pre-training corpus. Their **logit difference** is the signal
used by Divergence Decoding.

## How it's used

**Divergence Decoding (DD)** is state-of-the-art for Q&A unlearning and is designed to scale
to large datasets. Frontier to Point-in-Time uses it to unlearn all knowledge after the
cutoff of December 31, 2015. It requires the two auxiliary models and applies the following
inference-time adjustment to the large model's logits:

```
l̂_2015 = l_frontier + α · (l_aux-2015 − l_aux-2024)
```

where `l_frontier` are the frontier model's logits (Qwen 3.5 27B), `l_aux-2015` are this
model's, and `l_aux-2024` are the recent-era twin's
([Aux 2024](https://huggingface.co/fin-ai-lab/aux-2024)).

The auxiliary models are trained from scratch with the Qwen 3.5 tokenizer, then SFT'd on
samples distilled from the large model so they inherit its chat template and response style.
Unlike prior work, which saves checkpoints from a single training run as data is introduced
chronologically, each auxiliary model is trained independently.

See the [project page](https://frontiertopit.com/) for the full method — including the
feature-steering component — and the
[GitHub repository](https://github.com/fin-ai-lab/frontier-to-pit/) for production-ready
inference code.

## Model details

- **Architecture:** `MinistralDualRope` — a Ministral (Llama-family math: SiLU-gated MLP,
  RMSNorm, GQA, rotary, no biases) with a per-layer sliding/full attention pattern and a
  second, unscaled rotary for the sliding layers (the Gemma-3 dual-RoPE design).
- **Parameters:** ~3.4B — 28 layers, hidden size 3072, 24 attention heads, 8 KV heads,
  head dim 128, intermediate size 8192.
- **Attention:** sliding window 512, every 6th layer full attention.
- **Context length:** 131,072 (llama3 RoPE scaling, factor 64).
- **Vocab:** 248,320 (shared with the frontier model so DD can bridge logits).
- **Precision:** bf16.
- **Training:** temporal cooldown base followed by a partial supervised fine-tuning pass on
  samples distilled from the frontier model, so it inherits the frontier model's chat
  template and response style. The chat template supports optional thinking
  (`enable_thinking`).

## Usage

This is a custom architecture — load with `trust_remote_code=True`:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "fin-ai-lab/aux-2015", trust_remote_code=True, torch_dtype="bfloat16"
)
tok = AutoTokenizer.from_pretrained("fin-ai-lab/aux-2015", trust_remote_code=True)
```

## Citation

```bibtex
@inproceedings{
    merchant2026divergence,
    title={Divergence Decoding: Inference-Time Unlearning via Auxiliary Models},
    author={Humzah Merchant and Bradford Levy},
    booktitle={Forty-third International Conference on Machine Learning},
    year={2026},
    url={https://openreview.net/forum?id=JPbp2S9yTO}
}
@inproceedings{
    merchant2026a,
    title={A Fast and Effective Solution to the Problem of Look-ahead Bias in {LLM}s},
    author={Humzah Merchant and Bradford Levy},
    booktitle={NeurIPS 2025 Workshop: Generative AI in Finance},
    year={2026},
    url={https://openreview.net/forum?id=zYsLIPgM28}
}
@inproceedings{
    merchant2026forecasting,
    title={Forecasting With {LLM}s: Improved Generalization Through Feature Steering},
    author={Humzah Merchant and Bradford Levy},
    booktitle={Forecasting as a New Frontier of Intelligence},
    year={2026},
    url={https://openreview.net/forum?id=ppN6CmoNOk}
}
```

## License

Apache-2.0.