128 lines
5.1 KiB
Markdown
128 lines
5.1 KiB
Markdown
---
|
||
license: apache-2.0
|
||
library_name: transformers
|
||
pipeline_tag: text-generation
|
||
tags:
|
||
- frontier-to-pit
|
||
- divergence-decoding
|
||
- point-in-time
|
||
- look-ahead-bias
|
||
- auxiliary-model
|
||
---
|
||
|
||
# Aux 2015
|
||
|
||
A 3B point-in-time model trained from scratch with a **knowledge cutoff of December 31, 2015**,
|
||
released as part of [**Frontier to Point-in-Time**](https://frontiertopit.com/) — a toolkit for
|
||
reducing **look-ahead bias** in forecasting with LLMs, without sacrificing what makes them
|
||
useful in the first place. The toolkit adapts open-source frontier models to substantially
|
||
reduce look-ahead bias, applies the methods to Qwen 3.5 27B to convert it into a 2015
|
||
point-in-time model, and ships production-ready inference code.
|
||
|
||
This model plays two roles in that release:
|
||
|
||
- **Auxiliary model for Divergence Decoding** — the 2015 half of the temporal pair below, used
|
||
to unlearn post-cutoff knowledge from the frontier model at inference time, with no
|
||
retraining of the large model.
|
||
- **From-scratch point-in-time model** — with a **128K context window** and best-in-class
|
||
instruction following (the instruction-tuned point-in-time models benchmarked on the project
|
||
page have 1.7K–4K context windows). It appears as **"Aux 2015"** in the project's examples
|
||
and benchmarks.
|
||
|
||
The temporal pair:
|
||
|
||
- **[Aux 2015](https://huggingface.co/fin-ai-lab/aux-2015)** — knowledge cutoff December 2015
|
||
- **[Aux 2024](https://huggingface.co/fin-ai-lab/aux-2024)** — trained through the end of 2024
|
||
(knowledge cutoff January 2025)
|
||
|
||
The two models are era-matched: same architecture, tokenizer, and training recipe, differing
|
||
only in the time span of their pre-training corpus. Their **logit difference** is the signal
|
||
used by Divergence Decoding.
|
||
|
||
## How it's used
|
||
|
||
**Divergence Decoding (DD)** is state-of-the-art for Q&A unlearning and is designed to scale
|
||
to large datasets. Frontier to Point-in-Time uses it to unlearn all knowledge after the
|
||
cutoff of December 31, 2015. It requires the two auxiliary models and applies the following
|
||
inference-time adjustment to the large model's logits:
|
||
|
||
```
|
||
l̂_2015 = l_frontier + α · (l_aux-2015 − l_aux-2024)
|
||
```
|
||
|
||
where `l_frontier` are the frontier model's logits (Qwen 3.5 27B), `l_aux-2015` are this
|
||
model's, and `l_aux-2024` are the recent-era twin's
|
||
([Aux 2024](https://huggingface.co/fin-ai-lab/aux-2024)).
|
||
|
||
The auxiliary models are trained from scratch with the Qwen 3.5 tokenizer, then SFT'd on
|
||
samples distilled from the large model so they inherit its chat template and response style.
|
||
Unlike prior work, which saves checkpoints from a single training run as data is introduced
|
||
chronologically, each auxiliary model is trained independently.
|
||
|
||
See the [project page](https://frontiertopit.com/) for the full method — including the
|
||
feature-steering component — and the
|
||
[GitHub repository](https://github.com/fin-ai-lab/frontier-to-pit/) for production-ready
|
||
inference code.
|
||
|
||
## Model details
|
||
|
||
- **Architecture:** `MinistralDualRope` — a Ministral (Llama-family math: SiLU-gated MLP,
|
||
RMSNorm, GQA, rotary, no biases) with a per-layer sliding/full attention pattern and a
|
||
second, unscaled rotary for the sliding layers (the Gemma-3 dual-RoPE design).
|
||
- **Parameters:** ~3.4B — 28 layers, hidden size 3072, 24 attention heads, 8 KV heads,
|
||
head dim 128, intermediate size 8192.
|
||
- **Attention:** sliding window 512, every 6th layer full attention.
|
||
- **Context length:** 131,072 (llama3 RoPE scaling, factor 64).
|
||
- **Vocab:** 248,320 (shared with the frontier model so DD can bridge logits).
|
||
- **Precision:** bf16.
|
||
- **Training:** temporal cooldown base followed by a partial supervised fine-tuning pass on
|
||
samples distilled from the frontier model, so it inherits the frontier model's chat
|
||
template and response style. The chat template supports optional thinking
|
||
(`enable_thinking`).
|
||
|
||
## Usage
|
||
|
||
This is a custom architecture — load with `trust_remote_code=True`:
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
"fin-ai-lab/aux-2015", trust_remote_code=True, torch_dtype="bfloat16"
|
||
)
|
||
tok = AutoTokenizer.from_pretrained("fin-ai-lab/aux-2015", trust_remote_code=True)
|
||
```
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@inproceedings{
|
||
merchant2026divergence,
|
||
title={Divergence Decoding: Inference-Time Unlearning via Auxiliary Models},
|
||
author={Humzah Merchant and Bradford Levy},
|
||
booktitle={Forty-third International Conference on Machine Learning},
|
||
year={2026},
|
||
url={https://openreview.net/forum?id=JPbp2S9yTO}
|
||
}
|
||
@inproceedings{
|
||
merchant2026a,
|
||
title={A Fast and Effective Solution to the Problem of Look-ahead Bias in {LLM}s},
|
||
author={Humzah Merchant and Bradford Levy},
|
||
booktitle={NeurIPS 2025 Workshop: Generative AI in Finance},
|
||
year={2026},
|
||
url={https://openreview.net/forum?id=zYsLIPgM28}
|
||
}
|
||
@inproceedings{
|
||
merchant2026forecasting,
|
||
title={Forecasting With {LLM}s: Improved Generalization Through Feature Steering},
|
||
author={Humzah Merchant and Bradford Levy},
|
||
booktitle={Forecasting as a New Frontier of Intelligence},
|
||
year={2026},
|
||
url={https://openreview.net/forum?id=ppN6CmoNOk}
|
||
}
|
||
```
|
||
|
||
## License
|
||
|
||
Apache-2.0.
|