初始化项目，由ModelHub XC社区提供模型

Model: fin-ai-lab/aux-2024 Source: Original Platform
2026-08-05 22:33:17 +08:00
commit 25e4aaaf69
9 changed files with 788 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,123 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - frontier-to-pit
+  - divergence-decoding
+  - point-in-time
+  - look-ahead-bias
+  - auxiliary-model
+---
+
+# Aux 2024
+
+A 3B model trained from scratch **through the end of 2024** (knowledge cutoff January 2025),
+released as part of [**Frontier to Point-in-Time**](https://frontiertopit.com/) — a toolkit for
+reducing **look-ahead bias** in forecasting with LLMs, without sacrificing what makes them
+useful in the first place. The toolkit adapts open-source frontier models to substantially
+reduce look-ahead bias, applies the methods to Qwen 3.5 27B to convert it into a 2015
+point-in-time model, and ships production-ready inference code.
+
+This model is the recent-era half of the temporal pair used by **Divergence Decoding**: its
+logits stand in for the frontier model's post-2015 knowledge, which the method cancels at
+inference time, with no retraining of the large model. Like its twin, it is a from-scratch
+model with a **128K context window** and best-in-class instruction following; the 2015 twin
+appears as **"Aux 2015"** in the project's examples and benchmarks.
+
+The temporal pair:
+
+- **[Aux 2015](https://huggingface.co/fin-ai-lab/aux-2015)** — knowledge cutoff December 2015
+- **[Aux 2024](https://huggingface.co/fin-ai-lab/aux-2024)** — trained through the end of 2024
+  (knowledge cutoff January 2025)
+
+The two models are era-matched: same architecture, tokenizer, and training recipe, differing
+only in the time span of their pre-training corpus. Their **logit difference** is the signal
+used by Divergence Decoding.
+
+## How it's used
+
+**Divergence Decoding (DD)** is state-of-the-art for Q&A unlearning and is designed to scale
+to large datasets. Frontier to Point-in-Time uses it to unlearn all knowledge after the
+cutoff of December 31, 2015. It requires the two auxiliary models and applies the following
+inference-time adjustment to the large model's logits:
+
+```
+l̂_2015 = l_frontier + α · (l_aux-2015 − l_aux-2024)
+```
+
+where `l_frontier` are the frontier model's logits (Qwen 3.5 27B), `l_aux-2024` are this
+model's, and `l_aux-2015` are the 2015 twin's
+([Aux 2015](https://huggingface.co/fin-ai-lab/aux-2015)).
+
+The auxiliary models are trained from scratch with the Qwen 3.5 tokenizer, then SFT'd on
+samples distilled from the large model so they inherit its chat template and response style.
+Unlike prior work, which saves checkpoints from a single training run as data is introduced
+chronologically, each auxiliary model is trained independently.
+
+See the [project page](https://frontiertopit.com/) for the full method — including the
+feature-steering component — and the
+[GitHub repository](https://github.com/fin-ai-lab/frontier-to-pit/) for production-ready
+inference code.
+
+## Model details
+
+- **Architecture:** `MinistralDualRope` — a Ministral (Llama-family math: SiLU-gated MLP,
+  RMSNorm, GQA, rotary, no biases) with a per-layer sliding/full attention pattern and a
+  second, unscaled rotary for the sliding layers (the Gemma-3 dual-RoPE design).
+- **Parameters:** ~3.4B — 28 layers, hidden size 3072, 24 attention heads, 8 KV heads,
+  head dim 128, intermediate size 8192.
+- **Attention:** sliding window 512, every 6th layer full attention.
+- **Context length:** 131,072 (llama3 RoPE scaling, factor 64).
+- **Vocab:** 248,320 (shared with the frontier model so DD can bridge logits).
+- **Precision:** bf16.
+- **Training:** temporal cooldown base followed by a partial supervised fine-tuning pass on
+  samples distilled from the frontier model, so it inherits the frontier model's chat
+  template and response style. The chat template supports optional thinking
+  (`enable_thinking`).
+
+## Usage
+
+This is a custom architecture — load with `trust_remote_code=True`:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+    "fin-ai-lab/aux-2024", trust_remote_code=True, torch_dtype="bfloat16"
+)
+tok = AutoTokenizer.from_pretrained("fin-ai-lab/aux-2024", trust_remote_code=True)
+```
+
+## Citation
+
+```bibtex
+@inproceedings{
+    merchant2026divergence,
+    title={Divergence Decoding: Inference-Time Unlearning via Auxiliary Models},
+    author={Humzah Merchant and Bradford Levy},
+    booktitle={Forty-third International Conference on Machine Learning},
+    year={2026},
+    url={https://openreview.net/forum?id=JPbp2S9yTO}
+}
+@inproceedings{
+    merchant2026a,
+    title={A Fast and Effective Solution to the Problem of Look-ahead Bias in {LLM}s},
+    author={Humzah Merchant and Bradford Levy},
+    booktitle={NeurIPS 2025 Workshop: Generative AI in Finance},
+    year={2026},
+    url={https://openreview.net/forum?id=zYsLIPgM28}
+}
+@inproceedings{
+    merchant2026forecasting,
+    title={Forecasting With {LLM}s: Improved Generalization Through Feature Steering},
+    author={Humzah Merchant and Bradford Levy},
+    booktitle={Forecasting as a New Frontier of Intelligence},
+    year={2026},
+    url={https://openreview.net/forum?id=ppN6CmoNOk}
+}
+```
+
+## License
+
+Apache-2.0.