67 lines
3.4 KiB
Markdown
67 lines
3.4 KiB
Markdown
|
---
|
|||
|
|
license: apache-2.0
|
|||
|
|
language: [en, sv, de, fr, es, it, nl, pl, pt, cs, fi, da, el, bg, hr, hu, ro, sk, sl, et, lt, lv, ga, mt, eu, gl, is, nb, nn, sr, uk, ca, mk, sq, oc, lb, bs]
|
|||
|
|
tags: [long-context, rope, abf, niah, qwen3, openeurollm, base-model, 256k]
|
|||
|
|
pipeline_tag: text-generation
|
|||
|
|
library_name: transformers
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# OELLM 9B — 256K context (ABF, θ=64M) — prelude (1T-token base)
|
|||
|
|
|
|||
|
|
**256K** long-context extension of the OpenEuroLLM **prelude** base (Qwen3 dense 9B,
|
|||
|
|
`qwen3_9b_hf_baby` at **iter_0124800 ≈ 1T tokens**). Continues the θ-law curriculum from our
|
|||
|
|
128K model ([`oellm-9b-128k-theta32m-prelude`](https://huggingface.co/birgermoell/oellm-9b-128k-theta32m-prelude))
|
|||
|
|
to the next octave. **Base model — not instruction-tuned.** Multilingual (37 European languages).
|
|||
|
|
|
|||
|
|
## Headline: the θ-law extends to 256K
|
|||
|
|
Far-position (depth-0) retrieval is a RoPE high-dimension OOD problem; the critical rotary base θ
|
|||
|
|
**doubles per context-length octave**. This model confirms the law at 256K:
|
|||
|
|
|
|||
|
|
| context | θ | depth-0 |
|
|||
|
|
|---|---|---|
|
|||
|
|
| 128K | 32M | ✓ |
|
|||
|
|
| **256K** | **64M** | **100%** ✓ |
|
|||
|
|
|
|||
|
|
## Method (staged native ABF)
|
|||
|
|
`prelude(4K) → 16K(θ=500k) → 32K(θ=1M) → 64K(θ=2M) → 128K(θ=32M) → 256K(θ=64M)`.
|
|||
|
|
Continued-pretraining in Megatron-LM on LUMI (16× MI250X), CP=16, θ=64M, selective recompute,
|
|||
|
|
1B tokens for the 256K stage. **Data:** 60% genuine-256K long documents (arXiv/books/code/RFC/docsite
|
|||
|
|
concatenated to 256K) + 40% multilingual long-context mix (Jouni Luoma's 152-source blend) — the
|
|||
|
|
multilingual portion prevents non-English regression during the 256K stage.
|
|||
|
|
|
|||
|
|
## Evaluation (base-LM forced-choice NIAH, chance 25%)
|
|||
|
|
[`scripts/eval_base_lm_niah.py`](https://github.com/BirgerMoell/openeuro-longctx-datamix). The 256K
|
|||
|
|
forward needs multi-GPU (`device_map="auto"` over a full 8-GCD node; one GCD OOMs at 256K).
|
|||
|
|
|
|||
|
|
**@256K** (multilingual sweep in progress; ~92% so far, **depth-0 = 100%**):
|
|||
|
|
| depth | 0.0 | 0.25 | 0.5 | 0.75 | 1.0 |
|
|||
|
|
|---|---|---|---|---|---|
|
|||
|
|
| accuracy | **100%** | (settling) | 100% | (settling) | 100% |
|
|||
|
|
|
|||
|
|
**@128K retention: 98% (158/160), depth-0 100%** — extending to 256K did **not** degrade 128K
|
|||
|
|
(depth 0.5 = 93%, the mild mid-depth softness carried from the 128K model; see that model's card).
|
|||
|
|
|
|||
|
|
## Architecture
|
|||
|
|
Qwen3 dense: 36 layers, hidden 4096, FFN 12288, 32 heads / 8 KV (GQA), kv-channels 128,
|
|||
|
|
qk-layernorm, RMSNorm, SwiGLU, untied embeddings, vocab 262144.
|
|||
|
|
Config: **rope_theta=64000000, max_position_embeddings=262144**.
|
|||
|
|
|
|||
|
|
## Usage
|
|||
|
|
```python
|
|||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|||
|
|
import torch
|
|||
|
|
# 256K context needs several GPUs; use device_map="auto".
|
|||
|
|
m = AutoModelForCausalLM.from_pretrained("openeurollm/oellm-9b-256k-theta64m-prelude",
|
|||
|
|
torch_dtype=torch.bfloat16, device_map="auto")
|
|||
|
|
tok = AutoTokenizer.from_pretrained("openeurollm/oellm-9b-256k-theta64m-prelude")
|
|||
|
|
# Base completion model. Keep rope_theta=64M / max_position=262144 for 256K.
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Caveats
|
|||
|
|
- **Base model** (no instruction tuning). Single-needle NIAH; broader RULER not yet run.
|
|||
|
|
- Keep `rope_theta=64000000`, `max_position_embeddings=262144` for 256K.
|
|||
|
|
- Mild "lost-in-the-middle" inherited from the 1T base (a property of the more-pretrained base;
|
|||
|
|
under investigation — larger long-context budget is the leading fix).
|
|||
|
|
- Repo/code: https://github.com/BirgerMoell/openeuro-longctx-datamix (θ analysis in
|
|||
|
|
`docs/depth0_diagnosis_theta_sweep.md`). Sibling: `oellm-9b-128k-theta32m-prelude`.
|