初始化项目，由ModelHub XC社区提供模型

Model: birgermoell/oellm-9b-128k-theta32m-prelude Source: Original Platform
2026-07-05 23:16:19 +08:00
commit 4833f18281
9 changed files with 1424 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,123 @@
+---
+license: apache-2.0
+language: [en, sv, de, fr, es, it, nl, pl, pt, cs, fi, da, el, bg, hr, hu, ro, sk, sl, et, lt, lv, ga, mt, eu, gl, is, nb, nn, sr, uk, ca, mk, sq, oc, lb, bs]
+tags: [long-context, rope, abf, niah, qwen3, openeurollm, base-model, 128k]
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+# OELLM 9B — 128K context (ABF, θ=32M) — prelude (1T-token base)
+
+128K long-context extension of the OpenEuroLLM **prelude** base — Qwen3 dense 9B
+(`qwen3_9b_hf_baby` at **iter_0124800 ≈ 1 trillion tokens**, the strongest available base
+checkpoint). Same validated θ=32M recipe as
+[`oellm-9b-128k-theta32m-v3`](https://huggingface.co/birgermoell/oellm-9b-128k-theta32m-v3)
+(which used the earlier ~0.6T checkpoint), now on the upgraded 1T base.
+**Base model — not instruction-tuned.** Multilingual (37 European languages).
+
+---
+
+## Key result: depth-0 is a RoPE-θ problem, not data
+Standard ABF (uniform θ-scaling) leaves the model unable to retrieve from the **far start** of a
+long window ("depth-0"). We showed this is **not** fixable with more long-range data (two
+length-biased datasets gave depth-0 ≈ 0%), but **is** fixed by scaling RoPE θ to the target length
+— the high RoPE dimensions are out-of-distribution at long range (cf. LongRoPE2, arXiv:2502.20082).
+
+**Critical θ ≈ doubles per context-length octave:**
+| context | θ |
+|---|---|
+| 64K | 8M |
+| 128K | 16M (90%) → **32M (100%)** |
+| 256K | ~32–64M |
+θ ablation @128K depth-0: 2M/5M=0%, 8M=0% (fixes ≤64K), 16M=90%, **32M=100%**. This model uses **θ=32M**.
+
+## Evaluation
+**Method:** base-LM **forced-choice NIAH** (4-way, log-likelihood of answer tokens; no
+instruction-following needed). Distractor values are placed *in-context* (adversarial), so the
+chance floor is below 25%. Depth 0.0 = needle at the far start (max query distance); 1.0 = most
+recent. Script: [`scripts/eval_base_lm_niah.py`](https://github.com/BirgerMoell/openeuro-longctx-datamix/blob/main/scripts/eval_base_lm_niah.py).
+
+**Results @128K** — 15 languages, 900 trials. **Overall 96% (864/900).**
+
+By depth (needle position; 0.0 = far start):
+| depth | 0.0 | 0.25 | 0.5 | 0.75 | 1.0 |
+|---|---|---|---|---|---|
+| accuracy | **97%** | 96% | **88%** | 96% | 100% |
+
+By language: cs, da, en, es, pt, uk = 100%; el, fr, hu = 96%; it, nl, pl = 93%; de, fi, sv = 90%.
+depth-0 is 12/12 in every language except fi (8/12).
+
+**Read:** the θ=32M fix is confirmed — **far-position (depth-0) retrieval is solid (97%)**. The
+model is strong overall (96%) with a **mild "lost-in-the-middle"** (depth 0.5 = 88%). This is a
+slightly softer profile than the 0.6T-base sibling **v3** (which scored ~100% across depths on 12
+languages) — an honest, characterized difference, concentrated in the middle of the window rather
+than the far end. Candidate follow-ups: a larger 128K token budget, or investigating whether the
+mid-depth softness is a property of the longer-pretrained 1T base. (4K/16K/64K remain ~100%.)
+
+## How to reproduce
+
+### 1. Base model & architecture
+OpenEuroLLM `prelude` = `qwen3_9b_hf_baby` **iter_0124800** (~1T tokens). Qwen3 dense:
+36 layers, hidden 4096, FFN 12288, 32 attention heads / 8 KV groups (GQA), kv-channels 128,
+**qk-layernorm**, RMSNorm, SwiGLU, untied embeddings, vocab 262144, `openeurollm/tokenizer-256k`,
+native context 4K (rope θ=100000).
+
+### 2. Conversion HF → Megatron
+The prelude base ships in HF safetensors; converted to Megatron-core (torch_dist) via
+**Megatron-Bridge** (`convert_checkpoints.py import`) before extension.
+
+### 3. Extension recipe (staged native ABF)
+Continued-pretraining, raising `--seq-length` and `--rotary-base` each stage, `--finetune` from
+the previous stage:
+```
+prelude(4K, θ=100k) → 16K(θ=500k) → 32K(θ=1M) → 64K(θ=2M) → 128K(θ=32M)
+```
+Token budgets: 16K/32K ≈ 1B each, 64K = 3B, 128K = 2B.
+
+### 4. Training config (Megatron-LM, LUMI / 16× MI250X)
+```
+--rotary-base {500k|1M|2M|32M per stage} --seq-length {16384..131072} --use-flash-attn
+--tensor-model-parallel-size 8 --pipeline-model-parallel-size 1
+--context-parallel-size {1|1|2|8} --sequence-parallel --use-distributed-optimizer
+--micro-batch-size 1 --global-batch-size 64 --bf16
+--lr 1e-5 --min-lr 1e-6 --lr-decay-style cosine --weight-decay 0.1 --clip-grad 1.0
+--adam-beta1 0.9 --adam-beta2 0.95 --recompute-activations --recompute-granularity selective
+--qk-layernorm --normalization RMSNorm --swiglu --group-query-attention --num-query-groups 8
+--ckpt-format torch_dist --finetune --no-save-optim --no-save-rng --save-interval 100
+```
+Container: ROCm 6.4.4 / PyTorch 2.9 / TE 2.4 / FA 2.8. Throughput ≈ 500 tok/s/GPU at 128K.
+Megatron→HF export emits Qwen3Config + q_norm/k_norm.
+
+### 5. Data
+**Jouni Luoma's length-biased multilingual long-context mix** (token-proportional blend over 152
+sources): finepdfs (all langs + edu), dclm, hplt3 (38 langs), multisynth (synthetic multilingual),
+nemotron, megamath, starcoder, pes2o, arxiv, wiki — tiered short/medium/long (long_threshold=64K).
+The same data as v3. Note: **depth-0 is insensitive to the data mix — θ is the lever.** Data hosted
+on stable `/scratch` (the burst-buffer `/flash` proved unreliable mid-run).
+
+### 6. Reproduce the eval
+```
+python scripts/eval_base_lm_niah.py --model <this-model> \
+  --context-lengths 4096 16384 65536 131072 --depths 0.0 0.25 0.5 0.75 1.0 \
+  --languages en de fr es nl pl sv fi cs it pt el hu uk da --trials 6
+```
+
+## Usage
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+m = AutoModelForCausalLM.from_pretrained("openeurollm/oellm-9b-128k-theta32m-prelude",
+                                         torch_dtype=torch.bfloat16, device_map="auto")
+tok = AutoTokenizer.from_pretrained("openeurollm/oellm-9b-128k-theta32m-prelude")
+# Base completion model. Keep rope_theta=32M / max_position=131072 for 128K.
+```
+
+## Caveats
+- **Base model** (no instruction/chat tuning) — use as a completion model.
+- Evaluated with **single-needle** forced-choice NIAH; broader multi-task RULER not yet run.
+- Keep `rope_theta=32000000`, `max_position_embeddings=131072` for 128K.
+
+## Links / citation
+Code & write-ups: https://github.com/BirgerMoell/openeuro-longctx-datamix
+(`docs/depth0_diagnosis_theta_sweep.md` for the θ analysis). Sibling (0.6T base):
+`oellm-9b-128k-theta32m-v3`. RoPE-θ diagnosis informed by LongRoPE2 (arXiv:2502.20082).
--- a/added_tokens.json
+++ b/added_tokens.json
@@ -0,0 +1,3 @@
+{
+  "<pad>": 262144
+}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,68 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 131072,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 36,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 32000000,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.3",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 263168
+}
--- a/pytorch_model.bin
+++ b/pytorch_model.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6f07efb20b11e8b5a32c7167ee6310ae0be10287f8521444b5c850145227eb1
+size 18204021587
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,142 @@
+{
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>",
+    "<start_of_image>",
+    "<end_of_image>",
+    "<image_soft_token>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<tool_call>",
+    "</tool_call>",
+    "<unused_0>",
+    "<unused_1>",
+    "<unused_2>",
+    "<unused_3>",
+    "<unused_4>",
+    "<unused_5>",
+    "<unused_6>",
+    "<unused_7>",
+    "<unused_8>",
+    "<unused_9>",
+    "<unused_10>",
+    "<unused_11>",
+    "<unused_12>",
+    "<unused_13>",
+    "<unused_14>",
+    "<unused_15>",
+    "<unused_16>",
+    "<unused_17>",
+    "<unused_18>",
+    "<unused_19>",
+    "<unused_20>",
+    "<unused_21>",
+    "<unused_22>",
+    "<unused_23>",
+    "<unused_24>",
+    "<unused_25>",
+    "<unused_26>",
+    "<unused_27>",
+    "<unused_28>",
+    "<unused_29>",
+    "<unused_30>",
+    "<unused_31>",
+    "<unused_32>",
+    "<unused_33>",
+    "<unused_34>",
+    "<unused_35>",
+    "<unused_36>",
+    "<unused_37>",
+    "<unused_38>",
+    "<unused_39>",
+    "<unused_40>",
+    "<unused_41>",
+    "<unused_42>",
+    "<unused_43>",
+    "<unused_44>",
+    "<unused_45>",
+    "<unused_46>",
+    "<unused_47>",
+    "<unused_48>",
+    "<unused_49>",
+    "<unused_50>",
+    "<unused_51>",
+    "<unused_52>",
+    "<unused_53>",
+    "<unused_54>",
+    "<unused_55>",
+    "<unused_56>",
+    "<unused_57>",
+    "<unused_58>",
+    "<unused_59>",
+    "<unused_60>",
+    "<unused_61>",
+    "<unused_62>",
+    "<unused_63>",
+    "<unused_64>",
+    "<unused_65>",
+    "<unused_66>",
+    "<unused_67>",
+    "<unused_68>",
+    "<unused_69>",
+    "<unused_70>",
+    "<unused_71>",
+    "<unused_72>",
+    "<unused_73>",
+    "<unused_74>",
+    "<unused_75>",
+    "<unused_76>",
+    "<unused_77>",
+    "<unused_78>",
+    "<unused_79>",
+    "<unused_80>",
+    "<unused_81>",
+    "<unused_82>",
+    "<unused_83>",
+    "<unused_84>",
+    "<unused_85>",
+    "<unused_86>",
+    "<unused_87>",
+    "<unused_88>",
+    "<unused_89>",
+    "<unused_90>",
+    "<unused_91>",
+    "<unused_92>",
+    "<unused_93>",
+    "<unused_94>",
+    "<unused_95>",
+    "<unused_96>",
+    "<unused_97>",
+    "<unused_98>",
+    "<unused_99>"
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce296af846f114e330e8e62887df4c6abafd4fd682434fb0ced1a3abfa2e04c1
+size 37807304
--- a/tokenizer.model
+++ b/tokenizer.model
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac214401227404105817141c21c6693e3b17c878465a42765d2c345310e003ce
+size 4876269
--- a/tokenizer_config.json
+++ b/tokenizer_config.json