70 KiB
70 KiB
[MMLU] start Sun Aug 9 22:52:08 UTC 2026
[*] MicroLLM2 MMLU — model: /home/zeus/microllm2/microllm2-checkpoints/final_merged shots=5 limit=20
[*] GPT2-XL 1.5B 1024ctx vocab=50259 (ChatML) — MMLU via direct eval (no harness needed)
[*] lm-eval not installed — using lightweight direct MMLU eval (same logic, no harness)
[*] Install harness for official numbers: pip install lm-eval==0.4.4
[*] Loading tokenizer + model /home/zeus/microllm2/microllm2-checkpoints/final_merged ...
[+] Loaded on cuda:0 dtype=torch.bfloat16 — starting MMLU
MMLU subjects: 0%| | 0/57 [00:00<?, ?it/s]
============================================================
[>] abstract_algebra (shots=5)
MMLU subjects: 2%|▏ | 1/57 [00:03<03:40, 3.94s/it][=] abstract_algebra: 6/20 = 30.0% (running avg 30.0%)
============================================================
[>] anatomy (shots=5)
Downloading data: 100%|██████████| 20.1k/20.1k [00:00<00:00, 60.8kB/s]
Downloading data: 100%|██████████| 5.28k/5.28k [00:00<00:00, 30.6kB/s]
Downloading data: 100%|██████████| 3.50k/3.50k [00:00<00:00, 18.7kB/s]
Generating test split: 100%|██████████| 135/135 [00:00<00:00, 20566.29 examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 3828.67 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1496.15 examples/s]
MMLU subjects: 4%|▎ | 2/57 [00:07<03:32, 3.87s/it][=] anatomy: 5/20 = 25.0% (running avg 27.5%)
============================================================
[>] astronomy (shots=5)
Downloading data: 100%|██████████| 28.3k/28.3k [00:00<00:00, 153kB/s]
Downloading data: 100%|██████████| 6.05k/6.05k [00:00<00:00, 33.4kB/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 28.2kB/s]
Generating test split: 100%|██████████| 152/152 [00:00<00:00, 30542.02 examples/s]
Generating validation split: 100%|██████████| 16/16 [00:00<00:00, 4092.75 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1423.63 examples/s]
MMLU subjects: 5%|▌ | 3/57 [00:11<03:27, 3.84s/it][=] astronomy: 7/20 = 35.0% (running avg 30.0%)
============================================================
[>] business_ethics (shots=5)
Downloading data: 100%|██████████| 21.6k/21.6k [00:00<00:00, 131kB/s]
Downloading data: 100%|██████████| 5.09k/5.09k [00:00<00:00, 33.8kB/s]
Downloading data: 100%|██████████| 4.96k/4.96k [00:00<00:00, 27.9kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 23243.58 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3018.27 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1394.75 examples/s]
MMLU subjects: 7%|▋ | 4/57 [00:15<03:28, 3.94s/it][=] business_ethics: 6/20 = 30.0% (running avg 30.0%)
============================================================
[>] clinical_knowledge (shots=5)
Downloading data: 100%|██████████| 40.5k/40.5k [00:00<00:00, 199kB/s]
Downloading data: 100%|██████████| 7.48k/7.48k [00:00<00:00, 40.9kB/s]
Downloading data: 100%|██████████| 3.67k/3.67k [00:00<00:00, 20.4kB/s]
Generating test split: 100%|██████████| 265/265 [00:00<00:00, 52433.75 examples/s]
Generating validation split: 100%|██████████| 29/29 [00:00<00:00, 7840.33 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1485.34 examples/s]
MMLU subjects: 9%|▉ | 5/57 [00:19<03:24, 3.92s/it][=] clinical_knowledge: 9/20 = 45.0% (running avg 33.0%)
============================================================
[>] college_biology (shots=5)
Downloading data: 100%|██████████| 31.8k/31.8k [00:00<00:00, 166kB/s]
Downloading data: 100%|██████████| 6.90k/6.90k [00:00<00:00, 50.7kB/s]
Downloading data: 100%|██████████| 4.27k/4.27k [00:00<00:00, 23.0kB/s]
Generating test split: 100%|██████████| 144/144 [00:00<00:00, 29006.81 examples/s]
Generating validation split: 100%|██████████| 16/16 [00:00<00:00, 4205.34 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1440.85 examples/s]
MMLU subjects: 11%|█ | 6/57 [00:23<03:14, 3.81s/it][=] college_biology: 9/20 = 45.0% (running avg 35.0%)
============================================================
[>] college_chemistry (shots=5)
Downloading data: 100%|██████████| 17.9k/17.9k [00:00<00:00, 96.4kB/s]
Downloading data: 100%|██████████| 4.87k/4.87k [00:00<00:00, 26.9kB/s]
Downloading data: 100%|██████████| 4.04k/4.04k [00:00<00:00, 10.2kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22971.16 examples/s]
Generating validation split: 100%|██████████| 8/8 [00:00<00:00, 2089.45 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1161.02 examples/s]
MMLU subjects: 12%|█▏ | 7/57 [00:27<03:14, 3.88s/it][=] college_chemistry: 3/20 = 15.0% (running avg 32.1%)
============================================================
[>] college_computer_science (shots=5)
Downloading data: 100%|██████████| 28.1k/28.1k [00:00<00:00, 163kB/s]
Downloading data: 100%|██████████| 6.25k/6.25k [00:00<00:00, 40.2kB/s]
Downloading data: 100%|██████████| 6.81k/6.81k [00:00<00:00, 46.6kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22972.42 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3038.75 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1146.55 examples/s]
MMLU subjects: 14%|█▍ | 8/57 [00:31<03:10, 3.88s/it][=] college_computer_science: 9/20 = 45.0% (running avg 33.8%)
============================================================
[>] college_mathematics (shots=5)
Downloading data: 100%|██████████| 16.6k/16.6k [00:00<00:00, 85.2kB/s]
Downloading data: 100%|██████████| 5.00k/5.00k [00:00<00:00, 26.8kB/s]
Downloading data: 100%|██████████| 5.16k/5.16k [00:00<00:00, 36.1kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22919.69 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3062.55 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1144.17 examples/s]
MMLU subjects: 16%|█▌ | 9/57 [00:34<03:01, 3.79s/it][=] college_mathematics: 7/20 = 35.0% (running avg 33.9%)
============================================================
[>] college_medicine (shots=5)
Downloading data: 100%|██████████| 42.5k/42.5k [00:00<00:00, 220kB/s]
Downloading data: 100%|██████████| 8.99k/8.99k [00:00<00:00, 53.9kB/s]
Downloading data: 100%|██████████| 4.84k/4.84k [00:00<00:00, 34.1kB/s]
Generating test split: 100%|██████████| 173/173 [00:00<00:00, 34671.95 examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 4797.23 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1361.43 examples/s]
MMLU subjects: 18%|█▊ | 10/57 [00:38<02:55, 3.74s/it][=] college_medicine: 6/20 = 30.0% (running avg 33.5%)
============================================================
[>] college_physics (shots=5)
Downloading data: 100%|██████████| 18.6k/18.6k [00:00<00:00, 102kB/s]
Downloading data: 100%|██████████| 6.39k/6.39k [00:00<00:00, 39.5kB/s]
Downloading data: 100%|██████████| 4.51k/4.51k [00:00<00:00, 23.9kB/s]
Generating test split: 100%|██████████| 102/102 [00:00<00:00, 17934.14 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 2286.18 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1061.58 examples/s]
MMLU subjects: 19%|█▉ | 11/57 [00:42<02:55, 3.82s/it][=] college_physics: 3/20 = 15.0% (running avg 31.8%)
============================================================
[>] computer_security (shots=5)
Downloading data: 100%|██████████| 19.1k/19.1k [00:00<00:00, 106kB/s]
Downloading data: 100%|██████████| 6.67k/6.67k [00:00<00:00, 46.6kB/s]
Downloading data: 100%|██████████| 4.33k/4.33k [00:00<00:00, 23.7kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22049.75 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 2974.88 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1465.82 examples/s]
MMLU subjects: 21%|██ | 12/57 [00:45<02:48, 3.75s/it][=] computer_security: 6/20 = 30.0% (running avg 31.7%)
============================================================
[>] conceptual_physics (shots=5)
Downloading data: 100%|██████████| 25.0k/25.0k [00:00<00:00, 152kB/s]
Downloading data: 100%|██████████| 5.98k/5.98k [00:00<00:00, 32.1kB/s]
Downloading data: 100%|██████████| 3.96k/3.96k [00:00<00:00, 22.3kB/s]
Generating test split: 100%|██████████| 235/235 [00:00<00:00, 51400.78 examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 6864.22 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1387.28 examples/s]
MMLU subjects: 23%|██▎ | 13/57 [00:49<02:40, 3.66s/it][=] conceptual_physics: 1/20 = 5.0% (running avg 29.6%)
============================================================
[>] econometrics (shots=5)
Downloading data: 100%|██████████| 24.5k/24.5k [00:00<00:00, 140kB/s]
Downloading data: 100%|██████████| 7.02k/7.02k [00:00<00:00, 26.7kB/s]
Downloading data: 100%|██████████| 4.54k/4.54k [00:00<00:00, 28.9kB/s]
Generating test split: 100%|██████████| 114/114 [00:00<00:00, 22894.45 examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 3162.13 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1402.87 examples/s]
MMLU subjects: 25%|██▍ | 14/57 [00:53<02:39, 3.70s/it][=] econometrics: 6/20 = 30.0% (running avg 29.6%)
============================================================
[>] electrical_engineering (shots=5)
Downloading data: 100%|██████████| 17.6k/17.6k [00:00<00:00, 107kB/s]
Downloading data: 100%|██████████| 5.08k/5.08k [00:00<00:00, 34.7kB/s]
Downloading data: 100%|██████████| 4.08k/4.08k [00:00<00:00, 22.7kB/s]
Generating test split: 100%|██████████| 145/145 [00:00<00:00, 33690.12 examples/s]
Generating validation split: 100%|██████████| 16/16 [00:00<00:00, 4531.63 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1402.68 examples/s]
MMLU subjects: 26%|██▋ | 15/57 [00:56<02:30, 3.59s/it][=] electrical_engineering: 4/20 = 20.0% (running avg 29.0%)
============================================================
[>] elementary_mathematics (shots=5)
Downloading data: 100%|██████████| 41.1k/41.1k [00:00<00:00, 222kB/s]
Downloading data: 100%|██████████| 9.38k/9.38k [00:00<00:00, 52.3kB/s]
Downloading data: 100%|██████████| 4.55k/4.55k [00:00<00:00, 21.6kB/s]
Generating test split: 100%|██████████| 378/378 [00:00<00:00, 78826.97 examples/s]
Generating validation split: 100%|██████████| 41/41 [00:00<00:00, 11570.88 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1528.09 examples/s]
MMLU subjects: 28%|██▊ | 16/57 [00:59<02:26, 3.56s/it][=] elementary_mathematics: 6/20 = 30.0% (running avg 29.1%)
============================================================
[>] formal_logic (shots=5)
Downloading data: 100%|██████████| 21.5k/21.5k [00:00<00:00, 110kB/s]
Downloading data: 100%|██████████| 6.56k/6.56k [00:00<00:00, 38.8kB/s]
Downloading data: 100%|██████████| 4.81k/4.81k [00:00<00:00, 22.3kB/s]
Generating test split: 100%|██████████| 126/126 [00:00<00:00, 28698.47 examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 3785.72 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1486.81 examples/s]
MMLU subjects: 30%|██▉ | 17/57 [01:03<02:23, 3.58s/it][=] formal_logic: 2/20 = 10.0% (running avg 27.9%)
============================================================
[>] global_facts (shots=5)
Downloading data: 100%|██████████| 11.5k/11.5k [00:00<00:00, 58.8kB/s]
Downloading data: 100%|██████████| 4.19k/4.19k [00:00<00:00, 22.5kB/s]
Downloading data: 100%|██████████| 3.58k/3.58k [00:00<00:00, 24.1kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22527.01 examples/s]
Generating validation split: 100%|██████████| 10/10 [00:00<00:00, 2616.37 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1165.93 examples/s]
MMLU subjects: 32%|███▏ | 18/57 [01:06<02:15, 3.47s/it][=] global_facts: 7/20 = 35.0% (running avg 28.3%)
============================================================
[>] high_school_biology (shots=5)
Downloading data: 100%|██████████| 62.7k/62.7k [00:00<00:00, 353kB/s]
Downloading data: 100%|██████████| 10.6k/10.6k [00:00<00:00, 58.2kB/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 29.2kB/s]
Generating test split: 100%|██████████| 310/310 [00:00<00:00, 60104.20 examples/s]
Generating validation split: 100%|██████████| 32/32 [00:00<00:00, 7002.18 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1401.37 examples/s]
MMLU subjects: 33%|███▎ | 19/57 [01:10<02:10, 3.42s/it][=] high_school_biology: 9/20 = 45.0% (running avg 29.2%)
============================================================
[>] high_school_chemistry (shots=5)
Downloading data: 100%|██████████| 33.3k/33.3k [00:00<00:00, 185kB/s]
Downloading data: 100%|██████████| 8.31k/8.31k [00:00<00:00, 59.2kB/s]
Downloading data: 100%|██████████| 4.16k/4.16k [00:00<00:00, 20.7kB/s]
Generating test split: 100%|██████████| 203/203 [00:00<00:00, 43367.99 examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 5422.50 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1380.34 examples/s]
MMLU subjects: 35%|███▌ | 20/57 [01:13<02:05, 3.40s/it][=] high_school_chemistry: 7/20 = 35.0% (running avg 29.5%)
============================================================
[>] high_school_computer_science (shots=5)
Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 173kB/s]
Downloading data: 100%|██████████| 5.28k/5.28k [00:00<00:00, 36.0kB/s]
Downloading data: 100%|██████████| 6.54k/6.54k [00:00<00:00, 36.0kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22050.91 examples/s]
Generating validation split: 100%|██████████| 9/9 [00:00<00:00, 2392.04 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1416.80 examples/s]
MMLU subjects: 37%|███▋ | 21/57 [01:16<02:03, 3.43s/it][=] high_school_computer_science: 7/20 = 35.0% (running avg 29.8%)
============================================================
[>] high_school_european_history (shots=5)
Downloading data: 100%|██████████| 142k/142k [00:00<00:00, 678kB/s]
Downloading data: 100%|██████████| 31.6k/31.6k [00:00<00:00, 201kB/s]
Downloading data: 100%|██████████| 22.2k/22.2k [00:00<00:00, 112kB/s]
Generating test split: 100%|██████████| 165/165 [00:00<00:00, 31161.25 examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 4798.67 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1341.75 examples/s]
MMLU subjects: 39%|███▊ | 22/57 [01:20<01:57, 3.36s/it][=] high_school_european_history: 4/20 = 20.0% (running avg 29.3%)
============================================================
[>] high_school_geography (shots=5)
Downloading data: 100%|██████████| 28.2k/28.2k [00:00<00:00, 145kB/s]
Downloading data: 100%|██████████| 6.16k/6.16k [00:00<00:00, 40.6kB/s]
Downloading data: 100%|██████████| 3.93k/3.93k [00:00<00:00, 28.4kB/s]
Generating test split: 100%|██████████| 198/198 [00:00<00:00, 40891.83 examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 5636.82 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1341.75 examples/s]
MMLU subjects: 40%|████ | 23/57 [01:23<01:53, 3.32s/it][=] high_school_geography: 5/20 = 25.0% (running avg 29.1%)
============================================================
[>] high_school_government_and_politics (shots=5)
Downloading data: 100%|██████████| 40.2k/40.2k [00:00<00:00, 276kB/s]
Downloading data: 100%|██████████| 8.27k/8.27k [00:00<00:00, 53.9kB/s]
Downloading data: 100%|██████████| 4.47k/4.47k [00:00<00:00, 26.1kB/s]
Generating test split: 100%|██████████| 193/193 [00:00<00:00, 40098.11 examples/s]
Generating validation split: 100%|██████████| 21/21 [00:00<00:00, 5715.79 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1422.28 examples/s]
MMLU subjects: 42%|████▏ | 24/57 [01:26<01:46, 3.23s/it][=] high_school_government_and_politics: 4/20 = 20.0% (running avg 28.7%)
============================================================
[>] high_school_macroeconomics (shots=5)
Downloading data: 100%|██████████| 54.8k/54.8k [00:00<00:00, 362kB/s]
Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 50.2kB/s]
Downloading data: 100%|██████████| 4.04k/4.04k [00:00<00:00, 29.7kB/s]
Generating test split: 100%|██████████| 390/390 [00:00<00:00, 70428.77 examples/s]
Generating validation split: 100%|██████████| 43/43 [00:00<00:00, 10985.20 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1389.95 examples/s]
MMLU subjects: 44%|████▍ | 25/57 [01:29<01:43, 3.24s/it][=] high_school_macroeconomics: 0/20 = 0.0% (running avg 27.6%)
============================================================
[>] high_school_mathematics (shots=5)
Downloading data: 100%|██████████| 33.7k/33.7k [00:00<00:00, 218kB/s]
Downloading data: 100%|██████████| 6.99k/6.99k [00:00<00:00, 28.3kB/s]
Downloading data: 100%|██████████| 4.50k/4.50k [00:00<00:00, 21.2kB/s]
Generating test split: 100%|██████████| 270/270 [00:00<00:00, 59139.49 examples/s]
Generating validation split: 100%|██████████| 29/29 [00:00<00:00, 8262.11 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1471.58 examples/s]
MMLU subjects: 46%|████▌ | 26/57 [01:33<01:42, 3.29s/it][=] high_school_mathematics: 4/20 = 20.0% (running avg 27.3%)
============================================================
[>] high_school_microeconomics (shots=5)
Downloading data: 100%|██████████| 38.8k/38.8k [00:00<00:00, 266kB/s]
Downloading data: 100%|██████████| 7.22k/7.22k [00:00<00:00, 41.9kB/s]
Downloading data: 100%|██████████| 3.83k/3.83k [00:00<00:00, 25.0kB/s]
Generating test split: 100%|██████████| 238/238 [00:00<00:00, 49842.44 examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 7166.92 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1446.41 examples/s]
MMLU subjects: 47%|████▋ | 27/57 [01:36<01:37, 3.25s/it][=] high_school_microeconomics: 7/20 = 35.0% (running avg 27.6%)
============================================================
[>] high_school_physics (shots=5)
Downloading data: 100%|██████████| 33.0k/33.0k [00:00<00:00, 196kB/s]
Downloading data: 100%|██████████| 7.96k/7.96k [00:00<00:00, 42.2kB/s]
MMLU subjects: 49%|████▉ | 28/57 [01:38<01:27, 3.02s/it]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 51%|█████ | 29/57 [01:38<01:00, 2.15s/it]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 54%|█████▍ | 31/57 [01:39<00:31, 1.20s/it]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 56%|█████▌ | 32/57 [01:39<00:23, 1.08it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 58%|█████▊ | 33/57 [01:39<00:17, 1.40it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 60%|█████▉ | 34/57 [01:39<00:12, 1.83it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 61%|██████▏ | 35/57 [01:39<00:09, 2.36it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 63%|██████▎ | 36/57 [01:39<00:06, 3.02it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 65%|██████▍ | 37/57 [01:39<00:05, 3.78it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 67%|██████▋ | 38/57 [01:39<00:04, 4.61it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 68%|██████▊ | 39/57 [01:39<00:03, 5.47it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 70%|███████ | 40/57 [01:39<00:02, 6.29it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 74%|███████▎ | 42/57 [01:40<00:01, 7.54it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 75%|███████▌ | 43/57 [01:40<00:01, 7.82it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 77%|███████▋ | 44/57 [01:40<00:01, 8.27it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 81%|████████ | 46/57 [01:40<00:01, 8.87it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 82%|████████▏ | 47/57 [01:40<00:01, 9.04it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 84%|████████▍ | 48/57 [01:40<00:00, 9.18it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 86%|████████▌ | 49/57 [01:40<00:00, 9.33it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 88%|████████▊ | 50/57 [01:40<00:00, 9.46it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 89%|████████▉ | 51/57 [01:41<00:00, 9.49it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 91%|█████████ | 52/57 [01:41<00:00, 9.62it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 93%|█████████▎| 53/57 [01:41<00:00, 9.51it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 95%|█████████▍| 54/57 [01:41<00:00, 9.56it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 96%|█████████▋| 55/57 [01:41<00:00, 9.52it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 98%|█████████▊| 56/57 [01:41<00:00, 9.56it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 100%|██████████| 57/57 [01:41<00:00, 1.78s/it]
[!] Skip high_school_physics: Could not load MMLU subject high_school_physics
============================================================
[>] high_school_psychology (shots=5)
[!] Skip high_school_psychology: Could not load MMLU subject high_school_psychology
============================================================
[>] high_school_statistics (shots=5)
[!] Skip high_school_statistics: Could not load MMLU subject high_school_statistics
============================================================
[>] high_school_us_history (shots=5)
[!] Skip high_school_us_history: Could not load MMLU subject high_school_us_history
============================================================
[>] high_school_world_history (shots=5)
[!] Skip high_school_world_history: Could not load MMLU subject high_school_world_history
============================================================
[>] human_aging (shots=5)
[!] Skip human_aging: Could not load MMLU subject human_aging
============================================================
[>] human_sexuality (shots=5)
[!] Skip human_sexuality: Could not load MMLU subject human_sexuality
============================================================
[>] international_law (shots=5)
[!] Skip international_law: Could not load MMLU subject international_law
============================================================
[>] jurisprudence (shots=5)
[!] Skip jurisprudence: Could not load MMLU subject jurisprudence
============================================================
[>] logical_fallacies (shots=5)
[!] Skip logical_fallacies: Could not load MMLU subject logical_fallacies
============================================================
[>] machine_learning (shots=5)
[!] Skip machine_learning: Could not load MMLU subject machine_learning
============================================================
[>] management (shots=5)
[!] Skip management: Could not load MMLU subject management
============================================================
[>] marketing (shots=5)
[!] Skip marketing: Could not load MMLU subject marketing
============================================================
[>] medical_genetics (shots=5)
[!] Skip medical_genetics: Could not load MMLU subject medical_genetics
============================================================
[>] miscellaneous (shots=5)
[!] Skip miscellaneous: Could not load MMLU subject miscellaneous
============================================================
[>] moral_disputes (shots=5)
[!] Skip moral_disputes: Could not load MMLU subject moral_disputes
============================================================
[>] moral_scenarios (shots=5)
[!] Skip moral_scenarios: Could not load MMLU subject moral_scenarios
============================================================
[>] nutrition (shots=5)
[!] Skip nutrition: Could not load MMLU subject nutrition
============================================================
[>] philosophy (shots=5)
[!] Skip philosophy: Could not load MMLU subject philosophy
============================================================
[>] prehistory (shots=5)
[!] Skip prehistory: Could not load MMLU subject prehistory
============================================================
[>] professional_accounting (shots=5)
[!] Skip professional_accounting: Could not load MMLU subject professional_accounting
============================================================
[>] professional_law (shots=5)
[!] Skip professional_law: Could not load MMLU subject professional_law
============================================================
[>] professional_medicine (shots=5)
[!] Skip professional_medicine: Could not load MMLU subject professional_medicine
============================================================
[>] professional_psychology (shots=5)
[!] Skip professional_psychology: Could not load MMLU subject professional_psychology
============================================================
[>] public_relations (shots=5)
[!] Skip public_relations: Could not load MMLU subject public_relations
============================================================
[>] security_studies (shots=5)
[!] Skip security_studies: Could not load MMLU subject security_studies
============================================================
[>] sociology (shots=5)
[!] Skip sociology: Could not load MMLU subject sociology
============================================================
[>] us_foreign_policy (shots=5)
[!] Skip us_foreign_policy: Could not load MMLU subject us_foreign_policy
============================================================
[>] virology (shots=5)
[!] Skip virology: Could not load MMLU subject virology
============================================================
[>] world_religions (shots=5)
[!] Skip world_religions: Could not load MMLU subject world_religions
============================================================
MMLU RESULT — /home/zeus/microllm2/microllm2-checkpoints/final_merged
Shots: 5 Subjects: 27/57
abstract_algebra 30.0% (6/20)
anatomy 25.0% (5/20)
astronomy 35.0% (7/20)
business_ethics 30.0% (6/20)
clinical_knowledge 45.0% (9/20)
college_biology 45.0% (9/20)
college_chemistry 15.0% (3/20)
college_computer_science 45.0% (9/20)
college_mathematics 35.0% (7/20)
college_medicine 30.0% (6/20)
college_physics 15.0% (3/20)
computer_security 30.0% (6/20)
conceptual_physics 5.0% (1/20)
econometrics 30.0% (6/20)
electrical_engineering 20.0% (4/20)
elementary_mathematics 30.0% (6/20)
formal_logic 10.0% (2/20)
global_facts 35.0% (7/20)
high_school_biology 45.0% (9/20)
high_school_chemistry 35.0% (7/20)
high_school_computer_science 35.0% (7/20)
high_school_european_history 20.0% (4/20)
high_school_geography 25.0% (5/20)
high_school_government_and_politics 20.0% (4/20)
high_school_macroeconomics 0.0% (0/20)
high_school_mathematics 20.0% (4/20)
high_school_microeconomics 35.0% (7/20)
OVERALL: 149/540 = 27.59%
============================================================
[+] Saved mmlu_results.json
Note: GPT2-XL base ~24-26% MMLU (random 25%). MicroLLM2 distilled should be 25-30% —
MMLU is knowledge-heavy; GPT2 1.5B 1024ctx cannot match 7B+ models. Use as sanity check, not SOTA claim.
[MMLU] exit 0 at Sun Aug 9 22:53:53 UTC 2026
"total": 20,
"accuracy": 0.45
},
"college_mathematics": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"college_medicine": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"college_physics": {
"correct": 3,
"total": 20,
"accuracy": 0.15
},
"computer_security": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"conceptual_physics": {
"correct": 1,
"total": 20,
"accuracy": 0.05
},
"econometrics": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"electrical_engineering": {
"correct": 4,
"total": 20,
"accuracy": 0.2
},
"elementary_mathematics": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"formal_logic": {
"correct": 2,
"total": 20,
"accuracy": 0.1
},
"global_facts": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"high_school_biology": {
"correct": 9,
"total": 20,
"accuracy": 0.45
},
"high_school_chemistry": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"high_school_computer_science": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"high_school_european_history": {
"correct": 4,
"total": 20,
"accuracy": 0.2
},
"high_school_geography": {
"correct": 5,
"total": 20,
"accuracy": 0.25
},
"high_school_government_and_politics": {
"correct": 4,
"total": 20,
"accuracy": 0.2
},
"high_school_macroeconomics": {
"correct": 0,
"total": 20,
"accuracy": 0.0
},
"high_school_mathematics": {
"correct": 4,
"total": 20,
"accuracy": 0.2
},
"high_school_microeconomics": {
"correct": 7,
"total": 20,
"accuracy": 0.35
}
}
}'mmlu_results.json' -> '/teamspace/studios/this_studio/microllm2/mmlu_results.json'
'mmlu_run.log' -> '/teamspace/studios/this_studio/microllm2/mmlu_run.log'
[*] MicroLLM2 MMLU — model: /home/zeus/microllm2/microllm2-checkpoints/final_merged shots=5 limit=20
[*] GPT2-XL 1.5B 1024ctx vocab=50259 (ChatML) — MMLU via direct eval (no harness needed)
[*] lm-eval not installed — using lightweight direct MMLU eval (same logic, no harness)
[*] Install harness for official numbers: pip install lm-eval==0.4.4
[*] Loading tokenizer + model /home/zeus/microllm2/microllm2-checkpoints/final_merged ...
[+] Loaded on cuda:0 dtype=torch.bfloat16 — starting MMLU
MMLU subjects: 0%| | 0/57 [00:00<?, ?it/s]
============================================================
[>] abstract_algebra (shots=5)
MMLU subjects: 2%|▏ | 1/57 [00:03<03:40, 3.94s/it][=] abstract_algebra: 6/20 = 30.0% (running avg 30.0%)
============================================================
[>] anatomy (shots=5)
Downloading data: 100%|██████████| 20.1k/20.1k [00:00<00:00, 60.8kB/s]
Downloading data: 100%|██████████| 5.28k/5.28k [00:00<00:00, 30.6kB/s]
Downloading data: 100%|██████████| 3.50k/3.50k [00:00<00:00, 18.7kB/s]
Generating test split: 100%|██████████| 135/135 [00:00<00:00, 20566.29 examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 3828.67 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1496.15 examples/s]
MMLU subjects: 4%|▎ | 2/57 [00:07<03:32, 3.87s/it][=] anatomy: 5/20 = 25.0% (running avg 27.5%)
============================================================
[>] astronomy (shots=5)
Downloading data: 100%|██████████| 28.3k/28.3k [00:00<00:00, 153kB/s]
Downloading data: 100%|██████████| 6.05k/6.05k [00:00<00:00, 33.4kB/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 28.2kB/s]
Generating test split: 100%|██████████| 152/152 [00:00<00:00, 30542.02 examples/s]
Generating validation split: 100%|██████████| 16/16 [00:00<00:00, 4092.75 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1423.63 examples/s]
MMLU subjects: 5%|▌ | 3/57 [00:11<03:27, 3.84s/it][=] astronomy: 7/20 = 35.0% (running avg 30.0%)
============================================================
[>] business_ethics (shots=5)
Downloading data: 100%|██████████| 21.6k/21.6k [00:00<00:00, 131kB/s]
Downloading data: 100%|██████████| 5.09k/5.09k [00:00<00:00, 33.8kB/s]
Downloading data: 100%|██████████| 4.96k/4.96k [00:00<00:00, 27.9kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 23243.58 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3018.27 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1394.75 examples/s]
MMLU subjects: 7%|▋ | 4/57 [00:15<03:28, 3.94s/it][=] business_ethics: 6/20 = 30.0% (running avg 30.0%)
============================================================
[>] clinical_knowledge (shots=5)
Downloading data: 100%|██████████| 40.5k/40.5k [00:00<00:00, 199kB/s]
Downloading data: 100%|██████████| 7.48k/7.48k [00:00<00:00, 40.9kB/s]
Downloading data: 100%|██████████| 3.67k/3.67k [00:00<00:00, 20.4kB/s]
Generating test split: 100%|██████████| 265/265 [00:00<00:00, 52433.75 examples/s]
Generating validation split: 100%|██████████| 29/29 [00:00<00:00, 7840.33 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1485.34 examples/s]
MMLU subjects: 9%|▉ | 5/57 [00:19<03:24, 3.92s/it][=] clinical_knowledge: 9/20 = 45.0% (running avg 33.0%)
============================================================
[>] college_biology (shots=5)
Downloading data: 100%|██████████| 31.8k/31.8k [00:00<00:00, 166kB/s]
Downloading data: 100%|██████████| 6.90k/6.90k [00:00<00:00, 50.7kB/s]
Downloading data: 100%|██████████| 4.27k/4.27k [00:00<00:00, 23.0kB/s]
Generating test split: 100%|██████████| 144/144 [00:00<00:00, 29006.81 examples/s]
Generating validation split: 100%|██████████| 16/16 [00:00<00:00, 4205.34 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1440.85 examples/s]
MMLU subjects: 11%|█ | 6/57 [00:23<03:14, 3.81s/it][=] college_biology: 9/20 = 45.0% (running avg 35.0%)
============================================================
[>] college_chemistry (shots=5)
Downloading data: 100%|██████████| 17.9k/17.9k [00:00<00:00, 96.4kB/s]
Downloading data: 100%|██████████| 4.87k/4.87k [00:00<00:00, 26.9kB/s]
Downloading data: 100%|██████████| 4.04k/4.04k [00:00<00:00, 10.2kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22971.16 examples/s]
Generating validation split: 100%|██████████| 8/8 [00:00<00:00, 2089.45 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1161.02 examples/s]
MMLU subjects: 12%|█▏ | 7/57 [00:27<03:14, 3.88s/it][=] college_chemistry: 3/20 = 15.0% (running avg 32.1%)
============================================================
[>] college_computer_science (shots=5)
Downloading data: 100%|██████████| 28.1k/28.1k [00:00<00:00, 163kB/s]
Downloading data: 100%|██████████| 6.25k/6.25k [00:00<00:00, 40.2kB/s]
Downloading data: 100%|██████████| 6.81k/6.81k [00:00<00:00, 46.6kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22972.42 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3038.75 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1146.55 examples/s]
MMLU subjects: 14%|█▍ | 8/57 [00:31<03:10, 3.88s/it][=] college_computer_science: 9/20 = 45.0% (running avg 33.8%)
============================================================
[>] college_mathematics (shots=5)
Downloading data: 100%|██████████| 16.6k/16.6k [00:00<00:00, 85.2kB/s]
Downloading data: 100%|██████████| 5.00k/5.00k [00:00<00:00, 26.8kB/s]
Downloading data: 100%|██████████| 5.16k/5.16k [00:00<00:00, 36.1kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22919.69 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3062.55 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1144.17 examples/s]
MMLU subjects: 16%|█▌ | 9/57 [00:34<03:01, 3.79s/it][=] college_mathematics: 7/20 = 35.0% (running avg 33.9%)
============================================================
[>] college_medicine (shots=5)
Downloading data: 100%|██████████| 42.5k/42.5k [00:00<00:00, 220kB/s]
Downloading data: 100%|██████████| 8.99k/8.99k [00:00<00:00, 53.9kB/s]
Downloading data: 100%|██████████| 4.84k/4.84k [00:00<00:00, 34.1kB/s]
Generating test split: 100%|██████████| 173/173 [00:00<00:00, 34671.95 examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 4797.23 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1361.43 examples/s]
MMLU subjects: 18%|█▊ | 10/57 [00:38<02:55, 3.74s/it][=] college_medicine: 6/20 = 30.0% (running avg 33.5%)
============================================================
[>] college_physics (shots=5)
Downloading data: 100%|██████████| 18.6k/18.6k [00:00<00:00, 102kB/s]
Downloading data: 100%|██████████| 6.39k/6.39k [00:00<00:00, 39.5kB/s]
Downloading data: 100%|██████████| 4.51k/4.51k [00:00<00:00, 23.9kB/s]
Generating test split: 100%|██████████| 102/102 [00:00<00:00, 17934.14 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 2286.18 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1061.58 examples/s]
MMLU subjects: 19%|█▉ | 11/57 [00:42<02:55, 3.82s/it][=] college_physics: 3/20 = 15.0% (running avg 31.8%)
============================================================
[>] computer_security (shots=5)
Downloading data: 100%|██████████| 19.1k/19.1k [00:00<00:00, 106kB/s]
Downloading data: 100%|██████████| 6.67k/6.67k [00:00<00:00, 46.6kB/s]
Downloading data: 100%|██████████| 4.33k/4.33k [00:00<00:00, 23.7kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22049.75 examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 2974.88 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1465.82 examples/s]
MMLU subjects: 21%|██ | 12/57 [00:45<02:48, 3.75s/it][=] computer_security: 6/20 = 30.0% (running avg 31.7%)
============================================================
[>] conceptual_physics (shots=5)
Downloading data: 100%|██████████| 25.0k/25.0k [00:00<00:00, 152kB/s]
Downloading data: 100%|██████████| 5.98k/5.98k [00:00<00:00, 32.1kB/s]
Downloading data: 100%|██████████| 3.96k/3.96k [00:00<00:00, 22.3kB/s]
Generating test split: 100%|██████████| 235/235 [00:00<00:00, 51400.78 examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 6864.22 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1387.28 examples/s]
MMLU subjects: 23%|██▎ | 13/57 [00:49<02:40, 3.66s/it][=] conceptual_physics: 1/20 = 5.0% (running avg 29.6%)
============================================================
[>] econometrics (shots=5)
Downloading data: 100%|██████████| 24.5k/24.5k [00:00<00:00, 140kB/s]
Downloading data: 100%|██████████| 7.02k/7.02k [00:00<00:00, 26.7kB/s]
Downloading data: 100%|██████████| 4.54k/4.54k [00:00<00:00, 28.9kB/s]
Generating test split: 100%|██████████| 114/114 [00:00<00:00, 22894.45 examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 3162.13 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1402.87 examples/s]
MMLU subjects: 25%|██▍ | 14/57 [00:53<02:39, 3.70s/it][=] econometrics: 6/20 = 30.0% (running avg 29.6%)
============================================================
[>] electrical_engineering (shots=5)
Downloading data: 100%|██████████| 17.6k/17.6k [00:00<00:00, 107kB/s]
Downloading data: 100%|██████████| 5.08k/5.08k [00:00<00:00, 34.7kB/s]
Downloading data: 100%|██████████| 4.08k/4.08k [00:00<00:00, 22.7kB/s]
Generating test split: 100%|██████████| 145/145 [00:00<00:00, 33690.12 examples/s]
Generating validation split: 100%|██████████| 16/16 [00:00<00:00, 4531.63 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1402.68 examples/s]
MMLU subjects: 26%|██▋ | 15/57 [00:56<02:30, 3.59s/it][=] electrical_engineering: 4/20 = 20.0% (running avg 29.0%)
============================================================
[>] elementary_mathematics (shots=5)
Downloading data: 100%|██████████| 41.1k/41.1k [00:00<00:00, 222kB/s]
Downloading data: 100%|██████████| 9.38k/9.38k [00:00<00:00, 52.3kB/s]
Downloading data: 100%|██████████| 4.55k/4.55k [00:00<00:00, 21.6kB/s]
Generating test split: 100%|██████████| 378/378 [00:00<00:00, 78826.97 examples/s]
Generating validation split: 100%|██████████| 41/41 [00:00<00:00, 11570.88 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1528.09 examples/s]
MMLU subjects: 28%|██▊ | 16/57 [00:59<02:26, 3.56s/it][=] elementary_mathematics: 6/20 = 30.0% (running avg 29.1%)
============================================================
[>] formal_logic (shots=5)
Downloading data: 100%|██████████| 21.5k/21.5k [00:00<00:00, 110kB/s]
Downloading data: 100%|██████████| 6.56k/6.56k [00:00<00:00, 38.8kB/s]
Downloading data: 100%|██████████| 4.81k/4.81k [00:00<00:00, 22.3kB/s]
Generating test split: 100%|██████████| 126/126 [00:00<00:00, 28698.47 examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 3785.72 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1486.81 examples/s]
MMLU subjects: 30%|██▉ | 17/57 [01:03<02:23, 3.58s/it][=] formal_logic: 2/20 = 10.0% (running avg 27.9%)
============================================================
[>] global_facts (shots=5)
Downloading data: 100%|██████████| 11.5k/11.5k [00:00<00:00, 58.8kB/s]
Downloading data: 100%|██████████| 4.19k/4.19k [00:00<00:00, 22.5kB/s]
Downloading data: 100%|██████████| 3.58k/3.58k [00:00<00:00, 24.1kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22527.01 examples/s]
Generating validation split: 100%|██████████| 10/10 [00:00<00:00, 2616.37 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1165.93 examples/s]
MMLU subjects: 32%|███▏ | 18/57 [01:06<02:15, 3.47s/it][=] global_facts: 7/20 = 35.0% (running avg 28.3%)
============================================================
[>] high_school_biology (shots=5)
Downloading data: 100%|██████████| 62.7k/62.7k [00:00<00:00, 353kB/s]
Downloading data: 100%|██████████| 10.6k/10.6k [00:00<00:00, 58.2kB/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 29.2kB/s]
Generating test split: 100%|██████████| 310/310 [00:00<00:00, 60104.20 examples/s]
Generating validation split: 100%|██████████| 32/32 [00:00<00:00, 7002.18 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1401.37 examples/s]
MMLU subjects: 33%|███▎ | 19/57 [01:10<02:10, 3.42s/it][=] high_school_biology: 9/20 = 45.0% (running avg 29.2%)
============================================================
[>] high_school_chemistry (shots=5)
Downloading data: 100%|██████████| 33.3k/33.3k [00:00<00:00, 185kB/s]
Downloading data: 100%|██████████| 8.31k/8.31k [00:00<00:00, 59.2kB/s]
Downloading data: 100%|██████████| 4.16k/4.16k [00:00<00:00, 20.7kB/s]
Generating test split: 100%|██████████| 203/203 [00:00<00:00, 43367.99 examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 5422.50 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1380.34 examples/s]
MMLU subjects: 35%|███▌ | 20/57 [01:13<02:05, 3.40s/it][=] high_school_chemistry: 7/20 = 35.0% (running avg 29.5%)
============================================================
[>] high_school_computer_science (shots=5)
Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 173kB/s]
Downloading data: 100%|██████████| 5.28k/5.28k [00:00<00:00, 36.0kB/s]
Downloading data: 100%|██████████| 6.54k/6.54k [00:00<00:00, 36.0kB/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 22050.91 examples/s]
Generating validation split: 100%|██████████| 9/9 [00:00<00:00, 2392.04 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1416.80 examples/s]
MMLU subjects: 37%|███▋ | 21/57 [01:16<02:03, 3.43s/it][=] high_school_computer_science: 7/20 = 35.0% (running avg 29.8%)
============================================================
[>] high_school_european_history (shots=5)
Downloading data: 100%|██████████| 142k/142k [00:00<00:00, 678kB/s]
Downloading data: 100%|██████████| 31.6k/31.6k [00:00<00:00, 201kB/s]
Downloading data: 100%|██████████| 22.2k/22.2k [00:00<00:00, 112kB/s]
Generating test split: 100%|██████████| 165/165 [00:00<00:00, 31161.25 examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 4798.67 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1341.75 examples/s]
MMLU subjects: 39%|███▊ | 22/57 [01:20<01:57, 3.36s/it][=] high_school_european_history: 4/20 = 20.0% (running avg 29.3%)
============================================================
[>] high_school_geography (shots=5)
Downloading data: 100%|██████████| 28.2k/28.2k [00:00<00:00, 145kB/s]
Downloading data: 100%|██████████| 6.16k/6.16k [00:00<00:00, 40.6kB/s]
Downloading data: 100%|██████████| 3.93k/3.93k [00:00<00:00, 28.4kB/s]
Generating test split: 100%|██████████| 198/198 [00:00<00:00, 40891.83 examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 5636.82 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1341.75 examples/s]
MMLU subjects: 40%|████ | 23/57 [01:23<01:53, 3.32s/it][=] high_school_geography: 5/20 = 25.0% (running avg 29.1%)
============================================================
[>] high_school_government_and_politics (shots=5)
Downloading data: 100%|██████████| 40.2k/40.2k [00:00<00:00, 276kB/s]
Downloading data: 100%|██████████| 8.27k/8.27k [00:00<00:00, 53.9kB/s]
Downloading data: 100%|██████████| 4.47k/4.47k [00:00<00:00, 26.1kB/s]
Generating test split: 100%|██████████| 193/193 [00:00<00:00, 40098.11 examples/s]
Generating validation split: 100%|██████████| 21/21 [00:00<00:00, 5715.79 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1422.28 examples/s]
MMLU subjects: 42%|████▏ | 24/57 [01:26<01:46, 3.23s/it][=] high_school_government_and_politics: 4/20 = 20.0% (running avg 28.7%)
============================================================
[>] high_school_macroeconomics (shots=5)
Downloading data: 100%|██████████| 54.8k/54.8k [00:00<00:00, 362kB/s]
Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 50.2kB/s]
Downloading data: 100%|██████████| 4.04k/4.04k [00:00<00:00, 29.7kB/s]
Generating test split: 100%|██████████| 390/390 [00:00<00:00, 70428.77 examples/s]
Generating validation split: 100%|██████████| 43/43 [00:00<00:00, 10985.20 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1389.95 examples/s]
MMLU subjects: 44%|████▍ | 25/57 [01:29<01:43, 3.24s/it][=] high_school_macroeconomics: 0/20 = 0.0% (running avg 27.6%)
============================================================
[>] high_school_mathematics (shots=5)
Downloading data: 100%|██████████| 33.7k/33.7k [00:00<00:00, 218kB/s]
Downloading data: 100%|██████████| 6.99k/6.99k [00:00<00:00, 28.3kB/s]
Downloading data: 100%|██████████| 4.50k/4.50k [00:00<00:00, 21.2kB/s]
Generating test split: 100%|██████████| 270/270 [00:00<00:00, 59139.49 examples/s]
Generating validation split: 100%|██████████| 29/29 [00:00<00:00, 8262.11 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1471.58 examples/s]
MMLU subjects: 46%|████▌ | 26/57 [01:33<01:42, 3.29s/it][=] high_school_mathematics: 4/20 = 20.0% (running avg 27.3%)
============================================================
[>] high_school_microeconomics (shots=5)
Downloading data: 100%|██████████| 38.8k/38.8k [00:00<00:00, 266kB/s]
Downloading data: 100%|██████████| 7.22k/7.22k [00:00<00:00, 41.9kB/s]
Downloading data: 100%|██████████| 3.83k/3.83k [00:00<00:00, 25.0kB/s]
Generating test split: 100%|██████████| 238/238 [00:00<00:00, 49842.44 examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 7166.92 examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1446.41 examples/s]
MMLU subjects: 47%|████▋ | 27/57 [01:36<01:37, 3.25s/it][=] high_school_microeconomics: 7/20 = 35.0% (running avg 27.6%)
============================================================
[>] high_school_physics (shots=5)
Downloading data: 100%|██████████| 33.0k/33.0k [00:00<00:00, 196kB/s]
Downloading data: 100%|██████████| 7.96k/7.96k [00:00<00:00, 42.2kB/s]
MMLU subjects: 49%|████▉ | 28/57 [01:38<01:27, 3.02s/it]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 51%|█████ | 29/57 [01:38<01:00, 2.15s/it]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 54%|█████▍ | 31/57 [01:39<00:31, 1.20s/it]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 56%|█████▌ | 32/57 [01:39<00:23, 1.08it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 58%|█████▊ | 33/57 [01:39<00:17, 1.40it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 60%|█████▉ | 34/57 [01:39<00:12, 1.83it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 61%|██████▏ | 35/57 [01:39<00:09, 2.36it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 63%|██████▎ | 36/57 [01:39<00:06, 3.02it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 65%|██████▍ | 37/57 [01:39<00:05, 3.78it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 67%|██████▋ | 38/57 [01:39<00:04, 4.61it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 68%|██████▊ | 39/57 [01:39<00:03, 5.47it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 70%|███████ | 40/57 [01:39<00:02, 6.29it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 74%|███████▎ | 42/57 [01:40<00:01, 7.54it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 75%|███████▌ | 43/57 [01:40<00:01, 7.82it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 77%|███████▋ | 44/57 [01:40<00:01, 8.27it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 81%|████████ | 46/57 [01:40<00:01, 8.87it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 82%|████████▏ | 47/57 [01:40<00:01, 9.04it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 84%|████████▍ | 48/57 [01:40<00:00, 9.18it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 86%|████████▌ | 49/57 [01:40<00:00, 9.33it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 88%|████████▊ | 50/57 [01:40<00:00, 9.46it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 89%|████████▉ | 51/57 [01:41<00:00, 9.49it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 91%|█████████ | 52/57 [01:41<00:00, 9.62it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 93%|█████████▎| 53/57 [01:41<00:00, 9.51it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 95%|█████████▍| 54/57 [01:41<00:00, 9.56it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 96%|█████████▋| 55/57 [01:41<00:00, 9.52it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 98%|█████████▊| 56/57 [01:41<00:00, 9.56it/s]Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub
MMLU subjects: 100%|██████████| 57/57 [01:41<00:00, 1.78s/it]
[!] Skip high_school_physics: Could not load MMLU subject high_school_physics
============================================================
[>] high_school_psychology (shots=5)
[!] Skip high_school_psychology: Could not load MMLU subject high_school_psychology
============================================================
[>] high_school_statistics (shots=5)
[!] Skip high_school_statistics: Could not load MMLU subject high_school_statistics
============================================================
[>] high_school_us_history (shots=5)
[!] Skip high_school_us_history: Could not load MMLU subject high_school_us_history
============================================================
[>] high_school_world_history (shots=5)
[!] Skip high_school_world_history: Could not load MMLU subject high_school_world_history
============================================================
[>] human_aging (shots=5)
[!] Skip human_aging: Could not load MMLU subject human_aging
============================================================
[>] human_sexuality (shots=5)
[!] Skip human_sexuality: Could not load MMLU subject human_sexuality
============================================================
[>] international_law (shots=5)
[!] Skip international_law: Could not load MMLU subject international_law
============================================================
[>] jurisprudence (shots=5)
[!] Skip jurisprudence: Could not load MMLU subject jurisprudence
============================================================
[>] logical_fallacies (shots=5)
[!] Skip logical_fallacies: Could not load MMLU subject logical_fallacies
============================================================
[>] machine_learning (shots=5)
[!] Skip machine_learning: Could not load MMLU subject machine_learning
============================================================
[>] management (shots=5)
[!] Skip management: Could not load MMLU subject management
============================================================
[>] marketing (shots=5)
[!] Skip marketing: Could not load MMLU subject marketing
============================================================
[>] medical_genetics (shots=5)
[!] Skip medical_genetics: Could not load MMLU subject medical_genetics
============================================================
[>] miscellaneous (shots=5)
[!] Skip miscellaneous: Could not load MMLU subject miscellaneous
============================================================
[>] moral_disputes (shots=5)
[!] Skip moral_disputes: Could not load MMLU subject moral_disputes
============================================================
[>] moral_scenarios (shots=5)
[!] Skip moral_scenarios: Could not load MMLU subject moral_scenarios
============================================================
[>] nutrition (shots=5)
[!] Skip nutrition: Could not load MMLU subject nutrition
============================================================
[>] philosophy (shots=5)
[!] Skip philosophy: Could not load MMLU subject philosophy
============================================================
[>] prehistory (shots=5)
[!] Skip prehistory: Could not load MMLU subject prehistory
============================================================
[>] professional_accounting (shots=5)
[!] Skip professional_accounting: Could not load MMLU subject professional_accounting
============================================================
[>] professional_law (shots=5)
[!] Skip professional_law: Could not load MMLU subject professional_law
============================================================
[>] professional_medicine (shots=5)
[!] Skip professional_medicine: Could not load MMLU subject professional_medicine
============================================================
[>] professional_psychology (shots=5)
[!] Skip professional_psychology: Could not load MMLU subject professional_psychology
============================================================
[>] public_relations (shots=5)
[!] Skip public_relations: Could not load MMLU subject public_relations
============================================================
[>] security_studies (shots=5)
[!] Skip security_studies: Could not load MMLU subject security_studies
============================================================
[>] sociology (shots=5)
[!] Skip sociology: Could not load MMLU subject sociology
============================================================
[>] us_foreign_policy (shots=5)
[!] Skip us_foreign_policy: Could not load MMLU subject us_foreign_policy
============================================================
[>] virology (shots=5)
[!] Skip virology: Could not load MMLU subject virology
============================================================
[>] world_religions (shots=5)
[!] Skip world_religions: Could not load MMLU subject world_religions
============================================================
MMLU RESULT — /home/zeus/microllm2/microllm2-checkpoints/final_merged
Shots: 5 Subjects: 27/57
abstract_algebra 30.0% (6/20)
anatomy 25.0% (5/20)
astronomy 35.0% (7/20)
business_ethics 30.0% (6/20)
clinical_knowledge 45.0% (9/20)
college_biology 45.0% (9/20)
college_chemistry 15.0% (3/20)
college_computer_science 45.0% (9/20)
college_mathematics 35.0% (7/20)
college_medicine 30.0% (6/20)
college_physics 15.0% (3/20)
computer_security 30.0% (6/20)
conceptual_physics 5.0% (1/20)
econometrics 30.0% (6/20)
electrical_engineering 20.0% (4/20)
elementary_mathematics 30.0% (6/20)
formal_logic 10.0% (2/20)
global_facts 35.0% (7/20)
high_school_biology 45.0% (9/20)
high_school_chemistry 35.0% (7/20)
high_school_computer_science 35.0% (7/20)
high_school_european_history 20.0% (4/20)
high_school_geography 25.0% (5/20)
high_school_government_and_politics 20.0% (4/20)
high_school_macroeconomics 0.0% (0/20)
high_school_mathematics 20.0% (4/20)
high_school_microeconomics 35.0% (7/20)
OVERALL: 149/540 = 27.59%
============================================================
[+] Saved mmlu_results.json
Note: GPT2-XL base ~24-26% MMLU (random 25%). MicroLLM2 distilled should be 25-30% —
MMLU is knowledge-heavy; GPT2 1.5B 1024ctx cannot match 7B+ models. Use as sanity check, not SOTA claim.
[MMLU] exit 0 at Sun Aug 9 22:53:53 UTC 2026
"total": 20,
"accuracy": 0.45
},
"college_mathematics": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"college_medicine": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"college_physics": {
"correct": 3,
"total": 20,
"accuracy": 0.15
},
"computer_security": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"conceptual_physics": {
"correct": 1,
"total": 20,
"accuracy": 0.05
},
"econometrics": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"electrical_engineering": {
"correct": 4,
"total": 20,
"accuracy": 0.2
},
"elementary_mathematics": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"formal_logic": {
"correct": 2,
"total": 20,
"accuracy": 0.1
},
"global_facts": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"high_school_biology": {
"correct": 9,
"total": 20,
"accuracy": 0.45
},
"high_school_chemistry": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"high_school_computer_science": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"high_school_european_history": {
"correct": 4,
"total": 20,
"accuracy": 0.2
},
"high_school_geography": {
"correct": 5,
"total": 20,
"accuracy": 0.25
},
"high_school_government_and_politics": {
"correct": 4,
"total": 20,
"accuracy": 0.2
},
"high_school_macroeconomics": {
"correct": 0,
"total": 20,
"accuracy": 0.0
},
"high_school_mathematics": {
"correct": 4,
"total": 20,
"accuracy": 0.2
},
"high_school_microeconomics": {
"correct": 7,
"total": 20,
"accuracy": 0.35
}
}
}'mmlu_results.json' -> '/teamspace/studios/this_studio/microllm2/mmlu_results.json'
'mmlu_run.log' -> '/teamspace/studios/this_studio/microllm2/mmlu_run.log'