MLVXN/MicroLLM2
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
f9ca54119641735fe890de4baf3efd6e9c9806c4
MicroLLM2/mmlu57_token.log
ModelHub XC f9ca541196 初始化项目，由ModelHub XC社区提供模型 
Model: MLVXN/MicroLLM2
Source: Original Platform
2026-09-14 14:54:18 +08:00

1025 lines
72 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

[MMLU57-TOKEN] start Sun Aug 9 23:10:07 UTC 2026 token=hf_oUMhDnr...
[*] MicroLLM2 MMLU — model: /home/zeus/microllm2/microllm2-checkpoints/final_merged shots=5 limit=20
[*] GPT2-XL 1.5B 1024ctx vocab=50259 (ChatML) — MMLU via direct eval (no harness needed)
[*] lm-eval not installed — using lightweight direct MMLU eval (same logic, no harness)
[*] Install harness for official numbers: pip install lm-eval==0.4.4
[*] Loading tokenizer + model /home/zeus/microllm2/microllm2-checkpoints/final_merged ...
[+] Loaded on cuda:0 dtype=torch.bfloat16 — starting MMLU
MMLU subjects: 0%| | 0/57 [00:00<?, ?it/s]
============================================================
[>] abstract_algebra (shots=5)
abstract_algebra: 0%| | 0/20 [00:00<?, ?it/s]
abstract_algebra: 5%|▌ | 1/20 [00:01<00:21, 1.15s/it]
abstract_algebra: 15%|█▌ | 3/20 [00:01<00:05, 2.92it/s]
abstract_algebra: 25%|██▌ | 5/20 [00:01<00:02, 5.08it/s]
abstract_algebra: 40%|████ | 8/20 [00:01<00:01, 8.67it/s]
abstract_algebra: 50%|█████ | 10/20 [00:01<00:00, 10.56it/s]
abstract_algebra: 70%|███████ | 14/20 [00:01<00:00, 15.27it/s]
abstract_algebra: 85%|████████▌ | 17/20 [00:01<00:00, 17.18it/s]
abstract_algebra: 100%|██████████| 20/20 [00:02<00:00, 17.57it/s]

MMLU subjects: 2%|▏ | 1/57 [00:03<03:29, 3.73s/it][=] abstract_algebra: 6/20 = 30.0% (running avg 30.0%)
============================================================
[>] anatomy (shots=5)
anatomy: 0%| | 0/20 [00:00<?, ?it/s]
anatomy: 10%|█ | 2/20 [00:00<00:01, 13.20it/s]
anatomy: 30%|███ | 6/20 [00:00<00:00, 25.74it/s]
anatomy: 45%|████▌ | 9/20 [00:00<00:00, 24.91it/s]
anatomy: 60%|██████ | 12/20 [00:00<00:00, 24.58it/s]
anatomy: 75%|███████▌ | 15/20 [00:00<00:00, 23.82it/s]
anatomy: 90%|█████████ | 18/20 [00:00<00:00, 21.69it/s]

MMLU subjects: 4%|▎ | 2/57 [00:06<02:43, 2.97s/it][=] anatomy: 5/20 = 25.0% (running avg 27.5%)
============================================================
[>] astronomy (shots=5)
astronomy: 0%| | 0/20 [00:00<?, ?it/s]
astronomy: 10%|█ | 2/20 [00:00<00:01, 12.23it/s]
astronomy: 20%|██ | 4/20 [00:00<00:01, 14.44it/s]
astronomy: 30%|███ | 6/20 [00:00<00:00, 15.75it/s]
astronomy: 40%|████ | 8/20 [00:00<00:00, 16.45it/s]
astronomy: 50%|█████ | 10/20 [00:00<00:00, 16.60it/s]
astronomy: 60%|██████ | 12/20 [00:00<00:00, 17.09it/s]
astronomy: 70%|███████ | 14/20 [00:00<00:00, 17.48it/s]
astronomy: 85%|████████▌ | 17/20 [00:00<00:00, 19.61it/s]
astronomy: 95%|█████████▌| 19/20 [00:01<00:00, 19.15it/s]

MMLU subjects: 5%|▌ | 3/57 [00:08<02:23, 2.66s/it][=] astronomy: 7/20 = 35.0% (running avg 30.0%)
============================================================
[>] business_ethics (shots=5)
business_ethics: 0%| | 0/20 [00:00<?, ?it/s]
business_ethics: 10%|█ | 2/20 [00:00<00:01, 12.90it/s]
business_ethics: 20%|██ | 4/20 [00:00<00:01, 15.43it/s]
business_ethics: 40%|████ | 8/20 [00:00<00:00, 24.39it/s]
business_ethics: 55%|█████▌ | 11/20 [00:00<00:00, 24.16it/s]
business_ethics: 70%|███████ | 14/20 [00:00<00:00, 23.61it/s]
business_ethics: 85%|████████▌ | 17/20 [00:00<00:00, 23.27it/s]
business_ethics: 100%|██████████| 20/20 [00:00<00:00, 22.96it/s]

MMLU subjects: 7%|▋ | 4/57 [00:10<02:10, 2.45s/it][=] business_ethics: 6/20 = 30.0% (running avg 30.0%)
============================================================
[>] clinical_knowledge (shots=5)
clinical_knowledge: 0%| | 0/20 [00:00<?, ?it/s]
clinical_knowledge: 10%|█ | 2/20 [00:00<00:00, 18.14it/s]
clinical_knowledge: 20%|██ | 4/20 [00:00<00:00, 18.00it/s]
clinical_knowledge: 30%|███ | 6/20 [00:00<00:00, 17.94it/s]
clinical_knowledge: 40%|████ | 8/20 [00:00<00:00, 17.50it/s]
clinical_knowledge: 50%|█████ | 10/20 [00:00<00:00, 17.77it/s]
clinical_knowledge: 60%|██████ | 12/20 [00:00<00:00, 17.75it/s]
clinical_knowledge: 75%|███████▌ | 15/20 [00:00<00:00, 19.69it/s]
clinical_knowledge: 90%|█████████ | 18/20 [00:00<00:00, 20.96it/s]

MMLU subjects: 9%|▉ | 5/57 [00:12<02:05, 2.42s/it][=] clinical_knowledge: 9/20 = 45.0% (running avg 33.0%)
============================================================
[>] college_biology (shots=5)
college_biology: 0%| | 0/20 [00:00<?, ?it/s]
college_biology: 10%|█ | 2/20 [00:00<00:01, 13.18it/s]
college_biology: 20%|██ | 4/20 [00:00<00:01, 15.56it/s]
college_biology: 35%|███▌ | 7/20 [00:00<00:00, 19.08it/s]
college_biology: 45%|████▌ | 9/20 [00:00<00:00, 18.60it/s]
college_biology: 65%|██████▌ | 13/20 [00:00<00:00, 21.91it/s]
college_biology: 80%|████████ | 16/20 [00:00<00:00, 22.47it/s]
college_biology: 95%|█████████▌| 19/20 [00:00<00:00, 22.98it/s]

MMLU subjects: 11%|█ | 6/57 [00:15<01:59, 2.35s/it][=] college_biology: 9/20 = 45.0% (running avg 35.0%)
============================================================
[>] college_chemistry (shots=5)
college_chemistry: 0%| | 0/20 [00:00<?, ?it/s]
college_chemistry: 10%|█ | 2/20 [00:00<00:01, 12.84it/s]
college_chemistry: 25%|██▌ | 5/20 [00:00<00:00, 18.26it/s]
college_chemistry: 40%|████ | 8/20 [00:00<00:00, 20.43it/s]
college_chemistry: 55%|█████▌ | 11/20 [00:00<00:00, 19.37it/s]
college_chemistry: 65%|██████▌ | 13/20 [00:00<00:00, 18.98it/s]
college_chemistry: 85%|████████▌ | 17/20 [00:00<00:00, 21.96it/s]

MMLU subjects: 12%|█▏ | 7/57 [00:17<01:56, 2.33s/it][=] college_chemistry: 3/20 = 15.0% (running avg 32.1%)
============================================================
[>] college_computer_science (shots=5)
college_computer_science: 0%| | 0/20 [00:00<?, ?it/s]
college_computer_science: 5%|▌ | 1/20 [00:00<00:02, 9.28it/s]
college_computer_science: 15%|█▌ | 3/20 [00:00<00:01, 13.08it/s]
college_computer_science: 25%|██▌ | 5/20 [00:00<00:01, 14.86it/s]
college_computer_science: 35%|███▌ | 7/20 [00:00<00:00, 15.74it/s]
college_computer_science: 45%|████▌ | 9/20 [00:00<00:00, 16.53it/s]
college_computer_science: 55%|█████▌ | 11/20 [00:00<00:00, 17.01it/s]
college_computer_science: 65%|██████▌ | 13/20 [00:00<00:00, 17.24it/s]
college_computer_science: 75%|███████▌ | 15/20 [00:00<00:00, 17.33it/s]
college_computer_science: 85%|████████▌ | 17/20 [00:01<00:00, 17.21it/s]
college_computer_science: 95%|█████████▌| 19/20 [00:01<00:00, 17.23it/s]

MMLU subjects: 14%|█▍ | 8/57 [00:19<01:55, 2.37s/it][=] college_computer_science: 9/20 = 45.0% (running avg 33.8%)
============================================================
[>] college_mathematics (shots=5)
college_mathematics: 0%| | 0/20 [00:00<?, ?it/s]
college_mathematics: 10%|█ | 2/20 [00:00<00:01, 12.83it/s]
college_mathematics: 20%|██ | 4/20 [00:00<00:01, 15.33it/s]
college_mathematics: 35%|███▌ | 7/20 [00:00<00:00, 18.80it/s]
college_mathematics: 60%|██████ | 12/20 [00:00<00:00, 26.76it/s]
college_mathematics: 80%|████████ | 16/20 [00:00<00:00, 26.72it/s]
college_mathematics: 95%|█████████▌| 19/20 [00:00<00:00, 23.14it/s]

MMLU subjects: 16%|█▌ | 9/57 [00:22<01:52, 2.34s/it][=] college_mathematics: 7/20 = 35.0% (running avg 33.9%)
============================================================
[>] college_medicine (shots=5)
college_medicine: 0%| | 0/20 [00:00<?, ?it/s]
college_medicine: 10%|█ | 2/20 [00:00<00:01, 17.54it/s]
college_medicine: 20%|██ | 4/20 [00:00<00:00, 17.37it/s]
college_medicine: 30%|███ | 6/20 [00:00<00:00, 17.41it/s]
college_medicine: 45%|████▌ | 9/20 [00:00<00:00, 19.51it/s]
college_medicine: 55%|█████▌ | 11/20 [00:00<00:00, 18.71it/s]
college_medicine: 65%|██████▌ | 13/20 [00:00<00:00, 18.52it/s]
college_medicine: 75%|███████▌ | 15/20 [00:00<00:00, 18.49it/s]
college_medicine: 100%|██████████| 20/20 [00:00<00:00, 25.98it/s]

MMLU subjects: 18%|█▊ | 10/57 [00:24<01:47, 2.28s/it][=] college_medicine: 6/20 = 30.0% (running avg 33.5%)
============================================================
[>] college_physics (shots=5)
college_physics: 0%| | 0/20 [00:00<?, ?it/s]
college_physics: 10%|█ | 2/20 [00:00<00:01, 12.81it/s]
college_physics: 20%|██ | 4/20 [00:00<00:01, 15.22it/s]
college_physics: 40%|████ | 8/20 [00:00<00:00, 21.09it/s]
college_physics: 60%|██████ | 12/20 [00:00<00:00, 23.72it/s]
college_physics: 80%|████████ | 16/20 [00:00<00:00, 27.98it/s]
college_physics: 95%|█████████▌| 19/20 [00:00<00:00, 26.64it/s]

MMLU subjects: 19%|█▉ | 11/57 [00:26<01:42, 2.23s/it][=] college_physics: 3/20 = 15.0% (running avg 31.8%)
============================================================
[>] computer_security (shots=5)
computer_security: 0%| | 0/20 [00:00<?, ?it/s]
computer_security: 10%|█ | 2/20 [00:00<00:00, 18.55it/s]
computer_security: 20%|██ | 4/20 [00:00<00:00, 18.25it/s]
computer_security: 35%|███▌ | 7/20 [00:00<00:00, 20.93it/s]
computer_security: 55%|█████▌ | 11/20 [00:00<00:00, 24.12it/s]
computer_security: 70%|███████ | 14/20 [00:00<00:00, 23.97it/s]
computer_security: 90%|█████████ | 18/20 [00:00<00:00, 28.34it/s]

MMLU subjects: 21%|██ | 12/57 [00:28<01:37, 2.17s/it][=] computer_security: 6/20 = 30.0% (running avg 31.7%)
============================================================
[>] conceptual_physics (shots=5)
conceptual_physics: 0%| | 0/20 [00:00<?, ?it/s]
conceptual_physics: 5%|▌ | 1/20 [00:00<00:02, 9.44it/s]
conceptual_physics: 15%|█▌ | 3/20 [00:00<00:01, 14.31it/s]
conceptual_physics: 30%|███ | 6/20 [00:00<00:00, 18.53it/s]
conceptual_physics: 45%|████▌ | 9/20 [00:00<00:00, 20.38it/s]
conceptual_physics: 65%|██████▌ | 13/20 [00:00<00:00, 22.74it/s]
conceptual_physics: 85%|████████▌ | 17/20 [00:00<00:00, 24.44it/s]
conceptual_physics: 100%|██████████| 20/20 [00:00<00:00, 24.12it/s]

MMLU subjects: 23%|██▎ | 13/57 [00:30<01:37, 2.22s/it][=] conceptual_physics: 1/20 = 5.0% (running avg 29.6%)
============================================================
[>] econometrics (shots=5)
econometrics: 0%| | 0/20 [00:00<?, ?it/s]
econometrics: 20%|██ | 4/20 [00:00<00:00, 27.82it/s]
econometrics: 35%|███▌ | 7/20 [00:00<00:00, 21.59it/s]
econometrics: 55%|█████▌ | 11/20 [00:00<00:00, 23.65it/s]
econometrics: 80%|████████ | 16/20 [00:00<00:00, 28.86it/s]
econometrics: 95%|█████████▌| 19/20 [00:00<00:00, 26.62it/s]

MMLU subjects: 25%|██▍ | 14/57 [00:32<01:33, 2.18s/it][=] econometrics: 6/20 = 30.0% (running avg 29.6%)
============================================================
[>] electrical_engineering (shots=5)
electrical_engineering: 0%| | 0/20 [00:00<?, ?it/s]
electrical_engineering: 10%|█ | 2/20 [00:00<00:01, 17.36it/s]
electrical_engineering: 30%|███ | 6/20 [00:00<00:00, 23.79it/s]
electrical_engineering: 45%|████▌ | 9/20 [00:00<00:00, 20.65it/s]
electrical_engineering: 65%|██████▌ | 13/20 [00:00<00:00, 25.92it/s]
electrical_engineering: 95%|█████████▌| 19/20 [00:00<00:00, 35.89it/s]

MMLU subjects: 26%|██▋ | 15/57 [00:34<01:30, 2.15s/it][=] electrical_engineering: 4/20 = 20.0% (running avg 29.0%)
============================================================
[>] elementary_mathematics (shots=5)
elementary_mathematics: 0%| | 0/20 [00:00<?, ?it/s]
elementary_mathematics: 10%|█ | 2/20 [00:00<00:01, 12.90it/s]
elementary_mathematics: 20%|██ | 4/20 [00:00<00:01, 15.49it/s]
elementary_mathematics: 30%|███ | 6/20 [00:00<00:00, 16.54it/s]
elementary_mathematics: 50%|█████ | 10/20 [00:00<00:00, 23.85it/s]
elementary_mathematics: 70%|███████ | 14/20 [00:00<00:00, 28.23it/s]
elementary_mathematics: 85%|████████▌ | 17/20 [00:00<00:00, 26.39it/s]

MMLU subjects: 28%|██▊ | 16/57 [00:36<01:26, 2.11s/it][=] elementary_mathematics: 6/20 = 30.0% (running avg 29.1%)
============================================================
[>] formal_logic (shots=5)
formal_logic: 0%| | 0/20 [00:00<?, ?it/s]
formal_logic: 10%|█ | 2/20 [00:00<00:01, 13.10it/s]
formal_logic: 20%|██ | 4/20 [00:00<00:01, 15.61it/s]
formal_logic: 30%|███ | 6/20 [00:00<00:00, 16.51it/s]
formal_logic: 45%|████▌ | 9/20 [00:00<00:00, 19.07it/s]
formal_logic: 65%|██████▌ | 13/20 [00:00<00:00, 22.24it/s]
formal_logic: 80%|████████ | 16/20 [00:00<00:00, 22.59it/s]
formal_logic: 95%|█████████▌| 19/20 [00:00<00:00, 22.94it/s]

MMLU subjects: 30%|██▉ | 17/57 [00:39<01:24, 2.12s/it][=] formal_logic: 2/20 = 10.0% (running avg 27.9%)
============================================================
[>] global_facts (shots=5)
global_facts: 0%| | 0/20 [00:00<?, ?it/s]
global_facts: 10%|█ | 2/20 [00:00<00:01, 17.09it/s]
global_facts: 30%|███ | 6/20 [00:00<00:00, 28.49it/s]
global_facts: 50%|█████ | 10/20 [00:00<00:00, 32.74it/s]
global_facts: 70%|███████ | 14/20 [00:00<00:00, 34.73it/s]

MMLU subjects: 32%|███▏ | 18/57 [00:40<01:18, 2.00s/it][=] global_facts: 7/20 = 35.0% (running avg 28.3%)
============================================================
[>] high_school_biology (shots=5)
high_school_biology: 0%| | 0/20 [00:00<?, ?it/s]
high_school_biology: 15%|█▌ | 3/20 [00:00<00:00, 24.19it/s]
high_school_biology: 35%|███▌ | 7/20 [00:00<00:00, 31.92it/s]
high_school_biology: 55%|█████▌ | 11/20 [00:00<00:00, 26.11it/s]
high_school_biology: 75%|███████▌ | 15/20 [00:00<00:00, 26.59it/s]
high_school_biology: 95%|█████████▌| 19/20 [00:00<00:00, 29.76it/s]

MMLU subjects: 33%|███▎ | 19/57 [00:43<01:17, 2.04s/it][=] high_school_biology: 9/20 = 45.0% (running avg 29.2%)
============================================================
[>] high_school_chemistry (shots=5)
high_school_chemistry: 0%| | 0/20 [00:00<?, ?it/s]
high_school_chemistry: 10%|█ | 2/20 [00:00<00:01, 17.03it/s]
high_school_chemistry: 40%|████ | 8/20 [00:00<00:00, 40.46it/s]
high_school_chemistry: 65%|██████▌ | 13/20 [00:00<00:00, 40.38it/s]
high_school_chemistry: 90%|█████████ | 18/20 [00:00<00:00, 40.22it/s]

MMLU subjects: 35%|███▌ | 20/57 [00:44<01:13, 1.99s/it][=] high_school_chemistry: 7/20 = 35.0% (running avg 29.5%)
============================================================
[>] high_school_computer_science (shots=5)
high_school_computer_science: 0%| | 0/20 [00:00<?, ?it/s]
high_school_computer_science: 5%|▌ | 1/20 [00:00<00:02, 8.41it/s]
high_school_computer_science: 25%|██▌ | 5/20 [00:00<00:00, 20.00it/s]
high_school_computer_science: 45%|████▌ | 9/20 [00:00<00:00, 26.45it/s]
high_school_computer_science: 60%|██████ | 12/20 [00:00<00:00, 22.36it/s]
high_school_computer_science: 80%|████████ | 16/20 [00:00<00:00, 23.75it/s]
high_school_computer_science: 95%|█████████▌| 19/20 [00:00<00:00, 23.32it/s]

MMLU subjects: 37%|███▋ | 21/57 [00:46<01:12, 2.01s/it][=] high_school_computer_science: 7/20 = 35.0% (running avg 29.8%)
============================================================
[>] high_school_european_history (shots=5)
high_school_european_history: 0%| | 0/20 [00:00<?, ?it/s]
high_school_european_history: 15%|█▌ | 3/20 [00:00<00:00, 29.59it/s]
high_school_european_history: 40%|████ | 8/20 [00:00<00:00, 39.83it/s]
high_school_european_history: 65%|██████▌ | 13/20 [00:00<00:00, 42.99it/s]
high_school_european_history: 90%|█████████ | 18/20 [00:00<00:00, 44.36it/s]

MMLU subjects: 39%|███▊ | 22/57 [00:48<01:07, 1.94s/it][=] high_school_european_history: 4/20 = 20.0% (running avg 29.3%)
============================================================
[>] high_school_geography (shots=5)
high_school_geography: 0%| | 0/20 [00:00<?, ?it/s]
high_school_geography: 15%|█▌ | 3/20 [00:00<00:00, 22.09it/s]
high_school_geography: 45%|████▌ | 9/20 [00:00<00:00, 34.56it/s]
high_school_geography: 65%|██████▌ | 13/20 [00:00<00:00, 30.54it/s]
high_school_geography: 85%|████████▌ | 17/20 [00:00<00:00, 32.49it/s]

MMLU subjects: 40%|████ | 23/57 [00:50<01:05, 1.91s/it][=] high_school_geography: 5/20 = 25.0% (running avg 29.1%)
============================================================
[>] high_school_government_and_politics (shots=5)
high_school_government_and_politics: 0%| | 0/20 [00:00<?, ?it/s]
high_school_government_and_politics: 10%|█ | 2/20 [00:00<00:01, 14.50it/s]
high_school_government_and_politics: 30%|███ | 6/20 [00:00<00:00, 22.09it/s]
high_school_government_and_politics: 60%|██████ | 12/20 [00:00<00:00, 35.58it/s]
high_school_government_and_politics: 90%|█████████ | 18/20 [00:00<00:00, 38.62it/s]

MMLU subjects: 42%|████▏ | 24/57 [00:52<01:02, 1.89s/it][=] high_school_government_and_politics: 4/20 = 20.0% (running avg 28.7%)
============================================================
[>] high_school_macroeconomics (shots=5)
high_school_macroeconomics: 0%| | 0/20 [00:00<?, ?it/s]
high_school_macroeconomics: 20%|██ | 4/20 [00:00<00:00, 39.59it/s]
high_school_macroeconomics: 40%|████ | 8/20 [00:00<00:00, 38.27it/s]
high_school_macroeconomics: 60%|██████ | 12/20 [00:00<00:00, 32.61it/s]
high_school_macroeconomics: 90%|█████████ | 18/20 [00:00<00:00, 37.33it/s]

MMLU subjects: 44%|████▍ | 25/57 [00:54<00:58, 1.84s/it][=] high_school_macroeconomics: 0/20 = 0.0% (running avg 27.6%)
============================================================
[>] high_school_mathematics (shots=5)
high_school_mathematics: 0%| | 0/20 [00:00<?, ?it/s]
high_school_mathematics: 10%|█ | 2/20 [00:00<00:01, 12.89it/s]
high_school_mathematics: 30%|███ | 6/20 [00:00<00:00, 21.24it/s]
high_school_mathematics: 60%|██████ | 12/20 [00:00<00:00, 35.06it/s]
high_school_mathematics: 90%|█████████ | 18/20 [00:00<00:00, 43.42it/s]

MMLU subjects: 46%|████▌ | 26/57 [00:55<00:57, 1.85s/it][=] high_school_mathematics: 4/20 = 20.0% (running avg 27.3%)
============================================================
[>] high_school_microeconomics (shots=5)
high_school_microeconomics: 0%| | 0/20 [00:00<?, ?it/s]
high_school_microeconomics: 15%|█▌ | 3/20 [00:00<00:00, 18.39it/s]
high_school_microeconomics: 35%|███▌ | 7/20 [00:00<00:00, 27.64it/s]
high_school_microeconomics: 55%|█████▌ | 11/20 [00:00<00:00, 31.83it/s]
high_school_microeconomics: 75%|███████▌ | 15/20 [00:00<00:00, 33.78it/s]
high_school_microeconomics: 95%|█████████▌| 19/20 [00:00<00:00, 31.23it/s]

MMLU subjects: 47%|████▋ | 27/57 [00:57<00:55, 1.86s/it][=] high_school_microeconomics: 7/20 = 35.0% (running avg 27.6%)
============================================================
[>] high_school_physics (shots=5)
high_school_physics: 0%| | 0/20 [00:00<?, ?it/s]
high_school_physics: 10%|█ | 2/20 [00:00<00:00, 18.31it/s]
high_school_physics: 30%|███ | 6/20 [00:00<00:00, 29.27it/s]
high_school_physics: 50%|█████ | 10/20 [00:00<00:00, 28.30it/s]
high_school_physics: 70%|███████ | 14/20 [00:00<00:00, 31.21it/s]
high_school_physics: 90%|█████████ | 18/20 [00:00<00:00, 33.23it/s]

MMLU subjects: 49%|████▉ | 28/57 [00:59<00:55, 1.90s/it][=] high_school_physics: 4/20 = 20.0% (running avg 27.3%)
============================================================
[>] high_school_psychology (shots=5)
high_school_psychology: 0%| | 0/20 [00:00<?, ?it/s]
high_school_psychology: 20%|██ | 4/20 [00:00<00:00, 28.00it/s]
high_school_psychology: 40%|████ | 8/20 [00:00<00:00, 33.07it/s]
high_school_psychology: 60%|██████ | 12/20 [00:00<00:00, 34.99it/s]
high_school_psychology: 80%|████████ | 16/20 [00:00<00:00, 35.93it/s]
high_school_psychology: 100%|██████████| 20/20 [00:00<00:00, 36.42it/s]

MMLU subjects: 51%|█████ | 29/57 [01:01<00:52, 1.87s/it][=] high_school_psychology: 5/20 = 25.0% (running avg 27.2%)
============================================================
[>] high_school_statistics (shots=5)
high_school_statistics: 0%| | 0/20 [00:00<?, ?it/s]
high_school_statistics: 10%|█ | 2/20 [00:00<00:01, 12.71it/s]
high_school_statistics: 25%|██▌ | 5/20 [00:00<00:00, 18.45it/s]
high_school_statistics: 35%|███▌ | 7/20 [00:00<00:00, 18.23it/s]
high_school_statistics: 45%|████▌ | 9/20 [00:00<00:00, 18.11it/s]
high_school_statistics: 55%|█████▌ | 11/20 [00:00<00:00, 17.87it/s]
high_school_statistics: 65%|██████▌ | 13/20 [00:00<00:00, 17.72it/s]
high_school_statistics: 75%|███████▌ | 15/20 [00:00<00:00, 17.67it/s]
high_school_statistics: 85%|████████▌ | 17/20 [00:00<00:00, 17.60it/s]
high_school_statistics: 95%|█████████▌| 19/20 [00:01<00:00, 17.72it/s]

MMLU subjects: 53%|█████▎ | 30/57 [01:04<00:54, 2.04s/it][=] high_school_statistics: 8/20 = 40.0% (running avg 27.7%)
============================================================
[>] high_school_us_history (shots=5)
high_school_us_history: 0%| | 0/20 [00:00<?, ?it/s]
high_school_us_history: 20%|██ | 4/20 [00:00<00:00, 33.99it/s]
high_school_us_history: 45%|████▌ | 9/20 [00:00<00:00, 42.16it/s]
high_school_us_history: 70%|███████ | 14/20 [00:00<00:00, 44.91it/s]
high_school_us_history: 95%|█████████▌| 19/20 [00:00<00:00, 45.94it/s]

MMLU subjects: 54%|█████▍ | 31/57 [01:05<00:49, 1.92s/it][=] high_school_us_history: 4/20 = 20.0% (running avg 27.4%)
============================================================
[>] high_school_world_history (shots=5)
high_school_world_history: 0%| | 0/20 [00:00<?, ?it/s]
high_school_world_history: 15%|█▌ | 3/20 [00:00<00:00, 20.31it/s]
high_school_world_history: 40%|████ | 8/20 [00:00<00:00, 32.41it/s]
high_school_world_history: 65%|██████▌ | 13/20 [00:00<00:00, 39.35it/s]
high_school_world_history: 90%|█████████ | 18/20 [00:00<00:00, 43.13it/s]

MMLU subjects: 56%|█████▌ | 32/57 [01:07<00:46, 1.86s/it][=] high_school_world_history: 7/20 = 35.0% (running avg 27.7%)
============================================================
[>] human_aging (shots=5)
human_aging: 0%| | 0/20 [00:00<?, ?it/s]
human_aging: 5%|▌ | 1/20 [00:00<00:01, 9.91it/s]
human_aging: 20%|██ | 4/20 [00:00<00:00, 18.51it/s]
human_aging: 40%|████ | 8/20 [00:00<00:00, 23.30it/s]
human_aging: 55%|█████▌ | 11/20 [00:00<00:00, 23.62it/s]
human_aging: 70%|███████ | 14/20 [00:00<00:00, 23.48it/s]
human_aging: 90%|█████████ | 18/20 [00:00<00:00, 25.27it/s]

MMLU subjects: 58%|█████▊ | 33/57 [01:09<00:45, 1.90s/it][=] human_aging: 8/20 = 40.0% (running avg 28.0%)
============================================================
[>] human_sexuality (shots=5)
human_sexuality: 0%| | 0/20 [00:00<?, ?it/s]
human_sexuality: 10%|█ | 2/20 [00:00<00:01, 17.26it/s]
human_sexuality: 20%|██ | 4/20 [00:00<00:00, 17.97it/s]
human_sexuality: 50%|█████ | 10/20 [00:00<00:00, 31.12it/s]
human_sexuality: 70%|███████ | 14/20 [00:00<00:00, 33.83it/s]
human_sexuality: 90%|█████████ | 18/20 [00:00<00:00, 35.24it/s]

MMLU subjects: 60%|█████▉ | 34/57 [01:11<00:44, 1.94s/it][=] human_sexuality: 3/20 = 15.0% (running avg 27.6%)
============================================================
[>] international_law (shots=5)
international_law: 0%| | 0/20 [00:00<?, ?it/s]
international_law: 10%|█ | 2/20 [00:00<00:01, 13.41it/s]
international_law: 30%|███ | 6/20 [00:00<00:00, 21.45it/s]
international_law: 60%|██████ | 12/20 [00:00<00:00, 34.70it/s]
international_law: 80%|████████ | 16/20 [00:00<00:00, 28.56it/s]
international_law: 100%|██████████| 20/20 [00:00<00:00, 25.67it/s]

MMLU subjects: 61%|██████▏ | 35/57 [01:13<00:43, 1.99s/it][=] international_law: 7/20 = 35.0% (running avg 27.9%)
============================================================
[>] jurisprudence (shots=5)
jurisprudence: 0%| | 0/20 [00:00<?, ?it/s]
jurisprudence: 10%|█ | 2/20 [00:00<00:01, 17.11it/s]
jurisprudence: 40%|████ | 8/20 [00:00<00:00, 32.97it/s]
jurisprudence: 60%|██████ | 12/20 [00:00<00:00, 34.56it/s]
jurisprudence: 90%|█████████ | 18/20 [00:00<00:00, 37.91it/s]

MMLU subjects: 63%|██████▎ | 36/57 [01:15<00:40, 1.92s/it][=] jurisprudence: 8/20 = 40.0% (running avg 28.2%)
============================================================
[>] logical_fallacies (shots=5)
logical_fallacies: 0%| | 0/20 [00:00<?, ?it/s]
logical_fallacies: 20%|██ | 4/20 [00:00<00:00, 38.49it/s]
logical_fallacies: 50%|█████ | 10/20 [00:00<00:00, 49.03it/s]
logical_fallacies: 75%|███████▌ | 15/20 [00:00<00:00, 38.28it/s]
logical_fallacies: 100%|██████████| 20/20 [00:00<00:00, 38.86it/s]

MMLU subjects: 65%|██████▍ | 37/57 [01:17<00:37, 1.85s/it][=] logical_fallacies: 7/20 = 35.0% (running avg 28.4%)
============================================================
[>] machine_learning (shots=5)
Downloading data: 0%| | 0.00/5.25k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.25k/5.25k [00:00<00:00, 19.1kB/s]
Downloading data: 100%|██████████| 5.25k/5.25k [00:00<00:00, 19.0kB/s]
Generating test split: 0%| | 0/112 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 112/112 [00:00<00:00, 19509.20 examples/s]
Generating validation split: 0%| | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3115.91 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1538.18 examples/s]
machine_learning: 0%| | 0/20 [00:00<?, ?it/s]
machine_learning: 10%|█ | 2/20 [00:00<00:01, 13.64it/s]
machine_learning: 30%|███ | 6/20 [00:00<00:00, 25.60it/s]
machine_learning: 50%|█████ | 10/20 [00:00<00:00, 30.54it/s]
machine_learning: 80%|████████ | 16/20 [00:00<00:00, 35.58it/s]
machine_learning: 100%|██████████| 20/20 [00:00<00:00, 32.40it/s]

MMLU subjects: 67%|██████▋ | 38/57 [01:19<00:38, 2.03s/it][=] machine_learning: 10/20 = 50.0% (running avg 28.9%)
============================================================
[>] management (shots=5)
Downloading data: 0%| | 0.00/14.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 14.7k/14.7k [00:00<00:00, 71.1kB/s]
Downloading data: 100%|██████████| 14.7k/14.7k [00:00<00:00, 70.8kB/s]
Downloading data: 0%| | 0.00/4.50k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.50k/4.50k [00:00<00:00, 25.8kB/s]
Downloading data: 100%|██████████| 4.50k/4.50k [00:00<00:00, 25.7kB/s]
Downloading data: 0%| | 0.00/3.61k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 3.61k/3.61k [00:00<00:00, 19.8kB/s]
Downloading data: 100%|██████████| 3.61k/3.61k [00:00<00:00, 19.7kB/s]
Generating test split: 0%| | 0/103 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 103/103 [00:00<00:00, 23526.29 examples/s]
Generating validation split: 0%| | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3129.65 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1433.95 examples/s]
management: 0%| | 0/20 [00:00<?, ?it/s]
management: 10%|█ | 2/20 [00:00<00:01, 14.08it/s]
management: 20%|██ | 4/20 [00:00<00:00, 16.16it/s]
management: 35%|███▌ | 7/20 [00:00<00:00, 19.49it/s]
management: 45%|████▌ | 9/20 [00:00<00:00, 18.65it/s]
management: 60%|██████ | 12/20 [00:00<00:00, 20.56it/s]
management: 75%|███████▌ | 15/20 [00:00<00:00, 19.77it/s]

MMLU subjects: 68%|██████▊ | 39/57 [01:23<00:45, 2.55s/it][=] management: 4/20 = 20.0% (running avg 28.7%)
============================================================
[>] marketing (shots=5)
Downloading data: 0%| | 0.00/37.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 37.3k/37.3k [00:00<00:00, 216kB/s]
Downloading data: 100%|██████████| 37.3k/37.3k [00:00<00:00, 214kB/s]
Downloading data: 0%| | 0.00/8.21k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 8.21k/8.21k [00:00<00:00, 40.3kB/s]
Downloading data: 100%|██████████| 8.21k/8.21k [00:00<00:00, 40.1kB/s]
Downloading data: 0%| | 0.00/4.28k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.28k/4.28k [00:00<00:00, 13.5kB/s]
Downloading data: 100%|██████████| 4.28k/4.28k [00:00<00:00, 13.5kB/s]
Generating test split: 0%| | 0/234 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 234/234 [00:00<00:00, 47706.56 examples/s]
Generating validation split: 0%| | 0/25 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 25/25 [00:00<00:00, 6806.73 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1449.31 examples/s]
marketing: 0%| | 0/20 [00:00<?, ?it/s]
marketing: 25%|██▌ | 5/20 [00:00<00:00, 43.68it/s]
marketing: 55%|█████▌ | 11/20 [00:00<00:00, 52.50it/s]
marketing: 85%|████████▌ | 17/20 [00:00<00:00, 55.39it/s]

MMLU subjects: 70%|███████ | 40/57 [01:26<00:47, 2.77s/it][=] marketing: 7/20 = 35.0% (running avg 28.9%)
============================================================
[>] medical_genetics (shots=5)
Downloading data: 0%| | 0.00/16.4k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 16.4k/16.4k [00:00<00:00, 84.6kB/s]
Downloading data: 100%|██████████| 16.4k/16.4k [00:00<00:00, 84.2kB/s]
Downloading data: 0%| | 0.00/5.63k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.63k/5.63k [00:00<00:00, 25.0kB/s]
Downloading data: 100%|██████████| 5.63k/5.63k [00:00<00:00, 24.9kB/s]
Downloading data: 0%| | 0.00/3.77k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 3.77k/3.77k [00:00<00:00, 22.5kB/s]
Downloading data: 100%|██████████| 3.77k/3.77k [00:00<00:00, 22.4kB/s]
Generating test split: 0%| | 0/100 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 23050.69 examples/s]
Generating validation split: 0%| | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 3044.36 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1133.23 examples/s]
medical_genetics: 0%| | 0/20 [00:00<?, ?it/s]
medical_genetics: 15%|█▌ | 3/20 [00:00<00:00, 25.31it/s]
medical_genetics: 50%|█████ | 10/20 [00:00<00:00, 45.69it/s]
medical_genetics: 75%|███████▌ | 15/20 [00:00<00:00, 42.92it/s]

MMLU subjects: 72%|███████▏ | 41/57 [01:29<00:47, 2.94s/it][=] medical_genetics: 8/20 = 40.0% (running avg 29.1%)
============================================================
[>] miscellaneous (shots=5)
Downloading data: 0%| | 0.00/98.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 98.6k/98.6k [00:00<00:00, 387kB/s]
Downloading data: 100%|██████████| 98.6k/98.6k [00:00<00:00, 386kB/s]
Downloading data: 0%| | 0.00/13.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 13.2k/13.2k [00:00<00:00, 73.2kB/s]
Downloading data: 100%|██████████| 13.2k/13.2k [00:00<00:00, 72.8kB/s]
Downloading data: 0%| | 0.00/3.37k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 3.37k/3.37k [00:00<00:00, 18.5kB/s]
Downloading data: 100%|██████████| 3.37k/3.37k [00:00<00:00, 18.4kB/s]
Generating test split: 0%| | 0/783 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 783/783 [00:00<00:00, 137394.47 examples/s]
Generating validation split: 0%| | 0/86 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 86/86 [00:00<00:00, 21921.01 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1380.52 examples/s]
miscellaneous: 0%| | 0/20 [00:00<?, ?it/s]
miscellaneous: 15%|█▌ | 3/20 [00:00<00:00, 26.43it/s]
miscellaneous: 35%|███▌ | 7/20 [00:00<00:00, 33.14it/s]
miscellaneous: 55%|█████▌ | 11/20 [00:00<00:00, 23.92it/s]
miscellaneous: 70%|███████ | 14/20 [00:00<00:00, 23.88it/s]
miscellaneous: 90%|█████████ | 18/20 [00:00<00:00, 28.17it/s]

MMLU subjects: 74%|███████▎ | 42/57 [01:33<00:46, 3.09s/it][=] miscellaneous: 6/20 = 30.0% (running avg 29.2%)
============================================================
[>] moral_disputes (shots=5)
Downloading data: 0%| | 0.00/60.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 299kB/s]
Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 298kB/s]
Downloading data: 0%| | 0.00/10.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 51.6kB/s]
Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 51.4kB/s]
Downloading data: 0%| | 0.00/4.41k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 26.7kB/s]
Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 26.5kB/s]
Generating test split: 0%| | 0/346 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 346/346 [00:00<00:00, 62921.83 examples/s]
Generating validation split: 0%| | 0/38 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 38/38 [00:00<00:00, 9795.56 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1382.98 examples/s]
moral_disputes: 0%| | 0/20 [00:00<?, ?it/s]
moral_disputes: 25%|██▌ | 5/20 [00:00<00:00, 43.64it/s]
moral_disputes: 50%|█████ | 10/20 [00:00<00:00, 41.80it/s]
moral_disputes: 80%|████████ | 16/20 [00:00<00:00, 49.01it/s]

MMLU subjects: 75%|███████▌ | 43/57 [01:36<00:43, 3.11s/it][=] moral_disputes: 4/20 = 20.0% (running avg 29.0%)
============================================================
[>] moral_scenarios (shots=5)
Downloading data: 0%| | 0.00/89.8k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 502kB/s]
Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 499kB/s]
Downloading data: 0%| | 0.00/14.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 97.2kB/s]
Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 96.7kB/s]
Downloading data: 0%| | 0.00/5.14k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 27.5kB/s]
Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 27.4kB/s]
Generating test split: 0%| | 0/895 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 895/895 [00:00<00:00, 133752.66 examples/s]
Generating validation split: 0%| | 0/100 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 100/100 [00:00<00:00, 20927.57 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1409.47 examples/s]
moral_scenarios: 0%| | 0/20 [00:00<?, ?it/s]
moral_scenarios: 15%|█▌ | 3/20 [00:00<00:00, 24.61it/s]
moral_scenarios: 35%|███▌ | 7/20 [00:00<00:00, 31.12it/s]
moral_scenarios: 60%|██████ | 12/20 [00:00<00:00, 34.85it/s]
moral_scenarios: 80%|████████ | 16/20 [00:00<00:00, 35.45it/s]

MMLU subjects: 77%|███████▋ | 44/57 [01:39<00:40, 3.15s/it][=] moral_scenarios: 3/20 = 15.0% (running avg 28.6%)
============================================================
[>] nutrition (shots=5)
Downloading data: 0%| | 0.00/55.0k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 55.0k/55.0k [00:00<00:00, 173kB/s]
Downloading data: 100%|██████████| 55.0k/55.0k [00:00<00:00, 172kB/s]
Downloading data: 0%| | 0.00/9.02k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 9.02k/9.02k [00:00<00:00, 58.6kB/s]
Downloading data: 100%|██████████| 9.02k/9.02k [00:00<00:00, 58.3kB/s]
Downloading data: 0%| | 0.00/4.99k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.99k/4.99k [00:00<00:00, 28.0kB/s]
Downloading data: 100%|██████████| 4.99k/4.99k [00:00<00:00, 27.9kB/s]
Generating test split: 0%| | 0/306 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 306/306 [00:00<00:00, 60429.26 examples/s]
Generating validation split: 0%| | 0/33 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 33/33 [00:00<00:00, 9086.33 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1449.21 examples/s]
nutrition: 0%| | 0/20 [00:00<?, ?it/s]
nutrition: 10%|█ | 2/20 [00:00<00:00, 19.34it/s]
nutrition: 30%|███ | 6/20 [00:00<00:00, 24.76it/s]
nutrition: 45%|████▌ | 9/20 [00:00<00:00, 24.27it/s]
nutrition: 65%|██████▌ | 13/20 [00:00<00:00, 25.76it/s]
nutrition: 85%|████████▌ | 17/20 [00:00<00:00, 29.38it/s]

MMLU subjects: 79%|███████▉ | 45/57 [01:43<00:40, 3.34s/it][=] nutrition: 4/20 = 20.0% (running avg 28.4%)
============================================================
[>] philosophy (shots=5)
philosophy: 0%| | 0/20 [00:00<?, ?it/s]
philosophy: 10%|█ | 2/20 [00:00<00:01, 17.21it/s]
philosophy: 45%|████▌ | 9/20 [00:00<00:00, 43.07it/s]
philosophy: 80%|████████ | 16/20 [00:00<00:00, 51.44it/s]

MMLU subjects: 81%|████████ | 46/57 [01:45<00:31, 2.87s/it][=] philosophy: 3/20 = 15.0% (running avg 28.2%)
============================================================
[>] prehistory (shots=5)
Downloading data: 0%| | 0.00/54.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 336kB/s]
Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 334kB/s]
Downloading data: 0%| | 0.00/9.89k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 50.2kB/s]
Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 49.9kB/s]
Downloading data: 0%| | 0.00/4.62k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 21.8kB/s]
Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 21.7kB/s]
Generating test split: 0%| | 0/324 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 324/324 [00:00<00:00, 63532.23 examples/s]
Generating validation split: 0%| | 0/35 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 35/35 [00:00<00:00, 9094.33 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1395.22 examples/s]
prehistory: 0%| | 0/20 [00:00<?, ?it/s]
prehistory: 25%|██▌ | 5/20 [00:00<00:00, 43.02it/s]
prehistory: 50%|█████ | 10/20 [00:00<00:00, 41.15it/s]
prehistory: 75%|███████▌ | 15/20 [00:00<00:00, 35.37it/s]

MMLU subjects: 82%|████████▏ | 47/57 [01:48<00:30, 3.04s/it][=] prehistory: 5/20 = 25.0% (running avg 28.1%)
============================================================
[>] professional_accounting (shots=5)
Downloading data: 0%| | 0.00/69.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 69.5k/69.5k [00:00<00:00, 356kB/s]
Downloading data: 100%|██████████| 69.5k/69.5k [00:00<00:00, 354kB/s]
Downloading data: 0%| | 0.00/12.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 12.9k/12.9k [00:00<00:00, 61.9kB/s]
Downloading data: 100%|██████████| 12.9k/12.9k [00:00<00:00, 61.7kB/s]
Downloading data: 0%| | 0.00/4.89k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.89k/4.89k [00:00<00:00, 27.4kB/s]
Downloading data: 100%|██████████| 4.89k/4.89k [00:00<00:00, 27.3kB/s]
Generating test split: 0%| | 0/282 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 282/282 [00:00<00:00, 54398.83 examples/s]
Generating validation split: 0%| | 0/31 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 31/31 [00:00<00:00, 8148.87 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1535.93 examples/s]
professional_accounting: 0%| | 0/20 [00:00<?, ?it/s]
professional_accounting: 10%|█ | 2/20 [00:00<00:00, 18.76it/s]
professional_accounting: 30%|███ | 6/20 [00:00<00:00, 24.19it/s]
professional_accounting: 50%|█████ | 10/20 [00:00<00:00, 29.18it/s]
professional_accounting: 70%|███████ | 14/20 [00:00<00:00, 31.16it/s]
professional_accounting: 95%|█████████▌| 19/20 [00:00<00:00, 33.78it/s]

MMLU subjects: 84%|████████▍ | 48/57 [01:52<00:28, 3.19s/it][=] professional_accounting: 6/20 = 30.0% (running avg 28.1%)
============================================================
[>] professional_law (shots=5)
Downloading data: 0%| | 0.00/1.04M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 3.96MB/s]
Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 3.93MB/s]
Downloading data: 0%| | 0.00/116k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 603kB/s]
Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 600kB/s]
Downloading data: 0%| | 0.00/15.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 90.5kB/s]
Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 90.1kB/s]
Generating test split: 0%| | 0/1534 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1534/1534 [00:00<00:00, 72607.74 examples/s]
Generating validation split: 0%| | 0/170 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 170/170 [00:00<00:00, 29159.27 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1370.87 examples/s]
professional_law: 0%| | 0/20 [00:00<?, ?it/s]
professional_law: 20%|██ | 4/20 [00:00<00:00, 36.28it/s]
professional_law: 50%|█████ | 10/20 [00:00<00:00, 44.88it/s]
professional_law: 75%|███████▌ | 15/20 [00:00<00:00, 46.92it/s]

MMLU subjects: 86%|████████▌ | 49/57 [01:55<00:25, 3.17s/it][=] professional_law: 7/20 = 35.0% (running avg 28.3%)
============================================================
[>] professional_medicine (shots=5)
Downloading data: 0%| | 0.00/125k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 125k/125k [00:00<00:00, 666kB/s]
Downloading data: 100%|██████████| 125k/125k [00:00<00:00, 663kB/s]
Downloading data: 0%| | 0.00/19.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 19.9k/19.9k [00:00<00:00, 134kB/s]
Downloading data: 100%|██████████| 19.9k/19.9k [00:00<00:00, 134kB/s]
Downloading data: 0%| | 0.00/8.45k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 8.45k/8.45k [00:00<00:00, 52.0kB/s]
Downloading data: 100%|██████████| 8.45k/8.45k [00:00<00:00, 51.7kB/s]
Generating test split: 0%| | 0/272 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 272/272 [00:00<00:00, 37004.56 examples/s]
Generating validation split: 0%| | 0/31 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 31/31 [00:00<00:00, 7737.65 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1409.19 examples/s]
professional_medicine: 0%| | 0/20 [00:00<?, ?it/s]
professional_medicine: 20%|██ | 4/20 [00:00<00:00, 37.94it/s]
professional_medicine: 50%|█████ | 10/20 [00:00<00:00, 47.05it/s]
professional_medicine: 80%|████████ | 16/20 [00:00<00:00, 49.68it/s]

MMLU subjects: 88%|████████▊ | 50/57 [01:58<00:21, 3.13s/it][=] professional_medicine: 1/20 = 5.0% (running avg 27.8%)
============================================================
[>] professional_psychology (shots=5)
Downloading data: 0%| | 0.00/133k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 599kB/s]
Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 596kB/s]
Downloading data: 0%| | 0.00/22.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 81.8kB/s]
Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 81.6kB/s]
Downloading data: 0%| | 0.00/4.69k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 30.9kB/s]
Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 30.8kB/s]
Generating test split: 0%| | 0/612 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 612/612 [00:00<00:00, 95884.13 examples/s]
Generating validation split: 0%| | 0/69 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 69/69 [00:00<00:00, 16709.41 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1311.87 examples/s]
professional_psychology: 0%| | 0/20 [00:00<?, ?it/s]
professional_psychology: 20%|██ | 4/20 [00:00<00:00, 36.84it/s]
professional_psychology: 40%|████ | 8/20 [00:00<00:00, 36.65it/s]
professional_psychology: 60%|██████ | 12/20 [00:00<00:00, 27.62it/s]
professional_psychology: 80%|████████ | 16/20 [00:00<00:00, 30.48it/s]
professional_psychology: 100%|██████████| 20/20 [00:00<00:00, 29.12it/s]

MMLU subjects: 89%|████████▉ | 51/57 [02:01<00:19, 3.24s/it][=] professional_psychology: 9/20 = 45.0% (running avg 28.1%)
============================================================
[>] public_relations (shots=5)
Downloading data: 0%| | 0.00/20.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 125kB/s]
Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 125kB/s]
Downloading data: 0%| | 0.00/6.45k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 35.2kB/s]
Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 35.0kB/s]
Downloading data: 0%| | 0.00/4.43k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 29.9kB/s]
Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 29.7kB/s]
Generating test split: 0%| | 0/110 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 110/110 [00:00<00:00, 24437.15 examples/s]
Generating validation split: 0%| | 0/12 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 3341.85 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1466.54 examples/s]
public_relations: 0%| | 0/20 [00:00<?, ?it/s]
public_relations: 10%|█ | 2/20 [00:00<00:00, 19.12it/s]
public_relations: 30%|███ | 6/20 [00:00<00:00, 29.51it/s]
public_relations: 55%|█████▌ | 11/20 [00:00<00:00, 34.93it/s]
public_relations: 75%|███████▌ | 15/20 [00:00<00:00, 35.89it/s]

MMLU subjects: 91%|█████████ | 52/57 [02:05<00:16, 3.20s/it][=] public_relations: 9/20 = 45.0% (running avg 28.5%)
============================================================
[>] security_studies (shots=5)
Downloading data: 0%| | 0.00/114k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 675kB/s]
Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 671kB/s]
Downloading data: 0%| | 0.00/18.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 106kB/s]
Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 106kB/s]
Downloading data: 0%| | 0.00/7.49k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 45.0kB/s]
Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 44.8kB/s]
Generating test split: 0%| | 0/245 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 245/245 [00:00<00:00, 37800.42 examples/s]
Generating validation split: 0%| | 0/27 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 27/27 [00:00<00:00, 6508.78 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1451.42 examples/s]
security_studies: 0%| | 0/20 [00:00<?, ?it/s]
security_studies: 20%|██ | 4/20 [00:00<00:00, 37.50it/s]
security_studies: 50%|█████ | 10/20 [00:00<00:00, 46.09it/s]
security_studies: 80%|████████ | 16/20 [00:00<00:00, 48.87it/s]

MMLU subjects: 93%|█████████▎| 53/57 [02:08<00:12, 3.18s/it][=] security_studies: 5/20 = 25.0% (running avg 28.4%)
============================================================
[>] sociology (shots=5)
Downloading data: 0%| | 0.00/43.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 228kB/s]
Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 227kB/s]
Downloading data: 0%| | 0.00/8.36k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 51.2kB/s]
Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 50.9kB/s]
Downloading data: 0%| | 0.00/4.21k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 27.3kB/s]
Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 27.1kB/s]
Generating test split: 0%| | 0/201 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 201/201 [00:00<00:00, 40145.48 examples/s]
Generating validation split: 0%| | 0/22 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 5844.98 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1353.79 examples/s]
sociology: 0%| | 0/20 [00:00<?, ?it/s]
sociology: 10%|█ | 2/20 [00:00<00:01, 13.70it/s]
sociology: 30%|███ | 6/20 [00:00<00:00, 25.24it/s]
sociology: 60%|██████ | 12/20 [00:00<00:00, 38.57it/s]
sociology: 90%|█████████ | 18/20 [00:00<00:00, 45.52it/s]

MMLU subjects: 95%|█████████▍| 54/57 [02:11<00:09, 3.13s/it][=] sociology: 4/20 = 20.0% (running avg 28.2%)
============================================================
[>] us_foreign_policy (shots=5)
Downloading data: 0%| | 0.00/19.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 114kB/s]
Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 113kB/s]
Downloading data: 0%| | 0.00/5.27k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 34.0kB/s]
Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 33.8kB/s]
Downloading data: 0%| | 0.00/4.22k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 22.7kB/s]
Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 22.6kB/s]
Generating test split: 0%| | 0/100 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 23196.02 examples/s]
Generating validation split: 0%| | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 2971.81 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1376.35 examples/s]
us_foreign_policy: 0%| | 0/20 [00:00<?, ?it/s]
us_foreign_policy: 25%|██▌ | 5/20 [00:00<00:00, 44.09it/s]
us_foreign_policy: 55%|█████▌ | 11/20 [00:00<00:00, 52.64it/s]
us_foreign_policy: 85%|████████▌ | 17/20 [00:00<00:00, 55.54it/s]

MMLU subjects: 96%|█████████▋| 55/57 [02:14<00:06, 3.20s/it][=] us_foreign_policy: 5/20 = 25.0% (running avg 28.2%)
============================================================
[>] virology (shots=5)
Downloading data: 0%| | 0.00/27.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 176kB/s]
Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 175kB/s]
Downloading data: 0%| | 0.00/7.05k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 7.05k/7.05k [00:00<00:00, 42.4kB/s]
Downloading data: 100%|██████████| 7.05k/7.05k [00:00<00:00, 42.2kB/s]
Downloading data: 0%| | 0.00/3.87k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 3.87k/3.87k [00:00<00:00, 23.4kB/s]
Downloading data: 100%|██████████| 3.87k/3.87k [00:00<00:00, 23.3kB/s]
Generating test split: 0%| | 0/166 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 166/166 [00:00<00:00, 42164.02 examples/s]
Generating validation split: 0%| | 0/18 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 5863.43 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1653.77 examples/s]
virology: 0%| | 0/20 [00:00<?, ?it/s]
virology: 15%|█▌ | 3/20 [00:00<00:00, 27.33it/s]
virology: 45%|████▌ | 9/20 [00:00<00:00, 45.23it/s]
virology: 70%|███████ | 14/20 [00:00<00:00, 42.99it/s]
virology: 100%|██████████| 20/20 [00:00<00:00, 48.97it/s]

MMLU subjects: 98%|█████████▊| 56/57 [02:17<00:03, 3.17s/it][=] virology: 5/20 = 25.0% (running avg 28.1%)
============================================================
[>] world_religions (shots=5)
Downloading data: 0%| | 0.00/18.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 106kB/s]
Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 106kB/s]
Downloading data: 0%| | 0.00/4.94k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 30.3kB/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 30.2kB/s]
Downloading data: 0%| | 0.00/3.30k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 21.2kB/s]
Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 21.1kB/s]
Generating test split: 0%| | 0/171 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 171/171 [00:00<00:00, 39673.97 examples/s]
Generating validation split: 0%| | 0/19 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 19/19 [00:00<00:00, 5365.37 examples/s]
Generating dev split: 0%| | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 1399.69 examples/s]
world_religions: 0%| | 0/20 [00:00<?, ?it/s]
world_religions: 10%|█ | 2/20 [00:00<00:01, 13.20it/s]
world_religions: 25%|██▌ | 5/20 [00:00<00:00, 18.83it/s]
world_religions: 40%|████ | 8/20 [00:00<00:00, 20.93it/s]
world_religions: 60%|██████ | 12/20 [00:00<00:00, 23.89it/s]
world_religions: 75%|███████▌ | 15/20 [00:00<00:00, 23.91it/s]
world_religions: 95%|█████████▌| 19/20 [00:00<00:00, 28.23it/s]

MMLU subjects: 100%|██████████| 57/57 [02:20<00:00, 3.22s/it]
MMLU subjects: 100%|██████████| 57/57 [02:20<00:00, 2.47s/it]
[=] world_religions: 3/20 = 15.0% (running avg 27.9%)
============================================================
MMLU RESULT — /home/zeus/microllm2/microllm2-checkpoints/final_merged
Shots: 5 Subjects: 57/57
abstract_algebra 30.0% (6/20)
anatomy 25.0% (5/20)
astronomy 35.0% (7/20)
business_ethics 30.0% (6/20)
clinical_knowledge 45.0% (9/20)
college_biology 45.0% (9/20)
college_chemistry 15.0% (3/20)
college_computer_science 45.0% (9/20)
college_mathematics 35.0% (7/20)
college_medicine 30.0% (6/20)
college_physics 15.0% (3/20)
computer_security 30.0% (6/20)
conceptual_physics 5.0% (1/20)
econometrics 30.0% (6/20)
electrical_engineering 20.0% (4/20)
elementary_mathematics 30.0% (6/20)
formal_logic 10.0% (2/20)
global_facts 35.0% (7/20)
high_school_biology 45.0% (9/20)
high_school_chemistry 35.0% (7/20)
high_school_computer_science 35.0% (7/20)
high_school_european_history 20.0% (4/20)
high_school_geography 25.0% (5/20)
high_school_government_and_politics 20.0% (4/20)
high_school_macroeconomics 0.0% (0/20)
high_school_mathematics 20.0% (4/20)
high_school_microeconomics 35.0% (7/20)
high_school_physics 20.0% (4/20)
high_school_psychology 25.0% (5/20)
high_school_statistics 40.0% (8/20)
high_school_us_history 20.0% (4/20)
high_school_world_history 35.0% (7/20)
human_aging 40.0% (8/20)
human_sexuality 15.0% (3/20)
international_law 35.0% (7/20)
jurisprudence 40.0% (8/20)
logical_fallacies 35.0% (7/20)
machine_learning 50.0% (10/20)
management 20.0% (4/20)
marketing 35.0% (7/20)
medical_genetics 40.0% (8/20)
miscellaneous 30.0% (6/20)
moral_disputes 20.0% (4/20)
moral_scenarios 15.0% (3/20)
nutrition 20.0% (4/20)
philosophy 15.0% (3/20)
prehistory 25.0% (5/20)
professional_accounting 30.0% (6/20)
professional_law 35.0% (7/20)
professional_medicine 5.0% (1/20)
professional_psychology 45.0% (9/20)
public_relations 45.0% (9/20)
security_studies 25.0% (5/20)
sociology 20.0% (4/20)
us_foreign_policy 25.0% (5/20)
virology 25.0% (5/20)
world_religions 15.0% (3/20)
OVERALL: 318/1140 = 27.89%
============================================================
[+] Saved mmlu_results.json
Note: GPT2-XL base ~24-26% MMLU (random 25%). MicroLLM2 distilled should be 25-30% —
MMLU is knowledge-heavy; GPT2 1.5B 1024ctx cannot match 7B+ models. Use as sanity check, not SOTA claim.
[MMLU57-TOKEN] exit 0 at Sun Aug 9 23:12:32 UTC 2026
{
"model": "/home/zeus/microllm2/microllm2-checkpoints/final_merged",
"shots": 5,
"limit": 20,
"overall": {
"correct": 318,
"total": 1140,
"accuracy": 0.2789473684210526
},
"subjects": {
"abstract_algebra": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
"anatomy": {
"correct": 5,
"total": 20,
"accuracy": 0.25
},
"astronomy": {
"correct": 7,
"total": 20,
"accuracy": 0.35
},
"business_ethics": {
"correct": 6,
"total": 20,
"accuracy": 0.3
},
FINAL57 318 / 1140 27.89 57
'mmlu_results.json' -> 'mmlu57_results.json'
'mmlu_results.json' -> '/teamspace/studios/this_studio/microllm2/mmlu57_results.json'
'mmlu57_token.log' -> '/teamspace/studios/this_studio/microllm2/mmlu57_token.log'
Reference in New Issue View Git Blame Copy Permalink