Yaseal/llama3_1b_instruct_vallina_full_sft_30k
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Yaseal/llama3_1b_instruct_vallina_full_sft_30k
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-15 13:05:49 +08:00
commit 60b4306d96
253 changed files with 49440 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
.gitattributes vendored Normal file
View File

@@ -0,0 +1,57 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/predictions/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/reviews/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306/predictions/llama3_1b_instruct_vallina_full_sft_30k/agieval_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/predictions/llama3_1b_instruct_vallina_full_sft_30k/gsm8k_main.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_3/AIME24/20260315_211239/predictions/llama3_1b_instruct_vallina_full_sft_30k/aime24_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_2/MMLUProNoMath/20260315_200444/predictions/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/reviews/llama3_1b_instruct_vallina_full_sft_30k/gsm8k_main.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_2/MMLUProNoMath/20260315_200444/reviews/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343/predictions/llama3_1b_instruct_vallina_full_sft_30k/agieval_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_2/GSM8K/20260315_185404/predictions/llama3_1b_instruct_vallina_full_sft_30k/gsm8k_main.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_2/GSM8K/20260315_185404/reviews/llama3_1b_instruct_vallina_full_sft_30k/gsm8k_main.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_2/AIME24/20260315_170659/predictions/llama3_1b_instruct_vallina_full_sft_30k/aime24_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_2/GPQA/20260315_193901/predictions/llama3_1b_instruct_vallina_full_sft_30k/gpqa_extend_gpqa.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_1/AIME24/20260314_004254/predictions/llama3_1b_instruct_vallina_full_sft_30k/aime24_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_1/GPQA/20260315_152909/predictions/llama3_1b_instruct_vallina_full_sft_30k/gpqa_extend_gpqa.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_3/MMLUProNoMath/20260316_000556/predictions/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_3/MMLUProNoMath/20260316_000556/reviews/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_3/AGIEvalMath/20260315_220259/predictions/llama3_1b_instruct_vallina_full_sft_30k/agieval_math_default.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_3/GSM8K/20260315_225422/predictions/llama3_1b_instruct_vallina_full_sft_30k/gsm8k_main.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_3/GPQA/20260315_233853/predictions/llama3_1b_instruct_vallina_full_sft_30k/gpqa_extend_gpqa.jsonl filter=lfs diff=lfs merge=lfs -text
evalscope/version_20260314_002459/run_3/GSM8K/20260315_225422/reviews/llama3_1b_instruct_vallina_full_sft_30k/gsm8k_main.jsonl filter=lfs diff=lfs merge=lfs -text

66
README.md Normal file
View File

@@ -0,0 +1,66 @@
---
library_name: transformers
license: other
base_model: LLM-Research/Llama-3.2-1B-Instruct
tags:
- llama-factory
- full
- generated_from_trainer
model-index:
- name: llama3_1b_instruct_vallina_full_sft_30k
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# llama3_1b_instruct_vallina_full_sft_30k
This model is a fine-tuned version of [LLM-Research/Llama-3.2-1B-Instruct](https://huggingface.co/LLM-Research/Llama-3.2-1B-Instruct) on the deepmath_plain_30k_train dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5749
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 16
- total_eval_batch_size: 2
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 2.0
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 0.5103 | 1.7182 | 1000 | 0.5760 |
### Framework versions
- Transformers 4.52.4
- Pytorch 2.6.0+cu124
- Datasets 3.6.0
- Tokenizers 0.21.1

12
all_results.json Normal file
View File

@@ -0,0 +1,12 @@
{
"epoch": 2.0,
"eval_loss": 0.5749341249465942,
"eval_runtime": 9.7992,
"eval_samples_per_second": 11.021,
"eval_steps_per_second": 5.511,
"total_flos": 9.78689173969961e+17,
"train_loss": 0.5901876624507183,
"train_runtime": 6155.6604,
"train_samples_per_second": 3.026,
"train_steps_per_second": 0.189
}

93
chat_template.jinja Normal file
View File

@@ -0,0 +1,93 @@
{{- bos_token }}
{%- if custom_tools is defined %}
{%- set tools = custom_tools %}
{%- endif %}
{%- if not tools_in_user_message is defined %}
{%- set tools_in_user_message = true %}
{%- endif %}
{%- if not date_string is defined %}
{%- if strftime_now is defined %}
{%- set date_string = strftime_now("%d %b %Y") %}
{%- else %}
{%- set date_string = "26 Jul 2024" %}
{%- endif %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = none %}
{%- endif %}
{#- This block extracts the system message, so we can slot it into the right place. #}
{%- if messages[0]['role'] == 'system' %}
{%- set system_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{%- set system_message = "" %}
{%- endif %}
{#- System message #}
{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
{%- if tools is not none %}
{{- "Environment: ipython\n" }}
{%- endif %}
{{- "Cutting Knowledge Date: December 2023\n" }}
{{- "Today Date: " + date_string + "\n\n" }}
{%- if tools is not none and not tools_in_user_message %}
{{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{%- endif %}
{{- system_message }}
{{- "<|eot_id|>" }}
{#- Custom tools are passed in a user message with some extra guidance #}
{%- if tools_in_user_message and not tools is none %}
{#- Extract the first user message so we can plug it in here #}
{%- if messages | length != 0 %}
{%- set first_user_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
{%- endif %}
{{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
{{- "Given the following functions, please respond with a JSON for a function call " }}
{{- "with its proper arguments that best answers the given prompt.\n\n" }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{{- first_user_message + "<|eot_id|>"}}
{%- endif %}
{%- for message in messages %}
{%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
{{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
{%- elif 'tool_calls' in message %}
{%- if not message.tool_calls|length == 1 %}
{{- raise_exception("This model only supports single tool-calls at once!") }}
{%- endif %}
{%- set tool_call = message.tool_calls[0].function %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
{{- '{"name": "' + tool_call.name + '", ' }}
{{- '"parameters": ' }}
{{- tool_call.arguments | tojson }}
{{- "}" }}
{{- "<|eot_id|>" }}
{%- elif message.role == "tool" or message.role == "ipython" %}
{{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
{%- if message.content is mapping or message.content is iterable %}
{{- message.content | tojson }}
{%- else %}
{{- message.content }}
{%- endif %}
{{- "<|eot_id|>" }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
{%- endif %}

39
config.json Normal file
View File

@@ -0,0 +1,39 @@
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 128000,
"eos_token_id": [
128001,
128008,
128009
],
"head_dim": 64,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 8192,
"max_position_embeddings": 131072,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 16,
"num_key_value_heads": 8,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": {
"factor": 32.0,
"high_freq_factor": 4.0,
"low_freq_factor": 1.0,
"original_max_position_embeddings": 8192,
"rope_type": "llama3"
},
"rope_theta": 500000.0,
"tie_word_embeddings": true,
"torch_dtype": "bfloat16",
"transformers_version": "4.52.4",
"use_cache": false,
"vocab_size": 128256
}

7
eval_results.json Normal file
View File

@@ -0,0 +1,7 @@
{
"epoch": 2.0,
"eval_loss": 0.5749341249465942,
"eval_runtime": 9.7992,
"eval_samples_per_second": 11.021,
"eval_steps_per_second": 5.511
}

371
evalscope/version_20260314_002459/multi_run_summary.json Normal file
View File

@@ -0,0 +1,371 @@
{
"num_runs": 3,
"version": "version_20260314_002459",
"config_file": "eval_configs/llama3_1b_instruct_vallina_full_sft_30k.yaml",
"results": [
{
"benchmark": "AIME25",
"dataset": "aime25/AIME2025-I",
"run_count": 3,
"mean": 0.0,
"std": 0.0,
"std_percent": 0,
"min": 0.0,
"max": 0.0,
"range": 0.0,
"run_1": 0.0,
"run_2": 0.0,
"run_3": 0.0
},
{
"benchmark": "AIME25",
"dataset": "aime25/AIME2025-II",
"run_count": 3,
"mean": 0.0,
"std": 0.0,
"std_percent": 0,
"min": 0.0,
"max": 0.0,
"range": 0.0,
"run_1": 0.0,
"run_2": 0.0,
"run_3": 0.0
},
{
"benchmark": "AIME25",
"dataset": "aime25/all",
"run_count": 3,
"mean": 0.0,
"std": 0.0,
"std_percent": 0,
"min": 0.0,
"max": 0.0,
"range": 0.0,
"run_1": 0.0,
"run_2": 0.0,
"run_3": 0.0
},
{
"benchmark": "AIME24",
"dataset": "aime24/default",
"run_count": 3,
"mean": 0.0028,
"std": 0.0048,
"std_percent": 173.21,
"min": 0.0,
"max": 0.0083,
"range": 0.0083,
"run_1": 0.0,
"run_2": 0.0083,
"run_3": 0.0
},
{
"benchmark": "AIME24",
"dataset": "aime24/all",
"run_count": 3,
"mean": 0.0028,
"std": 0.0048,
"std_percent": 173.21,
"min": 0.0,
"max": 0.0083,
"range": 0.0083,
"run_1": 0.0,
"run_2": 0.0083,
"run_3": 0.0
},
{
"benchmark": "MATH500",
"dataset": "math_500/Level 1",
"run_count": 3,
"mean": 0.4341,
"std": 0.1147,
"std_percent": 26.43,
"min": 0.3023,
"max": 0.5116,
"range": 0.2093,
"run_1": 0.5116,
"run_2": 0.3023,
"run_3": 0.4884
},
{
"benchmark": "MATH500",
"dataset": "math_500/Level 2",
"run_count": 3,
"mean": 0.2852,
"std": 0.0632,
"std_percent": 22.16,
"min": 0.2333,
"max": 0.3556,
"range": 0.1223,
"run_1": 0.3556,
"run_2": 0.2333,
"run_3": 0.2667
},
{
"benchmark": "MATH500",
"dataset": "math_500/Level 3",
"run_count": 3,
"mean": 0.1714,
"std": 0.0343,
"std_percent": 20.01,
"min": 0.1429,
"max": 0.2095,
"range": 0.0666,
"run_1": 0.2095,
"run_2": 0.1619,
"run_3": 0.1429
},
{
"benchmark": "MATH500",
"dataset": "math_500/Level 4",
"run_count": 3,
"mean": 0.0833,
"std": 0.0197,
"std_percent": 23.61,
"min": 0.0625,
"max": 0.1016,
"range": 0.0391,
"run_1": 0.0625,
"run_2": 0.1016,
"run_3": 0.0859
},
{
"benchmark": "MATH500",
"dataset": "math_500/Level 5",
"run_count": 3,
"mean": 0.0498,
"std": 0.0188,
"std_percent": 37.71,
"min": 0.0299,
"max": 0.0672,
"range": 0.0373,
"run_1": 0.0299,
"run_2": 0.0522,
"run_3": 0.0672
},
{
"benchmark": "MATH500",
"dataset": "math_500/all",
"run_count": 3,
"mean": 0.1593,
"std": 0.017,
"std_percent": 10.68,
"min": 0.142,
"max": 0.176,
"range": 0.034,
"run_1": 0.176,
"run_2": 0.142,
"run_3": 0.16
},
{
"benchmark": "AMC",
"dataset": "amc/amc22",
"run_count": 3,
"mean": 0.0388,
"std": 0.0484,
"std_percent": 124.83,
"min": 0.0,
"max": 0.093,
"range": 0.093,
"run_1": 0.093,
"run_2": 0.0233,
"run_3": 0.0
},
{
"benchmark": "AMC",
"dataset": "amc/amc23",
"run_count": 3,
"mean": 0.0797,
"std": 0.0332,
"std_percent": 41.64,
"min": 0.0435,
"max": 0.1087,
"range": 0.0652,
"run_1": 0.0435,
"run_2": 0.087,
"run_3": 0.1087
},
{
"benchmark": "AMC",
"dataset": "amc/amc24",
"run_count": 3,
"mean": 0.0222,
"std": 0.0385,
"std_percent": 173.21,
"min": 0.0,
"max": 0.0667,
"range": 0.0667,
"run_1": 0.0,
"run_2": 0.0,
"run_3": 0.0667
},
{
"benchmark": "AMC",
"dataset": "amc/all",
"run_count": 3,
"mean": 0.0473,
"std": 0.0114,
"std_percent": 24.12,
"min": 0.0373,
"max": 0.0597,
"range": 0.0224,
"run_1": 0.0448,
"run_2": 0.0373,
"run_3": 0.0597
},
{
"benchmark": "AGIEvalMath",
"dataset": "agieval_math/default",
"run_count": 3,
"mean": 0.1553,
"std": 0.0078,
"std_percent": 5.0,
"min": 0.149,
"max": 0.164,
"range": 0.015,
"run_1": 0.153,
"run_2": 0.149,
"run_3": 0.164
},
{
"benchmark": "AGIEvalMath",
"dataset": "agieval_math/all",
"run_count": 3,
"mean": 0.1553,
"std": 0.0078,
"std_percent": 5.0,
"min": 0.149,
"max": 0.164,
"range": 0.015,
"run_1": 0.153,
"run_2": 0.149,
"run_3": 0.164
},
{
"benchmark": "GSM8K",
"dataset": "gsm8k/main",
"run_count": 3,
"mean": 0.2631,
"std": 0.0104,
"std_percent": 3.97,
"min": 0.254,
"max": 0.2745,
"range": 0.0205,
"run_1": 0.2745,
"run_2": 0.254,
"run_3": 0.2608
},
{
"benchmark": "GSM8K",
"dataset": "gsm8k/all",
"run_count": 3,
"mean": 0.2631,
"std": 0.0104,
"std_percent": 3.97,
"min": 0.254,
"max": 0.2745,
"range": 0.0205,
"run_1": 0.2745,
"run_2": 0.254,
"run_3": 0.2608
},
{
"benchmark": "GPQA",
"dataset": "gpqa_extend/gpqa",
"run_count": 3,
"mean": 0.2283,
"std": 0.0184,
"std_percent": 8.07,
"min": 0.2088,
"max": 0.2454,
"range": 0.0366,
"run_1": 0.2454,
"run_2": 0.2088,
"run_3": 0.2308
},
{
"benchmark": "GPQA",
"dataset": "gpqa_extend/all",
"run_count": 3,
"mean": 0.2283,
"std": 0.0184,
"std_percent": 8.07,
"min": 0.2088,
"max": 0.2454,
"range": 0.0366,
"run_1": 0.2454,
"run_2": 0.2088,
"run_3": 0.2308
},
{
"benchmark": "MMLUProNoMath",
"dataset": "mmlu_pro_no_math/default",
"run_count": 3,
"mean": 0.162,
"std": 0.0031,
"std_percent": 1.91,
"min": 0.1584,
"max": 0.1638,
"range": 0.0054,
"run_1": 0.1637,
"run_2": 0.1584,
"run_3": 0.1638
},
{
"benchmark": "MMLUProNoMath",
"dataset": "mmlu_pro_no_math/all",
"run_count": 3,
"mean": 0.162,
"std": 0.0031,
"std_percent": 1.91,
"min": 0.1584,
"max": 0.1638,
"range": 0.0054,
"run_1": 0.1637,
"run_2": 0.1584,
"run_3": 0.1638
},
{
"benchmark": "ARC",
"dataset": "arc/ARC-Easy",
"run_count": 3,
"mean": 0.4836,
"std": 0.0038,
"std_percent": 0.78,
"min": 0.4806,
"max": 0.4878,
"range": 0.0072,
"run_1": 0.4823,
"run_2": 0.4806,
"run_3": 0.4878
},
{
"benchmark": "ARC",
"dataset": "arc/ARC-Challenge",
"run_count": 3,
"mean": 0.3874,
"std": 0.0097,
"std_percent": 2.5,
"min": 0.3763,
"max": 0.3942,
"range": 0.0179,
"run_1": 0.3916,
"run_2": 0.3942,
"run_3": 0.3763
},
{
"benchmark": "ARC",
"dataset": "arc/all",
"run_count": 3,
"mean": 0.4518,
"std": 0.0007,
"std_percent": 0.15,
"min": 0.451,
"max": 0.4523,
"range": 0.0013,
"run_1": 0.4523,
"run_2": 0.4521,
"run_3": 0.451
}
]
}

BIN
evalscope/version_20260314_002459/multi_run_summary.xlsx Normal file
View File

Binary file not shown.

6
evalscope/version_20260314_002459/progress.json Normal file
View File

@@ -0,0 +1,6 @@
{
"version": "version_20260314_002459",
"num_runs": 3,
"config_file": "eval_configs/llama3_1b_instruct_vallina_full_sft_30k.yaml",
"created_at": "2026-03-15T15:29:02.493735"
}

78
evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306/configs/task_config_067c52.yaml Normal file
View File

@@ -0,0 +1,78 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
agieval_math:
aggregation: mean
dataset_id: hails/agieval-math
default_subset: default
description: AGIEval-Math is a subset of AGIEval containing 1000 competition-level
math problems drawn from the MATH dataset, covering algebra, geometry, number
theory and more. Answers are clean numerical or symbolic expressions.
eval_split: test
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc:
numeric: true
name: agieval_math
output_types:
- generation
pretty_name: AGIEval-Math
prompt_template: '{question}
Please reason step by step, and put your final answer within \boxed{{}}.'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: Please reason step by step to solve the problem. Put your final
answer in \boxed{}.
tags:
- Math
- Reasoning
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- agieval_math
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 1
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config: {}
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306

316
evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306/logs/eval_log.log Normal file
View File

@@ -0,0 +1,316 @@
2026-03-14 01:33:06 - evalscope - INFO: Running with native backend
2026-03-14 01:33:06 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306/configs/task_config_067c52.yaml
2026-03-14 01:33:06 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"agieval_math"
],
"dataset_args": {
"agieval_math": {
"name": "agieval_math",
"dataset_id": "hails/agieval-math",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "test",
"prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
"few_shot_prompt_template": null,
"system_prompt": "Please reason step by step to solve the problem. Put your final answer in \\boxed{}.",
"query_template": null,
"pretty_name": "AGIEval-Math",
"description": "AGIEval-Math is a subset of AGIEval containing 1000 competition-level math problems drawn from the MATH dataset, covering algebra, geometry, number theory and more. Answers are clean numerical or symbolic expressions.",
"tags": [
"Math",
"Reasoning"
],
"filters": null,
"metric_list": [
{
"acc": {
"numeric": true
}
}
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 1,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {},
"evalscope_version": "1.4.2"
}
2026-03-14 01:33:06 - evalscope - INFO: Start loading benchmark dataset: agieval_math
2026-03-14 01:33:06 - evalscope - INFO: Start evaluating 1 subsets of the agieval_math: ['default']
2026-03-14 01:33:06 - evalscope - INFO: Evaluating subset: default
2026-03-14 01:33:06 - evalscope - INFO: Getting predictions for subset: default
2026-03-14 01:33:06 - evalscope - INFO: Processing 1000 samples, if data is large, it may take a while.
2026-03-14 01:33:06 - evalscope - INFO: Loading model for prediction...
2026-03-14 01:33:06 - evalscope - INFO: Model loaded successfully.
2026-03-14 01:33:06 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=1, inflight=1)
2026-03-14 01:33:06 - evalscope - INFO: Dispatcher: Worker-1 <- 1 prompts (pending=0, inflight=2)
2026-03-14 01:33:15 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=990, inflight=2)
2026-03-14 01:33:28 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=982, inflight=2)
2026-03-14 01:34:07 - evalscope - INFO: Predicting[agieval_math@default]: 0%| 2/1000 [Elapsed: 01:00 < Remaining: 3:09:18, 11.38s/it]
2026-03-14 01:34:09 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=974, inflight=2)
2026-03-14 01:34:22 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=966, inflight=2)
2026-03-14 01:35:00 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=958, inflight=2)
2026-03-14 01:35:02 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=950, inflight=2)
2026-03-14 01:35:07 - evalscope - INFO: Predicting[agieval_math@default]: 3%| 34/1000 [Elapsed: 02:00 < Remaining: 47:25, 2.95s/it]
2026-03-14 01:35:55 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=942, inflight=2)
2026-03-14 01:35:55 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=934, inflight=2)
2026-03-14 01:36:07 - evalscope - INFO: Predicting[agieval_math@default]: 5%| 50/1000 [Elapsed: 03:00 < Remaining: 1:08:23, 4.32s/it]
2026-03-14 01:36:41 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=926, inflight=2)
2026-03-14 01:36:50 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=918, inflight=2)
2026-03-14 01:37:07 - evalscope - INFO: Predicting[agieval_math@default]: 7%| 66/1000 [Elapsed: 04:00 < Remaining: 45:11, 2.90s/it]
2026-03-14 01:37:28 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=910, inflight=2)
2026-03-14 01:37:29 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=902, inflight=2)
2026-03-14 01:38:07 - evalscope - INFO: Predicting[agieval_math@default]: 8%| 82/1000 [Elapsed: 05:00 < Remaining: 37:58, 2.48s/it]
2026-03-14 01:38:18 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=894, inflight=2)
2026-03-14 01:38:18 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=886, inflight=2)
2026-03-14 01:38:56 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=878, inflight=2)
2026-03-14 01:38:58 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=870, inflight=2)
2026-03-14 01:39:08 - evalscope - INFO: Predicting[agieval_math@default]: 11%| 114/1000 [Elapsed: 06:00 < Remaining: 34:33, 2.34s/it]
2026-03-14 01:39:49 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=862, inflight=2)
2026-03-14 01:39:52 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=854, inflight=2)
2026-03-14 01:40:08 - evalscope - INFO: Predicting[agieval_math@default]: 13%| 130/1000 [Elapsed: 07:01 < Remaining: 37:20, 2.58s/it]
2026-03-14 01:40:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=846, inflight=2)
2026-03-14 01:40:41 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=838, inflight=2)
2026-03-14 01:41:08 - evalscope - INFO: Predicting[agieval_math@default]: 15%| 146/1000 [Elapsed: 08:01 < Remaining: 37:51, 2.66s/it]
2026-03-14 01:41:23 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=830, inflight=2)
2026-03-14 01:41:33 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=822, inflight=2)
2026-03-14 01:42:08 - evalscope - INFO: Predicting[agieval_math@default]: 16%| 162/1000 [Elapsed: 09:01 < Remaining: 38:50, 2.78s/it]
2026-03-14 01:42:16 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=814, inflight=2)
2026-03-14 01:42:17 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=806, inflight=2)
2026-03-14 01:42:48 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=798, inflight=2)
2026-03-14 01:43:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=790, inflight=2)
2026-03-14 01:43:08 - evalscope - INFO: Predicting[agieval_math@default]: 19%| 194/1000 [Elapsed: 10:01 < Remaining: 37:02, 2.76s/it]
2026-03-14 01:43:42 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=782, inflight=2)
2026-03-14 01:43:58 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=774, inflight=2)
2026-03-14 01:44:08 - evalscope - INFO: Predicting[agieval_math@default]: 21%| 210/1000 [Elapsed: 11:01 < Remaining: 38:19, 2.91s/it]
2026-03-14 01:44:14 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=766, inflight=2)
2026-03-14 01:44:50 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=758, inflight=2)
2026-03-14 01:45:07 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=750, inflight=2)
2026-03-14 01:45:08 - evalscope - INFO: Predicting[agieval_math@default]: 23%| 234/1000 [Elapsed: 12:01 < Remaining: 36:22, 2.85s/it]
2026-03-14 01:45:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=742, inflight=2)
2026-03-14 01:45:45 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=734, inflight=2)
2026-03-14 01:46:09 - evalscope - INFO: Predicting[agieval_math@default]: 25%| 250/1000 [Elapsed: 13:01 < Remaining: 30:14, 2.42s/it]
2026-03-14 01:46:35 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=726, inflight=2)
2026-03-14 01:46:37 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=718, inflight=2)
2026-03-14 01:47:09 - evalscope - INFO: Predicting[agieval_math@default]: 27%| 266/1000 [Elapsed: 14:02 < Remaining: 31:30, 2.58s/it]
2026-03-14 01:47:18 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=710, inflight=2)
2026-03-14 01:47:21 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=702, inflight=2)
2026-03-14 01:47:59 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=694, inflight=2)
2026-03-14 01:48:09 - evalscope - INFO: Predicting[agieval_math@default]: 29%| 290/1000 [Elapsed: 15:02 < Remaining: 36:54, 3.12s/it]
2026-03-14 01:48:12 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=686, inflight=2)
2026-03-14 01:48:52 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=678, inflight=2)
2026-03-14 01:49:04 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=670, inflight=2)
2026-03-14 01:49:09 - evalscope - INFO: Predicting[agieval_math@default]: 31%| 314/1000 [Elapsed: 16:02 < Remaining: 31:56, 2.79s/it]
2026-03-14 01:49:39 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=662, inflight=2)
2026-03-14 01:49:58 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=654, inflight=2)
2026-03-14 01:50:09 - evalscope - INFO: Predicting[agieval_math@default]: 33%| 330/1000 [Elapsed: 17:02 < Remaining: 33:24, 2.99s/it]
2026-03-14 01:50:26 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=646, inflight=2)
2026-03-14 01:50:49 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=638, inflight=2)
2026-03-14 01:51:09 - evalscope - INFO: Predicting[agieval_math@default]: 35%| 346/1000 [Elapsed: 18:02 < Remaining: 33:50, 3.10s/it]
2026-03-14 01:51:17 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=630, inflight=2)
2026-03-14 01:51:41 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=622, inflight=2)
2026-03-14 01:51:54 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=614, inflight=2)
2026-03-14 01:52:09 - evalscope - INFO: Predicting[agieval_math@default]: 37%| 370/1000 [Elapsed: 19:02 < Remaining: 28:04, 2.67s/it]
2026-03-14 01:52:34 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=606, inflight=2)
2026-03-14 01:52:52 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=598, inflight=2)
2026-03-14 01:53:09 - evalscope - INFO: Predicting[agieval_math@default]: 39%| 386/1000 [Elapsed: 20:02 < Remaining: 31:05, 3.04s/it]
2026-03-14 01:53:28 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=590, inflight=2)
2026-03-14 01:53:46 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=582, inflight=2)
2026-03-14 01:54:10 - evalscope - INFO: Predicting[agieval_math@default]: 40%| 402/1000 [Elapsed: 21:03 < Remaining: 31:01, 3.11s/it]
2026-03-14 01:54:22 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=574, inflight=2)
2026-03-14 01:54:36 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=566, inflight=2)
2026-03-14 01:55:10 - evalscope - INFO: Predicting[agieval_math@default]: 42%| 418/1000 [Elapsed: 22:03 < Remaining: 29:04, 3.00s/it]
2026-03-14 01:55:13 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=558, inflight=2)
2026-03-14 01:55:29 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=550, inflight=2)
2026-03-14 01:56:04 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=542, inflight=2)
2026-03-14 01:56:10 - evalscope - INFO: Predicting[agieval_math@default]: 44%| 442/1000 [Elapsed: 23:03 < Remaining: 31:48, 3.42s/it]
2026-03-14 01:56:17 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=534, inflight=2)
2026-03-14 01:56:40 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=526, inflight=2)
2026-03-14 01:56:55 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=518, inflight=2)
2026-03-14 01:57:10 - evalscope - INFO: Predicting[agieval_math@default]: 47%| 466/1000 [Elapsed: 24:03 < Remaining: 22:52, 2.57s/it]
2026-03-14 01:57:35 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=510, inflight=2)
2026-03-14 01:57:51 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=502, inflight=2)
2026-03-14 01:58:10 - evalscope - INFO: Predicting[agieval_math@default]: 48%| 482/1000 [Elapsed: 25:03 < Remaining: 25:08, 2.91s/it]
2026-03-14 01:58:22 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=494, inflight=2)
2026-03-14 01:58:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=486, inflight=2)
2026-03-14 01:59:09 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=478, inflight=2)
2026-03-14 01:59:10 - evalscope - INFO: Predicting[agieval_math@default]: 51%| 506/1000 [Elapsed: 26:03 < Remaining: 25:30, 3.10s/it]
2026-03-14 01:59:33 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=470, inflight=2)
2026-03-14 02:00:03 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=462, inflight=2)
2026-03-14 02:00:11 - evalscope - INFO: Predicting[agieval_math@default]: 52%| 522/1000 [Elapsed: 27:03 < Remaining: 26:05, 3.28s/it]
2026-03-14 02:00:22 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=454, inflight=2)
2026-03-14 02:00:42 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=446, inflight=2)
2026-03-14 02:01:11 - evalscope - INFO: Predicting[agieval_math@default]: 54%| 538/1000 [Elapsed: 28:04 < Remaining: 21:42, 2.82s/it]
2026-03-14 02:01:17 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=438, inflight=2)
2026-03-14 02:01:28 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=430, inflight=2)
2026-03-14 02:02:00 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=422, inflight=2)
2026-03-14 02:02:11 - evalscope - INFO: Predicting[agieval_math@default]: 56%| 562/1000 [Elapsed: 29:04 < Remaining: 22:46, 3.12s/it]
2026-03-14 02:02:17 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=414, inflight=2)
2026-03-14 02:02:54 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=406, inflight=2)
2026-03-14 02:03:10 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=398, inflight=2)
2026-03-14 02:03:11 - evalscope - INFO: Predicting[agieval_math@default]: 58%| 579/1000 [Elapsed: 30:04 < Remaining: 20:44, 2.96s/it]
2026-03-14 02:03:47 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=390, inflight=2)
2026-03-14 02:04:00 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=382, inflight=2)
2026-03-14 02:04:11 - evalscope - INFO: Predicting[agieval_math@default]: 60%| 602/1000 [Elapsed: 31:04 < Remaining: 19:18, 2.91s/it]
2026-03-14 02:04:41 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=374, inflight=2)
2026-03-14 02:04:52 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=366, inflight=2)
2026-03-14 02:05:11 - evalscope - INFO: Predicting[agieval_math@default]: 62%| 618/1000 [Elapsed: 32:04 < Remaining: 18:38, 2.93s/it]
2026-03-14 02:05:22 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=358, inflight=2)
2026-03-14 02:05:44 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=350, inflight=2)
2026-03-14 02:06:09 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=342, inflight=2)
2026-03-14 02:06:11 - evalscope - INFO: Predicting[agieval_math@default]: 64%| 642/1000 [Elapsed: 33:04 < Remaining: 18:41, 3.13s/it]
2026-03-14 02:06:38 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=334, inflight=2)
2026-03-14 02:06:54 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=326, inflight=2)
2026-03-14 02:07:11 - evalscope - INFO: Predicting[agieval_math@default]: 66%| 658/1000 [Elapsed: 34:04 < Remaining: 16:35, 2.91s/it]
2026-03-14 02:07:32 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=318, inflight=2)
2026-03-14 02:07:42 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=310, inflight=2)
2026-03-14 02:08:11 - evalscope - INFO: Predicting[agieval_math@default]: 67%| 674/1000 [Elapsed: 35:04 < Remaining: 15:14, 2.81s/it]
2026-03-14 02:08:25 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=302, inflight=2)
2026-03-14 02:08:37 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=294, inflight=2)
2026-03-14 02:09:11 - evalscope - INFO: Predicting[agieval_math@default]: 69%| 690/1000 [Elapsed: 36:04 < Remaining: 15:09, 2.93s/it]
2026-03-14 02:09:18 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=286, inflight=2)
2026-03-14 02:09:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=278, inflight=2)
2026-03-14 02:10:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=270, inflight=2)
2026-03-14 02:10:11 - evalscope - INFO: Predicting[agieval_math@default]: 71%| 714/1000 [Elapsed: 37:04 < Remaining: 16:18, 3.42s/it]
2026-03-14 02:10:12 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=262, inflight=2)
2026-03-14 02:10:59 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=254, inflight=2)
2026-03-14 02:11:07 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=246, inflight=2)
2026-03-14 02:11:11 - evalscope - INFO: Predicting[agieval_math@default]: 74%| 738/1000 [Elapsed: 38:04 < Remaining: 12:18, 2.82s/it]
2026-03-14 02:11:50 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=238, inflight=2)
2026-03-14 02:11:50 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=230, inflight=2)
2026-03-14 02:12:12 - evalscope - INFO: Predicting[agieval_math@default]: 75%| 754/1000 [Elapsed: 39:04 < Remaining: 14:42, 3.59s/it]
2026-03-14 02:12:41 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=222, inflight=2)
2026-03-14 02:12:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=214, inflight=2)
2026-03-14 02:13:12 - evalscope - INFO: Predicting[agieval_math@default]: 77%| 770/1000 [Elapsed: 40:05 < Remaining: 09:55, 2.59s/it]
2026-03-14 02:13:35 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=206, inflight=2)
2026-03-14 02:13:36 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=198, inflight=2)
2026-03-14 02:14:12 - evalscope - INFO: Predicting[agieval_math@default]: 79%| 786/1000 [Elapsed: 41:05 < Remaining: 13:00, 3.65s/it]
2026-03-14 02:14:14 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=190, inflight=2)
2026-03-14 02:14:28 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=182, inflight=2)
2026-03-14 02:15:05 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=174, inflight=2)
2026-03-14 02:15:12 - evalscope - INFO: Predicting[agieval_math@default]: 81%| 810/1000 [Elapsed: 42:05 < Remaining: 10:20, 3.27s/it]
2026-03-14 02:15:22 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=166, inflight=2)
2026-03-14 02:15:47 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=158, inflight=2)
2026-03-14 02:16:12 - evalscope - INFO: Predicting[agieval_math@default]: 83%| 826/1000 [Elapsed: 43:05 < Remaining: 08:37, 2.98s/it]
2026-03-14 02:16:13 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=150, inflight=2)
2026-03-14 02:16:41 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=142, inflight=2)
2026-03-14 02:17:07 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=134, inflight=2)
2026-03-14 02:17:12 - evalscope - INFO: Predicting[agieval_math@default]: 85%| 850/1000 [Elapsed: 44:05 < Remaining: 07:59, 3.20s/it]
2026-03-14 02:17:19 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=126, inflight=2)
2026-03-14 02:18:00 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=118, inflight=2)
2026-03-14 02:18:12 - evalscope - INFO: Predicting[agieval_math@default]: 87%| 866/1000 [Elapsed: 45:05 < Remaining: 07:41, 3.44s/it]
2026-03-14 02:18:12 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=110, inflight=2)
2026-03-14 02:18:55 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=102, inflight=2)
2026-03-14 02:19:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=94, inflight=2)
2026-03-14 02:19:12 - evalscope - INFO: Predicting[agieval_math@default]: 89%| 890/1000 [Elapsed: 46:05 < Remaining: 05:20, 2.91s/it]
2026-03-14 02:19:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=86, inflight=2)
2026-03-14 02:19:58 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=78, inflight=2)
2026-03-14 02:20:12 - evalscope - INFO: Predicting[agieval_math@default]: 91%| 906/1000 [Elapsed: 47:05 < Remaining: 04:40, 2.99s/it]
2026-03-14 02:20:32 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=70, inflight=2)
2026-03-14 02:20:52 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=62, inflight=2)
2026-03-14 02:21:12 - evalscope - INFO: Predicting[agieval_math@default]: 92%| 922/1000 [Elapsed: 48:05 < Remaining: 03:59, 3.07s/it]
2026-03-14 02:21:23 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=54, inflight=2)
2026-03-14 02:21:44 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=46, inflight=2)
2026-03-14 02:22:12 - evalscope - INFO: Predicting[agieval_math@default]: 94%| 938/1000 [Elapsed: 49:05 < Remaining: 03:11, 3.10s/it]
2026-03-14 02:22:13 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=38, inflight=2)
2026-03-14 02:22:37 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=30, inflight=2)
2026-03-14 02:23:08 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=22, inflight=2)
2026-03-14 02:23:12 - evalscope - INFO: Predicting[agieval_math@default]: 96%| 962/1000 [Elapsed: 50:05 < Remaining: 02:08, 3.38s/it]
2026-03-14 02:23:29 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=14, inflight=2)
2026-03-14 02:23:41 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=6, inflight=2)
2026-03-14 02:24:12 - evalscope - INFO: Predicting[agieval_math@default]: 98%| 978/1000 [Elapsed: 51:05 < Remaining: 00:58, 2.66s/it]
2026-03-14 02:24:19 - evalscope - INFO: Dispatcher: Worker-0 <- 6 prompts (pending=0, inflight=2)
2026-03-14 02:25:06 - evalscope - INFO: Predicting[agieval_math@default]: 100%| 1000/1000 [Elapsed: 51:59 < Remaining: 00:00, 3.18s/it]
2026-03-14 02:25:06 - evalscope - INFO: Finished getting predictions for subset: default.
2026-03-14 02:25:06 - evalscope - INFO: Getting reviews for subset: default
2026-03-14 02:25:06 - evalscope - INFO: Reviewing 1000 samples, if data is large, it may take a while.
2026-03-14 02:25:16 - evalscope - INFO: Reviewing[agieval_math@default]: 100%| 1000/1000 [Elapsed: 00:10 < Remaining: 00:00, 104.78it/s]
2026-03-14 02:25:16 - evalscope - INFO: Finished reviewing subset: default. Total reviewed: 1000
2026-03-14 02:25:16 - evalscope - INFO: Aggregating scores for subset: default
2026-03-14 02:25:16 - evalscope - INFO: Evaluating [agieval_math] 100%| 1/1 [Elapsed: 52:09 < Remaining: 00:00, 3129.87s/subset]
2026-03-14 02:25:16 - evalscope - INFO: Generating report...
2026-03-14 02:25:16 - evalscope - INFO:
agieval_math report table:
+-----------------------------------------+--------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+==============+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | agieval_math | mean_acc | default | 1000 | 0.153 | default |
+-----------------------------------------+--------------+----------+----------+-------+---------+---------+
2026-03-14 02:25:16 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-14 02:25:16 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306/reports/llama3_1b_instruct_vallina_full_sft_30k/agieval_math.json
2026-03-14 02:25:16 - evalscope - INFO: Benchmark agieval_math evaluation finished.
2026-03-14 02:25:16 - evalscope - INFO: Overall report table:
+-----------------------------------------+--------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+==============+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | agieval_math | mean_acc | default | 1000 | 0.153 | default |
+-----------------------------------------+--------------+----------+----------+-------+---------+---------+
2026-03-14 02:25:17 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['agieval_math']
2026-03-14 02:25:17 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306
2026-03-14 02:25:17 - evalscope - INFO: [进度条] AGIEvalMath 评测完成 ✓
2026-03-14 02:25:17 - evalscope - INFO: [断点续传] AGIEvalMath 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AGIEvalMath_result.json
2026-03-14 02:25:17 - evalscope - INFO: 完成评测 AGIEvalMath (5/8)
2026-03-14 02:25:17 - evalscope - INFO:
==================================================
2026-03-14 02:25:17 - evalscope - INFO: 正在评估 GSM8K (repeat: 1次) (剩余: 2个)
2026-03-14 02:25:17 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-14 02:25:17 - evalscope - INFO: 开始创建 benchmark GSM8K 的 TaskConfig
2026-03-14 02:25:17 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-14 02:25:17 - evalscope - INFO: [GSM8K] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['gsm8k'], eval_batch_size=2048
2026-03-14 02:25:17 - evalscope - INFO: 开始评测 GSM8K...
2026-03-14 02:25:17 - evalscope - INFO: [进度条] GSM8K 开始评测
2026-03-14 02:25:17 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

3
evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306/predictions/llama3_1b_instruct_vallina_full_sft_30k/agieval_math_default.jsonl Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6a75f073819d4559ba0f841442f3ffb372bc1d4de0308d10c419febe4b4049bf
size 35184824

34
evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306/reports/llama3_1b_instruct_vallina_full_sft_30k/agieval_math.json Normal file
View File

@@ -0,0 +1,34 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@agieval_math",
"dataset_name": "agieval_math",
"dataset_pretty_name": "AGIEval-Math",
"dataset_description": "AGIEval-Math is a subset of AGIEval containing 1000 competition-level math problems drawn from the MATH dataset, covering algebra, geometry, number theory and more. Answers are clean numerical or symbolic expressions.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.153,
"metrics": [
{
"name": "mean_acc",
"num": 1000,
"score": 0.153,
"macro_score": 0.153,
"categories": [
{
"name": [
"default"
],
"num": 1000,
"score": 0.153,
"macro_score": 0.153,
"subsets": [
{
"name": "default",
"score": 0.153,
"num": 1000
}
]
}
]
}
],
"analysis": "N/A"
}

1000
evalscope/version_20260314_002459/run_1/AGIEvalMath/20260314_013306/reviews/llama3_1b_instruct_vallina_full_sft_30k/agieval_math_default.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

36
evalscope/version_20260314_002459/run_1/AGIEvalMath_result.json Normal file
View File

@@ -0,0 +1,36 @@
{
"agieval_math": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@agieval_math",
"dataset_name": "agieval_math",
"dataset_pretty_name": "AGIEval-Math",
"dataset_description": "AGIEval-Math is a subset of AGIEval containing 1000 competition-level math problems drawn from the MATH dataset, covering algebra, geometry, number theory and more. Answers are clean numerical or symbolic expressions.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.153,
"metrics": [
{
"name": "mean_acc",
"num": 1000,
"score": 0.153,
"macro_score": 0.153,
"categories": [
{
"name": [
"default"
],
"num": 1000,
"score": 0.153,
"macro_score": 0.153,
"subsets": [
{
"name": "default",
"score": 0.153,
"num": 1000
}
]
}
]
}
],
"analysis": "N/A"
}
}

79
evalscope/version_20260314_002459/run_1/AIME24/20260314_004254/configs/task_config_ebfcc5.yaml Normal file
View File

@@ -0,0 +1,79 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
aime24:
aggregation: mean
dataset_id: HuggingFaceH4/aime_2024
default_subset: default
description: The AIME 2024 benchmark is based on problems from the American Invitational
Mathematics Examination, a prestigious high school mathematics competition.
This benchmark tests a model's ability to solve challenging mathematics problems
by generating step-by-step solutions and providing the correct final answer.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc:
numeric: true
name: aime24
output_types:
- generation
pretty_name: AIME-2024
prompt_template: '{question}
Please reason step by step, and put your final answer within \boxed{{}}.'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Math
- Reasoning
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- aime24
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 14000
repetition_penalty: 1.0
temperature: 0.6
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 8
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME24/20260314_004254

185
evalscope/version_20260314_002459/run_1/AIME24/20260314_004254/logs/eval_log.log Normal file
View File

@@ -0,0 +1,185 @@
2026-03-14 00:42:54 - evalscope - INFO: Running with native backend
2026-03-14 00:42:54 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME24/20260314_004254/configs/task_config_ebfcc5.yaml
2026-03-14 00:42:54 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"aime24"
],
"dataset_args": {
"aime24": {
"name": "aime24",
"dataset_id": "HuggingFaceH4/aime_2024",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "AIME-2024",
"description": "The AIME 2024 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
"tags": [
"Math",
"Reasoning"
],
"filters": null,
"metric_list": [
{
"acc": {
"numeric": true
}
}
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 8,
"generation_config": {
"batch_size": 2048,
"max_tokens": 14000,
"top_p": 1.0,
"temperature": 0.6,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME24/20260314_004254",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 00:42:54 - evalscope - INFO: Start loading benchmark dataset: aime24
2026-03-14 00:42:54 - evalscope - INFO: Start evaluating 1 subsets of the aime24: ['default']
2026-03-14 00:42:54 - evalscope - INFO: Evaluating subset: default
2026-03-14 00:42:54 - evalscope - INFO: Getting predictions for subset: default
2026-03-14 00:42:54 - evalscope - INFO: Processing 240 samples, if data is large, it may take a while.
2026-03-14 00:42:54 - evalscope - INFO: Loading model for prediction...
2026-03-14 00:42:54 - evalscope - INFO: Model loaded successfully.
2026-03-14 00:42:54 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 00:42:54 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=208, inflight=2)
2026-03-14 00:43:10 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=223, inflight=2)
2026-03-14 00:43:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=215, inflight=2)
2026-03-14 00:43:47 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=207, inflight=2)
2026-03-14 00:43:54 - evalscope - INFO: Predicting[aime24@default]: 7%| 17/240 [Elapsed: 01:00 < Remaining: 16:00, 4.31s/it]
2026-03-14 00:44:19 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=199, inflight=2)
2026-03-14 00:44:30 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=191, inflight=2)
2026-03-14 00:44:54 - evalscope - INFO: Predicting[aime24@default]: 14%| 33/240 [Elapsed: 02:00 < Remaining: 09:52, 2.86s/it]
2026-03-14 00:45:05 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=183, inflight=2)
2026-03-14 00:45:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=175, inflight=2)
2026-03-14 00:45:54 - evalscope - INFO: Predicting[aime24@default]: 20%| 49/240 [Elapsed: 03:00 < Remaining: 09:37, 3.02s/it]
2026-03-14 00:46:08 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=167, inflight=2)
2026-03-14 00:46:25 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=159, inflight=2)
2026-03-14 00:46:54 - evalscope - INFO: Predicting[aime24@default]: 27%| 65/240 [Elapsed: 04:00 < Remaining: 09:39, 3.31s/it]
2026-03-14 00:47:07 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=151, inflight=2)
2026-03-14 00:47:27 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=143, inflight=2)
2026-03-14 00:47:54 - evalscope - INFO: Predicting[aime24@default]: 34%| 81/240 [Elapsed: 05:00 < Remaining: 09:20, 3.52s/it]
2026-03-14 00:48:14 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=135, inflight=2)
2026-03-14 00:48:28 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=127, inflight=2)
2026-03-14 00:48:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=119, inflight=2)
2026-03-14 00:48:54 - evalscope - INFO: Predicting[aime24@default]: 44%| 105/240 [Elapsed: 06:00 < Remaining: 06:50, 3.04s/it]
2026-03-14 00:49:25 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=111, inflight=2)
2026-03-14 00:49:36 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=103, inflight=2)
2026-03-14 00:49:54 - evalscope - INFO: Predicting[aime24@default]: 50%| 121/240 [Elapsed: 07:00 < Remaining: 05:50, 2.94s/it]
2026-03-14 00:50:27 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=95, inflight=2)
2026-03-14 00:50:35 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=87, inflight=2)
2026-03-14 00:50:54 - evalscope - INFO: Predicting[aime24@default]: 57%| 137/240 [Elapsed: 08:00 < Remaining: 05:16, 3.07s/it]
2026-03-14 00:51:15 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=79, inflight=2)
2026-03-14 00:51:32 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=71, inflight=2)
2026-03-14 00:51:54 - evalscope - INFO: Predicting[aime24@default]: 64%| 153/240 [Elapsed: 09:00 < Remaining: 04:37, 3.19s/it]
2026-03-14 00:52:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=63, inflight=2)
2026-03-14 00:52:40 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=55, inflight=2)
2026-03-14 00:52:54 - evalscope - INFO: Predicting[aime24@default]: 70%| 169/240 [Elapsed: 10:00 < Remaining: 04:11, 3.54s/it]
2026-03-14 00:53:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=47, inflight=2)
2026-03-14 00:53:41 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=39, inflight=2)
2026-03-14 00:53:54 - evalscope - INFO: Predicting[aime24@default]: 77%| 185/240 [Elapsed: 11:00 < Remaining: 03:14, 3.54s/it]
2026-03-14 00:54:25 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=31, inflight=2)
2026-03-14 00:54:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=23, inflight=2)
2026-03-14 00:54:55 - evalscope - INFO: Predicting[aime24@default]: 84%| 201/240 [Elapsed: 12:00 < Remaining: 02:19, 3.57s/it]
2026-03-14 00:55:22 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=15, inflight=2)
2026-03-14 00:55:41 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=7, inflight=2)
2026-03-14 00:55:55 - evalscope - INFO: Predicting[aime24@default]: 90%| 217/240 [Elapsed: 13:00 < Remaining: 01:20, 3.49s/it]
2026-03-14 00:56:22 - evalscope - INFO: Dispatcher: Worker-0 <- 7 prompts (pending=0, inflight=2)
2026-03-14 00:56:55 - evalscope - INFO: Predicting[aime24@default]: 97%| 233/240 [Elapsed: 14:00 < Remaining: 00:22, 3.16s/it]
2026-03-14 00:57:25 - evalscope - INFO: Predicting[aime24@default]: 100%| 240/240 [Elapsed: 14:31 < Remaining: 00:00, 4.19s/it]
2026-03-14 00:57:25 - evalscope - INFO: Finished getting predictions for subset: default.
2026-03-14 00:57:25 - evalscope - INFO: Getting reviews for subset: default
2026-03-14 00:57:25 - evalscope - INFO: Reviewing 240 samples, if data is large, it may take a while.
2026-03-14 00:57:28 - evalscope - INFO: Reviewing[aime24@default]: 100%| 240/240 [Elapsed: 00:02 < Remaining: 00:00, 16.24it/s]
2026-03-14 00:57:28 - evalscope - INFO: Finished reviewing subset: default. Total reviewed: 240
2026-03-14 00:57:28 - evalscope - INFO: Aggregating scores for subset: default
2026-03-14 00:57:28 - evalscope - INFO: Evaluating [aime24] 100%| 1/1 [Elapsed: 14:34 < Remaining: 00:00, 874.12s/subset]
2026-03-14 00:57:28 - evalscope - INFO: Generating report...
2026-03-14 00:57:28 - evalscope - INFO:
aime24 report table:
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | aime24 | mean_acc | default | 240 | 0 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
2026-03-14 00:57:28 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-14 00:57:28 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME24/20260314_004254/reports/llama3_1b_instruct_vallina_full_sft_30k/aime24.json
2026-03-14 00:57:28 - evalscope - INFO: Benchmark aime24 evaluation finished.
2026-03-14 00:57:28 - evalscope - INFO: Overall report table:
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | aime24 | mean_acc | default | 240 | 0 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
2026-03-14 00:57:28 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['aime24']
2026-03-14 00:57:28 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME24/20260314_004254
2026-03-14 00:57:28 - evalscope - INFO: [进度条] AIME24 评测完成 ✓
2026-03-14 00:57:28 - evalscope - INFO: [断点续传] AIME24 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME24_result.json
2026-03-14 00:57:28 - evalscope - INFO: 完成评测 AIME24 (2/8)
2026-03-14 00:57:28 - evalscope - INFO:
==================================================
2026-03-14 00:57:28 - evalscope - INFO: 正在评估 MATH500 (repeat: 1次) (剩余: 5个)
2026-03-14 00:57:28 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-14 00:57:28 - evalscope - INFO: 开始创建 benchmark MATH500 的 TaskConfig
2026-03-14 00:57:28 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-14 00:57:28 - evalscope - INFO: [MATH500] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['math_500'], eval_batch_size=2048
2026-03-14 00:57:28 - evalscope - INFO: 开始评测 MATH500...
2026-03-14 00:57:28 - evalscope - INFO: [进度条] MATH500 开始评测
2026-03-14 00:57:28 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

3
evalscope/version_20260314_002459/run_1/AIME24/20260314_004254/predictions/llama3_1b_instruct_vallina_full_sft_30k/aime24_default.jsonl Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:21bd5eb2fca8fef5466b5b4de1258aa19c816fe03adba3efc71bd955293785fb
size 11518718

34
evalscope/version_20260314_002459/run_1/AIME24/20260314_004254/reports/llama3_1b_instruct_vallina_full_sft_30k/aime24.json Normal file
View File

@@ -0,0 +1,34 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@aime24",
"dataset_name": "aime24",
"dataset_pretty_name": "AIME-2024",
"dataset_description": "The AIME 2024 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0,
"metrics": [
{
"name": "mean_acc",
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"categories": [
{
"name": [
"default"
],
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"subsets": [
{
"name": "default",
"score": 0.0,
"num": 240
}
]
}
]
}
],
"analysis": "N/A"
}

240
evalscope/version_20260314_002459/run_1/AIME24/20260314_004254/reviews/llama3_1b_instruct_vallina_full_sft_30k/aime24_default.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

36
evalscope/version_20260314_002459/run_1/AIME24_result.json Normal file
View File

@@ -0,0 +1,36 @@
{
"aime24": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@aime24",
"dataset_name": "aime24",
"dataset_pretty_name": "AIME-2024",
"dataset_description": "The AIME 2024 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0,
"metrics": [
{
"name": "mean_acc",
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"categories": [
{
"name": [
"default"
],
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"subsets": [
{
"name": "default",
"score": 0.0,
"num": 240
}
]
}
]
}
],
"analysis": "N/A"
}
}

86
evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/configs/task_config_49b228.yaml Normal file
View File

@@ -0,0 +1,86 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
aime25:
aggregation: mean
dataset_id: opencompass/AIME2025
default_subset: default
description: The AIME 2025 benchmark is based on problems from the American Invitational
Mathematics Examination, a prestigious high school mathematics competition.
This benchmark tests a model's ability to solve challenging mathematics problems
by generating step-by-step solutions and providing the correct final answer.
eval_split: test
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc:
numeric: true
name: aime25
output_types:
- generation
pretty_name: AIME-2025
prompt_template: '
Solve the following math problem step by step. Put your answer inside \boxed{{}}.
{question}
Remember to put your answer inside \boxed{{}}.'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- AIME2025-I
- AIME2025-II
system_prompt: null
tags:
- Math
- Reasoning
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- aime25
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 14000
repetition_penalty: 1.0
temperature: 0.6
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 8
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME25/20260314_002506

211
evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/logs/eval_log.log Normal file
View File

@@ -0,0 +1,211 @@
2026-03-14 00:25:06 - evalscope - INFO: Running with native backend
2026-03-14 00:25:06 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/configs/task_config_49b228.yaml
2026-03-14 00:25:06 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"aime25"
],
"dataset_args": {
"aime25": {
"name": "aime25",
"dataset_id": "opencompass/AIME2025",
"output_types": [
"generation"
],
"subset_list": [
"AIME2025-I",
"AIME2025-II"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "test",
"prompt_template": "\nSolve the following math problem step by step. Put your answer inside \\boxed{{}}.\n\n{question}\n\nRemember to put your answer inside \\boxed{{}}.",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "AIME-2025",
"description": "The AIME 2025 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
"tags": [
"Math",
"Reasoning"
],
"filters": null,
"metric_list": [
{
"acc": {
"numeric": true
}
}
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 8,
"generation_config": {
"batch_size": 2048,
"max_tokens": 14000,
"top_p": 1.0,
"temperature": 0.6,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME25/20260314_002506",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 00:25:06 - evalscope - INFO: Start loading benchmark dataset: aime25
2026-03-14 00:25:08 - evalscope - INFO: Start evaluating 2 subsets of the aime25: ['AIME2025-I', 'AIME2025-II']
2026-03-14 00:25:08 - evalscope - INFO: Evaluating subset: AIME2025-I
2026-03-14 00:25:08 - evalscope - INFO: Getting predictions for subset: AIME2025-I
2026-03-14 00:25:08 - evalscope - INFO: Processing 120 samples, if data is large, it may take a while.
2026-03-14 00:25:08 - evalscope - INFO: Loading model for prediction...
2026-03-14 00:25:08 - evalscope - INFO: Model loaded successfully.
2026-03-14 00:25:08 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=2, inflight=1)
2026-03-14 00:25:08 - evalscope - INFO: Dispatcher: Worker-1 <- 1 prompts (pending=1, inflight=2)
2026-03-14 00:26:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 0%| 0/120 [Elapsed: 01:00 < Remaining: ?, ?it/s]
2026-03-14 00:26:53 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=110, inflight=2)
2026-03-14 00:26:58 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=102, inflight=2)
2026-03-14 00:27:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 2%| 2/120 [Elapsed: 02:00 < Remaining: 1:30:30, 46.02s/it]
2026-03-14 00:27:33 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=94, inflight=2)
2026-03-14 00:28:02 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=86, inflight=2)
2026-03-14 00:28:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 15%| 18/120 [Elapsed: 03:00 < Remaining: 16:33, 9.74s/it]
2026-03-14 00:28:33 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=78, inflight=2)
2026-03-14 00:28:38 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=70, inflight=2)
2026-03-14 00:29:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 28%| 34/120 [Elapsed: 04:00 < Remaining: 05:41, 3.97s/it]
2026-03-14 00:29:36 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=62, inflight=2)
2026-03-14 00:29:42 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=54, inflight=2)
2026-03-14 00:30:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 42%| 50/120 [Elapsed: 05:00 < Remaining: 04:15, 3.64s/it]
2026-03-14 00:30:38 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=46, inflight=2)
2026-03-14 00:30:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=38, inflight=2)
2026-03-14 00:31:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 55%| 66/120 [Elapsed: 06:00 < Remaining: 03:03, 3.40s/it]
2026-03-14 00:31:19 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=30, inflight=2)
2026-03-14 00:31:47 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=22, inflight=2)
2026-03-14 00:32:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 68%| 82/120 [Elapsed: 07:00 < Remaining: 02:20, 3.70s/it]
2026-03-14 00:32:21 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=14, inflight=2)
2026-03-14 00:32:49 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=6, inflight=2)
2026-03-14 00:33:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 82%| 98/120 [Elapsed: 08:00 < Remaining: 01:22, 3.77s/it]
2026-03-14 00:33:24 - evalscope - INFO: Dispatcher: Worker-0 <- 6 prompts (pending=0, inflight=2)
2026-03-14 00:34:08 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 95%| 114/120 [Elapsed: 09:00 < Remaining: 00:23, 3.87s/it]
2026-03-14 00:34:27 - evalscope - INFO: Predicting[aime25@AIME2025-I]: 100%| 120/120 [Elapsed: 09:19 < Remaining: 00:00, 3.95s/it]
2026-03-14 00:34:27 - evalscope - INFO: Finished getting predictions for subset: AIME2025-I.
2026-03-14 00:34:27 - evalscope - INFO: Getting reviews for subset: AIME2025-I
2026-03-14 00:34:27 - evalscope - INFO: Reviewing 120 samples, if data is large, it may take a while.
2026-03-14 00:34:28 - evalscope - INFO: Reviewing[aime25@AIME2025-I]: 100%| 120/120 [Elapsed: 00:01 < Remaining: 00:00, 1.03s/it]
2026-03-14 00:34:28 - evalscope - INFO: Finished reviewing subset: AIME2025-I. Total reviewed: 120
2026-03-14 00:34:28 - evalscope - INFO: Aggregating scores for subset: AIME2025-I
2026-03-14 00:34:28 - evalscope - INFO: Evaluating [aime25] 50%| 1/2 [Elapsed: 09:20 < Remaining: 09:20, 560.79s/subset]
2026-03-14 00:34:28 - evalscope - INFO: Evaluating subset: AIME2025-II
2026-03-14 00:34:28 - evalscope - INFO: Getting predictions for subset: AIME2025-II
2026-03-14 00:34:28 - evalscope - INFO: Processing 120 samples, if data is large, it may take a while.
2026-03-14 00:34:28 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=1, inflight=1)
2026-03-14 00:34:28 - evalscope - INFO: Dispatcher: Worker-1 <- 1 prompts (pending=0, inflight=2)
2026-03-14 00:34:47 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=110, inflight=2)
2026-03-14 00:34:58 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=102, inflight=2)
2026-03-14 00:35:28 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 2%| 2/120 [Elapsed: 01:00 < Remaining: 28:07, 14.30s/it]
2026-03-14 00:35:49 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=94, inflight=2)
2026-03-14 00:35:59 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=86, inflight=2)
2026-03-14 00:36:28 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 15%| 18/120 [Elapsed: 02:00 < Remaining: 10:16, 6.05s/it]
2026-03-14 00:36:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=78, inflight=2)
2026-03-14 00:37:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=70, inflight=2)
2026-03-14 00:37:29 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 28%| 34/120 [Elapsed: 03:00 < Remaining: 06:43, 4.69s/it]
2026-03-14 00:37:33 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=62, inflight=2)
2026-03-14 00:38:07 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=54, inflight=2)
2026-03-14 00:38:29 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 42%| 50/120 [Elapsed: 04:00 < Remaining: 04:54, 4.21s/it]
2026-03-14 00:38:34 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=46, inflight=2)
2026-03-14 00:39:10 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=38, inflight=2)
2026-03-14 00:39:29 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 55%| 66/120 [Elapsed: 05:00 < Remaining: 03:42, 4.12s/it]
2026-03-14 00:39:33 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=30, inflight=2)
2026-03-14 00:40:14 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=22, inflight=2)
2026-03-14 00:40:29 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 68%| 82/120 [Elapsed: 06:00 < Remaining: 02:38, 4.16s/it]
2026-03-14 00:40:40 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=14, inflight=2)
2026-03-14 00:41:22 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=6, inflight=2)
2026-03-14 00:41:29 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 82%| 98/120 [Elapsed: 07:00 < Remaining: 01:33, 4.26s/it]
2026-03-14 00:41:49 - evalscope - INFO: Dispatcher: Worker-1 <- 6 prompts (pending=0, inflight=2)
2026-03-14 00:42:29 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 95%| 114/120 [Elapsed: 08:00 < Remaining: 00:25, 4.17s/it]
2026-03-14 00:42:48 - evalscope - INFO: Predicting[aime25@AIME2025-II]: 100%| 120/120 [Elapsed: 08:19 < Remaining: 00:00, 3.75s/it]
2026-03-14 00:42:48 - evalscope - INFO: Finished getting predictions for subset: AIME2025-II.
2026-03-14 00:42:48 - evalscope - INFO: Getting reviews for subset: AIME2025-II
2026-03-14 00:42:48 - evalscope - INFO: Reviewing 120 samples, if data is large, it may take a while.
2026-03-14 00:42:53 - evalscope - INFO: Reviewing[aime25@AIME2025-II]: 100%| 120/120 [Elapsed: 00:05 < Remaining: 00:00, 5.07s/it]
2026-03-14 00:42:53 - evalscope - INFO: Finished reviewing subset: AIME2025-II. Total reviewed: 120
2026-03-14 00:42:53 - evalscope - INFO: Aggregating scores for subset: AIME2025-II
2026-03-14 00:42:53 - evalscope - INFO: Evaluating [aime25] 100%| 2/2 [Elapsed: 17:45 < Remaining: 00:00, 527.92s/subset]
2026-03-14 00:42:53 - evalscope - INFO: Generating report...
2026-03-14 00:42:53 - evalscope - INFO:
aime25 report table:
+-----------------------------------------+-----------+----------+-------------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+=============+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | aime25 | mean_acc | AIME2025-I | 120 | 0 | default |
+-----------------------------------------+-----------+----------+-------------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | aime25 | mean_acc | AIME2025-II | 120 | 0 | default |
+-----------------------------------------+-----------+----------+-------------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | aime25 | mean_acc | OVERALL | 240 | 0 | - |
+-----------------------------------------+-----------+----------+-------------+-------+---------+---------+
2026-03-14 00:42:53 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-14 00:42:53 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/reports/llama3_1b_instruct_vallina_full_sft_30k/aime25.json
2026-03-14 00:42:53 - evalscope - INFO: Benchmark aime25 evaluation finished.
2026-03-14 00:42:53 - evalscope - INFO: Overall report table:
+-----------------------------------------+-----------+----------+-------------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+=============+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | aime25 | mean_acc | AIME2025-I | 120 | 0 | default |
+-----------------------------------------+-----------+----------+-------------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | aime25 | mean_acc | AIME2025-II | 120 | 0 | default |
+-----------------------------------------+-----------+----------+-------------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | aime25 | mean_acc | OVERALL | 240 | 0 | - |
+-----------------------------------------+-----------+----------+-------------+-------+---------+---------+
2026-03-14 00:42:54 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['aime25']
2026-03-14 00:42:54 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME25/20260314_002506
2026-03-14 00:42:54 - evalscope - INFO: [进度条] AIME25 评测完成 ✓
2026-03-14 00:42:54 - evalscope - INFO: [断点续传] AIME25 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AIME25_result.json
2026-03-14 00:42:54 - evalscope - INFO: 完成评测 AIME25 (1/8)
2026-03-14 00:42:54 - evalscope - INFO:
==================================================
2026-03-14 00:42:54 - evalscope - INFO: 正在评估 AIME24 (repeat: 8次) (剩余: 6个)
2026-03-14 00:42:54 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-14 00:42:54 - evalscope - INFO: 开始创建 benchmark AIME24 的 TaskConfig
2026-03-14 00:42:54 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-14 00:42:54 - evalscope - INFO: [AIME24] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['aime24'], eval_batch_size=2048
2026-03-14 00:42:54 - evalscope - INFO: 开始评测 AIME24...
2026-03-14 00:42:54 - evalscope - INFO: [进度条] AIME24 开始评测
2026-03-14 00:42:54 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

120
evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/predictions/llama3_1b_instruct_vallina_full_sft_30k/aime25_AIME2025-I.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

120
evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/predictions/llama3_1b_instruct_vallina_full_sft_30k/aime25_AIME2025-II.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

39
evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/reports/llama3_1b_instruct_vallina_full_sft_30k/aime25.json Normal file
View File

@@ -0,0 +1,39 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@aime25",
"dataset_name": "aime25",
"dataset_pretty_name": "AIME-2025",
"dataset_description": "The AIME 2025 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0,
"metrics": [
{
"name": "mean_acc",
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"categories": [
{
"name": [
"default"
],
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"subsets": [
{
"name": "AIME2025-I",
"score": 0.0,
"num": 120
},
{
"name": "AIME2025-II",
"score": 0.0,
"num": 120
}
]
}
]
}
],
"analysis": "N/A"
}

120
evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/reviews/llama3_1b_instruct_vallina_full_sft_30k/aime25_AIME2025-I.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

120
evalscope/version_20260314_002459/run_1/AIME25/20260314_002506/reviews/llama3_1b_instruct_vallina_full_sft_30k/aime25_AIME2025-II.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

41
evalscope/version_20260314_002459/run_1/AIME25_result.json Normal file
View File

@@ -0,0 +1,41 @@
{
"aime25": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@aime25",
"dataset_name": "aime25",
"dataset_pretty_name": "AIME-2025",
"dataset_description": "The AIME 2025 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0,
"metrics": [
{
"name": "mean_acc",
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"categories": [
{
"name": [
"default"
],
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"subsets": [
{
"name": "AIME2025-I",
"score": 0.0,
"num": 120
},
{
"name": "AIME2025-II",
"score": 0.0,
"num": 120
}
]
}
]
}
],
"analysis": "N/A"
}
}

78
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/configs/task_config_c6fe83.yaml Normal file
View File

@@ -0,0 +1,78 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
amc:
aggregation: mean
dataset_id: evalscope/amc_22-24
default_subset: default
description: AMC (American Mathematics Competitions) is a series of mathematics
competitions for high school students.
eval_split: null
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc:
numeric: true
name: amc
output_types:
- generation
pretty_name: AMC
prompt_template: '{question}
Please reason step by step, and put your final answer within \boxed{{}}.'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- amc22
- amc23
- amc24
system_prompt: null
tags:
- Math
- Reasoning
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- amc
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 1
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config: {}
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AMC/20260314_012458

202
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/logs/eval_log.log Normal file
View File

@@ -0,0 +1,202 @@
2026-03-14 01:24:58 - evalscope - INFO: Running with native backend
2026-03-14 01:24:58 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AMC/20260314_012458/configs/task_config_c6fe83.yaml
2026-03-14 01:24:58 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"amc"
],
"dataset_args": {
"amc": {
"name": "amc",
"dataset_id": "evalscope/amc_22-24",
"output_types": [
"generation"
],
"subset_list": [
"amc22",
"amc23",
"amc24"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": null,
"prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "AMC",
"description": "AMC (American Mathematics Competitions) is a series of mathematics competitions for high school students.",
"tags": [
"Math",
"Reasoning"
],
"filters": null,
"metric_list": [
{
"acc": {
"numeric": true
}
}
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AMC/20260314_012458",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 1,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {},
"evalscope_version": "1.4.2"
}
2026-03-14 01:24:58 - evalscope - INFO: Start loading benchmark dataset: amc
2026-03-14 01:24:58 - evalscope - INFO: Start evaluating 3 subsets of the amc: ['amc22', 'amc23', 'amc24']
2026-03-14 01:24:58 - evalscope - INFO: Evaluating subset: amc22
2026-03-14 01:24:58 - evalscope - INFO: Getting predictions for subset: amc22
2026-03-14 01:24:58 - evalscope - INFO: Processing 43 samples, if data is large, it may take a while.
2026-03-14 01:24:58 - evalscope - INFO: Loading model for prediction...
2026-03-14 01:24:58 - evalscope - INFO: Model loaded successfully.
2026-03-14 01:24:58 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 01:24:58 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=34, inflight=2)
2026-03-14 01:25:25 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=26, inflight=2)
2026-03-14 01:25:53 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=18, inflight=2)
2026-03-14 01:25:58 - evalscope - INFO: Predicting[amc@amc22]: 21%| 9/43 [Elapsed: 01:00 < Remaining: 15:52, 28.01s/it]
2026-03-14 01:26:16 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=10, inflight=2)
2026-03-14 01:26:48 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=2, inflight=2)
2026-03-14 01:26:58 - evalscope - INFO: Predicting[amc@amc22]: 58%| 25/43 [Elapsed: 02:00 < Remaining: 01:29, 4.97s/it]
2026-03-14 01:27:09 - evalscope - INFO: Dispatcher: Worker-0 <- 2 prompts (pending=0, inflight=2)
2026-03-14 01:27:44 - evalscope - INFO: Predicting[amc@amc22]: 100%| 43/43 [Elapsed: 02:45 < Remaining: 00:00, 4.05s/it]
2026-03-14 01:27:44 - evalscope - INFO: Finished getting predictions for subset: amc22.
2026-03-14 01:27:44 - evalscope - INFO: Getting reviews for subset: amc22
2026-03-14 01:27:44 - evalscope - INFO: Reviewing 43 samples, if data is large, it may take a while.
2026-03-14 01:27:44 - evalscope - INFO: Reviewing[amc@amc22]: 100%| 43/43 [Elapsed: 00:00 < Remaining: 00:00, 164.55it/s]
2026-03-14 01:27:44 - evalscope - INFO: Finished reviewing subset: amc22. Total reviewed: 43
2026-03-14 01:27:44 - evalscope - INFO: Aggregating scores for subset: amc22
2026-03-14 01:27:44 - evalscope - INFO: Evaluating [amc] 33%| 1/3 [Elapsed: 02:45 < Remaining: 05:31, 165.84s/subset]
2026-03-14 01:27:44 - evalscope - INFO: Evaluating subset: amc23
2026-03-14 01:27:44 - evalscope - INFO: Getting predictions for subset: amc23
2026-03-14 01:27:44 - evalscope - INFO: Processing 46 samples, if data is large, it may take a while.
2026-03-14 01:27:44 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 01:27:44 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=37, inflight=2)
2026-03-14 01:27:46 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=29, inflight=2)
2026-03-14 01:28:35 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=21, inflight=2)
2026-03-14 01:28:42 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=13, inflight=2)
2026-03-14 01:28:44 - evalscope - INFO: Predicting[amc@amc23]: 37%| 17/46 [Elapsed: 01:00 < Remaining: 02:15, 4.66s/it]
2026-03-14 01:29:08 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=5, inflight=2)
2026-03-14 01:29:14 - evalscope - INFO: Dispatcher: Worker-0 <- 5 prompts (pending=0, inflight=2)
2026-03-14 01:29:44 - evalscope - INFO: Predicting[amc@amc23]: 72%| 33/46 [Elapsed: 02:00 < Remaining: 00:31, 2.46s/it]
2026-03-14 01:30:03 - evalscope - INFO: Predicting[amc@amc23]: 100%| 46/46 [Elapsed: 02:18 < Remaining: 00:00, 3.08s/it]
2026-03-14 01:30:03 - evalscope - INFO: Finished getting predictions for subset: amc23.
2026-03-14 01:30:03 - evalscope - INFO: Getting reviews for subset: amc23
2026-03-14 01:30:03 - evalscope - INFO: Reviewing 46 samples, if data is large, it may take a while.
2026-03-14 01:30:03 - evalscope - INFO: Reviewing[amc@amc23]: 100%| 46/46 [Elapsed: 00:00 < Remaining: 00:00, 121.40it/s]
2026-03-14 01:30:03 - evalscope - INFO: Finished reviewing subset: amc23. Total reviewed: 46
2026-03-14 01:30:03 - evalscope - INFO: Aggregating scores for subset: amc23
2026-03-14 01:30:03 - evalscope - INFO: Evaluating [amc] 67%| 2/3 [Elapsed: 05:04 < Remaining: 02:30, 150.10s/subset]
2026-03-14 01:30:03 - evalscope - INFO: Evaluating subset: amc24
2026-03-14 01:30:03 - evalscope - INFO: Getting predictions for subset: amc24
2026-03-14 01:30:03 - evalscope - INFO: Processing 45 samples, if data is large, it may take a while.
2026-03-14 01:30:03 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 01:30:03 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=36, inflight=2)
2026-03-14 01:30:39 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=28, inflight=2)
2026-03-14 01:30:58 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=20, inflight=2)
2026-03-14 01:31:03 - evalscope - INFO: Predicting[amc@amc24]: 20%| 9/45 [Elapsed: 01:00 < Remaining: 15:37, 26.03s/it]
2026-03-14 01:31:33 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=12, inflight=2)
2026-03-14 01:31:53 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=4, inflight=2)
2026-03-14 01:32:03 - evalscope - INFO: Predicting[amc@amc24]: 56%| 25/45 [Elapsed: 02:00 < Remaining: 01:33, 4.66s/it]
2026-03-14 01:32:28 - evalscope - INFO: Dispatcher: Worker-0 <- 4 prompts (pending=0, inflight=2)
2026-03-14 01:33:03 - evalscope - INFO: Predicting[amc@amc24]: 91%| 41/45 [Elapsed: 03:00 < Remaining: 00:15, 3.86s/it]
2026-03-14 01:33:05 - evalscope - INFO: Predicting[amc@amc24]: 100%| 45/45 [Elapsed: 03:01 < Remaining: 00:00, 3.09s/it]
2026-03-14 01:33:05 - evalscope - INFO: Finished getting predictions for subset: amc24.
2026-03-14 01:33:05 - evalscope - INFO: Getting reviews for subset: amc24
2026-03-14 01:33:05 - evalscope - INFO: Reviewing 45 samples, if data is large, it may take a while.
2026-03-14 01:33:05 - evalscope - INFO: Reviewing[amc@amc24]: 100%| 45/45 [Elapsed: 00:00 < Remaining: 00:00, 118.49it/s]
2026-03-14 01:33:05 - evalscope - INFO: Finished reviewing subset: amc24. Total reviewed: 45
2026-03-14 01:33:05 - evalscope - INFO: Aggregating scores for subset: amc24
2026-03-14 01:33:05 - evalscope - INFO: Evaluating [amc] 100%| 3/3 [Elapsed: 08:07 < Remaining: 00:00, 164.74s/subset]
2026-03-14 01:33:05 - evalscope - INFO: Generating report...
2026-03-14 01:33:05 - evalscope - INFO:
amc report table:
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | amc | mean_acc | amc22 | 43 | 0.093 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | amc | mean_acc | amc23 | 46 | 0.0435 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | amc | mean_acc | amc24 | 45 | 0 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | amc | mean_acc | OVERALL | 134 | 0.0448 | - |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
2026-03-14 01:33:05 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-14 01:33:05 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AMC/20260314_012458/reports/llama3_1b_instruct_vallina_full_sft_30k/amc.json
2026-03-14 01:33:05 - evalscope - INFO: Benchmark amc evaluation finished.
2026-03-14 01:33:05 - evalscope - INFO: Overall report table:
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | amc | mean_acc | amc22 | 43 | 0.093 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | amc | mean_acc | amc23 | 46 | 0.0435 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | amc | mean_acc | amc24 | 45 | 0 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | amc | mean_acc | OVERALL | 134 | 0.0448 | - |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
2026-03-14 01:33:06 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['amc']
2026-03-14 01:33:06 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AMC/20260314_012458
2026-03-14 01:33:06 - evalscope - INFO: [进度条] AMC 评测完成 ✓
2026-03-14 01:33:06 - evalscope - INFO: [断点续传] AMC 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/AMC_result.json
2026-03-14 01:33:06 - evalscope - INFO: 完成评测 AMC (4/8)
2026-03-14 01:33:06 - evalscope - INFO:
==================================================
2026-03-14 01:33:06 - evalscope - INFO: 正在评估 AGIEvalMath (repeat: 1次) (剩余: 3个)
2026-03-14 01:33:06 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-14 01:33:06 - evalscope - INFO: 开始创建 benchmark AGIEvalMath 的 TaskConfig
2026-03-14 01:33:06 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-14 01:33:06 - evalscope - INFO: [AGIEvalMath] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['agieval_math'], eval_batch_size=2048
2026-03-14 01:33:06 - evalscope - INFO: 开始评测 AGIEvalMath...
2026-03-14 01:33:06 - evalscope - INFO: [进度条] AGIEvalMath 开始评测
2026-03-14 01:33:06 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

43
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/predictions/llama3_1b_instruct_vallina_full_sft_30k/amc_amc22.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

46
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/predictions/llama3_1b_instruct_vallina_full_sft_30k/amc_amc23.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

45
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/predictions/llama3_1b_instruct_vallina_full_sft_30k/amc_amc24.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

44
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/reports/llama3_1b_instruct_vallina_full_sft_30k/amc.json Normal file
View File

@@ -0,0 +1,44 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@amc",
"dataset_name": "amc",
"dataset_pretty_name": "AMC",
"dataset_description": "AMC (American Mathematics Competitions) is a series of mathematics competitions for high school students.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0448,
"metrics": [
{
"name": "mean_acc",
"num": 134,
"score": 0.0448,
"macro_score": 0.0448,
"categories": [
{
"name": [
"default"
],
"num": 134,
"score": 0.0448,
"macro_score": 0.0455,
"subsets": [
{
"name": "amc22",
"score": 0.093,
"num": 43
},
{
"name": "amc23",
"score": 0.0435,
"num": 46
},
{
"name": "amc24",
"score": 0.0,
"num": 45
}
]
}
]
}
],
"analysis": "N/A"
}

43
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/reviews/llama3_1b_instruct_vallina_full_sft_30k/amc_amc22.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

46
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/reviews/llama3_1b_instruct_vallina_full_sft_30k/amc_amc23.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

45
evalscope/version_20260314_002459/run_1/AMC/20260314_012458/reviews/llama3_1b_instruct_vallina_full_sft_30k/amc_amc24.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

46
evalscope/version_20260314_002459/run_1/AMC_result.json Normal file
View File

@@ -0,0 +1,46 @@
{
"amc": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@amc",
"dataset_name": "amc",
"dataset_pretty_name": "AMC",
"dataset_description": "AMC (American Mathematics Competitions) is a series of mathematics competitions for high school students.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0448,
"metrics": [
{
"name": "mean_acc",
"num": 134,
"score": 0.0448,
"macro_score": 0.0448,
"categories": [
{
"name": [
"default"
],
"num": 134,
"score": 0.0448,
"macro_score": 0.0455,
"subsets": [
{
"name": "amc22",
"score": 0.093,
"num": 43
},
{
"name": "amc23",
"score": 0.0435,
"num": 46
},
{
"name": "amc24",
"score": 0.0,
"num": 45
}
]
}
]
}
],
"analysis": "N/A"
}
}

84
evalscope/version_20260314_002459/run_1/ARC/20260315_164329/configs/task_config_656443.yaml Normal file
View File

@@ -0,0 +1,84 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
arc:
aggregation: mean
dataset_id: allenai/ai2_arc
default_subset: default
description: 'The ARC (AI2 Reasoning Challenge) benchmark is designed to evaluate
the reasoning capabilities of AI models through multiple-choice questions derived
from science exams. It includes two subsets: ARC-Easy and ARC-Challenge, which
vary in difficulty.'
eval_split: test
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: arc
output_types:
- generation
pretty_name: ARC
prompt_template: 'Answer the following multiple choice question. The entire content
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- ARC-Easy
- ARC-Challenge
system_prompt: null
tags:
- Reasoning
- MCQ
train_split: train
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- arc
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 1
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config: {}
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/ARC/20260315_164329

249
evalscope/version_20260314_002459/run_1/ARC/20260315_164329/logs/eval_log.log Normal file
View File

@@ -0,0 +1,249 @@
2026-03-15 16:43:29 - evalscope - INFO: Running with native backend
2026-03-15 16:43:29 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/ARC/20260315_164329/configs/task_config_656443.yaml
2026-03-15 16:43:29 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"arc"
],
"dataset_args": {
"arc": {
"name": "arc",
"dataset_id": "allenai/ai2_arc",
"output_types": [
"generation"
],
"subset_list": [
"ARC-Easy",
"ARC-Challenge"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": "train",
"eval_split": "test",
"prompt_template": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "ARC",
"description": "The ARC (AI2 Reasoning Challenge) benchmark is designed to evaluate the reasoning capabilities of AI models through multiple-choice questions derived from science exams. It includes two subsets: ARC-Easy and ARC-Challenge, which vary in difficulty.",
"tags": [
"Reasoning",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/ARC/20260315_164329",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 1,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {},
"evalscope_version": "1.4.2"
}
2026-03-15 16:43:29 - evalscope - INFO: Start loading benchmark dataset: arc
2026-03-15 16:43:30 - evalscope - INFO: Start evaluating 2 subsets of the arc: ['ARC-Easy', 'ARC-Challenge']
2026-03-15 16:43:30 - evalscope - INFO: Evaluating subset: ARC-Easy
2026-03-15 16:43:30 - evalscope - INFO: Getting predictions for subset: ARC-Easy
2026-03-15 16:43:30 - evalscope - INFO: Processing 2376 samples, if data is large, it may take a while.
2026-03-15 16:43:30 - evalscope - INFO: Loading model for prediction...
2026-03-15 16:43:30 - evalscope - INFO: Model loaded successfully.
2026-03-15 16:43:30 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=1, inflight=1)
2026-03-15 16:43:30 - evalscope - INFO: Dispatcher: Worker-1 <- 1 prompts (pending=0, inflight=2)
2026-03-15 16:43:30 - evalscope - INFO: Dispatcher: Worker-0 <- 109 prompts (pending=0, inflight=2)
2026-03-15 16:43:30 - evalscope - INFO: Dispatcher: Worker-1 <- 13 prompts (pending=0, inflight=2)
2026-03-15 16:43:32 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=1798, inflight=2)
2026-03-15 16:43:51 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=1683, inflight=2)
2026-03-15 16:43:54 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=1683, inflight=2)
2026-03-15 16:44:18 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=1664, inflight=2)
2026-03-15 16:44:31 - evalscope - INFO: Predicting[arc@ARC-Easy]: 16%| 380/2376 [Elapsed: 01:00 < Remaining: 05:53, 5.65it/s]
2026-03-15 16:44:36 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=1612, inflight=2)
2026-03-15 16:44:38 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=1484, inflight=2)
2026-03-15 16:44:54 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=1356, inflight=2)
2026-03-15 16:45:13 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=1228, inflight=2)
2026-03-15 16:45:28 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=1100, inflight=2)
2026-03-15 16:45:30 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=972, inflight=2)
2026-03-15 16:45:31 - evalscope - INFO: Predicting[arc@ARC-Easy]: 48%| 1148/2376 [Elapsed: 02:01 < Remaining: 01:46, 11.57it/s]
2026-03-15 16:45:43 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=844, inflight=2)
2026-03-15 16:46:09 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=716, inflight=2)
2026-03-15 16:46:10 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=588, inflight=2)
2026-03-15 16:46:32 - evalscope - INFO: Predicting[arc@ARC-Easy]: 64%| 1532/2376 [Elapsed: 03:01 < Remaining: 01:16, 11.03it/s]
2026-03-15 16:46:35 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=460, inflight=2)
2026-03-15 16:46:52 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=332, inflight=2)
2026-03-15 16:46:59 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=204, inflight=2)
2026-03-15 16:47:08 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=76, inflight=2)
2026-03-15 16:47:13 - evalscope - INFO: Dispatcher: Worker-0 <- 76 prompts (pending=0, inflight=2)
2026-03-15 16:47:33 - evalscope - INFO: Predicting[arc@ARC-Easy]: 91%| 2172/2376 [Elapsed: 04:02 < Remaining: 00:15, 13.12it/s]
2026-03-15 16:47:39 - evalscope - INFO: Predicting[arc@ARC-Easy]: 100%| 2376/2376 [Elapsed: 04:09 < Remaining: 00:00, 10.64it/s]
2026-03-15 16:47:39 - evalscope - INFO: Finished getting predictions for subset: ARC-Easy.
2026-03-15 16:47:39 - evalscope - INFO: Getting reviews for subset: ARC-Easy
2026-03-15 16:47:39 - evalscope - INFO: Reviewing 2376 samples, if data is large, it may take a while.
2026-03-15 16:47:42 - evalscope - INFO: Reviewing[arc@ARC-Easy]: 100%| 2376/2376 [Elapsed: 00:02 < Remaining: 00:00, 1036.42it/s]
2026-03-15 16:47:42 - evalscope - INFO: Finished reviewing subset: ARC-Easy. Total reviewed: 2376
2026-03-15 16:47:42 - evalscope - INFO: Aggregating scores for subset: ARC-Easy
2026-03-15 16:47:42 - evalscope - INFO: Evaluating [arc] 50%| 1/2 [Elapsed: 04:12 < Remaining: 04:12, 252.33s/subset]
2026-03-15 16:47:42 - evalscope - INFO: Evaluating subset: ARC-Challenge
2026-03-15 16:47:42 - evalscope - INFO: Getting predictions for subset: ARC-Challenge
2026-03-15 16:47:42 - evalscope - INFO: Processing 1172 samples, if data is large, it may take a while.
2026-03-15 16:47:42 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-15 16:47:42 - evalscope - INFO: Dispatcher: Worker-0 <- 44 prompts (pending=44, inflight=1)
2026-03-15 16:47:42 - evalscope - INFO: Dispatcher: Worker-1 <- 44 prompts (pending=0, inflight=2)
2026-03-15 16:47:52 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=955, inflight=2)
2026-03-15 16:48:07 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=827, inflight=2)
2026-03-15 16:48:19 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=699, inflight=2)
2026-03-15 16:48:32 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=571, inflight=2)
2026-03-15 16:48:39 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=443, inflight=2)
2026-03-15 16:48:43 - evalscope - INFO: Predicting[arc@ARC-Challenge]: 40%| 473/1172 [Elapsed: 01:00 < Remaining: 01:22, 8.47it/s]
2026-03-15 16:49:15 - evalscope - INFO: Dispatcher: Worker-1 <- 128 prompts (pending=315, inflight=2)
2026-03-15 16:49:23 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=187, inflight=2)
2026-03-15 16:49:43 - evalscope - INFO: Predicting[arc@ARC-Challenge]: 62%| 729/1172 [Elapsed: 02:01 < Remaining: 01:01, 7.24it/s]
2026-03-15 16:49:48 - evalscope - INFO: Dispatcher: Worker-0 <- 128 prompts (pending=59, inflight=2)
2026-03-15 16:49:56 - evalscope - INFO: Dispatcher: Worker-1 <- 59 prompts (pending=0, inflight=2)
2026-03-15 16:50:40 - evalscope - INFO: Predicting[arc@ARC-Challenge]: 100%| 1172/1172 [Elapsed: 02:57 < Remaining: 00:00, 5.63it/s]
2026-03-15 16:50:40 - evalscope - INFO: Finished getting predictions for subset: ARC-Challenge.
2026-03-15 16:50:40 - evalscope - INFO: Getting reviews for subset: ARC-Challenge
2026-03-15 16:50:40 - evalscope - INFO: Reviewing 1172 samples, if data is large, it may take a while.
2026-03-15 16:50:41 - evalscope - INFO: Reviewing[arc@ARC-Challenge]: 100%| 1172/1172 [Elapsed: 00:01 < Remaining: 00:00, 998.61it/s]
2026-03-15 16:50:41 - evalscope - INFO: Finished reviewing subset: ARC-Challenge. Total reviewed: 1172
2026-03-15 16:50:41 - evalscope - INFO: Aggregating scores for subset: ARC-Challenge
2026-03-15 16:50:41 - evalscope - INFO: Evaluating [arc] 100%| 2/2 [Elapsed: 07:11 < Remaining: 00:00, 209.43s/subset]
2026-03-15 16:50:41 - evalscope - INFO: Generating report...
2026-03-15 16:50:41 - evalscope - INFO:
arc report table:
+-----------------------------------------+-----------+----------+---------------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+===============+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | arc | mean_acc | ARC-Easy | 2376 | 0.4823 | default |
+-----------------------------------------+-----------+----------+---------------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | arc | mean_acc | ARC-Challenge | 1172 | 0.3916 | default |
+-----------------------------------------+-----------+----------+---------------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | arc | mean_acc | OVERALL | 3548 | 0.4523 | - |
+-----------------------------------------+-----------+----------+---------------+-------+---------+---------+
2026-03-15 16:50:41 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-15 16:50:41 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/ARC/20260315_164329/reports/llama3_1b_instruct_vallina_full_sft_30k/arc.json
2026-03-15 16:50:41 - evalscope - INFO: Benchmark arc evaluation finished.
2026-03-15 16:50:41 - evalscope - INFO: Overall report table:
+-----------------------------------------+-----------+----------+---------------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+===============+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | arc | mean_acc | ARC-Easy | 2376 | 0.4823 | default |
+-----------------------------------------+-----------+----------+---------------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | arc | mean_acc | ARC-Challenge | 1172 | 0.3916 | default |
+-----------------------------------------+-----------+----------+---------------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | arc | mean_acc | OVERALL | 3548 | 0.4523 | - |
+-----------------------------------------+-----------+----------+---------------+-------+---------+---------+
2026-03-15 16:50:42 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['arc']
2026-03-15 16:50:42 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/ARC/20260315_164329
2026-03-15 16:50:42 - evalscope - INFO: [进度条] ARC 评测完成 ✓
2026-03-15 16:50:42 - evalscope - INFO: [断点续传] ARC 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/ARC_result.json
2026-03-15 16:50:42 - evalscope - INFO: 完成评测 ARC (9/9)
2026-03-15 16:50:43 - evalscope - INFO: Excel汇总结果已保存至: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/summary.xlsx
2026-03-15 16:50:43 - evalscope - INFO: 详细结果已保存至: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/detailed_results.json
2026-03-15 16:50:43 - evalscope - INFO: 第 1 次运行完成
2026-03-15 16:50:43 - evalscope - INFO: 等待 5 秒后开始下一次运行...
2026-03-15 16:50:48 - evalscope - INFO:
######################################################################
2026-03-15 16:50:48 - evalscope - INFO: # 第 2/3 次运行
2026-03-15 16:50:48 - evalscope - INFO: ######################################################################
2026-03-15 16:50:48 - evalscope - INFO:
============================================================
2026-03-15 16:50:48 - evalscope - INFO: 开始运行评估 (第 2 次运行) - 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-15 16:50:48 - evalscope - INFO: 配置文件: eval_configs/llama3_1b_instruct_vallina_full_sft_30k.yaml
2026-03-15 16:50:48 - evalscope - INFO: 输出目录: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_2
2026-03-15 16:50:48 - evalscope - INFO: ============================================================
2026-03-15 16:50:48 - evalscope - INFO: 配置已保存至: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_2/config.yaml
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark AIME25 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [AIME25] worker_chunk_size=8
2026-03-15 16:50:48 - evalscope - INFO: [AIME25] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['aime25'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark AIME24 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [AIME24] worker_chunk_size=8
2026-03-15 16:50:48 - evalscope - INFO: [AIME24] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['aime24'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark MATH500 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [MATH500] worker_chunk_size=8
2026-03-15 16:50:48 - evalscope - INFO: [MATH500] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['math_500'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark AMC 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [AMC] worker_chunk_size=8
2026-03-15 16:50:48 - evalscope - INFO: [AMC] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['amc'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark AGIEvalMath 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [AGIEvalMath] worker_chunk_size=8
2026-03-15 16:50:48 - evalscope - INFO: [AGIEvalMath] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['agieval_math'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark GSM8K 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [GSM8K] worker_chunk_size=8
2026-03-15 16:50:48 - evalscope - INFO: [GSM8K] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['gsm8k'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark GPQA 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [GPQA] worker_chunk_size=8
2026-03-15 16:50:48 - evalscope - INFO: [GPQA] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['gpqa_extend'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark MMLUProNoMath 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [MMLUProNoMath] worker_chunk_size=256
2026-03-15 16:50:48 - evalscope - INFO: [MMLUProNoMath] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['mmlu_pro_no_math'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark ARC 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [ARC] worker_chunk_size=128
2026-03-15 16:50:48 - evalscope - INFO: [ARC] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['arc'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 有效 benchmarks: ['AIME25', 'AIME24', 'MATH500', 'AMC', 'AGIEvalMath', 'GSM8K', 'GPQA', 'MMLUProNoMath', 'ARC']
2026-03-15 16:50:48 - evalscope - INFO:
==================================================
2026-03-15 16:50:48 - evalscope - INFO: 正在评估 AIME25 (repeat: 8次) (剩余: 8个)
2026-03-15 16:50:48 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-15 16:50:48 - evalscope - INFO: 开始创建 benchmark AIME25 的 TaskConfig
2026-03-15 16:50:48 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:50:48 - evalscope - INFO: [AIME25] worker_chunk_size=8
2026-03-15 16:50:48 - evalscope - INFO: [AIME25] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['aime25'], eval_batch_size=2048
2026-03-15 16:50:48 - evalscope - INFO: 开始评测 AIME25...
2026-03-15 16:50:48 - evalscope - INFO: [进度条] AIME25 开始评测
2026-03-15 16:50:48 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

1172
evalscope/version_20260314_002459/run_1/ARC/20260315_164329/predictions/llama3_1b_instruct_vallina_full_sft_30k/arc_ARC-Challenge.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

2376
evalscope/version_20260314_002459/run_1/ARC/20260315_164329/predictions/llama3_1b_instruct_vallina_full_sft_30k/arc_ARC-Easy.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

39
evalscope/version_20260314_002459/run_1/ARC/20260315_164329/reports/llama3_1b_instruct_vallina_full_sft_30k/arc.json Normal file
View File

@@ -0,0 +1,39 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@arc",
"dataset_name": "arc",
"dataset_pretty_name": "ARC",
"dataset_description": "The ARC (AI2 Reasoning Challenge) benchmark is designed to evaluate the reasoning capabilities of AI models through multiple-choice questions derived from science exams. It includes two subsets: ARC-Easy and ARC-Challenge, which vary in difficulty.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.4523,
"metrics": [
{
"name": "mean_acc",
"num": 3548,
"score": 0.4523,
"macro_score": 0.4523,
"categories": [
{
"name": [
"default"
],
"num": 3548,
"score": 0.4523,
"macro_score": 0.437,
"subsets": [
{
"name": "ARC-Easy",
"score": 0.4823,
"num": 2376
},
{
"name": "ARC-Challenge",
"score": 0.3916,
"num": 1172
}
]
}
]
}
],
"analysis": "N/A"
}

1172
evalscope/version_20260314_002459/run_1/ARC/20260315_164329/reviews/llama3_1b_instruct_vallina_full_sft_30k/arc_ARC-Challenge.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

2376
evalscope/version_20260314_002459/run_1/ARC/20260315_164329/reviews/llama3_1b_instruct_vallina_full_sft_30k/arc_ARC-Easy.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

41
evalscope/version_20260314_002459/run_1/ARC_result.json Normal file
View File

@@ -0,0 +1,41 @@
{
"arc": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@arc",
"dataset_name": "arc",
"dataset_pretty_name": "ARC",
"dataset_description": "The ARC (AI2 Reasoning Challenge) benchmark is designed to evaluate the reasoning capabilities of AI models through multiple-choice questions derived from science exams. It includes two subsets: ARC-Easy and ARC-Challenge, which vary in difficulty.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.4523,
"metrics": [
{
"name": "mean_acc",
"num": 3548,
"score": 0.4523,
"macro_score": 0.4523,
"categories": [
{
"name": [
"default"
],
"num": 3548,
"score": 0.4523,
"macro_score": 0.437,
"subsets": [
{
"name": "ARC-Easy",
"score": 0.4823,
"num": 2376
},
{
"name": "ARC-Challenge",
"score": 0.3916,
"num": 1172
}
]
}
]
}
],
"analysis": "N/A"
}
}

82
evalscope/version_20260314_002459/run_1/GPQA/20260314_030908/configs/task_config_a749d7.yaml Normal file
View File

@@ -0,0 +1,82 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gpqa_extend:
aggregation: mean
dataset_id: modelscope/gpqa
default_subset: default
description: GPQA Extended dataset for evaluating reasoning on graduate-level
science problems.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: gpqa_extend
output_types:
- generation
pretty_name: GPQA-Extended
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Knowledge
- MCQ
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gpqa_extend
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_030908

91
evalscope/version_20260314_002459/run_1/GPQA/20260314_030908/logs/eval_log.log Normal file
View File

@@ -0,0 +1,91 @@
2026-03-14 03:09:08 - evalscope - INFO: Running with native backend
2026-03-14 03:09:08 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_030908/configs/task_config_a749d7.yaml
2026-03-14 03:09:08 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gpqa_extend"
],
"dataset_args": {
"gpqa_extend": {
"name": "gpqa_extend",
"dataset_id": "modelscope/gpqa",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "GPQA-Extended",
"description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_030908",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 03:09:08 - evalscope - INFO: Start loading benchmark dataset: gpqa_extend
2026-03-14 03:09:08 - evalscope - INFO: Loading dataset modelscope/gpqa from modelscope > subset: default > split: train ...
2026-03-14 03:09:12 - evalscope - ERROR: [进度条] GPQA 评测失败 ✗
2026-03-14 03:09:12 - evalscope - ERROR: 评测过程中出错: 'default'
2026-03-14 03:09:15 - evalscope - INFO: Ray Workers 已关闭

82
evalscope/version_20260314_002459/run_1/GPQA/20260314_103605/configs/task_config_a4dc61.yaml Normal file
View File

@@ -0,0 +1,82 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gpqa_extend:
aggregation: mean
dataset_id: modelscope/gpqa
default_subset: default
description: GPQA Extended dataset for evaluating reasoning on graduate-level
science problems.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: gpqa_extend
output_types:
- generation
pretty_name: GPQA-Extended
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Knowledge
- MCQ
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gpqa_extend
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103605

91
evalscope/version_20260314_002459/run_1/GPQA/20260314_103605/logs/eval_log.log Normal file
View File

@@ -0,0 +1,91 @@
2026-03-14 10:36:05 - evalscope - INFO: Running with native backend
2026-03-14 10:36:05 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103605/configs/task_config_a4dc61.yaml
2026-03-14 10:36:06 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gpqa_extend"
],
"dataset_args": {
"gpqa_extend": {
"name": "gpqa_extend",
"dataset_id": "modelscope/gpqa",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "GPQA-Extended",
"description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103605",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 10:36:06 - evalscope - INFO: Start loading benchmark dataset: gpqa_extend
2026-03-14 10:36:06 - evalscope - INFO: Loading dataset modelscope/gpqa from modelscope > subset: default > split: train ...
2026-03-14 10:36:09 - evalscope - ERROR: [进度条] GPQA 评测失败 ✗
2026-03-14 10:36:09 - evalscope - ERROR: 评测过程中出错: 'default'
2026-03-14 10:36:11 - evalscope - INFO: Ray Workers 已关闭

82
evalscope/version_20260314_002459/run_1/GPQA/20260314_103803/configs/task_config_9cc141.yaml Normal file
View File

@@ -0,0 +1,82 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gpqa_extend:
aggregation: mean
dataset_id: modelscope/gpqa
default_subset: default
description: GPQA Extended dataset for evaluating reasoning on graduate-level
science problems.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: gpqa_extend
output_types:
- generation
pretty_name: GPQA-Extended
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Knowledge
- MCQ
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gpqa_extend
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103803

91
evalscope/version_20260314_002459/run_1/GPQA/20260314_103803/logs/eval_log.log Normal file
View File

@@ -0,0 +1,91 @@
2026-03-14 10:38:03 - evalscope - INFO: Running with native backend
2026-03-14 10:38:03 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103803/configs/task_config_9cc141.yaml
2026-03-14 10:38:03 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gpqa_extend"
],
"dataset_args": {
"gpqa_extend": {
"name": "gpqa_extend",
"dataset_id": "modelscope/gpqa",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "GPQA-Extended",
"description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103803",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 10:38:03 - evalscope - INFO: Start loading benchmark dataset: gpqa_extend
2026-03-14 10:38:03 - evalscope - INFO: Loading dataset modelscope/gpqa from modelscope > subset: default > split: train ...
2026-03-14 10:38:08 - evalscope - ERROR: [进度条] GPQA 评测失败 ✗
2026-03-14 10:38:08 - evalscope - ERROR: 评测过程中出错: 'default'
2026-03-14 10:38:09 - evalscope - INFO: Ray Workers 已关闭

82
evalscope/version_20260314_002459/run_1/GPQA/20260314_103944/configs/task_config_809913.yaml Normal file
View File

@@ -0,0 +1,82 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gpqa_extend:
aggregation: mean
dataset_id: modelscope/gpqa
default_subset: default
description: GPQA Extended dataset for evaluating reasoning on graduate-level
science problems.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: gpqa_extend
output_types:
- generation
pretty_name: GPQA-Extended
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Knowledge
- MCQ
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gpqa_extend
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103944

91
evalscope/version_20260314_002459/run_1/GPQA/20260314_103944/logs/eval_log.log Normal file
View File

@@ -0,0 +1,91 @@
2026-03-14 10:39:44 - evalscope - INFO: Running with native backend
2026-03-14 10:39:44 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103944/configs/task_config_809913.yaml
2026-03-14 10:39:44 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gpqa_extend"
],
"dataset_args": {
"gpqa_extend": {
"name": "gpqa_extend",
"dataset_id": "modelscope/gpqa",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "GPQA-Extended",
"description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_103944",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 10:39:44 - evalscope - INFO: Start loading benchmark dataset: gpqa_extend
2026-03-14 10:39:45 - evalscope - INFO: Loading dataset modelscope/gpqa from modelscope > subset: default > split: train ...
2026-03-14 10:39:48 - evalscope - ERROR: [进度条] GPQA 评测失败 ✗
2026-03-14 10:39:48 - evalscope - ERROR: 评测过程中出错: 'default'
2026-03-14 10:39:49 - evalscope - INFO: Ray Workers 已关闭

82
evalscope/version_20260314_002459/run_1/GPQA/20260314_104708/configs/task_config_cb3bb4.yaml Normal file
View File

@@ -0,0 +1,82 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gpqa_extend:
aggregation: mean
dataset_id: modelscope/gpqa
default_subset: default
description: GPQA Extended dataset for evaluating reasoning on graduate-level
science problems.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: gpqa_extend
output_types:
- generation
pretty_name: GPQA-Extended
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Knowledge
- MCQ
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gpqa_extend
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_104708

91
evalscope/version_20260314_002459/run_1/GPQA/20260314_104708/logs/eval_log.log Normal file
View File

@@ -0,0 +1,91 @@
2026-03-14 10:47:08 - evalscope - INFO: Running with native backend
2026-03-14 10:47:08 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_104708/configs/task_config_cb3bb4.yaml
2026-03-14 10:47:08 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gpqa_extend"
],
"dataset_args": {
"gpqa_extend": {
"name": "gpqa_extend",
"dataset_id": "modelscope/gpqa",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "GPQA-Extended",
"description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_104708",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 10:47:08 - evalscope - INFO: Start loading benchmark dataset: gpqa_extend
2026-03-14 10:47:09 - evalscope - INFO: Loading dataset modelscope/gpqa from modelscope > subset: default > split: train ...
2026-03-14 10:47:12 - evalscope - ERROR: [进度条] GPQA 评测失败 ✗
2026-03-14 10:47:12 - evalscope - ERROR: 评测过程中出错: 'default'
2026-03-14 10:47:13 - evalscope - INFO: Ray Workers 已关闭

82
evalscope/version_20260314_002459/run_1/GPQA/20260314_105828/configs/task_config_1d0d4e.yaml Normal file
View File

@@ -0,0 +1,82 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gpqa_extend:
aggregation: mean
dataset_id: modelscope/gpqa
default_subset: default
description: GPQA Extended dataset for evaluating reasoning on graduate-level
science problems.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: gpqa_extend
output_types:
- generation
pretty_name: GPQA-Extended
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Knowledge
- MCQ
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gpqa_extend
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_105828

91
evalscope/version_20260314_002459/run_1/GPQA/20260314_105828/logs/eval_log.log Normal file
View File

@@ -0,0 +1,91 @@
2026-03-14 10:58:28 - evalscope - INFO: Running with native backend
2026-03-14 10:58:28 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_105828/configs/task_config_1d0d4e.yaml
2026-03-14 10:58:28 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gpqa_extend"
],
"dataset_args": {
"gpqa_extend": {
"name": "gpqa_extend",
"dataset_id": "modelscope/gpqa",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "GPQA-Extended",
"description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_105828",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 10:58:28 - evalscope - INFO: Start loading benchmark dataset: gpqa_extend
2026-03-14 10:58:28 - evalscope - INFO: Loading dataset modelscope/gpqa from modelscope > subset: default > split: train ...
2026-03-14 10:58:32 - evalscope - ERROR: [进度条] GPQA 评测失败 ✗
2026-03-14 10:58:32 - evalscope - ERROR: 评测过程中出错: 'default'
2026-03-14 10:58:33 - evalscope - INFO: Ray Workers 已关闭

82
evalscope/version_20260314_002459/run_1/GPQA/20260314_110206/configs/task_config_62d51a.yaml Normal file
View File

@@ -0,0 +1,82 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gpqa_extend:
aggregation: mean
dataset_id: modelscope/gpqa
default_subset: gpqa
description: GPQA Extended dataset for evaluating reasoning on graduate-level
science problems.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: gpqa_extend
output_types:
- generation
pretty_name: GPQA-Extended
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Knowledge
- MCQ
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gpqa_extend
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_110206

91
evalscope/version_20260314_002459/run_1/GPQA/20260314_110206/logs/eval_log.log Normal file
View File

@@ -0,0 +1,91 @@
2026-03-14 11:02:06 - evalscope - INFO: Running with native backend
2026-03-14 11:02:06 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_110206/configs/task_config_62d51a.yaml
2026-03-14 11:02:06 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gpqa_extend"
],
"dataset_args": {
"gpqa_extend": {
"name": "gpqa_extend",
"dataset_id": "modelscope/gpqa",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "gpqa",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "GPQA-Extended",
"description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260314_110206",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-14 11:02:06 - evalscope - INFO: Start loading benchmark dataset: gpqa_extend
2026-03-14 11:02:06 - evalscope - INFO: Loading dataset modelscope/gpqa from modelscope > subset: default > split: train ...
2026-03-14 11:02:10 - evalscope - ERROR: [进度条] GPQA 评测失败 ✗
2026-03-14 11:02:10 - evalscope - ERROR: 评测过程中出错: 'default'
2026-03-14 11:02:11 - evalscope - INFO: Ray Workers 已关闭

82
evalscope/version_20260314_002459/run_1/GPQA/20260315_152909/configs/task_config_7454cf.yaml Normal file
View File

@@ -0,0 +1,82 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gpqa_extend:
aggregation: mean
dataset_id: modelscope/gpqa
default_subset: gpqa
description: GPQA Extended dataset for evaluating reasoning on graduate-level
science problems.
eval_split: train
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: gpqa_extend
output_types:
- generation
pretty_name: GPQA-Extended
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- Knowledge
- MCQ
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gpqa_extend
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 5
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260315_152909

235
evalscope/version_20260314_002459/run_1/GPQA/20260315_152909/logs/eval_log.log Normal file
View File

@@ -0,0 +1,235 @@
2026-03-15 15:29:09 - evalscope - INFO: Running with native backend
2026-03-15 15:29:09 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260315_152909/configs/task_config_7454cf.yaml
2026-03-15 15:29:09 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gpqa_extend"
],
"dataset_args": {
"gpqa_extend": {
"name": "gpqa_extend",
"dataset_id": "modelscope/gpqa",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "gpqa",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "train",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "GPQA-Extended",
"description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"tags": [
"Knowledge",
"MCQ"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260315_152909",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 5,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {
"work_dir": "/fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox"
},
"evalscope_version": "1.4.2"
}
2026-03-15 15:29:09 - evalscope - INFO: Start loading benchmark dataset: gpqa_extend
2026-03-15 15:29:09 - evalscope - INFO: Start evaluating 1 subsets of the gpqa_extend: ['gpqa']
2026-03-15 15:29:09 - evalscope - INFO: Evaluating subset: gpqa
2026-03-15 15:29:09 - evalscope - INFO: Getting predictions for subset: gpqa
2026-03-15 15:29:09 - evalscope - INFO: Processing 546 samples, if data is large, it may take a while.
2026-03-15 15:29:09 - evalscope - INFO: Loading model for prediction...
2026-03-15 15:29:09 - evalscope - INFO: Model loaded successfully.
2026-03-15 15:29:10 - evalscope - INFO: Dispatcher: Worker-0 <- 3 prompts (pending=4, inflight=1)
2026-03-15 15:29:10 - evalscope - INFO: Dispatcher: Worker-1 <- 3 prompts (pending=1, inflight=2)
2026-03-15 15:30:10 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 0%| 0/546 [Elapsed: 01:00 < Remaining: ?, ?it/s]
2026-03-15 15:31:10 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 0%| 0/546 [Elapsed: 02:00 < Remaining: ?, ?it/s]
2026-03-15 15:31:59 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=532, inflight=2)
2026-03-15 15:32:10 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 1%| 3/546 [Elapsed: 03:00 < Remaining: 25:40:35, 170.23s/it]
2026-03-15 15:32:14 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=524, inflight=2)
2026-03-15 15:32:50 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=516, inflight=2)
2026-03-15 15:33:05 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=508, inflight=2)
2026-03-15 15:33:10 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 4%| 22/546 [Elapsed: 04:00 < Remaining: 1:17:08, 8.83s/it]
2026-03-15 15:33:35 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=500, inflight=2)
2026-03-15 15:33:39 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=492, inflight=2)
2026-03-15 15:34:01 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=484, inflight=2)
2026-03-15 15:34:10 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 8%| 46/546 [Elapsed: 05:00 < Remaining: 29:29, 3.54s/it]
2026-03-15 15:34:15 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=476, inflight=2)
2026-03-15 15:34:53 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=468, inflight=2)
2026-03-15 15:34:55 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=460, inflight=2)
2026-03-15 15:35:10 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 13%| 70/546 [Elapsed: 06:00 < Remaining: 19:29, 2.46s/it]
2026-03-15 15:35:42 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=452, inflight=2)
2026-03-15 15:35:47 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=444, inflight=2)
2026-03-15 15:36:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 16%| 86/546 [Elapsed: 07:00 < Remaining: 20:11, 2.63s/it]
2026-03-15 15:36:20 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=436, inflight=2)
2026-03-15 15:36:39 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=428, inflight=2)
2026-03-15 15:37:01 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=420, inflight=2)
2026-03-15 15:37:04 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=412, inflight=2)
2026-03-15 15:37:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 22%| 118/546 [Elapsed: 08:00 < Remaining: 14:56, 2.09s/it]
2026-03-15 15:37:52 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=404, inflight=2)
2026-03-15 15:37:54 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=396, inflight=2)
2026-03-15 15:38:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 25%| 134/546 [Elapsed: 09:00 < Remaining: 16:14, 2.36s/it]
2026-03-15 15:38:27 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=388, inflight=2)
2026-03-15 15:38:37 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=380, inflight=2)
2026-03-15 15:39:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 27%| 150/546 [Elapsed: 10:01 < Remaining: 15:51, 2.40s/it]
2026-03-15 15:39:20 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=372, inflight=2)
2026-03-15 15:39:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=364, inflight=2)
2026-03-15 15:40:10 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=356, inflight=2)
2026-03-15 15:40:11 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=348, inflight=2)
2026-03-15 15:40:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 31%| 167/546 [Elapsed: 11:01 < Remaining: 21:51, 3.46s/it]
2026-03-15 15:40:51 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=340, inflight=2)
2026-03-15 15:41:01 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=332, inflight=2)
2026-03-15 15:41:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 36%| 198/546 [Elapsed: 12:01 < Remaining: 15:06, 2.61s/it]
2026-03-15 15:41:38 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=324, inflight=2)
2026-03-15 15:41:43 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=316, inflight=2)
2026-03-15 15:42:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 39%| 214/546 [Elapsed: 13:01 < Remaining: 13:41, 2.48s/it]
2026-03-15 15:42:20 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=308, inflight=2)
2026-03-15 15:42:34 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=300, inflight=2)
2026-03-15 15:43:11 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=292, inflight=2)
2026-03-15 15:43:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 42%| 231/546 [Elapsed: 14:01 < Remaining: 17:06, 3.26s/it]
2026-03-15 15:43:25 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=284, inflight=2)
2026-03-15 15:44:01 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=276, inflight=2)
2026-03-15 15:44:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 47%| 254/546 [Elapsed: 15:01 < Remaining: 16:19, 3.35s/it]
2026-03-15 15:44:16 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=268, inflight=2)
2026-03-15 15:44:53 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=260, inflight=2)
2026-03-15 15:45:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 49%| 270/546 [Elapsed: 16:01 < Remaining: 15:38, 3.40s/it]
2026-03-15 15:45:15 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=252, inflight=2)
2026-03-15 15:45:44 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=244, inflight=2)
2026-03-15 15:46:09 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=236, inflight=2)
2026-03-15 15:46:11 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 54%| 294/546 [Elapsed: 17:01 < Remaining: 13:54, 3.31s/it]
2026-03-15 15:46:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=228, inflight=2)
2026-03-15 15:47:02 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=220, inflight=2)
2026-03-15 15:47:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 57%| 310/546 [Elapsed: 18:01 < Remaining: 13:07, 3.34s/it]
2026-03-15 15:47:20 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=212, inflight=2)
2026-03-15 15:47:52 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=204, inflight=2)
2026-03-15 15:48:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 60%| 326/546 [Elapsed: 19:01 < Remaining: 12:08, 3.31s/it]
2026-03-15 15:48:14 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=196, inflight=2)
2026-03-15 15:48:41 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=188, inflight=2)
2026-03-15 15:48:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=180, inflight=2)
2026-03-15 15:48:59 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=172, inflight=2)
2026-03-15 15:49:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 66%| 358/546 [Elapsed: 20:01 < Remaining: 06:55, 2.21s/it]
2026-03-15 15:49:36 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=164, inflight=2)
2026-03-15 15:49:43 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=156, inflight=2)
2026-03-15 15:50:07 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=148, inflight=2)
2026-03-15 15:50:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 70%| 382/546 [Elapsed: 21:01 < Remaining: 06:53, 2.52s/it]
2026-03-15 15:50:28 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=140, inflight=2)
2026-03-15 15:50:45 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=132, inflight=2)
2026-03-15 15:51:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 73%| 398/546 [Elapsed: 22:01 < Remaining: 05:53, 2.39s/it]
2026-03-15 15:51:13 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=124, inflight=2)
2026-03-15 15:51:18 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=116, inflight=2)
2026-03-15 15:51:56 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=108, inflight=2)
2026-03-15 15:52:05 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=100, inflight=2)
2026-03-15 15:52:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 79%| 430/546 [Elapsed: 23:02 < Remaining: 04:37, 2.39s/it]
2026-03-15 15:52:21 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=92, inflight=2)
2026-03-15 15:52:56 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=84, inflight=2)
2026-03-15 15:53:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 82%| 446/546 [Elapsed: 24:02 < Remaining: 04:47, 2.87s/it]
2026-03-15 15:53:12 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=76, inflight=2)
2026-03-15 15:53:40 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=68, inflight=2)
2026-03-15 15:54:01 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=60, inflight=2)
2026-03-15 15:54:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 86%| 470/546 [Elapsed: 25:02 < Remaining: 03:32, 2.79s/it]
2026-03-15 15:54:27 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=52, inflight=2)
2026-03-15 15:54:44 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=44, inflight=2)
2026-03-15 15:55:00 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=36, inflight=2)
2026-03-15 15:55:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 90%| 494/546 [Elapsed: 26:02 < Remaining: 02:10, 2.50s/it]
2026-03-15 15:55:33 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=28, inflight=2)
2026-03-15 15:55:51 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=20, inflight=2)
2026-03-15 15:56:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 93%| 510/546 [Elapsed: 27:02 < Remaining: 01:38, 2.74s/it]
2026-03-15 15:56:24 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=12, inflight=2)
2026-03-15 15:56:42 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=4, inflight=2)
2026-03-15 15:57:12 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 96%| 526/546 [Elapsed: 28:02 < Remaining: 00:57, 2.89s/it]
2026-03-15 15:57:13 - evalscope - INFO: Dispatcher: Worker-0 <- 4 prompts (pending=0, inflight=2)
2026-03-15 15:58:02 - evalscope - INFO: Predicting[gpqa_extend@gpqa]: 100%| 546/546 [Elapsed: 28:52 < Remaining: 00:00, 3.34s/it]
2026-03-15 15:58:02 - evalscope - INFO: Finished getting predictions for subset: gpqa.
2026-03-15 15:58:02 - evalscope - INFO: Getting reviews for subset: gpqa
2026-03-15 15:58:02 - evalscope - INFO: Reviewing 546 samples, if data is large, it may take a while.
2026-03-15 15:58:03 - evalscope - INFO: Reviewing[gpqa_extend@gpqa]: 100%| 546/546 [Elapsed: 00:00 < Remaining: 00:00, 551.57it/s]
2026-03-15 15:58:03 - evalscope - INFO: Finished reviewing subset: gpqa. Total reviewed: 546
2026-03-15 15:58:03 - evalscope - INFO: Aggregating scores for subset: gpqa
2026-03-15 15:58:03 - evalscope - INFO: Evaluating [gpqa_extend] 100%| 1/1 [Elapsed: 28:53 < Remaining: 00:00, 1733.67s/subset]
2026-03-15 15:58:03 - evalscope - INFO: Generating report...
2026-03-15 15:58:03 - evalscope - INFO:
gpqa_extend report table:
+-----------------------------------------+-------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+=============+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | gpqa_extend | mean_acc | gpqa | 546 | 0.2454 | default |
+-----------------------------------------+-------------+----------+----------+-------+---------+---------+
2026-03-15 15:58:03 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-15 15:58:03 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260315_152909/reports/llama3_1b_instruct_vallina_full_sft_30k/gpqa_extend.json
2026-03-15 15:58:03 - evalscope - INFO: Benchmark gpqa_extend evaluation finished.
2026-03-15 15:58:03 - evalscope - INFO: Overall report table:
+-----------------------------------------+-------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+=============+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | gpqa_extend | mean_acc | gpqa | 546 | 0.2454 | default |
+-----------------------------------------+-------------+----------+----------+-------+---------+---------+
2026-03-15 15:58:10 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['gpqa_extend']
2026-03-15 15:58:10 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA/20260315_152909
2026-03-15 15:58:10 - evalscope - INFO: [进度条] GPQA 评测完成 ✓
2026-03-15 15:58:10 - evalscope - INFO: [断点续传] GPQA 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GPQA_result.json
2026-03-15 15:58:10 - evalscope - INFO: 完成评测 GPQA (7/9)
2026-03-15 15:58:10 - evalscope - INFO:
==================================================
2026-03-15 15:58:10 - evalscope - INFO: 正在评估 MMLUProNoMath (repeat: 1次) (剩余: 1个)
2026-03-15 15:58:10 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-15 15:58:10 - evalscope - INFO: 开始创建 benchmark MMLUProNoMath 的 TaskConfig
2026-03-15 15:58:10 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 15:58:10 - evalscope - INFO: [MMLUProNoMath] worker_chunk_size=256
2026-03-15 15:58:10 - evalscope - INFO: [MMLUProNoMath] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['mmlu_pro_no_math'], eval_batch_size=2048
2026-03-15 15:58:10 - evalscope - INFO: 开始评测 MMLUProNoMath...
2026-03-15 15:58:10 - evalscope - INFO: [进度条] MMLUProNoMath 开始评测
2026-03-15 15:58:10 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

3
evalscope/version_20260314_002459/run_1/GPQA/20260315_152909/predictions/llama3_1b_instruct_vallina_full_sft_30k/gpqa_extend_gpqa.jsonl Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:39039025e11930a27cd2f804f86f145e1b449c016ad7cf17477ae6347b0d0bf4
size 17847666

34
evalscope/version_20260314_002459/run_1/GPQA/20260315_152909/reports/llama3_1b_instruct_vallina_full_sft_30k/gpqa_extend.json Normal file
View File

@@ -0,0 +1,34 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@gpqa_extend",
"dataset_name": "gpqa_extend",
"dataset_pretty_name": "GPQA-Extended",
"dataset_description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.2454,
"metrics": [
{
"name": "mean_acc",
"num": 546,
"score": 0.2454,
"macro_score": 0.2454,
"categories": [
{
"name": [
"default"
],
"num": 546,
"score": 0.2454,
"macro_score": 0.2454,
"subsets": [
{
"name": "gpqa",
"score": 0.2454,
"num": 546
}
]
}
]
}
],
"analysis": "N/A"
}

546
evalscope/version_20260314_002459/run_1/GPQA/20260315_152909/reviews/llama3_1b_instruct_vallina_full_sft_30k/gpqa_extend_gpqa.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

36
evalscope/version_20260314_002459/run_1/GPQA_result.json Normal file
View File

@@ -0,0 +1,36 @@
{
"gpqa_extend": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@gpqa_extend",
"dataset_name": "gpqa_extend",
"dataset_pretty_name": "GPQA-Extended",
"dataset_description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.2454,
"metrics": [
{
"name": "mean_acc",
"num": 546,
"score": 0.2454,
"macro_score": 0.2454,
"categories": [
{
"name": [
"default"
],
"num": 546,
"score": 0.2454,
"macro_score": 0.2454,
"subsets": [
{
"name": "gpqa",
"score": 0.2454,
"num": 546
}
]
}
]
}
],
"analysis": "N/A"
}
}

84
evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/configs/task_config_85b44b.yaml Normal file
View File

@@ -0,0 +1,84 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
gsm8k:
aggregation: mean
dataset_id: AI-ModelScope/gsm8k
default_subset: default
description: GSM8K (Grade School Math 8K) is a dataset of grade school math problems,
designed to evaluate the mathematical reasoning abilities of AI models.
eval_split: test
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: 'Here are some examples of how to solve similar problems:
{fewshot}
{question}
Please reason step by step, and put your final answer within \boxed{{}}.'
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc:
numeric: true
name: gsm8k
output_types:
- generation
pretty_name: GSM8K
prompt_template: '{question}
Please reason step by step, and put your final answer within \boxed{{}}.'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- main
system_prompt: null
tags:
- Math
- Reasoning
train_split: train
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- gsm8k
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 1
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config: {}
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517

347
evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/logs/eval_log.log Normal file
View File

@@ -0,0 +1,347 @@
2026-03-14 02:25:17 - evalscope - INFO: Running with native backend
2026-03-14 02:25:17 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/configs/task_config_85b44b.yaml
2026-03-14 02:25:17 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"gsm8k"
],
"dataset_args": {
"gsm8k": {
"name": "gsm8k",
"dataset_id": "AI-ModelScope/gsm8k",
"output_types": [
"generation"
],
"subset_list": [
"main"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": "train",
"eval_split": "test",
"prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
"few_shot_prompt_template": "Here are some examples of how to solve similar problems:\n\n{fewshot}\n\n{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
"system_prompt": null,
"query_template": null,
"pretty_name": "GSM8K",
"description": "GSM8K (Grade School Math 8K) is a dataset of grade school math problems, designed to evaluate the mathematical reasoning abilities of AI models.",
"tags": [
"Math",
"Reasoning"
],
"filters": null,
"metric_list": [
{
"acc": {
"numeric": true
}
}
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 1,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {},
"evalscope_version": "1.4.2"
}
2026-03-14 02:25:17 - evalscope - INFO: Start loading benchmark dataset: gsm8k
2026-03-14 02:25:17 - evalscope - INFO: Start evaluating 1 subsets of the gsm8k: ['main']
2026-03-14 02:25:17 - evalscope - INFO: Evaluating subset: main
2026-03-14 02:25:17 - evalscope - INFO: Getting predictions for subset: main
2026-03-14 02:25:17 - evalscope - INFO: Processing 1319 samples, if data is large, it may take a while.
2026-03-14 02:25:17 - evalscope - INFO: Loading model for prediction...
2026-03-14 02:25:17 - evalscope - INFO: Model loaded successfully.
2026-03-14 02:25:17 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 02:25:17 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=202, inflight=2)
2026-03-14 02:25:18 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1302, inflight=2)
2026-03-14 02:25:46 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1294, inflight=2)
2026-03-14 02:26:03 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1286, inflight=2)
2026-03-14 02:26:07 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1278, inflight=2)
2026-03-14 02:26:18 - evalscope - INFO: Predicting[gsm8k@main]: 2%| 25/1319 [Elapsed: 01:00 < Remaining: 44:57, 2.08s/it]
2026-03-14 02:26:35 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1270, inflight=2)
2026-03-14 02:26:38 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1262, inflight=2)
2026-03-14 02:27:07 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1254, inflight=2)
2026-03-14 02:27:18 - evalscope - INFO: Predicting[gsm8k@main]: 4%| 49/1319 [Elapsed: 02:00 < Remaining: 52:06, 2.46s/it]
2026-03-14 02:27:27 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1246, inflight=2)
2026-03-14 02:27:48 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1238, inflight=2)
2026-03-14 02:27:55 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1230, inflight=2)
2026-03-14 02:28:18 - evalscope - INFO: Predicting[gsm8k@main]: 6%| 73/1319 [Elapsed: 03:00 < Remaining: 41:39, 2.01s/it]
2026-03-14 02:28:39 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1222, inflight=2)
2026-03-14 02:28:47 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1214, inflight=2)
2026-03-14 02:28:51 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1206, inflight=2)
2026-03-14 02:29:18 - evalscope - INFO: Predicting[gsm8k@main]: 7%| 97/1319 [Elapsed: 04:00 < Remaining: 38:14, 1.88s/it]
2026-03-14 02:29:21 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1198, inflight=2)
2026-03-14 02:29:40 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1190, inflight=2)
2026-03-14 02:29:51 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1182, inflight=2)
2026-03-14 02:30:01 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1174, inflight=2)
2026-03-14 02:30:18 - evalscope - INFO: Predicting[gsm8k@main]: 10%| 129/1319 [Elapsed: 05:01 < Remaining: 36:22, 1.83s/it]
2026-03-14 02:30:20 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1166, inflight=2)
2026-03-14 02:30:25 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1158, inflight=2)
2026-03-14 02:30:53 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1150, inflight=2)
2026-03-14 02:30:57 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1142, inflight=2)
2026-03-14 02:31:17 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1134, inflight=2)
2026-03-14 02:31:19 - evalscope - INFO: Predicting[gsm8k@main]: 13%| 169/1319 [Elapsed: 06:01 < Remaining: 36:56, 1.93s/it]
2026-03-14 02:31:29 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1126, inflight=2)
2026-03-14 02:31:53 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1118, inflight=2)
2026-03-14 02:32:03 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1110, inflight=2)
2026-03-14 02:32:19 - evalscope - INFO: Predicting[gsm8k@main]: 15%| 193/1319 [Elapsed: 07:01 < Remaining: 34:55, 1.86s/it]
2026-03-14 02:32:33 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1102, inflight=2)
2026-03-14 02:32:53 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1094, inflight=2)
2026-03-14 02:32:55 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1086, inflight=2)
2026-03-14 02:33:16 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1078, inflight=2)
2026-03-14 02:33:19 - evalscope - INFO: Predicting[gsm8k@main]: 17%| 225/1319 [Elapsed: 08:01 < Remaining: 37:44, 2.07s/it]
2026-03-14 02:33:34 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1070, inflight=2)
2026-03-14 02:33:42 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1062, inflight=2)
2026-03-14 02:33:53 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1054, inflight=2)
2026-03-14 02:34:12 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1046, inflight=2)
2026-03-14 02:34:19 - evalscope - INFO: Predicting[gsm8k@main]: 19%| 257/1319 [Elapsed: 09:02 < Remaining: 33:04, 1.87s/it]
2026-03-14 02:34:24 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1038, inflight=2)
2026-03-14 02:34:47 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1030, inflight=2)
2026-03-14 02:34:57 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1022, inflight=2)
2026-03-14 02:35:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=1014, inflight=2)
2026-03-14 02:35:19 - evalscope - INFO: Predicting[gsm8k@main]: 22%| 289/1319 [Elapsed: 10:02 < Remaining: 27:58, 1.63s/it]
2026-03-14 02:35:32 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1006, inflight=2)
2026-03-14 02:35:50 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=998, inflight=2)
2026-03-14 02:36:16 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=990, inflight=2)
2026-03-14 02:36:20 - evalscope - INFO: Predicting[gsm8k@main]: 24%| 313/1319 [Elapsed: 11:02 < Remaining: 41:07, 2.45s/it]
2026-03-14 02:36:25 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=982, inflight=2)
2026-03-14 02:36:48 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=974, inflight=2)
2026-03-14 02:37:06 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=966, inflight=2)
2026-03-14 02:37:08 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=958, inflight=2)
2026-03-14 02:37:20 - evalscope - INFO: Predicting[gsm8k@main]: 26%| 345/1319 [Elapsed: 12:02 < Remaining: 27:46, 1.71s/it]
2026-03-14 02:37:34 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=950, inflight=2)
2026-03-14 02:37:38 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=942, inflight=2)
2026-03-14 02:38:08 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=934, inflight=2)
2026-03-14 02:38:20 - evalscope - INFO: Predicting[gsm8k@main]: 28%| 369/1319 [Elapsed: 13:02 < Remaining: 36:35, 2.31s/it]
2026-03-14 02:38:28 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=926, inflight=2)
2026-03-14 02:38:46 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=918, inflight=2)
2026-03-14 02:39:05 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=910, inflight=2)
2026-03-14 02:39:17 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=902, inflight=2)
2026-03-14 02:39:20 - evalscope - INFO: Predicting[gsm8k@main]: 30%| 401/1319 [Elapsed: 14:02 < Remaining: 32:04, 2.10s/it]
2026-03-14 02:39:34 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=894, inflight=2)
2026-03-14 02:39:54 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=886, inflight=2)
2026-03-14 02:40:10 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=878, inflight=2)
2026-03-14 02:40:13 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=870, inflight=2)
2026-03-14 02:40:20 - evalscope - INFO: Predicting[gsm8k@main]: 33%| 433/1319 [Elapsed: 15:03 < Remaining: 23:48, 1.61s/it]
2026-03-14 02:41:00 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=862, inflight=2)
2026-03-14 02:41:03 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=854, inflight=2)
2026-03-14 02:41:21 - evalscope - INFO: Predicting[gsm8k@main]: 34%| 449/1319 [Elapsed: 16:03 < Remaining: 31:01, 2.14s/it]
2026-03-14 02:41:23 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=846, inflight=2)
2026-03-14 02:41:51 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=838, inflight=2)
2026-03-14 02:41:57 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=830, inflight=2)
2026-03-14 02:42:12 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=822, inflight=2)
2026-03-14 02:42:21 - evalscope - INFO: Predicting[gsm8k@main]: 36%| 481/1319 [Elapsed: 17:03 < Remaining: 28:10, 2.02s/it]
2026-03-14 02:42:34 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=814, inflight=2)
2026-03-14 02:42:39 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=806, inflight=2)
2026-03-14 02:43:05 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=798, inflight=2)
2026-03-14 02:43:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=790, inflight=2)
2026-03-14 02:43:21 - evalscope - INFO: Predicting[gsm8k@main]: 39%| 513/1319 [Elapsed: 18:03 < Remaining: 21:20, 1.59s/it]
2026-03-14 02:43:24 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=782, inflight=2)
2026-03-14 02:43:38 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=774, inflight=2)
2026-03-14 02:43:53 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=766, inflight=2)
2026-03-14 02:43:58 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=758, inflight=2)
2026-03-14 02:44:08 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=750, inflight=2)
2026-03-14 02:44:15 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=742, inflight=2)
2026-03-14 02:44:21 - evalscope - INFO: Predicting[gsm8k@main]: 43%| 561/1319 [Elapsed: 19:03 < Remaining: 15:48, 1.25s/it]
2026-03-14 02:44:43 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=734, inflight=2)
2026-03-14 02:45:07 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=726, inflight=2)
2026-03-14 02:45:14 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=718, inflight=2)
2026-03-14 02:45:21 - evalscope - INFO: Predicting[gsm8k@main]: 44%| 585/1319 [Elapsed: 20:03 < Remaining: 22:16, 1.82s/it]
2026-03-14 02:45:30 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=710, inflight=2)
2026-03-14 02:45:57 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=702, inflight=2)
2026-03-14 02:46:02 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=694, inflight=2)
2026-03-14 02:46:12 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=686, inflight=2)
2026-03-14 02:46:21 - evalscope - INFO: Predicting[gsm8k@main]: 47%| 617/1319 [Elapsed: 21:04 < Remaining: 19:17, 1.65s/it]
2026-03-14 02:46:24 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=678, inflight=2)
2026-03-14 02:46:38 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=670, inflight=2)
2026-03-14 02:46:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=662, inflight=2)
2026-03-14 02:46:53 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=654, inflight=2)
2026-03-14 02:47:12 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=646, inflight=2)
2026-03-14 02:47:14 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=638, inflight=2)
2026-03-14 02:47:22 - evalscope - INFO: Predicting[gsm8k@main]: 50%| 665/1319 [Elapsed: 22:04 < Remaining: 13:10, 1.21s/it]
2026-03-14 02:47:35 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=630, inflight=2)
2026-03-14 02:47:47 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=622, inflight=2)
2026-03-14 02:47:56 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=614, inflight=2)
2026-03-14 02:48:10 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=606, inflight=2)
2026-03-14 02:48:22 - evalscope - INFO: Predicting[gsm8k@main]: 53%| 697/1319 [Elapsed: 23:04 < Remaining: 16:01, 1.55s/it]
2026-03-14 02:48:28 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=598, inflight=2)
2026-03-14 02:48:48 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=590, inflight=2)
2026-03-14 02:48:49 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=582, inflight=2)
2026-03-14 02:49:22 - evalscope - INFO: Predicting[gsm8k@main]: 55%| 721/1319 [Elapsed: 24:04 < Remaining: 19:52, 1.99s/it]
2026-03-14 02:49:24 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=574, inflight=2)
2026-03-14 02:49:25 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=566, inflight=2)
2026-03-14 02:49:34 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=558, inflight=2)
2026-03-14 02:49:51 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=550, inflight=2)
2026-03-14 02:50:02 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=542, inflight=2)
2026-03-14 02:50:21 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=534, inflight=2)
2026-03-14 02:50:22 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=526, inflight=2)
2026-03-14 02:50:22 - evalscope - INFO: Predicting[gsm8k@main]: 58%| 770/1319 [Elapsed: 25:04 < Remaining: 11:54, 1.30s/it]
2026-03-14 02:50:49 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=518, inflight=2)
2026-03-14 02:50:57 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=510, inflight=2)
2026-03-14 02:51:13 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=502, inflight=2)
2026-03-14 02:51:22 - evalscope - INFO: Predicting[gsm8k@main]: 61%| 801/1319 [Elapsed: 26:04 < Remaining: 15:25, 1.79s/it]
2026-03-14 02:51:31 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=494, inflight=2)
2026-03-14 02:51:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=486, inflight=2)
2026-03-14 02:52:00 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=478, inflight=2)
2026-03-14 02:52:09 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=470, inflight=2)
2026-03-14 02:52:22 - evalscope - INFO: Predicting[gsm8k@main]: 63%| 833/1319 [Elapsed: 27:04 < Remaining: 13:12, 1.63s/it]
2026-03-14 02:52:29 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=462, inflight=2)
2026-03-14 02:52:39 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=454, inflight=2)
2026-03-14 02:52:50 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=446, inflight=2)
2026-03-14 02:53:16 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=438, inflight=2)
2026-03-14 02:53:22 - evalscope - INFO: Predicting[gsm8k@main]: 66%| 865/1319 [Elapsed: 28:05 < Remaining: 15:47, 2.09s/it]
2026-03-14 02:53:27 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=430, inflight=2)
2026-03-14 02:53:33 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=422, inflight=2)
2026-03-14 02:54:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=414, inflight=2)
2026-03-14 02:54:22 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=406, inflight=2)
2026-03-14 02:54:22 - evalscope - INFO: Predicting[gsm8k@main]: 67%| 890/1319 [Elapsed: 29:05 < Remaining: 15:53, 2.22s/it]
2026-03-14 02:54:23 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=398, inflight=2)
2026-03-14 02:54:41 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=390, inflight=2)
2026-03-14 02:54:52 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=382, inflight=2)
2026-03-14 02:55:12 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=374, inflight=2)
2026-03-14 02:55:23 - evalscope - INFO: Predicting[gsm8k@main]: 70%| 929/1319 [Elapsed: 30:05 < Remaining: 12:28, 1.92s/it]
2026-03-14 02:55:28 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=366, inflight=2)
2026-03-14 02:55:43 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=358, inflight=2)
2026-03-14 02:55:51 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=350, inflight=2)
2026-03-14 02:56:23 - evalscope - INFO: Predicting[gsm8k@main]: 72%| 953/1319 [Elapsed: 31:05 < Remaining: 10:03, 1.65s/it]
2026-03-14 02:56:23 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=342, inflight=2)
2026-03-14 02:56:33 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=334, inflight=2)
2026-03-14 02:57:12 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=326, inflight=2)
2026-03-14 02:57:23 - evalscope - INFO: Predicting[gsm8k@main]: 74%| 977/1319 [Elapsed: 32:05 < Remaining: 16:12, 2.84s/it]
2026-03-14 02:57:24 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=318, inflight=2)
2026-03-14 02:57:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=310, inflight=2)
2026-03-14 02:57:36 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=302, inflight=2)
2026-03-14 02:58:15 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=294, inflight=2)
2026-03-14 02:58:20 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=286, inflight=2)
2026-03-14 02:58:23 - evalscope - INFO: Predicting[gsm8k@main]: 77%| 1017/1319 [Elapsed: 33:05 < Remaining: 10:01, 1.99s/it]
2026-03-14 02:58:51 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=278, inflight=2)
2026-03-14 02:59:01 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=270, inflight=2)
2026-03-14 02:59:23 - evalscope - INFO: Predicting[gsm8k@main]: 78%| 1033/1319 [Elapsed: 34:05 < Remaining: 10:11, 2.14s/it]
2026-03-14 02:59:40 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=262, inflight=2)
2026-03-14 02:59:50 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=254, inflight=2)
2026-03-14 03:00:05 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=246, inflight=2)
2026-03-14 03:00:11 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=238, inflight=2)
2026-03-14 03:00:23 - evalscope - INFO: Predicting[gsm8k@main]: 81%| 1065/1319 [Elapsed: 35:05 < Remaining: 07:51, 1.86s/it]
2026-03-14 03:00:38 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=230, inflight=2)
2026-03-14 03:01:04 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=222, inflight=2)
2026-03-14 03:01:23 - evalscope - INFO: Predicting[gsm8k@main]: 82%| 1081/1319 [Elapsed: 36:05 < Remaining: 10:08, 2.56s/it]
2026-03-14 03:01:26 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=214, inflight=2)
2026-03-14 03:01:37 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=206, inflight=2)
2026-03-14 03:01:50 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=198, inflight=2)
2026-03-14 03:01:52 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=190, inflight=2)
2026-03-14 03:02:10 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=182, inflight=2)
2026-03-14 03:02:23 - evalscope - INFO: Predicting[gsm8k@main]: 85%| 1121/1319 [Elapsed: 37:05 < Remaining: 05:50, 1.77s/it]
2026-03-14 03:02:26 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=174, inflight=2)
2026-03-14 03:02:39 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=166, inflight=2)
2026-03-14 03:02:57 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=158, inflight=2)
2026-03-14 03:02:58 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=150, inflight=2)
2026-03-14 03:03:23 - evalscope - INFO: Predicting[gsm8k@main]: 87%| 1153/1319 [Elapsed: 38:05 < Remaining: 05:20, 1.93s/it]
2026-03-14 03:03:47 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=142, inflight=2)
2026-03-14 03:03:48 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=134, inflight=2)
2026-03-14 03:04:05 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=126, inflight=2)
2026-03-14 03:04:23 - evalscope - INFO: Predicting[gsm8k@main]: 89%| 1177/1319 [Elapsed: 39:06 < Remaining: 04:35, 1.94s/it]
2026-03-14 03:04:29 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=118, inflight=2)
2026-03-14 03:04:53 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=110, inflight=2)
2026-03-14 03:05:14 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=102, inflight=2)
2026-03-14 03:05:19 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=94, inflight=2)
2026-03-14 03:05:23 - evalscope - INFO: Predicting[gsm8k@main]: 92%| 1209/1319 [Elapsed: 40:06 < Remaining: 03:35, 1.96s/it]
2026-03-14 03:05:56 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=86, inflight=2)
2026-03-14 03:06:03 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=78, inflight=2)
2026-03-14 03:06:15 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=70, inflight=2)
2026-03-14 03:06:23 - evalscope - INFO: Predicting[gsm8k@main]: 93%| 1233/1319 [Elapsed: 41:06 < Remaining: 02:53, 2.02s/it]
2026-03-14 03:06:33 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=62, inflight=2)
2026-03-14 03:06:38 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=54, inflight=2)
2026-03-14 03:06:57 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=46, inflight=2)
2026-03-14 03:07:15 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=38, inflight=2)
2026-03-14 03:07:24 - evalscope - INFO: Predicting[gsm8k@main]: 96%| 1265/1319 [Elapsed: 42:06 < Remaining: 01:46, 1.97s/it]
2026-03-14 03:07:28 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=30, inflight=2)
2026-03-14 03:07:48 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=22, inflight=2)
2026-03-14 03:07:54 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=14, inflight=2)
2026-03-14 03:08:15 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=6, inflight=2)
2026-03-14 03:08:16 - evalscope - INFO: Dispatcher: Worker-0 <- 6 prompts (pending=0, inflight=2)
2026-03-14 03:08:24 - evalscope - INFO: Predicting[gsm8k@main]: 99%| 1305/1319 [Elapsed: 43:06 < Remaining: 00:19, 1.41s/it]
2026-03-14 03:09:05 - evalscope - INFO: Predicting[gsm8k@main]: 100%| 1319/1319 [Elapsed: 43:47 < Remaining: 00:00, 2.01s/it]
2026-03-14 03:09:05 - evalscope - INFO: Finished getting predictions for subset: main.
2026-03-14 03:09:05 - evalscope - INFO: Getting reviews for subset: main
2026-03-14 03:09:05 - evalscope - INFO: Reviewing 1319 samples, if data is large, it may take a while.
2026-03-14 03:09:07 - evalscope - INFO: Reviewing[gsm8k@main]: 100%| 1319/1319 [Elapsed: 00:02 < Remaining: 00:00, 577.46it/s]
2026-03-14 03:09:07 - evalscope - INFO: Finished reviewing subset: main. Total reviewed: 1319
2026-03-14 03:09:07 - evalscope - INFO: Aggregating scores for subset: main
2026-03-14 03:09:07 - evalscope - INFO: Evaluating [gsm8k] 100%| 1/1 [Elapsed: 43:50 < Remaining: 00:00, 2630.58s/subset]
2026-03-14 03:09:07 - evalscope - INFO: Generating report...
2026-03-14 03:09:07 - evalscope - INFO:
gsm8k report table:
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | gsm8k | mean_acc | main | 1319 | 0.2745 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
2026-03-14 03:09:07 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-14 03:09:07 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/reports/llama3_1b_instruct_vallina_full_sft_30k/gsm8k.json
2026-03-14 03:09:07 - evalscope - INFO: Benchmark gsm8k evaluation finished.
2026-03-14 03:09:08 - evalscope - INFO: Overall report table:
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | gsm8k | mean_acc | main | 1319 | 0.2745 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
2026-03-14 03:09:08 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['gsm8k']
2026-03-14 03:09:08 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517
2026-03-14 03:09:08 - evalscope - INFO: [进度条] GSM8K 评测完成 ✓
2026-03-14 03:09:08 - evalscope - INFO: [断点续传] GSM8K 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/GSM8K_result.json
2026-03-14 03:09:08 - evalscope - INFO: 完成评测 GSM8K (6/8)
2026-03-14 03:09:08 - evalscope - INFO:
==================================================
2026-03-14 03:09:08 - evalscope - INFO: 正在评估 GPQA (repeat: 1次) (剩余: 1个)
2026-03-14 03:09:08 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-14 03:09:08 - evalscope - INFO: 开始创建 benchmark GPQA 的 TaskConfig
2026-03-14 03:09:08 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-14 03:09:08 - evalscope - INFO: [GPQA] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['gpqa_extend'], eval_batch_size=2048
2026-03-14 03:09:08 - evalscope - INFO: 开始评测 GPQA...
2026-03-14 03:09:08 - evalscope - INFO: [进度条] GPQA 开始评测
2026-03-14 03:09:08 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

3
evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/predictions/llama3_1b_instruct_vallina_full_sft_30k/gsm8k_main.jsonl Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2b4f039c7406e00e7279fef145528b807e54ac66c77df5197b3f9e0e1c6ed2a9
size 24861961

34
evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/reports/llama3_1b_instruct_vallina_full_sft_30k/gsm8k.json Normal file
View File

@@ -0,0 +1,34 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@gsm8k",
"dataset_name": "gsm8k",
"dataset_pretty_name": "GSM8K",
"dataset_description": "GSM8K (Grade School Math 8K) is a dataset of grade school math problems, designed to evaluate the mathematical reasoning abilities of AI models.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.2745,
"metrics": [
{
"name": "mean_acc",
"num": 1319,
"score": 0.2745,
"macro_score": 0.2745,
"categories": [
{
"name": [
"default"
],
"num": 1319,
"score": 0.2745,
"macro_score": 0.2745,
"subsets": [
{
"name": "main",
"score": 0.2745,
"num": 1319
}
]
}
]
}
],
"analysis": "N/A"
}

3
evalscope/version_20260314_002459/run_1/GSM8K/20260314_022517/reviews/llama3_1b_instruct_vallina_full_sft_30k/gsm8k_main.jsonl Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5110a89b9c979e14b239f33eeb8813ca071b404b83ae26c18fb9aaef27d8e189
size 12986139

36
evalscope/version_20260314_002459/run_1/GSM8K_result.json Normal file
View File

@@ -0,0 +1,36 @@
{
"gsm8k": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@gsm8k",
"dataset_name": "gsm8k",
"dataset_pretty_name": "GSM8K",
"dataset_description": "GSM8K (Grade School Math 8K) is a dataset of grade school math problems, designed to evaluate the mathematical reasoning abilities of AI models.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.2745,
"metrics": [
{
"name": "mean_acc",
"num": 1319,
"score": 0.2745,
"macro_score": 0.2745,
"categories": [
{
"name": [
"default"
],
"num": 1319,
"score": 0.2745,
"macro_score": 0.2745,
"subsets": [
{
"name": "main",
"score": 0.2745,
"num": 1319
}
]
}
]
}
],
"analysis": "N/A"
}
}

83
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/configs/task_config_e54541.yaml Normal file
View File

@@ -0,0 +1,83 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
math_500:
aggregation: mean
dataset_id: AI-ModelScope/MATH-500
default_subset: default
description: MATH-500 is a benchmark for evaluating mathematical reasoning capabilities
of AI models. It consists of 500 diverse math problems across five levels of
difficulty, designed to test a model's ability to solve complex mathematical
problems by generating step-by-step solutions and providing the correct final
answer.
eval_split: test
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc:
numeric: true
name: math_500
output_types:
- generation
pretty_name: MATH-500
prompt_template: '{question}
Please reason step by step, and put your final answer within \boxed{{}}.'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- Level 1
- Level 2
- Level 3
- Level 4
- Level 5
system_prompt: null
tags:
- Math
- Reasoning
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- math_500
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 1
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config: {}
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MATH500/20260314_005728

302
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/logs/eval_log.log Normal file
View File

@@ -0,0 +1,302 @@
2026-03-14 00:57:28 - evalscope - INFO: Running with native backend
2026-03-14 00:57:28 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/configs/task_config_e54541.yaml
2026-03-14 00:57:28 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"math_500"
],
"dataset_args": {
"math_500": {
"name": "math_500",
"dataset_id": "AI-ModelScope/MATH-500",
"output_types": [
"generation"
],
"subset_list": [
"Level 1",
"Level 2",
"Level 3",
"Level 4",
"Level 5"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "test",
"prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
"few_shot_prompt_template": null,
"system_prompt": null,
"query_template": null,
"pretty_name": "MATH-500",
"description": "MATH-500 is a benchmark for evaluating mathematical reasoning capabilities of AI models. It consists of 500 diverse math problems across five levels of difficulty, designed to test a model's ability to solve complex mathematical problems by generating step-by-step solutions and providing the correct final answer.",
"tags": [
"Math",
"Reasoning"
],
"filters": null,
"metric_list": [
{
"acc": {
"numeric": true
}
}
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MATH500/20260314_005728",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 1,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {},
"evalscope_version": "1.4.2"
}
2026-03-14 00:57:28 - evalscope - INFO: Start loading benchmark dataset: math_500
2026-03-14 00:57:28 - evalscope - INFO: Start evaluating 5 subsets of the math_500: ['Level 1', 'Level 2', 'Level 3', 'Level 4', 'Level 5']
2026-03-14 00:57:28 - evalscope - INFO: Evaluating subset: Level 1
2026-03-14 00:57:28 - evalscope - INFO: Getting predictions for subset: Level 1
2026-03-14 00:57:28 - evalscope - INFO: Processing 43 samples, if data is large, it may take a while.
2026-03-14 00:57:28 - evalscope - INFO: Loading model for prediction...
2026-03-14 00:57:28 - evalscope - INFO: Model loaded successfully.
2026-03-14 00:57:28 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=1, inflight=1)
2026-03-14 00:57:28 - evalscope - INFO: Dispatcher: Worker-1 <- 1 prompts (pending=0, inflight=2)
2026-03-14 00:57:37 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=33, inflight=2)
2026-03-14 00:57:42 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=25, inflight=2)
2026-03-14 00:58:00 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=17, inflight=2)
2026-03-14 00:58:18 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=9, inflight=2)
2026-03-14 00:58:28 - evalscope - INFO: Predicting[math_500@Level 1]: 42%| 18/43 [Elapsed: 01:00 < Remaining: 01:34, 3.79s/it]
2026-03-14 00:58:48 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1, inflight=2)
2026-03-14 00:58:51 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=2)
2026-03-14 00:59:28 - evalscope - INFO: Predicting[math_500@Level 1]: 98%| 42/43 [Elapsed: 02:00 < Remaining: 00:03, 3.14s/it]
2026-03-14 00:59:38 - evalscope - INFO: Predicting[math_500@Level 1]: 100%| 43/43 [Elapsed: 02:09 < Remaining: 00:00, 2.51s/it]
2026-03-14 00:59:38 - evalscope - INFO: Finished getting predictions for subset: Level 1.
2026-03-14 00:59:38 - evalscope - INFO: Getting reviews for subset: Level 1
2026-03-14 00:59:38 - evalscope - INFO: Reviewing 43 samples, if data is large, it may take a while.
2026-03-14 00:59:39 - evalscope - INFO: Reviewing[math_500@Level 1]: 100%| 43/43 [Elapsed: 00:00 < Remaining: 00:00, 129.63it/s]
2026-03-14 00:59:39 - evalscope - INFO: Finished reviewing subset: Level 1. Total reviewed: 43
2026-03-14 00:59:39 - evalscope - INFO: Aggregating scores for subset: Level 1
2026-03-14 00:59:39 - evalscope - INFO: Evaluating [math_500] 20%| 1/5 [Elapsed: 02:10 < Remaining: 08:41, 130.28s/subset]
2026-03-14 00:59:39 - evalscope - INFO: Evaluating subset: Level 2
2026-03-14 00:59:39 - evalscope - INFO: Getting predictions for subset: Level 2
2026-03-14 00:59:39 - evalscope - INFO: Processing 90 samples, if data is large, it may take a while.
2026-03-14 00:59:39 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 00:59:39 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=81, inflight=2)
2026-03-14 00:59:50 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=73, inflight=2)
2026-03-14 01:00:29 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=65, inflight=2)
2026-03-14 01:00:36 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=57, inflight=2)
2026-03-14 01:00:39 - evalscope - INFO: Predicting[math_500@Level 2]: 19%| 17/90 [Elapsed: 01:00 < Remaining: 05:19, 4.38s/it]
2026-03-14 01:01:20 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=49, inflight=2)
2026-03-14 01:01:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=41, inflight=2)
2026-03-14 01:01:39 - evalscope - INFO: Predicting[math_500@Level 2]: 37%| 33/90 [Elapsed: 02:00 < Remaining: 02:48, 2.96s/it]
2026-03-14 01:01:55 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=33, inflight=2)
2026-03-14 01:01:57 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=25, inflight=2)
2026-03-14 01:02:35 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=17, inflight=2)
2026-03-14 01:02:39 - evalscope - INFO: Predicting[math_500@Level 2]: 63%| 57/90 [Elapsed: 03:00 < Remaining: 01:42, 3.10s/it]
2026-03-14 01:02:47 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=9, inflight=2)
2026-03-14 01:03:14 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=1, inflight=2)
2026-03-14 01:03:28 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=2)
2026-03-14 01:03:39 - evalscope - INFO: Predicting[math_500@Level 2]: 90%| 81/90 [Elapsed: 04:00 < Remaining: 00:22, 2.50s/it]
2026-03-14 01:04:08 - evalscope - INFO: Predicting[math_500@Level 2]: 100%| 90/90 [Elapsed: 04:29 < Remaining: 00:00, 3.32s/it]
2026-03-14 01:04:08 - evalscope - INFO: Finished getting predictions for subset: Level 2.
2026-03-14 01:04:08 - evalscope - INFO: Getting reviews for subset: Level 2
2026-03-14 01:04:08 - evalscope - INFO: Reviewing 90 samples, if data is large, it may take a while.
2026-03-14 01:04:09 - evalscope - INFO: Reviewing[math_500@Level 2]: 100%| 90/90 [Elapsed: 00:00 < Remaining: 00:00, 98.18it/s]
2026-03-14 01:04:09 - evalscope - INFO: Finished reviewing subset: Level 2. Total reviewed: 90
2026-03-14 01:04:09 - evalscope - INFO: Aggregating scores for subset: Level 2
2026-03-14 01:04:09 - evalscope - INFO: Evaluating [math_500] 40%| 2/5 [Elapsed: 06:40 < Remaining: 10:37, 212.50s/subset]
2026-03-14 01:04:09 - evalscope - INFO: Evaluating subset: Level 3
2026-03-14 01:04:09 - evalscope - INFO: Getting predictions for subset: Level 3
2026-03-14 01:04:09 - evalscope - INFO: Processing 105 samples, if data is large, it may take a while.
2026-03-14 01:04:09 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 01:04:09 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=96, inflight=2)
2026-03-14 01:04:21 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=88, inflight=2)
2026-03-14 01:04:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=80, inflight=2)
2026-03-14 01:05:09 - evalscope - INFO: Predicting[math_500@Level 3]: 9%| 9/105 [Elapsed: 01:00 < Remaining: 29:34, 18.48s/it]
2026-03-14 01:05:11 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=72, inflight=2)
2026-03-14 01:05:36 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=64, inflight=2)
2026-03-14 01:06:02 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=56, inflight=2)
2026-03-14 01:06:09 - evalscope - INFO: Predicting[math_500@Level 3]: 31%| 33/105 [Elapsed: 02:00 < Remaining: 04:31, 3.78s/it]
2026-03-14 01:06:28 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=48, inflight=2)
2026-03-14 01:06:56 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=40, inflight=2)
2026-03-14 01:07:09 - evalscope - INFO: Predicting[math_500@Level 3]: 47%| 49/105 [Elapsed: 03:00 < Remaining: 03:16, 3.50s/it]
2026-03-14 01:07:12 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=32, inflight=2)
2026-03-14 01:07:35 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=24, inflight=2)
2026-03-14 01:07:57 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=16, inflight=2)
2026-03-14 01:08:09 - evalscope - INFO: Predicting[math_500@Level 3]: 70%| 73/105 [Elapsed: 04:00 < Remaining: 01:32, 2.88s/it]
2026-03-14 01:08:27 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=8, inflight=2)
2026-03-14 01:08:48 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=0, inflight=2)
2026-03-14 01:09:09 - evalscope - INFO: Predicting[math_500@Level 3]: 92%| 97/105 [Elapsed: 05:00 < Remaining: 00:21, 2.75s/it]
2026-03-14 01:09:30 - evalscope - INFO: Predicting[math_500@Level 3]: 100%| 105/105 [Elapsed: 05:21 < Remaining: 00:00, 2.82s/it]
2026-03-14 01:09:30 - evalscope - INFO: Finished getting predictions for subset: Level 3.
2026-03-14 01:09:30 - evalscope - INFO: Getting reviews for subset: Level 3
2026-03-14 01:09:30 - evalscope - INFO: Reviewing 105 samples, if data is large, it may take a while.
2026-03-14 01:09:31 - evalscope - INFO: Reviewing[math_500@Level 3]: 100%| 105/105 [Elapsed: 00:00 < Remaining: 00:00, 118.89it/s]
2026-03-14 01:09:31 - evalscope - INFO: Finished reviewing subset: Level 3. Total reviewed: 105
2026-03-14 01:09:31 - evalscope - INFO: Aggregating scores for subset: Level 3
2026-03-14 01:09:31 - evalscope - INFO: Evaluating [math_500] 60%| 3/5 [Elapsed: 12:02 < Remaining: 08:45, 262.53s/subset]
2026-03-14 01:09:31 - evalscope - INFO: Evaluating subset: Level 4
2026-03-14 01:09:31 - evalscope - INFO: Getting predictions for subset: Level 4
2026-03-14 01:09:31 - evalscope - INFO: Processing 128 samples, if data is large, it may take a while.
2026-03-14 01:09:31 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 01:09:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=119, inflight=2)
2026-03-14 01:09:57 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=111, inflight=2)
2026-03-14 01:10:22 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=103, inflight=2)
2026-03-14 01:10:31 - evalscope - INFO: Predicting[math_500@Level 4]: 7%| 9/128 [Elapsed: 01:00 < Remaining: 50:04, 25.25s/it]
2026-03-14 01:10:50 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=95, inflight=2)
2026-03-14 01:11:15 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=87, inflight=2)
2026-03-14 01:11:31 - evalscope - INFO: Predicting[math_500@Level 4]: 20%| 25/128 [Elapsed: 02:00 < Remaining: 07:53, 4.60s/it]
2026-03-14 01:11:41 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=79, inflight=2)
2026-03-14 01:12:07 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=71, inflight=2)
2026-03-14 01:12:31 - evalscope - INFO: Predicting[math_500@Level 4]: 32%| 41/128 [Elapsed: 03:00 < Remaining: 05:24, 3.73s/it]
2026-03-14 01:12:35 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=63, inflight=2)
2026-03-14 01:13:01 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=55, inflight=2)
2026-03-14 01:13:18 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=47, inflight=2)
2026-03-14 01:13:31 - evalscope - INFO: Predicting[math_500@Level 4]: 51%| 65/128 [Elapsed: 04:00 < Remaining: 03:10, 3.02s/it]
2026-03-14 01:13:52 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=39, inflight=2)
2026-03-14 01:14:15 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=31, inflight=2)
2026-03-14 01:14:31 - evalscope - INFO: Predicting[math_500@Level 4]: 63%| 81/128 [Elapsed: 05:00 < Remaining: 02:33, 3.27s/it]
2026-03-14 01:14:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=23, inflight=2)
2026-03-14 01:15:08 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=15, inflight=2)
2026-03-14 01:15:31 - evalscope - INFO: Predicting[math_500@Level 4]: 76%| 97/128 [Elapsed: 06:00 < Remaining: 01:40, 3.23s/it]
2026-03-14 01:15:38 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=7, inflight=2)
2026-03-14 01:15:49 - evalscope - INFO: Dispatcher: Worker-0 <- 7 prompts (pending=0, inflight=2)
2026-03-14 01:16:31 - evalscope - INFO: Predicting[math_500@Level 4]: 88%| 113/128 [Elapsed: 07:00 < Remaining: 00:42, 2.82s/it]
2026-03-14 01:16:44 - evalscope - INFO: Predicting[math_500@Level 4]: 100%| 128/128 [Elapsed: 07:13 < Remaining: 00:00, 2.93s/it]
2026-03-14 01:16:44 - evalscope - INFO: Finished getting predictions for subset: Level 4.
2026-03-14 01:16:44 - evalscope - INFO: Getting reviews for subset: Level 4
2026-03-14 01:16:44 - evalscope - INFO: Reviewing 128 samples, if data is large, it may take a while.
2026-03-14 01:16:46 - evalscope - INFO: Reviewing[math_500@Level 4]: 100%| 128/128 [Elapsed: 00:01 < Remaining: 00:00, 1.02s/it]
2026-03-14 01:16:46 - evalscope - INFO: Finished reviewing subset: Level 4. Total reviewed: 128
2026-03-14 01:16:46 - evalscope - INFO: Aggregating scores for subset: Level 4
2026-03-14 01:16:46 - evalscope - INFO: Evaluating [math_500] 80%| 4/5 [Elapsed: 19:17 < Remaining: 05:30, 330.67s/subset]
2026-03-14 01:16:46 - evalscope - INFO: Evaluating subset: Level 5
2026-03-14 01:16:46 - evalscope - INFO: Getting predictions for subset: Level 5
2026-03-14 01:16:46 - evalscope - INFO: Processing 134 samples, if data is large, it may take a while.
2026-03-14 01:16:46 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-14 01:16:46 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=125, inflight=2)
2026-03-14 01:17:08 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=117, inflight=2)
2026-03-14 01:17:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=109, inflight=2)
2026-03-14 01:17:46 - evalscope - INFO: Predicting[math_500@Level 5]: 7%| 9/134 [Elapsed: 01:00 < Remaining: 1:02:39, 30.07s/it]
2026-03-14 01:18:03 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=101, inflight=2)
2026-03-14 01:18:40 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=93, inflight=2)
2026-03-14 01:18:46 - evalscope - INFO: Predicting[math_500@Level 5]: 19%| 25/134 [Elapsed: 02:00 < Remaining: 09:36, 5.29s/it]
2026-03-14 01:19:01 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=85, inflight=2)
2026-03-14 01:19:35 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=77, inflight=2)
2026-03-14 01:19:46 - evalscope - INFO: Predicting[math_500@Level 5]: 31%| 41/134 [Elapsed: 03:00 < Remaining: 06:29, 4.18s/it]
2026-03-14 01:19:55 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=69, inflight=2)
2026-03-14 01:20:28 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=61, inflight=2)
2026-03-14 01:20:46 - evalscope - INFO: Predicting[math_500@Level 5]: 43%| 57/134 [Elapsed: 04:00 < Remaining: 04:50, 3.77s/it]
2026-03-14 01:20:49 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=53, inflight=2)
2026-03-14 01:21:22 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=45, inflight=2)
2026-03-14 01:21:46 - evalscope - INFO: Predicting[math_500@Level 5]: 54%| 73/134 [Elapsed: 05:00 < Remaining: 03:38, 3.59s/it]
2026-03-14 01:21:46 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=37, inflight=2)
2026-03-14 01:22:16 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=29, inflight=2)
2026-03-14 01:22:39 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=21, inflight=2)
2026-03-14 01:22:46 - evalscope - INFO: Predicting[math_500@Level 5]: 72%| 97/134 [Elapsed: 06:00 < Remaining: 02:02, 3.30s/it]
2026-03-14 01:23:09 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=13, inflight=2)
2026-03-14 01:23:33 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=5, inflight=2)
2026-03-14 01:23:46 - evalscope - INFO: Predicting[math_500@Level 5]: 84%| 113/134 [Elapsed: 07:00 < Remaining: 01:10, 3.34s/it]
2026-03-14 01:24:03 - evalscope - INFO: Dispatcher: Worker-1 <- 5 prompts (pending=0, inflight=2)
2026-03-14 01:24:46 - evalscope - INFO: Predicting[math_500@Level 5]: 96%| 129/134 [Elapsed: 08:00 < Remaining: 00:17, 3.49s/it]
2026-03-14 01:24:56 - evalscope - INFO: Predicting[math_500@Level 5]: 100%| 134/134 [Elapsed: 08:09 < Remaining: 00:00, 3.33s/it]
2026-03-14 01:24:56 - evalscope - INFO: Finished getting predictions for subset: Level 5.
2026-03-14 01:24:56 - evalscope - INFO: Getting reviews for subset: Level 5
2026-03-14 01:24:56 - evalscope - INFO: Reviewing 134 samples, if data is large, it may take a while.
2026-03-14 01:24:58 - evalscope - INFO: Reviewing[math_500@Level 5]: 100%| 134/134 [Elapsed: 00:01 < Remaining: 00:00, 1.01s/it]
2026-03-14 01:24:58 - evalscope - INFO: Finished reviewing subset: Level 5. Total reviewed: 134
2026-03-14 01:24:58 - evalscope - INFO: Aggregating scores for subset: Level 5
2026-03-14 01:24:58 - evalscope - INFO: Evaluating [math_500] 100%| 5/5 [Elapsed: 27:29 < Remaining: 00:00, 388.75s/subset]
2026-03-14 01:24:58 - evalscope - INFO: Generating report...
2026-03-14 01:24:58 - evalscope - INFO:
math_500 report table:
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 1 | 43 | 0.5116 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 2 | 90 | 0.3556 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 3 | 105 | 0.2095 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 4 | 128 | 0.0625 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 5 | 134 | 0.0299 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | OVERALL | 500 | 0.176 | - |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
2026-03-14 01:24:58 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-14 01:24:58 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/reports/llama3_1b_instruct_vallina_full_sft_30k/math_500.json
2026-03-14 01:24:58 - evalscope - INFO: Benchmark math_500 evaluation finished.
2026-03-14 01:24:58 - evalscope - INFO: Overall report table:
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+===========+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 1 | 43 | 0.5116 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 2 | 90 | 0.3556 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 3 | 105 | 0.2095 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 4 | 128 | 0.0625 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | Level 5 | 134 | 0.0299 | default |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
| llama3_1b_instruct_vallina_full_sft_30k | math_500 | mean_acc | OVERALL | 500 | 0.176 | - |
+-----------------------------------------+-----------+----------+----------+-------+---------+---------+
2026-03-14 01:24:58 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['math_500']
2026-03-14 01:24:58 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MATH500/20260314_005728
2026-03-14 01:24:58 - evalscope - INFO: [进度条] MATH500 评测完成 ✓
2026-03-14 01:24:58 - evalscope - INFO: [断点续传] MATH500 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MATH500_result.json
2026-03-14 01:24:58 - evalscope - INFO: 完成评测 MATH500 (3/8)
2026-03-14 01:24:58 - evalscope - INFO:
==================================================
2026-03-14 01:24:58 - evalscope - INFO: 正在评估 AMC (repeat: 1次) (剩余: 4个)
2026-03-14 01:24:58 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-14 01:24:58 - evalscope - INFO: 开始创建 benchmark AMC 的 TaskConfig
2026-03-14 01:24:58 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-14 01:24:58 - evalscope - INFO: [AMC] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['amc'], eval_batch_size=2048
2026-03-14 01:24:58 - evalscope - INFO: 开始评测 AMC...
2026-03-14 01:24:58 - evalscope - INFO: [进度条] AMC 开始评测
2026-03-14 01:24:58 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

43
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/predictions/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 1.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

90
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/predictions/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 2.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

105
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/predictions/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 3.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

128
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/predictions/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 4.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

134
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/predictions/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 5.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

54
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/reports/llama3_1b_instruct_vallina_full_sft_30k/math_500.json Normal file
View File

@@ -0,0 +1,54 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@math_500",
"dataset_name": "math_500",
"dataset_pretty_name": "MATH-500",
"dataset_description": "MATH-500 is a benchmark for evaluating mathematical reasoning capabilities of AI models. It consists of 500 diverse math problems across five levels of difficulty, designed to test a model's ability to solve complex mathematical problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.176,
"metrics": [
{
"name": "mean_acc",
"num": 500,
"score": 0.176,
"macro_score": 0.176,
"categories": [
{
"name": [
"default"
],
"num": 500,
"score": 0.176,
"macro_score": 0.2338,
"subsets": [
{
"name": "Level 1",
"score": 0.5116,
"num": 43
},
{
"name": "Level 2",
"score": 0.3556,
"num": 90
},
{
"name": "Level 3",
"score": 0.2095,
"num": 105
},
{
"name": "Level 4",
"score": 0.0625,
"num": 128
},
{
"name": "Level 5",
"score": 0.0299,
"num": 134
}
]
}
]
}
],
"analysis": "N/A"
}

43
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/reviews/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 1.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

90
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/reviews/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 2.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

105
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/reviews/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 3.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

128
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/reviews/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 4.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

134
evalscope/version_20260314_002459/run_1/MATH500/20260314_005728/reviews/llama3_1b_instruct_vallina_full_sft_30k/math_500_Level 5.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

56
evalscope/version_20260314_002459/run_1/MATH500_result.json Normal file
View File

@@ -0,0 +1,56 @@
{
"math_500": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@math_500",
"dataset_name": "math_500",
"dataset_pretty_name": "MATH-500",
"dataset_description": "MATH-500 is a benchmark for evaluating mathematical reasoning capabilities of AI models. It consists of 500 diverse math problems across five levels of difficulty, designed to test a model's ability to solve complex mathematical problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.176,
"metrics": [
{
"name": "mean_acc",
"num": 500,
"score": 0.176,
"macro_score": 0.176,
"categories": [
{
"name": [
"default"
],
"num": 500,
"score": 0.176,
"macro_score": 0.2338,
"subsets": [
{
"name": "Level 1",
"score": 0.5116,
"num": 43
},
{
"name": "Level 2",
"score": 0.3556,
"num": 90
},
{
"name": "Level 3",
"score": 0.2095,
"num": 105
},
{
"name": "Level 4",
"score": 0.0625,
"num": 128
},
{
"name": "Level 5",
"score": 0.0299,
"num": 134
}
]
}
]
}
],
"analysis": "N/A"
}
}

105
evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/configs/task_config_c118b7.yaml Normal file
View File

@@ -0,0 +1,105 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
mmlu_pro_no_math:
aggregation: mean
dataset_id: TIGER-Lab/MMLU-Pro
default_subset: default
description: MMLU-Pro benchmark excluding the math category.
eval_split: test
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: 'The following are multiple choice questions (with answers)
about {subject}. Think step by step and then finish your answer with ''ANSWER:
[LETTER]'' (without quotes) where [LETTER] is the correct letter choice.
{examples}
Answer the following multiple choice question. The last line of your response
should be of the following format: ''ANSWER: [LETTER]'' (without quotes) where
[LETTER] is one of {letters}. Think step by step before answering.
Question:
{question}
Options:
{choices}
'
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc
name: mmlu_pro_no_math
output_types:
- generation
pretty_name: MMLU-Pro (No Math)
prompt_template: 'Answer the following multiple choice question. The last line
of your response should be of the following format: ''ANSWER: [LETTER]'' (without
quotes) where [LETTER] is one of {letters}. Think step by step before answering.
Question:
{question}
Options:
{choices}
'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: null
tags:
- MCQ
- Knowledge
train_split: validation
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- mmlu_pro_no_math
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 1
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config: {}
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810

223
evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/logs/eval_log.log Normal file
View File

@@ -0,0 +1,223 @@
2026-03-15 15:58:10 - evalscope - INFO: Running with native backend
2026-03-15 15:58:10 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/configs/task_config_c118b7.yaml
2026-03-15 15:58:10 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"mmlu_pro_no_math"
],
"dataset_args": {
"mmlu_pro_no_math": {
"name": "mmlu_pro_no_math",
"dataset_id": "TIGER-Lab/MMLU-Pro",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": "validation",
"eval_split": "test",
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\nQuestion:\n{question}\nOptions:\n{choices}\n",
"few_shot_prompt_template": "The following are multiple choice questions (with answers) about {subject}. Think step by step and then finish your answer with 'ANSWER: [LETTER]' (without quotes) where [LETTER] is the correct letter choice.\n\n{examples}\nAnswer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\nQuestion:\n{question}\nOptions:\n{choices}\n",
"system_prompt": null,
"query_template": null,
"pretty_name": "MMLU-Pro (No Math)",
"description": "MMLU-Pro benchmark excluding the math category.",
"tags": [
"MCQ",
"Knowledge"
],
"filters": null,
"metric_list": [
"acc"
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 1,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {},
"evalscope_version": "1.4.2"
}
2026-03-15 15:58:10 - evalscope - INFO: Start loading benchmark dataset: mmlu_pro_no_math
2026-03-15 15:58:14 - evalscope - INFO: Start evaluating 1 subsets of the mmlu_pro_no_math: ['default']
2026-03-15 15:58:14 - evalscope - INFO: Evaluating subset: default
2026-03-15 15:58:14 - evalscope - INFO: Getting predictions for subset: default
2026-03-15 15:58:14 - evalscope - INFO: Processing 10681 samples, if data is large, it may take a while.
2026-03-15 15:58:14 - evalscope - INFO: Loading model for prediction...
2026-03-15 15:58:14 - evalscope - INFO: Model loaded successfully.
2026-03-15 15:58:14 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=1, inflight=1)
2026-03-15 15:58:14 - evalscope - INFO: Dispatcher: Worker-1 <- 1 prompts (pending=0, inflight=2)
2026-03-15 15:58:17 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1790, inflight=2)
2026-03-15 15:58:18 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1535, inflight=2)
2026-03-15 15:59:15 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 0%| 2/10681 [Elapsed: 01:00 < Remaining: 5:31:53, 1.86s/it]
2026-03-15 16:00:15 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 0%| 2/10681 [Elapsed: 02:00 < Remaining: 5:31:53, 1.86s/it]
2026-03-15 16:00:58 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:01:02 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:01:16 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 5%| 514/10681 [Elapsed: 03:01 < Remaining: 1:16:26, 2.22it/s]
2026-03-15 16:02:16 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 5%| 514/10681 [Elapsed: 04:01 < Remaining: 1:16:26, 2.22it/s]
2026-03-15 16:02:20 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:03:06 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:03:17 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 10%| 1026/10681 [Elapsed: 05:02 < Remaining: 45:22, 3.55it/s]
2026-03-15 16:03:24 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:04:01 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:04:11 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:04:17 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 17%| 1794/10681 [Elapsed: 06:02 < Remaining: 19:33, 7.57it/s]
2026-03-15 16:05:07 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:05:15 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:05:18 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 22%| 2306/10681 [Elapsed: 07:03 < Remaining: 16:34, 8.42it/s]
2026-03-15 16:05:55 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:06:18 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 24%| 2562/10681 [Elapsed: 08:03 < Remaining: 17:38, 7.67it/s]
2026-03-15 16:06:28 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:07:11 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:07:19 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 29%| 3074/10681 [Elapsed: 09:03 < Remaining: 17:54, 7.08it/s]
2026-03-15 16:08:01 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:08:19 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 31%| 3330/10681 [Elapsed: 10:03 < Remaining: 19:19, 6.34it/s]
2026-03-15 16:09:19 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 31%| 3330/10681 [Elapsed: 11:04 < Remaining: 19:19, 6.34it/s]
2026-03-15 16:10:10 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:10:20 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 34%| 3586/10681 [Elapsed: 12:04 < Remaining: 31:02, 3.81it/s]
2026-03-15 16:11:20 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 34%| 3586/10681 [Elapsed: 13:05 < Remaining: 31:02, 3.81it/s]
2026-03-15 16:12:03 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:12:20 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 36%| 3842/10681 [Elapsed: 14:05 < Remaining: 36:03, 3.16it/s]
2026-03-15 16:13:21 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 36%| 3842/10681 [Elapsed: 15:06 < Remaining: 36:03, 3.16it/s]
2026-03-15 16:13:50 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:14:21 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 38%| 4098/10681 [Elapsed: 16:06 < Remaining: 38:09, 2.88it/s]
2026-03-15 16:15:22 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 38%| 4098/10681 [Elapsed: 17:07 < Remaining: 38:09, 2.88it/s]
2026-03-15 16:15:54 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:16:22 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 41%| 4354/10681 [Elapsed: 18:07 < Remaining: 40:59, 2.57it/s]
2026-03-15 16:16:24 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:16:50 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:17:22 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 46%| 4866/10681 [Elapsed: 19:07 < Remaining: 23:49, 4.07it/s]
2026-03-15 16:17:40 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:18:10 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:18:23 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 50%| 5378/10681 [Elapsed: 20:07 < Remaining: 17:19, 5.10it/s]
2026-03-15 16:19:04 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:19:24 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 53%| 5634/10681 [Elapsed: 21:08 < Remaining: 16:56, 4.96it/s]
2026-03-15 16:19:26 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:20:12 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:20:13 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:20:24 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 60%| 6402/10681 [Elapsed: 22:09 < Remaining: 08:36, 8.29it/s]
2026-03-15 16:21:08 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:21:25 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 62%| 6658/10681 [Elapsed: 23:10 < Remaining: 09:57, 6.73it/s]
2026-03-15 16:21:40 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:22:26 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 65%| 6914/10681 [Elapsed: 24:11 < Remaining: 08:55, 7.03it/s]
2026-03-15 16:22:46 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:23:10 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:23:26 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 70%| 7426/10681 [Elapsed: 25:11 < Remaining: 08:12, 6.61it/s]
2026-03-15 16:24:26 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 70%| 7426/10681 [Elapsed: 26:11 < Remaining: 08:12, 6.61it/s]
2026-03-15 16:25:27 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 70%| 7426/10681 [Elapsed: 27:11 < Remaining: 08:12, 6.61it/s]
2026-03-15 16:25:48 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:26:27 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 72%| 7682/10681 [Elapsed: 28:12 < Remaining: 14:33, 3.43it/s]
2026-03-15 16:26:41 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:27:28 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 74%| 7938/10681 [Elapsed: 29:13 < Remaining: 12:08, 3.77it/s]
2026-03-15 16:28:28 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 74%| 7938/10681 [Elapsed: 30:13 < Remaining: 12:08, 3.77it/s]
2026-03-15 16:29:19 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:29:29 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 77%| 8194/10681 [Elapsed: 31:14 < Remaining: 15:23, 2.69it/s]
2026-03-15 16:30:28 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:30:30 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 79%| 8450/10681 [Elapsed: 32:14 < Remaining: 12:39, 2.94it/s]
2026-03-15 16:31:30 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 79%| 8450/10681 [Elapsed: 33:15 < Remaining: 12:39, 2.94it/s]
2026-03-15 16:32:30 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 79%| 8450/10681 [Elapsed: 34:15 < Remaining: 12:39, 2.94it/s]
2026-03-15 16:32:36 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=1280, inflight=2)
2026-03-15 16:33:00 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=1207, inflight=2)
2026-03-15 16:33:31 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 84%| 8962/10681 [Elapsed: 35:16 < Remaining: 08:36, 3.33it/s]
2026-03-15 16:34:15 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=951, inflight=2)
2026-03-15 16:34:32 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 86%| 9218/10681 [Elapsed: 36:16 < Remaining: 07:17, 3.35it/s]
2026-03-15 16:34:46 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=695, inflight=2)
2026-03-15 16:35:32 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 89%| 9474/10681 [Elapsed: 37:17 < Remaining: 03:30, 5.74it/s]
2026-03-15 16:35:36 - evalscope - INFO: Dispatcher: Worker-1 <- 256 prompts (pending=439, inflight=2)
2026-03-15 16:36:33 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 91%| 9730/10681 [Elapsed: 38:17 < Remaining: 04:00, 3.95it/s]
2026-03-15 16:37:33 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 91%| 9730/10681 [Elapsed: 39:17 < Remaining: 04:00, 3.95it/s]
2026-03-15 16:38:33 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 91%| 9730/10681 [Elapsed: 40:17 < Remaining: 04:00, 3.95it/s]
2026-03-15 16:39:01 - evalscope - INFO: Dispatcher: Worker-0 <- 256 prompts (pending=183, inflight=2)
2026-03-15 16:39:33 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 93%| 9986/10681 [Elapsed: 41:18 < Remaining: 05:20, 2.17it/s]
2026-03-15 16:39:57 - evalscope - INFO: Dispatcher: Worker-1 <- 183 prompts (pending=0, inflight=2)
2026-03-15 16:40:34 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 96%| 10242/10681 [Elapsed: 42:18 < Remaining: 02:44, 2.66it/s]
2026-03-15 16:41:34 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 96%| 10242/10681 [Elapsed: 43:19 < Remaining: 02:44, 2.66it/s]
2026-03-15 16:42:34 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 96%| 10242/10681 [Elapsed: 44:19 < Remaining: 02:44, 2.66it/s]
2026-03-15 16:43:14 - evalscope - INFO: Predicting[mmlu_pro_no_math@default]: 100%| 10681/10681 [Elapsed: 44:59 < Remaining: 00:00, 2.89it/s]
2026-03-15 16:43:14 - evalscope - INFO: Finished getting predictions for subset: default.
2026-03-15 16:43:14 - evalscope - INFO: Getting reviews for subset: default
2026-03-15 16:43:14 - evalscope - INFO: Reviewing 10681 samples, if data is large, it may take a while.
2026-03-15 16:43:29 - evalscope - INFO: Reviewing[mmlu_pro_no_math@default]: 100%| 10681/10681 [Elapsed: 00:14 < Remaining: 00:00, 890.28it/s]
2026-03-15 16:43:29 - evalscope - INFO: Finished reviewing subset: default. Total reviewed: 10681
2026-03-15 16:43:29 - evalscope - INFO: Aggregating scores for subset: default
2026-03-15 16:43:29 - evalscope - INFO: Evaluating [mmlu_pro_no_math] 100%| 1/1 [Elapsed: 45:14 < Remaining: 00:00, 2714.95s/subset]
2026-03-15 16:43:29 - evalscope - INFO: Generating report...
2026-03-15 16:43:29 - evalscope - INFO:
mmlu_pro_no_math report table:
+-----------------------------------------+------------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+==================+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | mmlu_pro_no_math | mean_acc | default | 10681 | 0.1637 | default |
+-----------------------------------------+------------------+----------+----------+-------+---------+---------+
2026-03-15 16:43:29 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-15 16:43:29 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/reports/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math.json
2026-03-15 16:43:29 - evalscope - INFO: Benchmark mmlu_pro_no_math evaluation finished.
2026-03-15 16:43:29 - evalscope - INFO: Overall report table:
+-----------------------------------------+------------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+==================+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | mmlu_pro_no_math | mean_acc | default | 10681 | 0.1637 | default |
+-----------------------------------------+------------------+----------+----------+-------+---------+---------+
2026-03-15 16:43:29 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['mmlu_pro_no_math']
2026-03-15 16:43:29 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810
2026-03-15 16:43:29 - evalscope - INFO: [进度条] MMLUProNoMath 评测完成 ✓
2026-03-15 16:43:29 - evalscope - INFO: [断点续传] MMLUProNoMath 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_1/MMLUProNoMath_result.json
2026-03-15 16:43:29 - evalscope - INFO: 完成评测 MMLUProNoMath (8/9)
2026-03-15 16:43:29 - evalscope - INFO:
==================================================
2026-03-15 16:43:29 - evalscope - INFO: 正在评估 ARC (repeat: 1次) (剩余: 0个)
2026-03-15 16:43:29 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-15 16:43:29 - evalscope - INFO: 开始创建 benchmark ARC 的 TaskConfig
2026-03-15 16:43:29 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 16:43:29 - evalscope - INFO: [ARC] worker_chunk_size=128
2026-03-15 16:43:29 - evalscope - INFO: [ARC] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['arc'], eval_batch_size=2048
2026-03-15 16:43:29 - evalscope - INFO: 开始评测 ARC...
2026-03-15 16:43:29 - evalscope - INFO: [进度条] ARC 开始评测
2026-03-15 16:43:29 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

3
evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/predictions/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math_default.jsonl Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:59fac3792b5cd91b97a02b9aafc9f0b8dc8b16563f60a5387dff9601662cf4cb
size 237778615

34
evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/reports/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math.json Normal file
View File

@@ -0,0 +1,34 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@mmlu_pro_no_math",
"dataset_name": "mmlu_pro_no_math",
"dataset_pretty_name": "MMLU-Pro (No Math)",
"dataset_description": "MMLU-Pro benchmark excluding the math category.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.1637,
"metrics": [
{
"name": "mean_acc",
"num": 10681,
"score": 0.1637,
"macro_score": 0.1637,
"categories": [
{
"name": [
"default"
],
"num": 10681,
"score": 0.1637,
"macro_score": 0.1637,
"subsets": [
{
"name": "default",
"score": 0.1637,
"num": 10681
}
]
}
]
}
],
"analysis": "N/A"
}

3
evalscope/version_20260314_002459/run_1/MMLUProNoMath/20260315_155810/reviews/llama3_1b_instruct_vallina_full_sft_30k/mmlu_pro_no_math_default.jsonl Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e26c32ddb7f8c36feb838e80c88f6961efe340678ead2d97a65325ce84786094
size 128493438

36
evalscope/version_20260314_002459/run_1/MMLUProNoMath_result.json Normal file
View File

@@ -0,0 +1,36 @@
{
"mmlu_pro_no_math": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@mmlu_pro_no_math",
"dataset_name": "mmlu_pro_no_math",
"dataset_pretty_name": "MMLU-Pro (No Math)",
"dataset_description": "MMLU-Pro benchmark excluding the math category.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.1637,
"metrics": [
{
"name": "mean_acc",
"num": 10681,
"score": 0.1637,
"macro_score": 0.1637,
"categories": [
{
"name": [
"default"
],
"num": 10681,
"score": 0.1637,
"macro_score": 0.1637,
"subsets": [
{
"name": "default",
"score": 0.1637,
"num": 10681
}
]
}
]
}
],
"analysis": "N/A"
}
}

73
evalscope/version_20260314_002459/run_1/config.yaml Normal file
View File

@@ -0,0 +1,73 @@
API_KEY: sk-9ae88192c0e54080831b8a7408d2da8d
BASE_URL: https://dashscope.aliyuncs.com/compatible-mode/v1
benchmarks:
AGIEvalMath:
dataset_name: agieval_math
few_shot_num: 0
max_tokens: 12000
repeats: 1
temperature: 1.0
AIME24:
dataset_name: aime24
few_shot_num: 0
max_tokens: 14000
repeats: 8
temperature: 0.6
use_sandbox: false
AIME25:
dataset_name: aime25
few_shot_num: 0
max_tokens: 14000
repeats: 8
temperature: 0.6
use_sandbox: false
AMC:
dataset_name: amc
few_shot_num: 0
max_tokens: 12000
repeats: 1
subset: amc23
temperature: 1.0
ARC:
dataset_name: arc
few_shot_num: 0
max_tokens: 12000
repeats: 1
temperature: 1.0
worker_chunk_size: 128
GPQA:
dataset_name: gpqa_extend
few_shot_num: 0
max_tokens: 12000
repeats: 1
temperature: 1.0
use_sandbox: false
GSM8K:
dataset_name: gsm8k
few_shot_num: 0
max_tokens: 12000
repeats: 1
temperature: 1.0
MATH500:
dataset_name: math_500
few_shot_num: 0
max_tokens: 12000
repeats: 1
temperature: 1.0
MMLUProNoMath:
dataset_name: mmlu_pro_no_math
few_shot_num: 0
max_tokens: 12000
repeats: 1
temperature: 1.0
worker_chunk_size: 256
model_path: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k
sandbox:
work_dir: /fs/fast/u20240075/luoyashuo/tmp/evalscope_sandbox
vllm:
dp: 2
enforce_eager: false
gpu_memory_utilization: 0.9
max_model_len: 16384
tp: 1
worker_chunk_size: 8

366
evalscope/version_20260314_002459/run_1/detailed_results.json Normal file
View File

@@ -0,0 +1,366 @@
{
"AIME25": {
"aime25": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@aime25",
"dataset_name": "aime25",
"dataset_pretty_name": "AIME-2025",
"dataset_description": "The AIME 2025 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0,
"metrics": [
{
"name": "mean_acc",
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"categories": [
{
"name": [
"default"
],
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"subsets": [
{
"name": "AIME2025-I",
"score": 0.0,
"num": 120
},
{
"name": "AIME2025-II",
"score": 0.0,
"num": 120
}
]
}
]
}
],
"analysis": "N/A"
}
},
"AIME24": {
"aime24": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@aime24",
"dataset_name": "aime24",
"dataset_pretty_name": "AIME-2024",
"dataset_description": "The AIME 2024 benchmark is based on problems from the American Invitational Mathematics Examination, a prestigious high school mathematics competition. This benchmark tests a model's ability to solve challenging mathematics problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0,
"metrics": [
{
"name": "mean_acc",
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"categories": [
{
"name": [
"default"
],
"num": 240,
"score": 0.0,
"macro_score": 0.0,
"subsets": [
{
"name": "default",
"score": 0.0,
"num": 240
}
]
}
]
}
],
"analysis": "N/A"
}
},
"MATH500": {
"math_500": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@math_500",
"dataset_name": "math_500",
"dataset_pretty_name": "MATH-500",
"dataset_description": "MATH-500 is a benchmark for evaluating mathematical reasoning capabilities of AI models. It consists of 500 diverse math problems across five levels of difficulty, designed to test a model's ability to solve complex mathematical problems by generating step-by-step solutions and providing the correct final answer.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.176,
"metrics": [
{
"name": "mean_acc",
"num": 500,
"score": 0.176,
"macro_score": 0.176,
"categories": [
{
"name": [
"default"
],
"num": 500,
"score": 0.176,
"macro_score": 0.2338,
"subsets": [
{
"name": "Level 1",
"score": 0.5116,
"num": 43
},
{
"name": "Level 2",
"score": 0.3556,
"num": 90
},
{
"name": "Level 3",
"score": 0.2095,
"num": 105
},
{
"name": "Level 4",
"score": 0.0625,
"num": 128
},
{
"name": "Level 5",
"score": 0.0299,
"num": 134
}
]
}
]
}
],
"analysis": "N/A"
}
},
"AMC": {
"amc": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@amc",
"dataset_name": "amc",
"dataset_pretty_name": "AMC",
"dataset_description": "AMC (American Mathematics Competitions) is a series of mathematics competitions for high school students.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.0448,
"metrics": [
{
"name": "mean_acc",
"num": 134,
"score": 0.0448,
"macro_score": 0.0448,
"categories": [
{
"name": [
"default"
],
"num": 134,
"score": 0.0448,
"macro_score": 0.0455,
"subsets": [
{
"name": "amc22",
"score": 0.093,
"num": 43
},
{
"name": "amc23",
"score": 0.0435,
"num": 46
},
{
"name": "amc24",
"score": 0.0,
"num": 45
}
]
}
]
}
],
"analysis": "N/A"
}
},
"AGIEvalMath": {
"agieval_math": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@agieval_math",
"dataset_name": "agieval_math",
"dataset_pretty_name": "AGIEval-Math",
"dataset_description": "AGIEval-Math is a subset of AGIEval containing 1000 competition-level math problems drawn from the MATH dataset, covering algebra, geometry, number theory and more. Answers are clean numerical or symbolic expressions.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.153,
"metrics": [
{
"name": "mean_acc",
"num": 1000,
"score": 0.153,
"macro_score": 0.153,
"categories": [
{
"name": [
"default"
],
"num": 1000,
"score": 0.153,
"macro_score": 0.153,
"subsets": [
{
"name": "default",
"score": 0.153,
"num": 1000
}
]
}
]
}
],
"analysis": "N/A"
}
},
"GSM8K": {
"gsm8k": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@gsm8k",
"dataset_name": "gsm8k",
"dataset_pretty_name": "GSM8K",
"dataset_description": "GSM8K (Grade School Math 8K) is a dataset of grade school math problems, designed to evaluate the mathematical reasoning abilities of AI models.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.2745,
"metrics": [
{
"name": "mean_acc",
"num": 1319,
"score": 0.2745,
"macro_score": 0.2745,
"categories": [
{
"name": [
"default"
],
"num": 1319,
"score": 0.2745,
"macro_score": 0.2745,
"subsets": [
{
"name": "main",
"score": 0.2745,
"num": 1319
}
]
}
]
}
],
"analysis": "N/A"
}
},
"GPQA": {
"gpqa_extend": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@gpqa_extend",
"dataset_name": "gpqa_extend",
"dataset_pretty_name": "GPQA-Extended",
"dataset_description": "GPQA Extended dataset for evaluating reasoning on graduate-level science problems.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.2454,
"metrics": [
{
"name": "mean_acc",
"num": 546,
"score": 0.2454,
"macro_score": 0.2454,
"categories": [
{
"name": [
"default"
],
"num": 546,
"score": 0.2454,
"macro_score": 0.2454,
"subsets": [
{
"name": "gpqa",
"score": 0.2454,
"num": 546
}
]
}
]
}
],
"analysis": "N/A"
}
},
"MMLUProNoMath": {
"mmlu_pro_no_math": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@mmlu_pro_no_math",
"dataset_name": "mmlu_pro_no_math",
"dataset_pretty_name": "MMLU-Pro (No Math)",
"dataset_description": "MMLU-Pro benchmark excluding the math category.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.1637,
"metrics": [
{
"name": "mean_acc",
"num": 10681,
"score": 0.1637,
"macro_score": 0.1637,
"categories": [
{
"name": [
"default"
],
"num": 10681,
"score": 0.1637,
"macro_score": 0.1637,
"subsets": [
{
"name": "default",
"score": 0.1637,
"num": 10681
}
]
}
]
}
],
"analysis": "N/A"
}
},
"ARC": {
"arc": {
"name": "llama3_1b_instruct_vallina_full_sft_30k@arc",
"dataset_name": "arc",
"dataset_pretty_name": "ARC",
"dataset_description": "The ARC (AI2 Reasoning Challenge) benchmark is designed to evaluate the reasoning capabilities of AI models through multiple-choice questions derived from science exams. It includes two subsets: ARC-Easy and ARC-Challenge, which vary in difficulty.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.4523,
"metrics": [
{
"name": "mean_acc",
"num": 3548,
"score": 0.4523,
"macro_score": 0.4523,
"categories": [
{
"name": [
"default"
],
"num": 3548,
"score": 0.4523,
"macro_score": 0.437,
"subsets": [
{
"name": "ARC-Easy",
"score": 0.4823,
"num": 2376
},
{
"name": "ARC-Challenge",
"score": 0.3916,
"num": 1172
}
]
}
]
}
],
"analysis": "N/A"
}
}
}

BIN
evalscope/version_20260314_002459/run_1/summary.xlsx Normal file
View File

Binary file not shown.

78
evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343/configs/task_config_2b162a.yaml Normal file
View File

@@ -0,0 +1,78 @@
analysis_report: false
api_url: null
chat_template: null
dataset_args:
agieval_math:
aggregation: mean
dataset_id: hails/agieval-math
default_subset: default
description: AGIEval-Math is a subset of AGIEval containing 1000 competition-level
math problems drawn from the MATH dataset, covering algebra, geometry, number
theory and more. Answers are clean numerical or symbolic expressions.
eval_split: test
extra_params: {}
few_shot_num: 0
few_shot_prompt_template: null
few_shot_random: false
filters: null
force_redownload: false
metric_list:
- acc:
numeric: true
name: agieval_math
output_types:
- generation
pretty_name: AGIEval-Math
prompt_template: '{question}
Please reason step by step, and put your final answer within \boxed{{}}.'
query_template: null
review_timeout: null
sandbox_config: {}
shuffle: false
shuffle_choices: false
subset_list:
- default
system_prompt: Please reason step by step to solve the problem. Put your final
answer in \boxed{}.
tags:
- Math
- Reasoning
train_split: null
dataset_dir: /fs/fast/u20240075/luoyashuo/modelscope_cache/datasets
dataset_hub: modelscope
datasets:
- agieval_math
debug: false
eval_backend: Native
eval_batch_size: 2048
eval_config: null
eval_type: mock_llm
evalscope_version: 1.4.2
generation_config:
batch_size: 2048
max_tokens: 12000
repetition_penalty: 1.0
temperature: 1.0
top_k: 50
top_p: 1.0
ignore_errors: false
judge_model_args: {}
judge_strategy: rule
judge_worker_num: 1
limit: null
model: VLLMOfflineModelAPI
model_args: {}
model_id: llama3_1b_instruct_vallina_full_sft_30k
model_task: text_generation
no_timestamp: false
repeats: 1
rerun_review: false
sandbox_manager_config: {}
sandbox_type: docker
seed: 42
stream: null
timeout: null
use_cache: null
use_sandbox: false
work_dir: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343

315
evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343/logs/eval_log.log Normal file
View File

@@ -0,0 +1,315 @@
2026-03-15 18:03:43 - evalscope - INFO: Running with native backend
2026-03-15 18:03:43 - evalscope - INFO: Dump task config to /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343/configs/task_config_2b162a.yaml
2026-03-15 18:03:43 - evalscope - INFO: {
"model": "VLLMOfflineModelAPI",
"model_id": "llama3_1b_instruct_vallina_full_sft_30k",
"model_args": {},
"model_task": "text_generation",
"chat_template": null,
"datasets": [
"agieval_math"
],
"dataset_args": {
"agieval_math": {
"name": "agieval_math",
"dataset_id": "hails/agieval-math",
"output_types": [
"generation"
],
"subset_list": [
"default"
],
"default_subset": "default",
"few_shot_num": 0,
"few_shot_random": false,
"train_split": null,
"eval_split": "test",
"prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
"few_shot_prompt_template": null,
"system_prompt": "Please reason step by step to solve the problem. Put your final answer in \\boxed{}.",
"query_template": null,
"pretty_name": "AGIEval-Math",
"description": "AGIEval-Math is a subset of AGIEval containing 1000 competition-level math problems drawn from the MATH dataset, covering algebra, geometry, number theory and more. Answers are clean numerical or symbolic expressions.",
"tags": [
"Math",
"Reasoning"
],
"filters": null,
"metric_list": [
{
"acc": {
"numeric": true
}
}
],
"aggregation": "mean",
"shuffle": false,
"shuffle_choices": false,
"force_redownload": false,
"review_timeout": null,
"extra_params": {},
"sandbox_config": {}
}
},
"dataset_dir": "/fs/fast/u20240075/luoyashuo/modelscope_cache/datasets",
"dataset_hub": "modelscope",
"repeats": 1,
"generation_config": {
"batch_size": 2048,
"max_tokens": 12000,
"top_p": 1.0,
"temperature": 1.0,
"repetition_penalty": 1.0,
"top_k": 50
},
"eval_type": "mock_llm",
"eval_backend": "Native",
"eval_config": null,
"limit": null,
"eval_batch_size": 2048,
"use_cache": null,
"rerun_review": false,
"work_dir": "/fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343",
"no_timestamp": false,
"ignore_errors": false,
"debug": false,
"seed": 42,
"api_url": null,
"timeout": null,
"stream": null,
"judge_strategy": "rule",
"judge_worker_num": 1,
"judge_model_args": {},
"analysis_report": false,
"use_sandbox": false,
"sandbox_type": "docker",
"sandbox_manager_config": {},
"evalscope_version": "1.4.2"
}
2026-03-15 18:03:43 - evalscope - INFO: Start loading benchmark dataset: agieval_math
2026-03-15 18:03:43 - evalscope - INFO: Start evaluating 1 subsets of the agieval_math: ['default']
2026-03-15 18:03:43 - evalscope - INFO: Evaluating subset: default
2026-03-15 18:03:43 - evalscope - INFO: Getting predictions for subset: default
2026-03-15 18:03:43 - evalscope - INFO: Processing 1000 samples, if data is large, it may take a while.
2026-03-15 18:03:43 - evalscope - INFO: Loading model for prediction...
2026-03-15 18:03:43 - evalscope - INFO: Model loaded successfully.
2026-03-15 18:03:43 - evalscope - INFO: Dispatcher: Worker-0 <- 1 prompts (pending=0, inflight=1)
2026-03-15 18:03:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=207, inflight=2)
2026-03-15 18:04:31 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=983, inflight=2)
2026-03-15 18:04:32 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=975, inflight=2)
2026-03-15 18:04:44 - evalscope - INFO: Predicting[agieval_math@default]: 1%| 9/1000 [Elapsed: 01:00 < Remaining: 5:39:54, 20.58s/it]
2026-03-15 18:05:18 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=967, inflight=2)
2026-03-15 18:05:24 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=959, inflight=2)
2026-03-15 18:05:44 - evalscope - INFO: Predicting[agieval_math@default]: 2%| 25/1000 [Elapsed: 02:00 < Remaining: 1:02:46, 3.86s/it]
2026-03-15 18:06:10 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=951, inflight=2)
2026-03-15 18:06:17 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=943, inflight=2)
2026-03-15 18:06:44 - evalscope - INFO: Predicting[agieval_math@default]: 4%| 41/1000 [Elapsed: 03:00 < Remaining: 50:58, 3.19s/it]
2026-03-15 18:07:01 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=935, inflight=2)
2026-03-15 18:07:08 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=927, inflight=2)
2026-03-15 18:07:44 - evalscope - INFO: Predicting[agieval_math@default]: 6%| 57/1000 [Elapsed: 04:00 < Remaining: 46:58, 2.99s/it]
2026-03-15 18:07:58 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=919, inflight=2)
2026-03-15 18:08:02 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=911, inflight=2)
2026-03-15 18:08:33 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=903, inflight=2)
2026-03-15 18:08:38 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=895, inflight=2)
2026-03-15 18:08:44 - evalscope - INFO: Predicting[agieval_math@default]: 9%| 89/1000 [Elapsed: 05:00 < Remaining: 36:32, 2.41s/it]
2026-03-15 18:09:16 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=887, inflight=2)
2026-03-15 18:09:25 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=879, inflight=2)
2026-03-15 18:09:45 - evalscope - INFO: Predicting[agieval_math@default]: 10%| 105/1000 [Elapsed: 06:01 < Remaining: 37:35, 2.52s/it]
2026-03-15 18:10:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=871, inflight=2)
2026-03-15 18:10:20 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=863, inflight=2)
2026-03-15 18:10:45 - evalscope - INFO: Predicting[agieval_math@default]: 12%| 121/1000 [Elapsed: 07:01 < Remaining: 41:48, 2.85s/it]
2026-03-15 18:10:53 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=855, inflight=2)
2026-03-15 18:10:59 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=847, inflight=2)
2026-03-15 18:11:45 - evalscope - INFO: Predicting[agieval_math@default]: 14%| 137/1000 [Elapsed: 08:01 < Remaining: 35:50, 2.49s/it]
2026-03-15 18:11:47 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=839, inflight=2)
2026-03-15 18:11:52 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=831, inflight=2)
2026-03-15 18:12:38 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=823, inflight=2)
2026-03-15 18:12:45 - evalscope - INFO: Predicting[agieval_math@default]: 16%| 161/1000 [Elapsed: 09:01 < Remaining: 49:55, 3.57s/it]
2026-03-15 18:12:46 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=815, inflight=2)
2026-03-15 18:13:29 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=807, inflight=2)
2026-03-15 18:13:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=799, inflight=2)
2026-03-15 18:13:45 - evalscope - INFO: Predicting[agieval_math@default]: 18%| 185/1000 [Elapsed: 10:01 < Remaining: 35:01, 2.58s/it]
2026-03-15 18:14:21 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=791, inflight=2)
2026-03-15 18:14:23 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=783, inflight=2)
2026-03-15 18:14:45 - evalscope - INFO: Predicting[agieval_math@default]: 20%| 201/1000 [Elapsed: 11:01 < Remaining: 35:21, 2.65s/it]
2026-03-15 18:14:55 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=775, inflight=2)
2026-03-15 18:15:13 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=767, inflight=2)
2026-03-15 18:15:42 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=759, inflight=2)
2026-03-15 18:15:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=751, inflight=2)
2026-03-15 18:15:45 - evalscope - INFO: Predicting[agieval_math@default]: 23%| 226/1000 [Elapsed: 12:01 < Remaining: 29:07, 2.26s/it]
2026-03-15 18:16:18 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=743, inflight=2)
2026-03-15 18:16:35 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=735, inflight=2)
2026-03-15 18:16:46 - evalscope - INFO: Predicting[agieval_math@default]: 25%| 249/1000 [Elapsed: 13:02 < Remaining: 32:43, 2.61s/it]
2026-03-15 18:17:11 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=727, inflight=2)
2026-03-15 18:17:32 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=719, inflight=2)
2026-03-15 18:17:46 - evalscope - INFO: Predicting[agieval_math@default]: 26%| 265/1000 [Elapsed: 14:02 < Remaining: 37:06, 3.03s/it]
2026-03-15 18:18:05 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=711, inflight=2)
2026-03-15 18:18:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=703, inflight=2)
2026-03-15 18:18:46 - evalscope - INFO: Predicting[agieval_math@default]: 28%| 281/1000 [Elapsed: 15:02 < Remaining: 36:17, 3.03s/it]
2026-03-15 18:18:49 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=695, inflight=2)
2026-03-15 18:19:15 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=687, inflight=2)
2026-03-15 18:19:42 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=679, inflight=2)
2026-03-15 18:19:46 - evalscope - INFO: Predicting[agieval_math@default]: 30%| 305/1000 [Elapsed: 16:02 < Remaining: 37:23, 3.23s/it]
2026-03-15 18:19:57 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=671, inflight=2)
2026-03-15 18:20:27 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=663, inflight=2)
2026-03-15 18:20:31 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=655, inflight=2)
2026-03-15 18:20:46 - evalscope - INFO: Predicting[agieval_math@default]: 33%| 329/1000 [Elapsed: 17:02 < Remaining: 25:39, 2.29s/it]
2026-03-15 18:21:17 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=647, inflight=2)
2026-03-15 18:21:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=639, inflight=2)
2026-03-15 18:21:46 - evalscope - INFO: Predicting[agieval_math@default]: 34%| 345/1000 [Elapsed: 18:02 < Remaining: 27:56, 2.56s/it]
2026-03-15 18:22:02 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=631, inflight=2)
2026-03-15 18:22:10 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=623, inflight=2)
2026-03-15 18:22:31 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=615, inflight=2)
2026-03-15 18:22:43 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=607, inflight=2)
2026-03-15 18:22:46 - evalscope - INFO: Predicting[agieval_math@default]: 38%| 377/1000 [Elapsed: 19:02 < Remaining: 23:33, 2.27s/it]
2026-03-15 18:23:28 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=599, inflight=2)
2026-03-15 18:23:35 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=591, inflight=2)
2026-03-15 18:23:47 - evalscope - INFO: Predicting[agieval_math@default]: 39%| 393/1000 [Elapsed: 20:03 < Remaining: 25:52, 2.56s/it]
2026-03-15 18:24:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=583, inflight=2)
2026-03-15 18:24:29 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=575, inflight=2)
2026-03-15 18:24:47 - evalscope - INFO: Predicting[agieval_math@default]: 41%| 409/1000 [Elapsed: 21:03 < Remaining: 27:07, 2.75s/it]
2026-03-15 18:24:59 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=567, inflight=2)
2026-03-15 18:25:21 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=559, inflight=2)
2026-03-15 18:25:47 - evalscope - INFO: Predicting[agieval_math@default]: 42%| 425/1000 [Elapsed: 22:03 < Remaining: 28:31, 2.98s/it]
2026-03-15 18:25:51 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=551, inflight=2)
2026-03-15 18:25:57 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=543, inflight=2)
2026-03-15 18:26:25 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=535, inflight=2)
2026-03-15 18:26:28 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=527, inflight=2)
2026-03-15 18:26:47 - evalscope - INFO: Predicting[agieval_math@default]: 46%| 457/1000 [Elapsed: 23:03 < Remaining: 18:26, 2.04s/it]
2026-03-15 18:27:18 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=519, inflight=2)
2026-03-15 18:27:19 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=511, inflight=2)
2026-03-15 18:27:47 - evalscope - INFO: Predicting[agieval_math@default]: 47%| 473/1000 [Elapsed: 24:03 < Remaining: 20:58, 2.39s/it]
2026-03-15 18:28:11 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=503, inflight=2)
2026-03-15 18:28:14 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=495, inflight=2)
2026-03-15 18:28:47 - evalscope - INFO: Predicting[agieval_math@default]: 49%| 489/1000 [Elapsed: 25:03 < Remaining: 22:34, 2.65s/it]
2026-03-15 18:28:51 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=487, inflight=2)
2026-03-15 18:29:06 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=479, inflight=2)
2026-03-15 18:29:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=471, inflight=2)
2026-03-15 18:29:47 - evalscope - INFO: Predicting[agieval_math@default]: 51%| 513/1000 [Elapsed: 26:03 < Remaining: 28:07, 3.46s/it]
2026-03-15 18:30:01 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=463, inflight=2)
2026-03-15 18:30:38 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=455, inflight=2)
2026-03-15 18:30:48 - evalscope - INFO: Predicting[agieval_math@default]: 53%| 529/1000 [Elapsed: 27:04 < Remaining: 27:20, 3.48s/it]
2026-03-15 18:30:48 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=447, inflight=2)
2026-03-15 18:31:22 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=439, inflight=2)
2026-03-15 18:31:46 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=431, inflight=2)
2026-03-15 18:31:48 - evalscope - INFO: Predicting[agieval_math@default]: 55%| 553/1000 [Elapsed: 28:04 < Remaining: 23:38, 3.17s/it]
2026-03-15 18:32:15 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=423, inflight=2)
2026-03-15 18:32:38 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=415, inflight=2)
2026-03-15 18:32:48 - evalscope - INFO: Predicting[agieval_math@default]: 57%| 569/1000 [Elapsed: 29:04 < Remaining: 22:35, 3.14s/it]
2026-03-15 18:33:07 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=407, inflight=2)
2026-03-15 18:33:31 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=399, inflight=2)
2026-03-15 18:33:48 - evalscope - INFO: Predicting[agieval_math@default]: 58%| 585/1000 [Elapsed: 30:04 < Remaining: 22:20, 3.23s/it]
2026-03-15 18:34:00 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=391, inflight=2)
2026-03-15 18:34:22 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=383, inflight=2)
2026-03-15 18:34:48 - evalscope - INFO: Predicting[agieval_math@default]: 60%| 601/1000 [Elapsed: 31:04 < Remaining: 20:50, 3.13s/it]
2026-03-15 18:34:54 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=375, inflight=2)
2026-03-15 18:35:09 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=367, inflight=2)
2026-03-15 18:35:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=359, inflight=2)
2026-03-15 18:35:48 - evalscope - INFO: Predicting[agieval_math@default]: 62%| 625/1000 [Elapsed: 32:04 < Remaining: 21:18, 3.41s/it]
2026-03-15 18:35:55 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=351, inflight=2)
2026-03-15 18:36:22 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=343, inflight=2)
2026-03-15 18:36:48 - evalscope - INFO: Predicting[agieval_math@default]: 64%| 641/1000 [Elapsed: 33:04 < Remaining: 17:34, 2.94s/it]
2026-03-15 18:36:48 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=335, inflight=2)
2026-03-15 18:37:13 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=327, inflight=2)
2026-03-15 18:37:43 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=319, inflight=2)
2026-03-15 18:37:48 - evalscope - INFO: Predicting[agieval_math@default]: 66%| 665/1000 [Elapsed: 34:04 < Remaining: 18:12, 3.26s/it]
2026-03-15 18:37:54 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=311, inflight=2)
2026-03-15 18:38:21 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=303, inflight=2)
2026-03-15 18:38:32 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=295, inflight=2)
2026-03-15 18:38:48 - evalscope - INFO: Predicting[agieval_math@default]: 69%| 689/1000 [Elapsed: 35:04 < Remaining: 12:40, 2.44s/it]
2026-03-15 18:39:11 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=287, inflight=2)
2026-03-15 18:39:27 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=279, inflight=2)
2026-03-15 18:39:46 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=271, inflight=2)
2026-03-15 18:39:48 - evalscope - INFO: Predicting[agieval_math@default]: 71%| 713/1000 [Elapsed: 36:04 < Remaining: 12:52, 2.69s/it]
2026-03-15 18:40:02 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=263, inflight=2)
2026-03-15 18:40:37 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=255, inflight=2)
2026-03-15 18:40:49 - evalscope - INFO: Predicting[agieval_math@default]: 73%| 729/1000 [Elapsed: 37:05 < Remaining: 13:44, 3.04s/it]
2026-03-15 18:40:50 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=247, inflight=2)
2026-03-15 18:41:31 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=239, inflight=2)
2026-03-15 18:41:41 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=231, inflight=2)
2026-03-15 18:41:49 - evalscope - INFO: Predicting[agieval_math@default]: 75%| 753/1000 [Elapsed: 38:05 < Remaining: 11:15, 2.74s/it]
2026-03-15 18:42:23 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=223, inflight=2)
2026-03-15 18:42:32 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=215, inflight=2)
2026-03-15 18:42:49 - evalscope - INFO: Predicting[agieval_math@default]: 77%| 769/1000 [Elapsed: 39:05 < Remaining: 10:42, 2.78s/it]
2026-03-15 18:43:03 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=207, inflight=2)
2026-03-15 18:43:24 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=199, inflight=2)
2026-03-15 18:43:49 - evalscope - INFO: Predicting[agieval_math@default]: 78%| 785/1000 [Elapsed: 40:05 < Remaining: 10:29, 2.93s/it]
2026-03-15 18:43:57 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=191, inflight=2)
2026-03-15 18:44:16 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=183, inflight=2)
2026-03-15 18:44:49 - evalscope - INFO: Predicting[agieval_math@default]: 80%| 801/1000 [Elapsed: 41:05 < Remaining: 10:05, 3.04s/it]
2026-03-15 18:44:49 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=175, inflight=2)
2026-03-15 18:45:09 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=167, inflight=2)
2026-03-15 18:45:36 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=159, inflight=2)
2026-03-15 18:45:49 - evalscope - INFO: Predicting[agieval_math@default]: 82%| 825/1000 [Elapsed: 42:05 < Remaining: 09:11, 3.15s/it]
2026-03-15 18:45:54 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=151, inflight=2)
2026-03-15 18:46:31 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=143, inflight=2)
2026-03-15 18:46:45 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=135, inflight=2)
2026-03-15 18:46:49 - evalscope - INFO: Predicting[agieval_math@default]: 85%| 849/1000 [Elapsed: 43:05 < Remaining: 07:22, 2.93s/it]
2026-03-15 18:47:22 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=127, inflight=2)
2026-03-15 18:47:38 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=119, inflight=2)
2026-03-15 18:47:45 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=111, inflight=2)
2026-03-15 18:47:49 - evalscope - INFO: Predicting[agieval_math@default]: 87%| 873/1000 [Elapsed: 44:05 < Remaining: 05:01, 2.38s/it]
2026-03-15 18:48:27 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=103, inflight=2)
2026-03-15 18:48:33 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=95, inflight=2)
2026-03-15 18:48:49 - evalscope - INFO: Predicting[agieval_math@default]: 89%| 889/1000 [Elapsed: 45:05 < Remaining: 04:32, 2.46s/it]
2026-03-15 18:49:18 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=87, inflight=2)
2026-03-15 18:49:26 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=79, inflight=2)
2026-03-15 18:49:49 - evalscope - INFO: Predicting[agieval_math@default]: 90%| 905/1000 [Elapsed: 46:05 < Remaining: 04:17, 2.71s/it]
2026-03-15 18:49:51 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=71, inflight=2)
2026-03-15 18:50:23 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=63, inflight=2)
2026-03-15 18:50:45 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=55, inflight=2)
2026-03-15 18:50:49 - evalscope - INFO: Predicting[agieval_math@default]: 93%| 929/1000 [Elapsed: 47:05 < Remaining: 03:35, 3.03s/it]
2026-03-15 18:50:56 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=47, inflight=2)
2026-03-15 18:51:35 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=39, inflight=2)
2026-03-15 18:51:49 - evalscope - INFO: Predicting[agieval_math@default]: 94%| 945/1000 [Elapsed: 48:05 < Remaining: 02:57, 3.23s/it]
2026-03-15 18:51:52 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=31, inflight=2)
2026-03-15 18:52:28 - evalscope - INFO: Dispatcher: Worker-0 <- 8 prompts (pending=23, inflight=2)
2026-03-15 18:52:44 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=15, inflight=2)
2026-03-15 18:52:49 - evalscope - INFO: Predicting[agieval_math@default]: 97%| 969/1000 [Elapsed: 49:05 < Remaining: 01:33, 3.00s/it]
2026-03-15 18:53:10 - evalscope - INFO: Dispatcher: Worker-1 <- 8 prompts (pending=7, inflight=2)
2026-03-15 18:53:20 - evalscope - INFO: Dispatcher: Worker-0 <- 7 prompts (pending=0, inflight=2)
2026-03-15 18:53:49 - evalscope - INFO: Predicting[agieval_math@default]: 98%| 985/1000 [Elapsed: 50:05 < Remaining: 00:37, 2.50s/it]
2026-03-15 18:53:53 - evalscope - INFO: Predicting[agieval_math@default]: 100%| 1000/1000 [Elapsed: 50:09 < Remaining: 00:00, 2.13s/it]
2026-03-15 18:53:53 - evalscope - INFO: Finished getting predictions for subset: default.
2026-03-15 18:53:53 - evalscope - INFO: Getting reviews for subset: default
2026-03-15 18:53:53 - evalscope - INFO: Reviewing 1000 samples, if data is large, it may take a while.
2026-03-15 18:54:04 - evalscope - INFO: Reviewing[agieval_math@default]: 100%| 1000/1000 [Elapsed: 00:10 < Remaining: 00:00, 62.57it/s]
2026-03-15 18:54:04 - evalscope - INFO: Finished reviewing subset: default. Total reviewed: 1000
2026-03-15 18:54:04 - evalscope - INFO: Aggregating scores for subset: default
2026-03-15 18:54:04 - evalscope - INFO: Evaluating [agieval_math] 100%| 1/1 [Elapsed: 50:20 < Remaining: 00:00, 3020.77s/subset]
2026-03-15 18:54:04 - evalscope - INFO: Generating report...
2026-03-15 18:54:04 - evalscope - INFO:
agieval_math report table:
+-----------------------------------------+--------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+==============+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | agieval_math | mean_acc | default | 1000 | 0.149 | default |
+-----------------------------------------+--------------+----------+----------+-------+---------+---------+
2026-03-15 18:54:04 - evalscope - INFO: Skipping report analysis (`analysis_report=False`).
2026-03-15 18:54:04 - evalscope - INFO: Dump report to: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343/reports/llama3_1b_instruct_vallina_full_sft_30k/agieval_math.json
2026-03-15 18:54:04 - evalscope - INFO: Benchmark agieval_math evaluation finished.
2026-03-15 18:54:04 - evalscope - INFO: Overall report table:
+-----------------------------------------+--------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=========================================+==============+==========+==========+=======+=========+=========+
| llama3_1b_instruct_vallina_full_sft_30k | agieval_math | mean_acc | default | 1000 | 0.149 | default |
+-----------------------------------------+--------------+----------+----------+-------+---------+---------+
2026-03-15 18:54:04 - evalscope - INFO: Finished evaluation for llama3_1b_instruct_vallina_full_sft_30k on ['agieval_math']
2026-03-15 18:54:04 - evalscope - INFO: Output directory: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343
2026-03-15 18:54:04 - evalscope - INFO: [进度条] AGIEvalMath 评测完成 ✓
2026-03-15 18:54:04 - evalscope - INFO: [断点续传] AGIEvalMath 结果已保存: /fs/fast/u20240075/luoyashuo/cdad/checkpoints/vallina_sft/llama3_1b_instruct_vallina_full_sft_30k/evalscope/version_20260314_002459/run_2/AGIEvalMath_result.json
2026-03-15 18:54:04 - evalscope - INFO: 完成评测 AGIEvalMath (5/9)
2026-03-15 18:54:04 - evalscope - INFO:
==================================================
2026-03-15 18:54:04 - evalscope - INFO: 正在评估 GSM8K (repeat: 1次) (剩余: 3个)
2026-03-15 18:54:04 - evalscope - INFO: 模型: llama3_1b_instruct_vallina_full_sft_30k
2026-03-15 18:54:04 - evalscope - INFO: 开始创建 benchmark GSM8K 的 TaskConfig
2026-03-15 18:54:04 - evalscope - INFO: No model is provided, using DummyCustomModel for testing.
2026-03-15 18:54:04 - evalscope - INFO: [GSM8K] worker_chunk_size=8
2026-03-15 18:54:04 - evalscope - INFO: [GSM8K] TaskConfig创建完成: model=llama3_1b_instruct_vallina_full_sft_30k, datasets=['gsm8k'], eval_batch_size=2048
2026-03-15 18:54:04 - evalscope - INFO: 开始评测 GSM8K...
2026-03-15 18:54:04 - evalscope - INFO: [进度条] GSM8K 开始评测
2026-03-15 18:54:04 - evalscope - INFO: Args: Task config is provided with TaskConfig type.

3
evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343/predictions/llama3_1b_instruct_vallina_full_sft_30k/agieval_math_default.jsonl Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ecd8b103085bba14a8e4fc16067c02ee2d110d7c83066b5ca889c6997c65c9a8
size 33753948

34
evalscope/version_20260314_002459/run_2/AGIEvalMath/20260315_180343/reports/llama3_1b_instruct_vallina_full_sft_30k/agieval_math.json Normal file
View File

@@ -0,0 +1,34 @@
{
"name": "llama3_1b_instruct_vallina_full_sft_30k@agieval_math",
"dataset_name": "agieval_math",
"dataset_pretty_name": "AGIEval-Math",
"dataset_description": "AGIEval-Math is a subset of AGIEval containing 1000 competition-level math problems drawn from the MATH dataset, covering algebra, geometry, number theory and more. Answers are clean numerical or symbolic expressions.",
"model_name": "llama3_1b_instruct_vallina_full_sft_30k",
"score": 0.149,
"metrics": [
{
"name": "mean_acc",
"num": 1000,
"score": 0.149,
"macro_score": 0.149,
"categories": [
{
"name": [
"default"
],
"num": 1000,
"score": 0.149,
"macro_score": 0.149,
"subsets": [
{
"name": "default",
"score": 0.149,
"num": 1000
}
]
}
]
}
],
"analysis": "N/A"
}

Some files were not shown because too many files have changed in this diff Show More