461 lines
57 KiB
Plaintext
461 lines
57 KiB
Plaintext
2026-07-18:14:38:06 WARNING [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
|
|
2026-07-18:14:38:10 INFO [_cli.run:376] Selected Tasks: ['babilong_longctx']
|
|
2026-07-18:14:38:11 INFO [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
|
2026-07-18:14:38:11 INFO [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
|
|
2026-07-18:14:38:15 INFO [models.huggingface:161] Using device 'cuda:0'
|
|
2026-07-18:14:38:15 INFO [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
|
|
2026-07-18:14:38:17 WARNING [api.task:856] babilong_qa5: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
|
|
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
|
|
|
|
|
|
README.md: 0%| | 0.00/12.0k [00:00<?, ?B/s][A
|
|
README.md: 100%|██████████| 12.0k/12.0k [00:00<00:00, 46.5MB/s]
|
|
|
|
|
|
8k/qa1-00000-of-00001.parquet: 0%| | 0.00/18.8M [00:00<?, ?B/s][A
|
|
8k/qa1-00000-of-00001.parquet: 100%|██████████| 18.8M/18.8M [00:00<00:00, 36.4MB/s]
|
|
|
|
|
|
8k/qa2-00000-of-00001.parquet: 0%| | 0.00/18.6M [00:00<?, ?B/s][A
|
|
8k/qa2-00000-of-00001.parquet: 100%|██████████| 18.6M/18.6M [00:00<00:00, 73.1MB/s]
|
|
|
|
|
|
8k/qa3-00000-of-00001.parquet: 0%| | 0.00/18.5M [00:00<?, ?B/s][A
|
|
8k/qa3-00000-of-00001.parquet: 100%|██████████| 18.5M/18.5M [00:00<00:00, 35.1MB/s]
|
|
|
|
|
|
8k/qa4-00000-of-00001.parquet: 0%| | 0.00/18.8M [00:00<?, ?B/s][A
|
|
8k/qa4-00000-of-00001.parquet: 100%|██████████| 18.8M/18.8M [00:00<00:00, 63.1MB/s]
|
|
|
|
|
|
8k/qa5-00000-of-00001.parquet: 0%| | 0.00/18.7M [00:00<?, ?B/s][A
|
|
8k/qa5-00000-of-00001.parquet: 100%|██████████| 18.7M/18.7M [00:00<00:00, 67.9MB/s]
|
|
|
|
|
|
Generating qa1 split: 0%| | 0/1000 [00:00<?, ? examples/s][A
|
|
Generating qa1 split: 100%|██████████| 1000/1000 [00:00<00:00, 5788.82 examples/s]
|
|
|
|
|
|
Generating qa2 split: 0%| | 0/999 [00:00<?, ? examples/s][A
|
|
Generating qa2 split: 100%|██████████| 999/999 [00:00<00:00, 9726.55 examples/s]
|
|
|
|
|
|
Generating qa3 split: 0%| | 0/999 [00:00<?, ? examples/s][A
|
|
Generating qa3 split: 100%|██████████| 999/999 [00:00<00:00, 10002.34 examples/s]
|
|
|
|
|
|
Generating qa4 split: 0%| | 0/999 [00:00<?, ? examples/s][A
|
|
Generating qa4 split: 100%|██████████| 999/999 [00:00<00:00, 10155.80 examples/s]
|
|
|
|
|
|
Generating qa5 split: 0%| | 0/999 [00:00<?, ? examples/s][A
|
|
Generating qa5 split: 100%|██████████| 999/999 [00:00<00:00, 11153.79 examples/s]
|
|
2026-07-18:14:38:20 WARNING [api.task:856] babilong_qa4: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
|
|
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
|
|
2026-07-18:14:38:20 WARNING [api.task:856] babilong_qa3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
|
|
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
|
|
2026-07-18:14:38:20 WARNING [api.task:856] babilong_qa2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
|
|
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
|
|
2026-07-18:14:38:20 WARNING [api.task:856] babilong_qa1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
|
|
For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
|
|
2026-07-18:14:38:20 INFO [tasks:700] Selected tasks:
|
|
2026-07-18:14:38:20 INFO [tasks:703] Group: babilong_longctx
|
|
2026-07-18:14:38:20 INFO [tasks:726] ConfigurableGroup(group=babilong_longctx,group_alias=None): {'babilong_qa1': ConfigurableTask(task_name=babilong_qa1,output_type=generate_until,num_fewshot=2,num_samples=1000), 'babilong_qa2': ConfigurableTask(task_name=babilong_qa2,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa3': ConfigurableTask(task_name=babilong_qa3,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa4': ConfigurableTask(task_name=babilong_qa4,output_type=generate_until,num_fewshot=2,num_samples=999), 'babilong_qa5': ConfigurableTask(task_name=babilong_qa5,output_type=generate_until,num_fewshot=2,num_samples=999)}
|
|
2026-07-18:14:38:20 INFO [evaluator:314] babilong_qa1: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
|
|
2026-07-18:14:38:20 WARNING [evaluator:333] Overwriting default num_fewshot of babilong_qa1 from 2 to 2
|
|
2026-07-18:14:38:20 INFO [evaluator:314] babilong_qa2: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
|
|
2026-07-18:14:38:20 WARNING [evaluator:333] Overwriting default num_fewshot of babilong_qa2 from 2 to 2
|
|
2026-07-18:14:38:20 INFO [evaluator:314] babilong_qa3: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
|
|
2026-07-18:14:38:20 WARNING [evaluator:333] Overwriting default num_fewshot of babilong_qa3 from 2 to 2
|
|
2026-07-18:14:38:20 INFO [evaluator:314] babilong_qa4: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
|
|
2026-07-18:14:38:20 WARNING [evaluator:333] Overwriting default num_fewshot of babilong_qa4 from 2 to 2
|
|
2026-07-18:14:38:20 INFO [evaluator:314] babilong_qa5: Using gen_kwargs: {'do_sample': False, 'temperature': 0.0, 'max_gen_toks': 16, 'until': []}
|
|
2026-07-18:14:38:20 WARNING [evaluator:333] Overwriting default num_fewshot of babilong_qa5 from 2 to 2
|
|
2026-07-18:14:38:20 INFO [api.task:311] Building contexts for babilong_qa1 on rank 0...
|
|
|
|
|
|
0%| | 0/50 [00:00<?, ?it/s][A
|
|
100%|██████████| 50/50 [00:00<00:00, 251.53it/s]
|
|
2026-07-18:14:38:20 INFO [api.task:311] Building contexts for babilong_qa2 on rank 0...
|
|
|
|
|
|
0%| | 0/50 [00:00<?, ?it/s][A
|
|
100%|██████████| 50/50 [00:00<00:00, 370.37it/s]
|
|
2026-07-18:14:38:21 INFO [api.task:311] Building contexts for babilong_qa3 on rank 0...
|
|
|
|
|
|
0%| | 0/50 [00:00<?, ?it/s][A
|
|
100%|██████████| 50/50 [00:00<00:00, 360.63it/s]
|
|
2026-07-18:14:38:21 INFO [api.task:311] Building contexts for babilong_qa4 on rank 0...
|
|
|
|
|
|
0%| | 0/50 [00:00<?, ?it/s][A
|
|
100%|██████████| 50/50 [00:00<00:00, 372.64it/s]
|
|
2026-07-18:14:38:21 INFO [api.task:311] Building contexts for babilong_qa5 on rank 0...
|
|
|
|
|
|
0%| | 0/50 [00:00<?, ?it/s][A
|
|
100%|██████████| 50/50 [00:00<00:00, 372.31it/s]
|
|
2026-07-18:14:38:21 INFO [evaluator:584] Running generate_until requests
|
|
|
|
|
|
Running generate_until requests: 0%| | 0/250 [00:00<?, ?it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 0%| | 1/250 [00:05<22:46, 5.49s/it][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 1%| | 3/250 [00:06<08:04, 1.96s/it][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 2%|▏ | 5/250 [00:08<05:24, 1.33s/it][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 3%|▎ | 7/250 [00:09<04:20, 1.07s/it][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 4%|▎ | 9/250 [00:11<03:46, 1.06it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 4%|▍ | 11/250 [00:12<03:26, 1.16it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 5%|▌ | 13/250 [00:14<03:14, 1.22it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 6%|▌ | 15/250 [00:15<03:05, 1.27it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 7%|▋ | 17/250 [00:17<02:59, 1.30it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 8%|▊ | 19/250 [00:18<02:54, 1.32it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 8%|▊ | 21/250 [00:20<02:50, 1.34it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 9%|▉ | 23/250 [00:21<02:48, 1.35it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 10%|█ | 25/250 [00:22<02:45, 1.36it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 11%|█ | 27/250 [00:24<02:43, 1.36it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 12%|█▏ | 29/250 [00:25<02:41, 1.37it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 12%|█▏ | 31/250 [00:27<02:39, 1.37it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 13%|█▎ | 33/250 [00:28<02:38, 1.37it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 14%|█▍ | 35/250 [00:30<02:36, 1.37it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 15%|█▍ | 37/250 [00:31<02:34, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 16%|█▌ | 39/250 [00:33<02:33, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 16%|█▋ | 41/250 [00:34<02:31, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 17%|█▋ | 43/250 [00:35<02:29, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 18%|█▊ | 45/250 [00:37<02:28, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 19%|█▉ | 47/250 [00:38<02:26, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 20%|█▉ | 49/250 [00:40<02:25, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 20%|██ | 51/250 [00:41<02:24, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 21%|██ | 53/250 [00:43<02:22, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 22%|██▏ | 55/250 [00:44<02:20, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 23%|██▎ | 57/250 [00:46<02:19, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 24%|██▎ | 59/250 [00:47<02:17, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 24%|██▍ | 61/250 [00:48<02:16, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 25%|██▌ | 63/250 [00:50<02:14, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 26%|██▌ | 65/250 [00:51<02:13, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 27%|██▋ | 67/250 [00:53<02:11, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 28%|██▊ | 69/250 [00:54<02:10, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 28%|██▊ | 71/250 [00:56<02:08, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 29%|██▉ | 73/250 [00:57<02:07, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 30%|███ | 75/250 [00:59<02:06, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 31%|███ | 77/250 [01:00<02:04, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 32%|███▏ | 79/250 [01:01<02:03, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 32%|███▏ | 81/250 [01:03<02:02, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 33%|███▎ | 83/250 [01:04<02:00, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 34%|███▍ | 85/250 [01:06<01:59, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 35%|███▍ | 87/250 [01:07<01:57, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 36%|███▌ | 89/250 [01:09<01:56, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 36%|███▋ | 91/250 [01:10<01:54, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 37%|███▋ | 93/250 [01:12<01:53, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 38%|███▊ | 95/250 [01:13<01:51, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 39%|███▉ | 97/250 [01:14<01:50, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 40%|███▉ | 99/250 [01:16<01:49, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 40%|████ | 101/250 [01:17<01:50, 1.34it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 41%|████ | 103/250 [01:19<01:48, 1.36it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 42%|████▏ | 105/250 [01:20<01:46, 1.36it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 43%|████▎ | 107/250 [01:22<01:44, 1.36it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 44%|████▎ | 109/250 [01:23<01:42, 1.37it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 44%|████▍ | 111/250 [01:25<01:41, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 45%|████▌ | 113/250 [01:26<01:39, 1.37it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 46%|████▌ | 115/250 [01:28<01:38, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 47%|████▋ | 117/250 [01:29<01:36, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 48%|████▊ | 119/250 [01:31<01:34, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 48%|████▊ | 121/250 [01:32<01:34, 1.37it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 49%|████▉ | 123/250 [01:33<01:32, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 50%|█████ | 125/250 [01:35<01:30, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 51%|█████ | 127/250 [01:36<01:29, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 52%|█████▏ | 129/250 [01:38<01:27, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 52%|█████▏ | 131/250 [01:39<01:26, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 53%|█████▎ | 133/250 [01:41<01:24, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 54%|█████▍ | 135/250 [01:42<01:23, 1.38it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 55%|█████▍ | 137/250 [01:44<01:21, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 56%|█████▌ | 139/250 [01:45<01:19, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 56%|█████▋ | 141/250 [01:46<01:18, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 57%|█████▋ | 143/250 [01:48<01:16, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 58%|█████▊ | 145/250 [01:49<01:15, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 59%|█████▉ | 147/250 [01:51<01:14, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 60%|█████▉ | 149/250 [01:52<01:12, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 60%|██████ | 151/250 [01:54<01:11, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 61%|██████ | 153/250 [01:55<01:09, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 62%|██████▏ | 155/250 [01:57<01:08, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 63%|██████▎ | 157/250 [01:58<01:06, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 64%|██████▎ | 159/250 [01:59<01:05, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 64%|██████▍ | 161/250 [02:01<01:04, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 65%|██████▌ | 163/250 [02:02<01:02, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 66%|██████▌ | 165/250 [02:04<01:01, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 67%|██████▋ | 167/250 [02:05<00:59, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 68%|██████▊ | 169/250 [02:07<00:58, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 68%|██████▊ | 171/250 [02:08<00:56, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 69%|██████▉ | 173/250 [02:09<00:55, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 70%|███████ | 175/250 [02:11<00:53, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 71%|███████ | 177/250 [02:12<00:52, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 72%|███████▏ | 179/250 [02:14<00:50, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 72%|███████▏ | 181/250 [02:15<00:49, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 73%|███████▎ | 183/250 [02:17<00:48, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 74%|███████▍ | 185/250 [02:18<00:46, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 75%|███████▍ | 187/250 [02:19<00:45, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 76%|███████▌ | 189/250 [02:21<00:43, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 76%|███████▋ | 191/250 [02:22<00:42, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 77%|███████▋ | 193/250 [02:24<00:40, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 78%|███████▊ | 195/250 [02:25<00:39, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 79%|███████▉ | 197/250 [02:27<00:37, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 80%|███████▉ | 199/250 [02:28<00:36, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 80%|████████ | 201/250 [02:30<00:35, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 81%|████████ | 203/250 [02:31<00:33, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 82%|████████▏ | 205/250 [02:32<00:32, 1.39it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 83%|████████▎ | 207/250 [02:34<00:30, 1.40it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 84%|████████▎ | 209/250 [02:35<00:29, 1.41it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 84%|████████▍ | 211/250 [02:37<00:27, 1.41it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 85%|████████▌ | 213/250 [02:38<00:26, 1.42it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 86%|████████▌ | 215/250 [02:39<00:24, 1.42it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 87%|████████▋ | 217/250 [02:41<00:23, 1.42it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 88%|████████▊ | 219/250 [02:42<00:21, 1.42it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 88%|████████▊ | 221/250 [02:44<00:20, 1.42it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 89%|████████▉ | 223/250 [02:45<00:18, 1.42it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 90%|█████████ | 225/250 [02:46<00:17, 1.43it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 91%|█████████ | 227/250 [02:48<00:16, 1.43it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 92%|█████████▏| 229/250 [02:49<00:14, 1.43it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 92%|█████████▏| 231/250 [02:51<00:13, 1.44it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 93%|█████████▎| 233/250 [02:52<00:11, 1.44it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 94%|█████████▍| 235/250 [02:53<00:10, 1.44it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 95%|█████████▍| 237/250 [02:55<00:08, 1.45it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 96%|█████████▌| 239/250 [02:56<00:07, 1.46it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 96%|█████████▋| 241/250 [02:57<00:06, 1.46it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 97%|█████████▋| 243/250 [02:59<00:04, 1.47it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 98%|█████████▊| 245/250 [03:00<00:03, 1.47it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 99%|█████████▉| 247/250 [03:01<00:02, 1.49it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
|
|
Running generate_until requests: 100%|█████████▉| 249/250 [03:03<00:00, 1.50it/s][AThe following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
|
|
|
|
Running generate_until requests: 100%|██████████| 250/250 [03:03<00:00, 1.36it/s]
|
|
2026-07-18:14:41:29 INFO [loggers.evaluation_tracker:247] Saving results aggregated
|
|
2026-07-18:14:41:29 INFO [loggers.evaluation_tracker:119] Saving per-task samples to /workspace/outputs/eval/vanilla/babilong8k_qa1_qa5_n50/__workspace/*.jsonl
|
|
hf ({'pretrained': '/workspace', 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}), gen_kwargs: ({}), limit: 50.0, num_fewshot: 2, batch_size: 1
|
|
| Tasks |Version|Filter|n-shot|Metric| |Value| |Stderr|
|
|
|----------------|------:|------|-----:|------|---|----:|---|-----:|
|
|
|babilong_longctx| 0|none | |acc |↑ | 0.60|± |0.0295|
|
|
| - babilong_qa1 | 0|none | 2|acc |↑ | 0.68|± |0.0666|
|
|
| - babilong_qa2 | 0|none | 2|acc |↑ | 0.48|± |0.0714|
|
|
| - babilong_qa3 | 0|none | 2|acc |↑ | 0.34|± |0.0677|
|
|
| - babilong_qa4 | 0|none | 2|acc |↑ | 0.72|± |0.0641|
|
|
| - babilong_qa5 | 0|none | 2|acc |↑ | 0.78|± |0.0592|
|
|
|
|
| Groups |Version|Filter|n-shot|Metric| |Value| |Stderr|
|
|
|----------------|------:|------|------|------|---|----:|---|-----:|
|
|
|babilong_longctx| 0|none | |acc |↑ | 0.6|± |0.0295|
|
|
|