29 lines
2.2 KiB
Plaintext
29 lines
2.2 KiB
Plaintext
|
2026-08-10:09:45:52 WARNING [config.evaluate_config:287] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
|
||
|
|
2026-08-10:09:45:59 INFO [_cli.run:388] Selected Tasks: ['arc_challenge', 'winogrande', 'piqa', 'boolq', 'openbookqa', 'truthfulqa_mc1', 'lambada_openai', 'copa', 'rte', 'wsc', 'mmlu', 'sciq']
|
||
|
|
2026-08-10:09:45:59 INFO [evaluator:214] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
|
||
|
|
2026-08-10:09:45:59 INFO [evaluator:239] Initializing gguf model, with arguments: {'pretrained': '/tmp/miniart-2.0-q4_k_m.gguf', 'n_ctx': 2048, 'n_threads': 2}
|
||
|
|
Traceback (most recent call last):
|
||
|
|
File "/opt/hostedtoolcache/Python/3.11.15/x64/bin/lm_eval", line 6, in <module>
|
||
|
|
sys.exit(cli_evaluate())
|
||
|
|
^^^^^^^^^^^^^^
|
||
|
|
File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/lm_eval/__main__.py", line 10, in cli_evaluate
|
||
|
|
parser.execute(args)
|
||
|
|
File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/lm_eval/_cli/harness.py", line 60, in execute
|
||
|
|
args.func(args)
|
||
|
|
File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/lm_eval/_cli/run.py", line 391, in _execute
|
||
|
|
results = simple_evaluate(
|
||
|
|
^^^^^^^^^^^^^^^^
|
||
|
|
File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/lm_eval/utils.py", line 575, in _wrapper
|
||
|
|
return fn(*args, **kwargs)
|
||
|
|
^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/lm_eval/evaluator.py", line 242, in simple_evaluate
|
||
|
|
lm = lm_eval.api.registry.get_model(model).create_from_arg_obj(
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/lm_eval/api/model.py", line 169, in create_from_arg_obj
|
||
|
|
return cls(**arg_dict, **additional_config)
|
||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/lm_eval/models/gguf.py", line 41, in __init__
|
||
|
|
assert self.base_url, "must pass `base_url` to use GGUF LM!"
|
||
|
|
^^^^^^^^^^^^^
|
||
|
|
AssertionError: must pass `base_url` to use GGUF LM!
|