115 lines
7.5 KiB
Markdown
115 lines
7.5 KiB
Markdown
---
|
|
license: apache-2.0
|
|
license_link: https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct/blob/main/LICENSE
|
|
language:
|
|
- zho
|
|
- eng
|
|
- fra
|
|
- spa
|
|
- por
|
|
- deu
|
|
- ita
|
|
- rus
|
|
- jpn
|
|
- kor
|
|
- vie
|
|
- tha
|
|
- ara
|
|
base_model: Qwen/Qwen2.5-0.5B
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- gptqmodel
|
|
- modelcloud
|
|
- chat
|
|
- qwen2
|
|
- instruct
|
|
- int4
|
|
- gptq
|
|
- 4bit
|
|
- W4A16
|
|
---
|
|
|
|
GPT-QModel quantized: [GPT-QModel](https://github.com/ModelCloud/GPTQModel).
|
|
|
|
```
|
|
| Metric | MARLIN |
|
|
|--------------------------------|----------|
|
|
| arc_challenge :: acc,none | 0.2884 |
|
|
| arc_challenge :: acc_norm,none | 0.3208 |
|
|
| mmlu :: acc,none | 0.442 |
|
|
|
|
| Tasks |Version|Filter|n-shot| Metric | |Value | |Stderr|
|
|
|----------------------------------------|------:|------|-----:|--------|---|-----:|---|-----:|
|
|
|arc_challenge | 1|none | 0|acc |↑ |0.2884|± |0.0132|
|
|
| | |none | 0|acc_norm|↑ |0.3208|± |0.0136|
|
|
|mmlu | 2|none | |acc |↑ |0.4420|± |0.0041|
|
|
|mmlu_humanities | 2|none | |acc |↑ |0.4066|± |0.0070|
|
|
|mmlu_formal_logic | 1|none | 0|acc |↑ |0.3413|± |0.0424|
|
|
|mmlu_high_school_european_history | 1|none | 0|acc |↑ |0.5576|± |0.0388|
|
|
|mmlu_high_school_us_history | 1|none | 0|acc |↑ |0.4804|± |0.0351|
|
|
|mmlu_high_school_world_history | 1|none | 0|acc |↑ |0.5612|± |0.0323|
|
|
|mmlu_international_law | 1|none | 0|acc |↑ |0.6364|± |0.0439|
|
|
|mmlu_jurisprudence | 1|none | 0|acc |↑ |0.5556|± |0.0480|
|
|
|mmlu_logical_fallacies | 1|none | 0|acc |↑ |0.4785|± |0.0392|
|
|
|mmlu_moral_disputes | 1|none | 0|acc |↑ |0.5289|± |0.0269|
|
|
|mmlu_moral_scenarios | 1|none | 0|acc |↑ |0.2380|± |0.0142|
|
|
|mmlu_philosophy | 1|none | 0|acc |↑ |0.4469|± |0.0282|
|
|
|mmlu_prehistory | 1|none | 0|acc |↑ |0.5062|± |0.0278|
|
|
|mmlu_professional_law | 1|none | 0|acc |↑ |0.3514|± |0.0122|
|
|
|mmlu_world_religions | 1|none | 0|acc |↑ |0.5497|± |0.0382|
|
|
|mmlu_other | 2|none | |acc |↑ |0.4841|± |0.0088|
|
|
|mmlu_business_ethics | 1|none | 0|acc |↑ |0.5000|± |0.0503|
|
|
|mmlu_clinical_knowledge | 1|none | 0|acc |↑ |0.4717|± |0.0307|
|
|
|mmlu_college_medicine | 1|none | 0|acc |↑ |0.4624|± |0.0380|
|
|
|mmlu_global_facts | 1|none | 0|acc |↑ |0.3000|± |0.0461|
|
|
|mmlu_human_aging | 1|none | 0|acc |↑ |0.4933|± |0.0336|
|
|
|mmlu_management | 1|none | 0|acc |↑ |0.6408|± |0.0475|
|
|
|mmlu_marketing | 1|none | 0|acc |↑ |0.7051|± |0.0299|
|
|
|mmlu_medical_genetics | 1|none | 0|acc |↑ |0.5000|± |0.0503|
|
|
|mmlu_miscellaneous | 1|none | 0|acc |↑ |0.5147|± |0.0179|
|
|
|mmlu_nutrition | 1|none | 0|acc |↑ |0.5196|± |0.0286|
|
|
|mmlu_professional_accounting | 1|none | 0|acc |↑ |0.3617|± |0.0287|
|
|
|mmlu_professional_medicine | 1|none | 0|acc |↑ |0.3419|± |0.0288|
|
|
|mmlu_virology | 1|none | 0|acc |↑ |0.4277|± |0.0385|
|
|
|mmlu_social_sciences | 2|none | |acc |↑ |0.5044|± |0.0089|
|
|
|mmlu_econometrics | 1|none | 0|acc |↑ |0.3158|± |0.0437|
|
|
|mmlu_high_school_geography | 1|none | 0|acc |↑ |0.5404|± |0.0355|
|
|
|mmlu_high_school_government_and_politics| 1|none | 0|acc |↑ |0.5130|± |0.0361|
|
|
|mmlu_high_school_macroeconomics | 1|none | 0|acc |↑ |0.4410|± |0.0252|
|
|
|mmlu_high_school_microeconomics | 1|none | 0|acc |↑ |0.4370|± |0.0322|
|
|
|mmlu_high_school_psychology | 1|none | 0|acc |↑ |0.6055|± |0.0210|
|
|
|mmlu_human_sexuality | 1|none | 0|acc |↑ |0.4580|± |0.0437|
|
|
|mmlu_professional_psychology | 1|none | 0|acc |↑ |0.4314|± |0.0200|
|
|
|mmlu_public_relations | 1|none | 0|acc |↑ |0.5273|± |0.0478|
|
|
|mmlu_security_studies | 1|none | 0|acc |↑ |0.5224|± |0.0320|
|
|
|mmlu_sociology | 1|none | 0|acc |↑ |0.6070|± |0.0345|
|
|
|mmlu_us_foreign_policy | 1|none | 0|acc |↑ |0.7200|± |0.0451|
|
|
|mmlu_stem | 2|none | |acc |↑ |0.3923|± |0.0086|
|
|
|mmlu_abstract_algebra | 1|none | 0|acc |↑ |0.3100|± |0.0465|
|
|
|mmlu_anatomy | 1|none | 0|acc |↑ |0.4519|± |0.0430|
|
|
|mmlu_astronomy | 1|none | 0|acc |↑ |0.5132|± |0.0407|
|
|
|mmlu_college_biology | 1|none | 0|acc |↑ |0.4167|± |0.0412|
|
|
|mmlu_college_chemistry | 1|none | 0|acc |↑ |0.3000|± |0.0461|
|
|
|mmlu_college_computer_science | 1|none | 0|acc |↑ |0.4200|± |0.0496|
|
|
|mmlu_college_mathematics | 1|none | 0|acc |↑ |0.3400|± |0.0476|
|
|
|mmlu_college_physics | 1|none | 0|acc |↑ |0.3039|± |0.0458|
|
|
|mmlu_computer_security | 1|none | 0|acc |↑ |0.6000|± |0.0492|
|
|
|mmlu_conceptual_physics | 1|none | 0|acc |↑ |0.3532|± |0.0312|
|
|
|mmlu_electrical_engineering | 1|none | 0|acc |↑ |0.5586|± |0.0414|
|
|
|mmlu_elementary_mathematics | 1|none | 0|acc |↑ |0.3386|± |0.0244|
|
|
|mmlu_high_school_biology | 1|none | 0|acc |↑ |0.5226|± |0.0284|
|
|
|mmlu_high_school_chemistry | 1|none | 0|acc |↑ |0.3941|± |0.0344|
|
|
|mmlu_high_school_computer_science | 1|none | 0|acc |↑ |0.4100|± |0.0494|
|
|
|mmlu_high_school_mathematics | 1|none | 0|acc |↑ |0.2926|± |0.0277|
|
|
|mmlu_high_school_physics | 1|none | 0|acc |↑ |0.2781|± |0.0366|
|
|
|mmlu_high_school_statistics | 1|none | 0|acc |↑ |0.3056|± |0.0314|
|
|
|mmlu_machine_learning | 1|none | 0|acc |↑ |0.4286|± |0.0470|
|
|
|
|
| Groups |Version|Filter|n-shot|Metric| |Value | |Stderr|
|
|
|--------------------|------:|------|------|------|---|-----:|---|-----:|
|
|
|mmlu | 2|none | |acc |↑ |0.4420|± |0.0041|
|
|
|mmlu_humanities | 2|none | |acc |↑ |0.4066|± |0.0070|
|
|
|mmlu_other | 2|none | |acc |↑ |0.4841|± |0.0088|
|
|
|mmlu_social_sciences| 2|none | |acc |↑ |0.5044|± |0.0089|
|
|
|mmlu_stem | 2|none | |acc |↑ |0.3923|± |0.0086|
|
|
``` |