初始化项目，由ModelHub XC社区提供模型
Model: lanawwas/ALLaM-7B-Instruct-preview Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
241
README.md
Normal file
241
README.md
Normal file
@@ -0,0 +1,241 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- ar
|
||||
- en
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- pytorch
|
||||
library_name: transformers
|
||||
---
|
||||
# ALLaM-7B-Instruct-preview
|
||||
|
||||
ALLaM is a series of powerful language models designed to advance Arabic Language Technology (ALT) developed by the National Center for Artificial Intelligence (NCAI) at the [Saudi Data and AI Authority (SDAIA)](https://sdaia.gov.sa/en/default.aspx). `ALLaM-AI/ALLaM-7B-Instruct-preview` is trained from scratch. Our pretraining from scratch recipe consists of two steps: training on 4T English tokens followed by training on 1.2T mixed Arabic/English tokens. This retains the English capabilities of the model without catastrophic forgetting, effectively transferring knowledge from one language distribution to another.
|
||||
|
||||
## Intended Use
|
||||
|
||||
`ALLaM` is specifically designed to expedite the research and development of ALT through Large Language Models (LLM). It serves as one of the foundational elements for building product offerings as well as facilitating experimental initiatives.
|
||||
|
||||
The ALLaM series models are designed to be a component of a larger AI system, and it is important for developers to incorporate safety measures when creating these systems. These safety measures are crucial for ensuring a balance between effectiveness and security, as well as minimizing potential risks, such as those resulting from the integration of the model with external tools.
|
||||
|
||||
## Model Details
|
||||
|
||||
ALLaM is a family of LLMs specially trained for Arabic. The main two paths followed for pretraining are:
|
||||
|
||||
- **ALLaM**: Pretraining models from scratch
|
||||
- **ALLaM-Adapted/ALLaM-(\*\*)/(\*\*)-ALLaM**/: Continued training from open source/weight models
|
||||
|
||||
For this release, we are providing our instruction-tuned 7B parameter generative model pretrained from scratch.
|
||||
|
||||
Some parameters for this model are provided in the following table:
|
||||
|
||||
| Size | Context Length | Pretraining Tokens | Instructions | Preference Pairs |
|
||||
|----------------|-----------------|--------------------|--------------|------------------|
|
||||
| 7B parameters | 4096 tokens |4T(en) + 1.2T(en+ar)| 7M | 260K |
|
||||
|
||||
#### Model Revisions Loading Tags:
|
||||
|
||||
- **Version 7b-alpha-v1.27.2.25 (Old Release)**: `revision = "v1"`
|
||||
- **Version 7b-alpha-v2.33.0.30 (Newest Releas)**: `revision = "v2"`
|
||||
|
||||
## Model Description
|
||||
|
||||
- **Developed by:** National Center for Artificial Intelligence at [SDAIA](https://sdaia.gov.sa/en/default.aspx)
|
||||
- **Model type:** Autoregressive Transformer
|
||||
- **Language(s):** Arabic, English
|
||||
- **License:** Please see the LICENSE file
|
||||
- **Input:** Text
|
||||
- **Output:** Text
|
||||
|
||||
|
||||
## Training Details
|
||||
|
||||
ALLaM-7B-Instruct-preview is pretrained on a total of 5.2 trillion tokens in English and Arabic, Our training codebase is built on [NVIDIA/MegatronLM](https://github.com/NVIDIA/Megatron-LM). Average MFU during training was ~42%. We trained our model using bf16-mixed precision.
|
||||
|
||||
|
||||
## Getting started
|
||||
|
||||
|
||||
### System Prompt
|
||||
|
||||
It is important to note that this model is optimized to function without a predefined system prompt.
|
||||
While Allam does not come with a default system prompt, it does provide the flexibility to add a custom system prompt.
|
||||
For instance, a well crafted system prompt could be:
|
||||
|
||||
“You are ALLaM, a bilingual English and Arabic AI assistant.”
|
||||
System prompts can also be in Arabic:
|
||||
|
||||
"أنت علام، مساعد ذكاء اصطناعي مطور من الهيئة السعودية للبيانات والذكاء الاصطناعي، تجيب على الأسئلة بطريقة مفيدة مع مراعاة القيم الثقافية المحلية."
|
||||
Alternatively, users can get creative with their prompts, such as:
|
||||
|
||||
“You are an AI assistant who responds to everything like a pirate.”
|
||||
|
||||
The system prompt is integrated inside the tokenizer config (accessed via `apply_chat_template()` module).
|
||||
|
||||
|
||||
### Example Usages
|
||||
|
||||
The weights for ALLaM model checkpoints can be accessed via [HuggingFace transformers](https://github.com/huggingface/transformers) (tested with `transformers>=4.40.1`). The following code snippet demonstrates how to load the model and generate text using the `ALLaM-AI/ALLaM-7B-Instruct-preview` model.
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
allam_model = AutoModelForCausalLM.from_pretrained("ALLaM-AI/ALLaM-7B-Instruct-preview")
|
||||
tokenizer = AutoTokenizer.from_pretrained("ALLaM-AI/ALLaM-7B-Instruct-preview")
|
||||
messages=[
|
||||
{"role": "user", "content": "كيف أجهز كوب شاهي؟"},
|
||||
]
|
||||
inputs = tokenizer.apply_chat_template(messages, tokenize=False)
|
||||
inputs = tokenizer(inputs, return_tensors='pt', return_token_type_ids=False)
|
||||
inputs = {k: v.to('cuda') for k,v in inputs.items()}
|
||||
allam_model = allam_model.to('cuda')
|
||||
response = allam_model.generate(**inputs, max_new_tokens=4096, do_sample=True, top_k=50, top_p=0.95, temperature=.6)
|
||||
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])
|
||||
```
|
||||
|
||||
|
||||
## Ethical Considerations and Limitations
|
||||
|
||||
ALLaM is a generative model that comes with inherent uncertainties. Trials cannot encompass every possible use case. Hence, predicting ALLaM's responses in every context is not possible, leading on occasion to incorrect or biased outputs. Developers must conduct thorough safety evaluations and make specific adjustments to ensure the model is suitable for the intended purposes.
|
||||
|
||||
*The output generated by this model is not considered a statement of NCAI, SDAIA, or any other organization.*
|
||||
|
||||
## Evaluation
|
||||
|
||||
### Automatic Benchmarks
|
||||
|
||||
#### Arabic Benchmarks
|
||||
**Massive Multitask Language Understanding** (MMLU) is a collection of many multiple-choice evaluation questions sourced from various academic levels (elementary to college level). These questions are typically related to humanities, STEM, or social sciences. It was originally an English dataset, but other variants were developed for Arabic:
|
||||
|
||||
<!-- - [Original English MMLU (MMLU-en)](https://github.com/hendrycks/test): A collection of 14,079 original English questions spanning 57 domains. -->
|
||||
- [Arabic MMLU](https://huggingface.co/datasets/MBZUAI/ArabicMMLU): A collection of 14,575 original Arabic questions spanning 40 domains published by MBZUAI.
|
||||
- [OpenAI MMLU-ar](https://huggingface.co/datasets/openai/MMMLU): A dataset comprising 14,042 questions, translated from the original MMLU benchmark published by OpenAI.
|
||||
|
||||
**Exams Arabic** ([Exams (Ar)](https://github.com/FreedomIntelligence/Arabic-eval/blob/main/LLM/benchmark_eval/benchmarks/EXAMS_Arabic/exam_test.jsonl)): A multiple choice question dataset with 537 samples, covering several domains e.g., Islamic studies, science, humanities, and physics.
|
||||
|
||||
**Arabic Cultural Alignment** ([ACVA](https://huggingface.co/datasets/FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment)): This dataset was generated by `gpt-3.5-turbo` and contains 8,710 True and False questions from 58 different areas.
|
||||
|
||||
**Education and Training Evaluation Commission** (ETEC): This dataset consists of Arabic-language multiple-choice questions, compiled by the ALLaM team in collaboration with [Saudi ETEC](https://acpd.etec.gov.sa/Home/index?csrt=5175167507218838843). It spans various educational levels, from elementary through post-college, with a total of 1,887 test samples.
|
||||
|
||||
**IEN**: This dataset was curated from the Ministry of Education's (MOE) [IEN platform](https://www.ientv.edu.sa/ar), organized by grade, topic, and difficulty level. It comprehensively covers the entire Saudi curriculum from 1st grade through high school, with 9990 multiple-choice questions and 5823 true/false questions.
|
||||
|
||||
**GAT**: The General Aptitude Test (GAT) dataset consists of approximately 16,000 Arabic multiple-choice questions, representing various sections of [the Qiyas General Aptitude Test](https://www.etec.gov.sa/en/service/Generalabilitytest/servicegoal). The sections include algebra, reading comprehension, analogies, arithmetic, associations, comparisons, completions, contextual understanding, and geometry.
|
||||
|
||||
**AraPro**: A curated collection of 5,001 multiple-choice questions (MCQs) authored by our domain experts. The dataset spans various subjects, including mathematics, science, and other relevant fields, providing a diverse set of questions for evaluation purpose.
|
||||
|
||||
**AraMath**: AraMath consists of 605 MCQs derived from [ArMath](https://github.com/reem-codes/ArMATH), which includes mathematical word problems, that was transformed to MCQs internally.
|
||||
|
||||
**Ar-IFEval**: an Arabic instruction-following (IF) evaluation dataset designed to automatically assess language models' compliance with specified instructions through verifiable methods. The dataset consists of 535 instances, each containing two to four verifiable instructions that can be validated using deterministic programming approaches.
|
||||
|
||||
All models were evaluated using our proprietary evaluation pipeline and [LM Evaluation Harness framework](https://github.com/EleutherAI/lm-evaluation-harness) to ensure fair comparisons. For API-based models, we used exact match evaluations of the generated outputs.
|
||||
|
||||
The evaluation scores of ALLaM can be found in JSON format [here](https://huggingface.co/ALLaM-AI/ALLaM-7B-Instruct-preview/tree/main/evaluation).
|
||||
|
||||
|
||||
| Model |AVG | ETEC <br>0 shot | IEN-MCQ <br>0 shot | IEN-TF <br>0 shot | AraPro <br>0 shot | AraMath <br>5 shot | Ar-IFEval <br>(prompt strict) <br>0 shot | Ar-IFEval <br>(inst strict) <br>0 shot | ExamsAR <br>5 shot | ACVA <br> 5 shot | Arabic MMLU <br>0 Shot | Openai MMLU <br>0 shot | GAT <br>0 shot |
|
||||
|:----------------------------|:----------|:---------|:-----------------|:----------------|:----------------|:-----------------|:-----------------------------------|:---------------------------------|:------------------|:--------------|:--------------------|:--------------------|:-----------------------------|
|
||||
| ALLaM-7B-Instruct-preview | 64.42 | 66.67 | **91.77** | 82.95 | 69.71 | 66.78 | 31.34 | 67.65 | 51.58 | 76.33 | 67.78 | 55.91 | 44.53 |
|
||||
| AceGPT-v2-8B-Chat | 52.67 | 56.81 | 77.01 | 75.91 | 63.51 | 41.49 | 10.26 | 39.25 | 51.96 | 72.69 | 57.02 | 49.99 | 36.15 |
|
||||
| AceGPT-v2-32B-Chat | 62.23 | 64.81 | 81.6 | 80.35 | 67.19 | 64.46 | 25.75 | 63.41 | 55.31 | 71.57 | 68.3 | 60.8 | 43.21 |
|
||||
| jais-family-6p7b-chat | 46.31 | 45.47 | 46.22 | 63.92 | 54.31 | 25.29 | 13.99 | 52.97 | 46.93 | 73.8 | 56.15 | 44.96 | 31.71 |
|
||||
| jais-family-13b-chat | 49.14 | 48.65 | 62.95 | 68.68 | 57.53 | 26.61 | 17.16 | 54.27 | 45.07 | 71.18 | 58.14 | 47.73 | 31.72 |
|
||||
| jais-family-30b-16k-chat | 52.54 | 53.31 | 74.88 | 68.76 | 62.79 | 41.49 | 16.6 | 54.95 | 49.72 | 60.08 | 62.04 | 50.98 | 34.85 |
|
||||
| jais-family-30b-8k-chat | 53.19 | 53.52 | 72.76 | 70.65 | 61.27 | 33.39 | 16.79 | 54.68 | 50.28 | 74.47 | 63.11 | 50.9 | 36.44 |
|
||||
| jais-adapted-7b-chat | 45.19 | 40.49 | 57.38 | 67.18 | 50.59 | 28.43 | 14.93 | 54.27 | 40.6 | 70.44 | 49.75 | 38.54 | 29.68 |
|
||||
| jais-adapted-13b-chat | 51.86 | 48.12 | 69.65 | 71.85 | 59.07 | 37.02 | 23.32 | 60.61 | 48.23 | 67.78 | 56.42 | 46.83 | 33.4 |
|
||||
| jais-adapted-70b-chat | 58.32 | 56.81 | 74.51 | 76.47 | 64.59 | 45.62 | 27.05 | 65.05 | 54.75 | 73.33 | 65.74 | 56.82 | 39.15 |
|
||||
| Qwen2.5-7B-Instruct | 60.55 | 64.12 | 66.38 | 78.46 | 64.63 | 71.74 | 28.17 | 65.19 | 50.65 | 78.17 | 61.54 | 56.1 | 41.42 |
|
||||
| Qwen2.5-14B-Instruct | 71.26 | 72.18 | 80.51 | 77.64 | 69.11 | 82.81 | 68.66 | 86.76 | 57.54 | 75.04 | 69.36 | 63.8 | 51.7 |
|
||||
| Qwen2.5-72B-Instruct | **76.91** | **78.7** | 86.88 | **86.62** | **74.69** | **92.89** | 67.72 | 87.51 | 60.71 | **79.92** | **74.1** | **73.59** | **59.54** |
|
||||
| Mistral-7B-Instruct-v0.3 | 43.05 | 35.67 | 53.59 | 63.4 | 43.85 | 27.11 | 30.41 | 64.03 | 34.08 | 60.25 | 45.27 | 32.3 | 26.65 |
|
||||
| Mistral-Nemo-Instruct-2407 | 53.79 | 49.28 | 68.43 | 71.78 | 57.61 | 40.0 | 35.82 | 70.58 | 47.49 | 76.92 | 55.97 | 46.15 | 25.44 |
|
||||
| Mistral-Small-Instruct-2409 | 51.11 | 40.96 | 60.64 | 63.66 | 47.73 | 44.46 | 51.12 | 78.16 | 38.73 | 68.93 | 50.43 | 39.63 | 28.82 |
|
||||
| Falcon3-7B-Instruct | 41.3 | 37.52 | 52.65 | 57.63 | 41.47 | 56.53 | 8.58 | 47.92 | 31.84 | 58.98 | 42.08 | 32.36 | 27.99 |
|
||||
| Meta-Llama-3.1-8B-Instruct | 54.08 | 45.68 | 59.23 | 71.7 | 52.51 | 34.38 | 51.87 | 79.11 | 52.51 | 69.93 | 56.43 | 44.67 | 30.9 |
|
||||
| Llama-3.3-70B-Instruct | 71.43 | 68.84 | 79.6 | 78.81 | 70.49 | 70.91 | **70.9** | **88.6** | **65.74** | 76.93 | 72.01 | 70.25 | 44.12 |
|
||||
|
||||
Closed models evaluations:
|
||||
|
||||
| Model | ETEC <br>0 shot | IEN-MCQ <br>0 shot | IEN-TF <br>0 shot | AraPro <br>0 shot | AraMath <br>5 shot | ARIFEval <br>(prompt strict) <br>0 shot | ARIFEval <br>(inst strict) <br>0 shot | ExamsAR <br>5 shot | ACVA <br> 5 shot | Arabicmmlu <br>0 Shot | Openai mmlu <br>0 shot | GAT 0 shot |
|
||||
|:---------------------------------------|:--------------|:-----------------|:----------------|:----------------|:-----------------|:----------------------------------|:--------------------------------|:-----------------|:-----------------------|:--------------------|:---------------------|:----------------------|
|
||||
| Azureml GPT4o (gpt-4o-900ptu) | 79.39 | **92.03** | 88.97 | 80.86 | 83.47 | 70.9 | 88.12 | 61.82 | 72.51 | 79.02 | **76.5** | 62.65 |
|
||||
| Claude Sonnet 3.5 (claude-3-5-sonnet-20241022) | **85.9** | 86.17 | **89.42** | **81.46** | 79.83 | 53.73 | 80.14 | **62.38** | **80.42** | 69.5 | 66.4 | **68.89** |
|
||||
| gemini pro 1.5 (gemini-1.5-pro) | 83.31 | 88.28 | 85.44 | 76.22 | **94.88** | **74.81** | **90.17** | 58.1 | 75.17 | **82.0** | 64.8 | 59.14 |
|
||||
|
||||
#### English Benchmarks
|
||||
|
||||
| model |Avg | AGIEval 0 Shot | Arc (challenge) 0 Shot | GPQA (main) 0 Shot | Hendrycks <br>ethics 0 Shot | Winogrande 0 Shot | HellaSwag 0 Shot | TriviaQa 5 Shot | MMLU Pro<br>5 Shot | Minerva Math <br>4 Shot | MMLU 0 Shot | TruthfulQA <br>(mc2) 0 Shot | IFEval <br>(prompt strict)<br>0 Shot | IFEval <br>(inst strict)<br>0 Shot | GSM8k 5 Shot |
|
||||
|:----------------------------------|:----------|:-----------------|:-----------------------|:--------------------------|:--------------------------|:--------------------|:-------------------|:------------------|:------------------|:----------------------|:--------------|:------------------------|:---------------------------------|:-------------------------------|:---------------|
|
||||
| ALLaM-7B-Instruct-preview | 46.85 | 41.99 | 51.28 | 22.77 | 73.17 | 70.48 | 76.26 | 16.07 | 30.4 | 17.3 | 59.6 | 46.67 | 38.08 | 50.0 | 61.79 |
|
||||
| AceGPT-v2-8B-Chat | 49.51 | 37.17 | 53.5 | 25.67 | 68.14 | 73.72 | 79.21 | 67.65 | 37.38 | 17.58 | 64.62 | 55.2 | 23.48 | 32.97 | 56.86 |
|
||||
| AceGPT-v2-32B-Chat | 57.14 | 56.01 | 53.92 | 32.8125 | 66.23 | 79.16 | 83.29 | 69.45 | 45.89 | 32.8 | 74.03 | 59.18 | 27.54 | 40.89 | 78.7 |
|
||||
| jais-family-6p7b-chat | 38.33 | 30.56 | 44.62 | 23.21 | 65.7 | 62.43 | 72.05 | 29.74 | 23.3 | 2.56 | 49.62 | 40.99 | 14.05 | 23.5 | 54.36 |
|
||||
| jais-family-13b-chat | 42.62 | 30.31 | 47.87 | 25.89 | 65.91 | 65.04 | 75.0 | 35.82 | 24.4 | 19.1 | 51.91 | 40.57 | 19.41 | 30.82 | 64.59 |
|
||||
| jais-family-30b-16k-chat | 45.15 | 31.85 | 48.46 | 23.88 | 69.44 | 68.19 | 76.21 | 43.99 | 29.11 | 22.3 | 58.5 | 44.78 | 18.3 | 29.14 | 67.93 |
|
||||
| jais-family-30b-8k-chat | 47.59 | 36.65 | 48.38 | 21.88 | 69.28 | 70.32 | 78.55 | 46.67 | 28.7 | 26.44 | 57.46 | 49.49 | 22.92 | 37.05 | 72.48 |
|
||||
| jais-adapted-7b-chat | 44.91 | 32.9 | 52.65 | 23.88 | 55.32 | 71.74 | 79.39 | 63.89 | 24.38 | 15.34 | 52.36 | 41.12 | 22.0 | 35.73 | 58.07 |
|
||||
| jais-adapted-13b-chat | 47.7 | 36.49 | 54.18 | 26.34 | 65.73 | 69.77 | 80.86 | 58.48 | 26.29 | 21.34 | 55.66 | 42.27 | 24.95 | 36.57 | 68.84 |
|
||||
| jais-adapted-70b-chat | 53.49 | 39.96 | 59.56 | 20.98 | 70.8 | 77.27 | 84.06 | 68.64 | 37.25 | 27.72 | 65.23 | 44.49 | 31.61 | 44.0 | 77.26 |
|
||||
| Qwen2.5-7B-Instruct | 54.68 | 59.2 | 51.28 | 26.56 | 73.76 | 69.38 | 79.55 | 50.59 | 44.92 | 12.04 | 70.56 | 58.93 | 57.3 | 68.23 | 43.29 |
|
||||
| Qwen2.5-14B-Instruct | 62.37 | 66.32 | 62.12 | 25.89 | 76.19 | 75.77 | 84.36 | 59.47 | 52.44 | 23.04 | 78.93 | 69.01 | 52.13 | 64.03 | 83.47 |
|
||||
| Qwen2.5-72B-Instruct | **70.06** | **71.09** | **63.48** | 25.67 | 78.33 | 76.24 | **87.41** | 70.9 | **62.77** | **54.04** | **83.44** | **69.54** | 67.65 | 77.1 | **93.25** |
|
||||
| Mistral-7B-Instruct-v0.3 | 51.98 | 36.45 | 58.87 | 23.21 | 72.58 | 73.95 | 82.93 | 67.97 | 33.18 | 13.44 | 59.74 | 59.69 | 42.51 | 54.8 | 48.37 |
|
||||
| Mistral-Nemo-Instruct-2407 | 54.0 | 39.65 | 59.04 | 24.33 | 67.86 | 74.66 | 82.35 | 72.77 | 44.27 | 29.62 | 65.56 | 54.88 | 30.13 | 38.97 | 71.95 |
|
||||
| Mistral-Small-Instruct-2409 | 61.65 | 40.76 | 60.49 | 25.89 | 72.27 | 78.53 | 85.35 | 79.11 | 47.47 | 39.42 | 69.42 | 56.35 | 58.23 | 68.35 | 81.43 |
|
||||
| Falcon3-7B-Instruct | 58.04 | 43.84 | 59.47 | **33.71** | 70.39 | 70.09 | 78.43 | 51.98 | 46.73 | 30.76 | 68.14 | 55.53 | 56.01 | 68.59 | 78.92 |
|
||||
| Meta-Llama-3.1-8B-Instruct | 56.5 | 42.39 | 55.12 | 27.23 | 66.69 | 73.95 | 79.28 | 70.05 | 40.641622 | 34.26 | 67.96 | 54.05 | 44.36 | 58.51 | 76.5 |
|
||||
| Llama-3.3-70B-Instruct | 67.7 | 55.44 | 63.4 | 25.89 | **81.05** | **79.24** | 84.39 | **81.7** | 60.51 | 46.42 | 81.99 | 60.91 | **63.22** | **72.78** | 90.83 |
|
||||
|
||||
### MT-bench
|
||||
|
||||
**Multi-Turn Bench** (MT-Bench): A challenging multi-turn benchmark that uses GPT-4o as a judge. MT-bench comprises 80 questions from 8 domains. Each question is presented to the model and the responses are submitted to GPT-4o to assign scores to each response. The judge returns a score for the first and second turn separately.
|
||||
This dataset was also automatically translated to Arabic and manually verified and culturally aligned.
|
||||
|
||||
| Model | AR Average | AR Turn 1 | AR Turn 2 | EN Average | EN Turn 1 | EN Turn 2 |
|
||||
|---------------------|------------|-----------|-----------|------------|-----------|-----------|
|
||||
| ALLaM-7B-Instruct-preview | 5.9 | **6.93**| 4.88 | 6.5 | 7.49 | 5.15 |
|
||||
| AceGPT-v1.5-13B-Chat | 4.61 | 5.28 | 3.93 | 4.86 | 5.56 | 4.17 |
|
||||
| AceGPT-v2-32B-Chat |5.43 | 6.61 | 4.26 | **6.5** | **7.41** | **5.58** |
|
||||
| jais-family-13b-chat | 4.89 | 5.37 | 4.41 | 4.77 | 5.57 | 3.97
|
||||
| jais-family-30b-16k-chat | 4.87 | 5.50 | 4.25 | 5.13 | 5.86 | 4.4 |
|
||||
| jais-adapted-70b-chat | 5.86 | 6.33 | **5.38** | 5.88 | 6.41 | 5.36 |
|
||||
|
||||
## Citation
|
||||
|
||||
If you found this work helpful or used any part of this work, please include the following citation:
|
||||
|
||||
```
|
||||
@inproceedings{
|
||||
bari2025allam,
|
||||
title={{ALL}aM: Large Language Models for Arabic and English},
|
||||
author={M Saiful Bari and Yazeed Alnumay and Norah A. Alzahrani and Nouf M. Alotaibi and Hisham Abdullah Alyahya and Sultan AlRashed and Faisal Abdulrahman Mirza and Shaykhah Z. Alsubaie and Hassan A. Alahmed and Ghadah Alabduljabbar and Raghad Alkhathran and Yousef Almushayqih and Raneem Alnajim and Salman Alsubaihi and Maryam Al Mansour and Saad Amin Hassan and Dr. Majed Alrubaian and Ali Alammari and Zaki Alawami and Abdulmohsen Al-Thubaity and Ahmed Abdelali and Jeril Kuriakose and Abdalghani Abujabal and Nora Al-Twairesh and Areeb Alowisheq and Haidar Khan},
|
||||
booktitle={The Thirteenth International Conference on Learning Representations},
|
||||
year={2025},
|
||||
url={https://openreview.net/forum?id=MscdsFVZrN}
|
||||
}
|
||||
```
|
||||
|
||||
Apart from the authors of the paper, the ALLaM team is rapidly growing, supported by a large and talented group of in-house professionals, whom we gratefully acknowledge here.
|
||||
|
||||
### Acknowledgment
|
||||
|
||||
This release is funded by **Saudi Data & AI Authority**. We thank **Mishari Almishari** and **Yaser Alonaizan** for leading the **ALLaM** project. We list below the different teams that supported the development of **ALLaM**.
|
||||
|
||||
#### Member of the Technical Staff:
|
||||
|
||||
Alhanoof Althnian, Iman Albilali, Jubair Sheikh, Ibrahim Mammu, Ghadah Alsaif, Abdulhameed Alothaimeen, Hadeel Alnegheimish, Emad Alghamdi, Khalid Almubarak, Atheer Al-Barqi, Batool Al-Otaibi, Muhammad Al-Hajji, Abdulraouf Al-Maasoumi, Abdulmajeed Alrowithi, Faisal Qarah, Sakhar Alkhereyf, Arwa Omayrah, Mohammed Al-Saleem, Amal Al-Shammari, Maryam Saif, Raeda Al-Marzooq, Lamyaa Alqhatani, Anas Al-Manie, Ghassan Al-Ward, Shahad Al-Zaidi, Batool Al-Ghoraibi, Alanoud Al-Shammari, Saad Alrazoq.
|
||||
|
||||
#### Data Acquisition, Management and Engineering:
|
||||
|
||||
Ali Alotaibi, Abdulrahman Alsaudi, Abdulaziz Asheyban, Areej Alokaili, Norah Alangari, Hamad Alnamazi, Fatima Alsalman, Saleh Alrusayyis, Moneerah Alballa, Omar AlGhamdi, Raneem Alqahtani, Amirah Albinhar, Nour Al-Aldahan, Khalid Alharbi, Hanouf Aljlayl, Fatimah Alzubaidi, Lama Aldossary, Rania Alzahrani, Shoug Alkhalaf.
|
||||
|
||||
#### Development and Infrastructure:
|
||||
|
||||
Saud AlHamoud, Fahad Alothaimeen, Ahmed Alrowatie, Saad Alajlan, Hassan Almitib, Abdinasir Mohamed, Sultan Alotaibi, Mohammed Alanazi, Mohammed Alsharani, Khalid Aljandal, Faisal Abulnaja, Ahmed Alosaimi, Muhannad Abu Hussain, Nasser Almoffarej, Muhammad Al-Hashem, Ahmed Al-Ghamdi, Amer Ashraf, Abeer Al-khars, Nawaf Babgy, Fevicks Kumar, Islam Gamal, Layla Al-Mutairi, Shroq Al-Ghamdi, Amjad Al-Zahrani, Tjad Clark, Ali bjorn, Meshari Alyami, Abdulrahman Bahkali, Reema Alomair.
|
||||
|
||||
#### Project Management:
|
||||
|
||||
Naif Shalhoub, Esshaq Almotawa, Sara AlRasheed, Mohammed Alshaalan, Mohammed Albreeh, Nezar Kaaki, Muna Alsahli, Abdullah Aldahami, Faisal Al-Tamimi, Tariq Alrouqi, Sahar Alghamdi, Raghad Abuznadah, Naif Almohammed, Oraib Alhemmyraine, Sarah Binlibdah, Shrooq Almohamdi.
|
||||
|
||||
#### Human Evaluators:
|
||||
|
||||
Amal Almohammadi, Badr almalki, Amjad Alsaeed, Alhanouf Alotaibi, Rajaa Almalki, Maryam Alasmari, Maha Awaji, Fatima Alshehri, Maryam ALshuwaiman, Ebtesam Alzahrani, Yasmeen Al-gahtani, Atheer Almusailem, Rehab Almalky, Shahad Alsulami, Abdullah Albalawi, Abeer Alqahtani, Lama Alrajeh, Shahad Aqeel, Yasir Alharbi, Rassil Al-Otaibi, Khulud Alawadh, Fatimah Almutairi, Suad Alkhamshi, Abdulrahman Alasmari, Goot Alqahtani, alhanouf Alfoaim, Rawan Aljohani, Aisha Almutairi.
|
||||
28
config.json
Normal file
28
config.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.006,
|
||||
"intermediate_size": 11008,
|
||||
"max_position_embeddings": 4096,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 32,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 32,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 1000000.0,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.39.3",
|
||||
"use_cache": true,
|
||||
"vocab_size": 64000,
|
||||
"internal_version": "7b-alpha-v2.33.0.30"
|
||||
}
|
||||
125
evaluations/ar/AceGPT-v2-32B-Chat/acva_5_shot.json
Normal file
125
evaluations/ar/AceGPT-v2-32B-Chat/acva_5_shot.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.7274397244546499,
|
||||
"acc_stderr,none": 0.004771397968508457,
|
||||
"acc_norm,none": 0.7157290470723306,
|
||||
"acc_norm_stderr,none": 0.004833440968499389
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1737779797.3395095,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.1",
|
||||
"upper_git_hash": "086919bd66f4e15fdcd4b792a7b27a698c1ba091",
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 26647.534977248,
|
||||
"end_time": 27360.084961217,
|
||||
"total_evaluation_time_seconds": "712.5499839689983"
|
||||
}
|
||||
142
evaluations/ar/AceGPT-v2-32B-Chat/ar_ifeval_0_shot.json
Normal file
142
evaluations/ar/AceGPT-v2-32B-Chat/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.2574626865671642,
|
||||
"prompt_level_strict_acc_stderr,none": 0.018903377119672635,
|
||||
"inst_level_strict_acc,none": 0.6341296928327645,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.31529850746268656,
|
||||
"prompt_level_loose_acc_stderr,none": 0.020087907677710036,
|
||||
"inst_level_loose_acc,none": 0.6764505119453925,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738794647.2071357,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "d0b91e989c8b697090db63bf498d8e2d8dd80815a595e5f22845a8425bff22fa"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
|
||||
"chat_template_sha": "af9c0233881b083b52ff773580215222b5440ac3d0beeeca99b76329b048f8db",
|
||||
"start_time": 1753623.131321269,
|
||||
"end_time": 1761093.682009075,
|
||||
"total_evaluation_time_seconds": "7470.550687805982"
|
||||
}
|
||||
126
evaluations/ar/AceGPT-v2-32B-Chat/araMath_v3_5_shot.json
Normal file
126
evaluations/ar/AceGPT-v2-32B-Chat/araMath_v3_5_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.6446280991735537,
|
||||
"acc_stderr,none": 0.019475010007284948,
|
||||
"acc_norm,none": 0.6446280991735537,
|
||||
"acc_norm_stderr,none": 0.019475010007284948
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u0645\u0646 \u0628\u064a\u0646 'A\u060c B\u060c C\u060c D' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738805225.8162587,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "17b2596f46d709ea107ed20bef044ca126de23a8e9bbc8ba0a9beef94fbc032d"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
|
||||
"chat_template_sha": "af9c0233881b083b52ff773580215222b5440ac3d0beeeca99b76329b048f8db",
|
||||
"start_time": 1764201.606664753,
|
||||
"end_time": 1764270.091855178,
|
||||
"total_evaluation_time_seconds": "68.48519042483531"
|
||||
}
|
||||
130
evaluations/ar/AceGPT-v2-32B-Chat/araPro_0_shot.json
Normal file
130
evaluations/ar/AceGPT-v2-32B-Chat/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.671865626874625,
|
||||
"acc_stderr,none": 0.006640213946839424,
|
||||
"acc_norm,none": 0.671865626874625,
|
||||
"acc_norm_stderr,none": 0.006640213946839424
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738802810.5474553,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"araPro": "2f706897ad0129e016cc8d6907f8bb4359c32403fc2d1b0a4e78717f424793da"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
|
||||
"chat_template_sha": "af9c0233881b083b52ff773580215222b5440ac3d0beeeca99b76329b048f8db",
|
||||
"start_time": 1761786.552693387,
|
||||
"end_time": 1761894.218775138,
|
||||
"total_evaluation_time_seconds": "107.66608175099827"
|
||||
}
|
||||
2051
evaluations/ar/AceGPT-v2-32B-Chat/arabicmmlu_0_shot.json
Normal file
2051
evaluations/ar/AceGPT-v2-32B-Chat/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/AceGPT-v2-32B-Chat/etec_v2_0_shot.json
Normal file
126
evaluations/ar/AceGPT-v2-32B-Chat/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.6481187069422364,
|
||||
"acc_stderr,none": 0.010996501146375258,
|
||||
"acc_norm,none": 0.6481187069422364,
|
||||
"acc_norm_stderr,none": 0.010996501146375258
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738805984.3189015,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"etec_v2": "697b8bfc7d6b0f85165e5cca6953182b09b7a2b0d79fa31e74cc3897f432de41"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
|
||||
"chat_template_sha": "af9c0233881b083b52ff773580215222b5440ac3d0beeeca99b76329b048f8db",
|
||||
"start_time": 1764960.166542801,
|
||||
"end_time": 1765035.801506021,
|
||||
"total_evaluation_time_seconds": "75.63496321998537"
|
||||
}
|
||||
127
evaluations/ar/AceGPT-v2-32B-Chat/exams_ar_5_shot.json
Normal file
127
evaluations/ar/AceGPT-v2-32B-Chat/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.553072625698324,
|
||||
"acc_stderr,none": 0.021474702941383872,
|
||||
"acc_norm,none": 0.553072625698324,
|
||||
"acc_norm_stderr,none": 0.021474702941383872
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
16
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1737780545.20475,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.1",
|
||||
"upper_git_hash": "086919bd66f4e15fdcd4b792a7b27a698c1ba091",
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 27395.295045238,
|
||||
"end_time": 27506.949709817,
|
||||
"total_evaluation_time_seconds": "111.65466457900038"
|
||||
}
|
||||
543
evaluations/ar/AceGPT-v2-32B-Chat/gat_0_shot.json
Normal file
543
evaluations/ar/AceGPT-v2-32B-Chat/gat_0_shot.json
Normal file
@@ -0,0 +1,543 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.4321459927254484,
|
||||
"acc_stderr,none": 0.0038347299693873033,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.3992578849721707,
|
||||
"acc_stderr,none": 0.009435653731651068
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.2867030965391621,
|
||||
"acc_stderr,none": 0.00863295163043938
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.3894000736105999,
|
||||
"acc_stderr,none": 0.009356458715331561
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.4143540669856459,
|
||||
"acc_stderr,none": 0.01524590184737997
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.34672131147540985,
|
||||
"acc_stderr,none": 0.013631312083187472
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.5793388429752067,
|
||||
"acc_stderr,none": 0.014197745251253151
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.522239263803681,
|
||||
"acc_stderr,none": 0.013837823280527494
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.5013698630136987,
|
||||
"acc_stderr,none": 0.026207022561245137
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.585633270321361,
|
||||
"acc_stderr,none": 0.009580200187530542
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.4321459927254484,
|
||||
"acc_stderr,none": 0.0038347299693873033,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "ef4b2026",
|
||||
"date": 1733932681.9722512,
|
||||
"pretty_env_info": "PyTorch version: 2.1.0a0+29c30b1\nIs debug build: False\nCUDA used to build PyTorch: 12.2\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.22.2\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.1.0a0+29c30b1\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.16.0a0\n[pip3] triton==2.0.0.dev20221202\n[conda] Could not collect",
|
||||
"transformers_version": "4.47.0",
|
||||
"upper_git_hash": "27ba526c4b16ee30604687f8bfd4c19680101dd1",
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 2367.995520754,
|
||||
"end_time": 5482.980996963,
|
||||
"total_evaluation_time_seconds": "3114.9854762089994"
|
||||
}
|
||||
127
evaluations/ar/AceGPT-v2-32B-Chat/moe_ien_mcq_0_shot.json
Normal file
127
evaluations/ar/AceGPT-v2-32B-Chat/moe_ien_mcq_0_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.816016016016016,
|
||||
"acc_stderr,none": 0.0038768441643790346,
|
||||
"acc_norm,none": 0.816016016016016,
|
||||
"acc_norm_stderr,none": 0.0038768441643790346
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738807582.4110897,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "e5422ff2f277b9bfffeb1b5ad185b714804b5a3d276dfff99a29eb88d9a41683"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
|
||||
"chat_template_sha": "af9c0233881b083b52ff773580215222b5440ac3d0beeeca99b76329b048f8db",
|
||||
"start_time": 1766558.431540363,
|
||||
"end_time": 1766704.504224634,
|
||||
"total_evaluation_time_seconds": "146.07268427102827"
|
||||
}
|
||||
129
evaluations/ar/AceGPT-v2-32B-Chat/moe_ien_tf_0_shot.json
Normal file
129
evaluations/ar/AceGPT-v2-32B-Chat/moe_ien_tf_0_shot.json
Normal file
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_tf": {
|
||||
"alias": "moe_ien_tf",
|
||||
"acc,none": 0.8035376953460416,
|
||||
"acc_stderr,none": 0.005207228603848848,
|
||||
"acc_norm,none": 0.8035376953460416,
|
||||
"acc_norm_stderr,none": 0.005207228603848848
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_tf": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_tf": {
|
||||
"task": "moe_ien_tf",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_tf/moe_ien_tf.py",
|
||||
"dataset_name": "moe_ien_tf",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n keys=[\"\u0635\u062d\u064a\u062d\u0629\",\n \"\u062e\u0627\u0637\u0626\u0629\"\n ]\n #keys =[\"\u0635\u0648\u0627\u0628\",\n # \"\u062e\u0637\u0623\"]\n target_key = int(doc[\"Answer\"])-1\n\n out_doc = {\n \"query\": \"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" +doc[\"Question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\", \n \"choices\": keys,\n \"gold\": target_key,\n }\n return out_doc\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{Subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d\u064a\u062d\u0629' \u0623\u0648 '\u062e\u0627\u0637\u0626\u0629' \u062f\u0648\u0646 \u0634\u0631\u062d ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_tf": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_tf": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_tf": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_tf": {
|
||||
"original": 5823,
|
||||
"effective": 5823
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-32B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 32512545792,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "1c0ca4fb3fa4c292ac3d1f64f330f210c9f184d4",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738809377.2163908,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"151643"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 151643,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"moe_ien_tf": "116cb28cd11c72b01c3d52d75d3918c312d0a4f569bfdb8b2219398ec576a3f4"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-32B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-32B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
|
||||
"chat_template_sha": "af9c0233881b083b52ff773580215222b5440ac3d0beeeca99b76329b048f8db",
|
||||
"start_time": 1768353.06839988,
|
||||
"end_time": 1768502.097875321,
|
||||
"total_evaluation_time_seconds": "149.0294754409697"
|
||||
}
|
||||
2660
evaluations/ar/AceGPT-v2-32B-Chat/openaimmlu_0_shot.json
Normal file
2660
evaluations/ar/AceGPT-v2-32B-Chat/openaimmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
123
evaluations/ar/AceGPT-v2-8B-Chat/acva_5_shot.json
Normal file
123
evaluations/ar/AceGPT-v2-8B-Chat/acva_5_shot.json
Normal file
@@ -0,0 +1,123 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.7415614236509759,
|
||||
"acc_stderr,none": 0.004691028694524559,
|
||||
"acc_norm,none": 0.7268656716417911,
|
||||
"acc_norm_stderr,none": 0.004774534958083965
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "562d0998c03c02d315e346f81650a43955711901",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "5e10e017",
|
||||
"date": 1736966813.484974,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.0",
|
||||
"upper_git_hash": "2e5cd5395faf76fea1afc96dd0f7161a9d3aa145",
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 2430.929540314,
|
||||
"end_time": 3025.204908665,
|
||||
"total_evaluation_time_seconds": "594.275368351"
|
||||
}
|
||||
142
evaluations/ar/AceGPT-v2-8B-Chat/ar_ifeval_0_shot.json
Normal file
142
evaluations/ar/AceGPT-v2-8B-Chat/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.10261194029850747,
|
||||
"prompt_level_strict_acc_stderr,none": 0.01311934649092474,
|
||||
"inst_level_strict_acc,none": 0.3924914675767918,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.12126865671641791,
|
||||
"prompt_level_loose_acc_stderr,none": 0.01411319854290401,
|
||||
"inst_level_loose_acc,none": 0.42389078498293514,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "562d0998c03c02d315e346f81650a43955711901",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739784109.8369951,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "9ce88f26b4b78e684512ecd933af67fe512192f41e27d2bedc62f288943db360"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 62023.729831301,
|
||||
"end_time": 66967.714743853,
|
||||
"total_evaluation_time_seconds": "4943.98491255199"
|
||||
}
|
||||
126
evaluations/ar/AceGPT-v2-8B-Chat/araMath_v3_5_shot.json
Normal file
126
evaluations/ar/AceGPT-v2-8B-Chat/araMath_v3_5_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.41487603305785126,
|
||||
"acc_stderr,none": 0.02004770429343817,
|
||||
"acc_norm,none": 0.41487603305785126,
|
||||
"acc_norm_stderr,none": 0.02004770429343817
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u0645\u0646 \u0628\u064a\u0646 'A\u060c B\u060c C\u060c D' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "562d0998c03c02d315e346f81650a43955711901",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739784015.8084505,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "4eebd1da6e6937fc09bb9f1871adb53192dbce96733f0f8ee76d406c2fc8cad5"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 61929.69246185,
|
||||
"end_time": 61980.464828513,
|
||||
"total_evaluation_time_seconds": "50.772366663004505"
|
||||
}
|
||||
130
evaluations/ar/AceGPT-v2-8B-Chat/araPro_0_shot.json
Normal file
130
evaluations/ar/AceGPT-v2-8B-Chat/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.6350729854029195,
|
||||
"acc_stderr,none": 0.006808161111700288,
|
||||
"acc_norm,none": 0.6350729854029195,
|
||||
"acc_norm_stderr,none": 0.006808161111700288
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "562d0998c03c02d315e346f81650a43955711901",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739782427.4652286,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {
|
||||
"araPro": "655c2f6626c4b10533bba45ff63f9d4501694dea7f65d0bb251390819154f901"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 60341.23142254,
|
||||
"end_time": 60939.383586887,
|
||||
"total_evaluation_time_seconds": "598.1521643470041"
|
||||
}
|
||||
2045
evaluations/ar/AceGPT-v2-8B-Chat/arabicmmlu_0_shot.json
Normal file
2045
evaluations/ar/AceGPT-v2-8B-Chat/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/AceGPT-v2-8B-Chat/etec_v2_0_shot.json
Normal file
126
evaluations/ar/AceGPT-v2-8B-Chat/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.5680975092739798,
|
||||
"acc_stderr,none": 0.011406002243769559,
|
||||
"acc_norm,none": 0.5680975092739798,
|
||||
"acc_norm_stderr,none": 0.011406002243769559
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "562d0998c03c02d315e346f81650a43955711901",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739783073.791851,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {
|
||||
"etec_v2": "d371135bd6f3e91b2eb292576c3b2fae24dc4c0d7cd2a5f6eacf1fe6bc062e76"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 60987.772646854,
|
||||
"end_time": 61072.230445773,
|
||||
"total_evaluation_time_seconds": "84.4577989190002"
|
||||
}
|
||||
119
evaluations/ar/AceGPT-v2-8B-Chat/exams_ar_5_shot.json
Normal file
119
evaluations/ar/AceGPT-v2-8B-Chat/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,119 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.5195530726256983,
|
||||
"acc_stderr,none": 0.02158019049784565,
|
||||
"acc_norm,none": 0.5195530726256983,
|
||||
"acc_norm_stderr,none": 0.02158019049784565
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,tensor_parallel_size=1,data_parallel_size=2,gpu_memory_utilization=0.4,download_dir=/tmp",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "8e1bd48d",
|
||||
"date": 1735747770.5687191,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.47.1",
|
||||
"upper_git_hash": "f64fe2f2a86055aaecced603b56097fd79201711",
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {},
|
||||
"model_source": "vllm",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 8055.848670643,
|
||||
"end_time": 8272.25518881,
|
||||
"total_evaluation_time_seconds": "216.40651816700029"
|
||||
}
|
||||
539
evaluations/ar/AceGPT-v2-8B-Chat/gat_0_shot.json
Normal file
539
evaluations/ar/AceGPT-v2-8B-Chat/gat_0_shot.json
Normal file
@@ -0,0 +1,539 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.3615326727706008,
|
||||
"acc_stderr,none": 0.003748588350676633,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.30241187384044527,
|
||||
"acc_stderr,none": 0.008849121616191958
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.3227686703096539,
|
||||
"acc_stderr,none": 0.008925286248200312
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.3213102686786897,
|
||||
"acc_stderr,none": 0.008960516811645579
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.39425837320574164,
|
||||
"acc_stderr,none": 0.01512460088966808
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.28114754098360656,
|
||||
"acc_stderr,none": 0.012876124676937594
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.46115702479338844,
|
||||
"acc_stderr,none": 0.014336474830596175
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.2983128834355828,
|
||||
"acc_stderr,none": 0.012674637536976358
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.3232876712328767,
|
||||
"acc_stderr,none": 0.024515791774351408
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.5183364839319471,
|
||||
"acc_stderr,none": 0.009717331969425425
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.3615326727706008,
|
||||
"acc_stderr,none": 0.003748588350676633,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,tensor_parallel_size=1,data_parallel_size=2,gpu_memory_utilization=0.4,download_dir=/tmp",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "8e1bd48d",
|
||||
"date": 1735749781.6371627,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.47.1",
|
||||
"upper_git_hash": "f64fe2f2a86055aaecced603b56097fd79201711",
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {},
|
||||
"model_source": "vllm",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 10066.91226392,
|
||||
"end_time": 10586.891967311,
|
||||
"total_evaluation_time_seconds": "519.9797033909999"
|
||||
}
|
||||
127
evaluations/ar/AceGPT-v2-8B-Chat/moe_ien_mcq_0_shot.json
Normal file
127
evaluations/ar/AceGPT-v2-8B-Chat/moe_ien_mcq_0_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.7700700700700701,
|
||||
"acc_stderr,none": 0.0042101916833611345,
|
||||
"acc_norm,none": 0.7700700700700701,
|
||||
"acc_norm_stderr,none": 0.0042101916833611345
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "562d0998c03c02d315e346f81650a43955711901",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739783202.062394,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "99731f9d1bb76d010da5a439ea1b0bb7695451459d680f708f7222f02ba8e831"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 61116.014324615,
|
||||
"end_time": 61463.567260828,
|
||||
"total_evaluation_time_seconds": "347.5529362130037"
|
||||
}
|
||||
129
evaluations/ar/AceGPT-v2-8B-Chat/moe_ien_tf_0_shot.json
Normal file
129
evaluations/ar/AceGPT-v2-8B-Chat/moe_ien_tf_0_shot.json
Normal file
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_tf": {
|
||||
"alias": "moe_ien_tf",
|
||||
"acc,none": 0.7590589043448395,
|
||||
"acc_stderr,none": 0.00560476076159517,
|
||||
"acc_norm,none": 0.7590589043448395,
|
||||
"acc_norm_stderr,none": 0.00560476076159517
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_tf": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_tf": {
|
||||
"task": "moe_ien_tf",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_tf/moe_ien_tf.py",
|
||||
"dataset_name": "moe_ien_tf",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n keys=[\"\u0635\u062d\u064a\u062d\u0629\",\n \"\u062e\u0627\u0637\u0626\u0629\"\n ]\n #keys =[\"\u0635\u0648\u0627\u0628\",\n # \"\u062e\u0637\u0623\"]\n target_key = int(doc[\"Answer\"])-1\n\n out_doc = {\n \"query\": \"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" +doc[\"Question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\", \n \"choices\": keys,\n \"gold\": target_key,\n }\n return out_doc\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{Subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d\u064a\u062d\u0629' \u0623\u0648 '\u062e\u0627\u0637\u0626\u0629' \u062f\u0648\u0646 \u0634\u0631\u062d ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_tf": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_tf": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_tf": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_tf": {
|
||||
"original": 5823,
|
||||
"effective": 5823
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=FreedomIntelligence/AceGPT-v2-8B-Chat,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.float16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "562d0998c03c02d315e346f81650a43955711901",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739783594.7150183,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|end_of_text|>",
|
||||
"128001"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128001,
|
||||
"max_length": 8192,
|
||||
"task_hashes": {
|
||||
"moe_ien_tf": "a8315c59ec304a82f04395ff5e7728d6586b1b0b5f569486840b7d29d76a8dd8"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "FreedomIntelligence/AceGPT-v2-8B-Chat",
|
||||
"model_name_sanitized": "FreedomIntelligence__AceGPT-v2-8B-Chat",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 61508.598662402,
|
||||
"end_time": 61883.458017876,
|
||||
"total_evaluation_time_seconds": "374.85935547400004"
|
||||
}
|
||||
2662
evaluations/ar/AceGPT-v2-8B-Chat/openaimmlu_0_shot.json
Normal file
2662
evaluations/ar/AceGPT-v2-8B-Chat/openaimmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
119
evaluations/ar/Allam-7b-instruct-preview/acva_5_shot.json
Normal file
119
evaluations/ar/Allam-7b-instruct-preview/acva_5_shot.json
Normal file
@@ -0,0 +1,119 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.7746268656716417,
|
||||
"acc_stderr,none": 0.004477269169728854,
|
||||
"acc_norm,none": 0.7632606199770379,
|
||||
"acc_norm_stderr,none": 0.004554991129754026
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,tensor_parallel_size=1,data_parallel_size=2,gpu_memory_utilization=0.8",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "8e1bd48d",
|
||||
"date": 1735662713.7617116,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.47.1",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"acva": "d007c508f0accdd697f549d7cbe7f960f1470c8f86f1a0969355a6ef33108edb"
|
||||
},
|
||||
"model_source": "vllm",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 3374.021232778,
|
||||
"end_time": 3578.563943596,
|
||||
"total_evaluation_time_seconds": "204.54271081800016"
|
||||
}
|
||||
142
evaluations/ar/Allam-7b-instruct-preview/ar_ifeval_0_shot.json
Normal file
142
evaluations/ar/Allam-7b-instruct-preview/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.31343283582089554,
|
||||
"prompt_level_strict_acc_stderr,none": 0.020055655889994813,
|
||||
"inst_level_strict_acc,none": 0.6764505119453925,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.3656716417910448,
|
||||
"prompt_level_loose_acc_stderr,none": 0.020822161638297296,
|
||||
"inst_level_loose_acc,none": 0.7051194539249147,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7000559616,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618378.981141,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "d0db7903ef270d7dc54efe4e7713be0de9864fc3a36c901c6e5777a6a5f69aa9"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\n' + system_message + '\\n<</SYS>>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + ' [INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}",
|
||||
"chat_template_sha": "f1dff938141b507da4a409b6bb3431382088a97a963acd246a41f2f344ae831f",
|
||||
"start_time": 1393068.333905473,
|
||||
"end_time": 1397143.169266589,
|
||||
"total_evaluation_time_seconds": "4074.8353611161"
|
||||
}
|
||||
126
evaluations/ar/Allam-7b-instruct-preview/araMath_v3_5_shot.json
Normal file
126
evaluations/ar/Allam-7b-instruct-preview/araMath_v3_5_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.6677685950413224,
|
||||
"acc_stderr,none": 0.019165266705090528,
|
||||
"acc_norm,none": 0.6677685950413224,
|
||||
"acc_norm_stderr,none": 0.019165266705090528
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u0645\u0646 \u0628\u064a\u0646 'A\u060c B\u060c C\u060c D' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7000559616,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618269.6292942,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "e7f60b63c44ee90c76a61f37207fa1f812622b6662200911fcfd7dabe78ada66"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\n' + system_message + '\\n<</SYS>>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + ' [INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}",
|
||||
"chat_template_sha": "f1dff938141b507da4a409b6bb3431382088a97a963acd246a41f2f344ae831f",
|
||||
"start_time": 1392959.193182268,
|
||||
"end_time": 1393012.133225703,
|
||||
"total_evaluation_time_seconds": "52.940043434966356"
|
||||
}
|
||||
130
evaluations/ar/Allam-7b-instruct-preview/araPro_0_shot.json
Normal file
130
evaluations/ar/Allam-7b-instruct-preview/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.6970605878824235,
|
||||
"acc_stderr,none": 0.006498724870364006,
|
||||
"acc_norm,none": 0.6970605878824235,
|
||||
"acc_norm_stderr,none": 0.006498724870364006
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7000559616,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617164.0204737,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"araPro": "01340c360a1565c46298c4c24dd3fdfe1ea614c6eef6e4d4f021f1da83da2584"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\n' + system_message + '\\n<</SYS>>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + ' [INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}",
|
||||
"chat_template_sha": "f1dff938141b507da4a409b6bb3431382088a97a963acd246a41f2f344ae831f",
|
||||
"start_time": 1391853.516943726,
|
||||
"end_time": 1392050.054185297,
|
||||
"total_evaluation_time_seconds": "196.5372415711172"
|
||||
}
|
||||
2086
evaluations/ar/Allam-7b-instruct-preview/arabicmmlu_0_shot.json
Normal file
2086
evaluations/ar/Allam-7b-instruct-preview/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/Allam-7b-instruct-preview/etec_v2_0_shot.json
Normal file
126
evaluations/ar/Allam-7b-instruct-preview/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.6666666666666666,
|
||||
"acc_stderr,none": 0.010854826817097195,
|
||||
"acc_norm,none": 0.6666666666666666,
|
||||
"acc_norm_stderr,none": 0.010854826817097195
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7000559616,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617421.4265695,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"etec_v2": "a0d87bf7eb82815b66ea544cb632aafb803526dee24b399f30fdc751be442b60"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\n' + system_message + '\\n<</SYS>>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + ' [INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}",
|
||||
"chat_template_sha": "f1dff938141b507da4a409b6bb3431382088a97a963acd246a41f2f344ae831f",
|
||||
"start_time": 1392110.980523203,
|
||||
"end_time": 1392198.883363127,
|
||||
"total_evaluation_time_seconds": "87.90283992397599"
|
||||
}
|
||||
121
evaluations/ar/Allam-7b-instruct-preview/exams_ar_5_shot.json
Normal file
121
evaluations/ar/Allam-7b-instruct-preview/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,121 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.515828677839851,
|
||||
"acc_stderr,none": 0.021585885942816244,
|
||||
"acc_norm,none": 0.515828677839851,
|
||||
"acc_norm_stderr,none": 0.021585885942816244
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,tensor_parallel_size=1,data_parallel_size=2,gpu_memory_utilization=0.8",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "8e1bd48d",
|
||||
"date": 1735662207.0830526,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.47.1",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"exams_ar": "b1561abd56354d570ac16bf64163b0ee8dc6c507234b05f678576b09c26c644a"
|
||||
},
|
||||
"model_source": "vllm",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 2867.397536365,
|
||||
"end_time": 2948.510496752,
|
||||
"total_evaluation_time_seconds": "81.11296038699993"
|
||||
}
|
||||
549
evaluations/ar/Allam-7b-instruct-preview/gat_0_shot.json
Normal file
549
evaluations/ar/Allam-7b-instruct-preview/gat_0_shot.json
Normal file
@@ -0,0 +1,549 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.4452527279568544,
|
||||
"acc_stderr,none": 0.0038711388833064567,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.40667903525046384,
|
||||
"acc_stderr,none": 0.009463939247454995
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.35919854280510016,
|
||||
"acc_stderr,none": 0.009158766245747282
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.40154582259845417,
|
||||
"acc_stderr,none": 0.009406284814832203
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.5464114832535886,
|
||||
"acc_stderr,none": 0.015407801869520031
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.34508196721311474,
|
||||
"acc_stderr,none": 0.013616100682624904
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.6057851239669422,
|
||||
"acc_stderr,none": 0.014054411207805699
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.3941717791411043,
|
||||
"acc_stderr,none": 0.013537713096332765
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.473972602739726,
|
||||
"acc_stderr,none": 0.026171590093068537
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.5727788279773157,
|
||||
"acc_stderr,none": 0.009620311542503682
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.4452527279568544,
|
||||
"acc_stderr,none": 0.0038711388833064567,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,tensor_parallel_size=1,data_parallel_size=2,gpu_memory_utilization=0.8",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "8e1bd48d",
|
||||
"date": 1735664096.2650902,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.47.1",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"gat_analogy": "ede28dec097bfebe8a85a19fa27d001696858276df66254bdb70fc63231f1a83",
|
||||
"gat_association": "5d82550d46c4f3cabf370185a8a23cc2eb5b08f1f0c5e210a8a712562a44bd08",
|
||||
"gat_completion": "fc3c19dd7f1896696fec1bffc21182804c9b2f1fb8d8c882428a6bb4bb61e370",
|
||||
"gat_reading": "93053b187a750d2e87f5488f2d0fda944f3da9195bb04d1c4dee9c4b56fa626a",
|
||||
"gat_algebra": "77832c595eaaf156775c3dbb27da0915ef600ebf46a7113ae32a202b0359e8a6",
|
||||
"gat_arithmetic": "6a498f75f5cc0ffd1b30f7a6293ba80d08f2a8876d5558d8e934bf57355ff0cc",
|
||||
"gat_comparisons": "acb80c0ed8dd07e916a471189aef3a546efc289824b2cc50a32c11dc4c97c9c1",
|
||||
"gat_contextual": "de063ed3b94011d74ee24a6532122c9d344fc15e42800db44f0849995a0bc37a",
|
||||
"gat_geometry": "3e482885559a4404ee9e97556edc6e49959770a499f4ae2c58f18ad85b91a363"
|
||||
},
|
||||
"model_source": "vllm",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 4756.376698655,
|
||||
"end_time": 5124.76942052,
|
||||
"total_evaluation_time_seconds": "368.39272186499966"
|
||||
}
|
||||
127
evaluations/ar/Allam-7b-instruct-preview/moe_ien_mcq_0_shot.json
Normal file
127
evaluations/ar/Allam-7b-instruct-preview/moe_ien_mcq_0_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.9177177177177177,
|
||||
"acc_stderr,none": 0.002749455634736978,
|
||||
"acc_norm,none": 0.9177177177177177,
|
||||
"acc_norm_stderr,none": 0.002749455634736978
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7000559616,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617571.8184838,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "504533b140426f12c89d975ef421328fc89d69af8719c420a1bf897ed4724191"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\n' + system_message + '\\n<</SYS>>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + ' [INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}",
|
||||
"chat_template_sha": "f1dff938141b507da4a409b6bb3431382088a97a963acd246a41f2f344ae831f",
|
||||
"start_time": 1392261.292633723,
|
||||
"end_time": 1392626.942167409,
|
||||
"total_evaluation_time_seconds": "365.64953368599527"
|
||||
}
|
||||
129
evaluations/ar/Allam-7b-instruct-preview/moe_ien_tf_0_shot.json
Normal file
129
evaluations/ar/Allam-7b-instruct-preview/moe_ien_tf_0_shot.json
Normal file
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_tf": {
|
||||
"alias": "moe_ien_tf",
|
||||
"acc,none": 0.8294693456980937,
|
||||
"acc_stderr,none": 0.004929073554117403,
|
||||
"acc_norm,none": 0.8294693456980937,
|
||||
"acc_norm_stderr,none": 0.004929073554117403
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_tf": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_tf": {
|
||||
"task": "moe_ien_tf",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_tf/moe_ien_tf.py",
|
||||
"dataset_name": "moe_ien_tf",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n keys=[\"\u0635\u062d\u064a\u062d\u0629\",\n \"\u062e\u0627\u0637\u0626\u0629\"\n ]\n #keys =[\"\u0635\u0648\u0627\u0628\",\n # \"\u062e\u0637\u0623\"]\n target_key = int(doc[\"Answer\"])-1\n\n out_doc = {\n \"query\": \"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" +doc[\"Question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\", \n \"choices\": keys,\n \"gold\": target_key,\n }\n return out_doc\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{Subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d\u064a\u062d\u0629' \u0623\u0648 '\u062e\u0627\u0637\u0626\u0629' \u062f\u0648\u0646 \u0634\u0631\u062d ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_tf": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_tf": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_tf": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_tf": {
|
||||
"original": 5823,
|
||||
"effective": 5823
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=/tmp/7b-alpha-v1.27.2.25,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7000559616,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617995.3462336,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 4096,
|
||||
"task_hashes": {
|
||||
"moe_ien_tf": "8701a646f6ea8b9bb96c028f817fbeabfb9031580f5054368b43d14d4a5a1270"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "/tmp/7b-alpha-v1.27.2.25",
|
||||
"model_name_sanitized": "__tmp__7b-alpha-v1.27.2.25",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<<SYS>>\\n' + system_message + '\\n<</SYS>>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ bos_token + ' [INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' + eos_token }}{% endif %}{% endfor %}",
|
||||
"chat_template_sha": "f1dff938141b507da4a409b6bb3431382088a97a963acd246a41f2f344ae831f",
|
||||
"start_time": 1392684.818305694,
|
||||
"end_time": 1392900.218863064,
|
||||
"total_evaluation_time_seconds": "215.40055736992508"
|
||||
}
|
||||
2707
evaluations/ar/Allam-7b-instruct-preview/openaimmlu_0_shot.json
Normal file
2707
evaluations/ar/Allam-7b-instruct-preview/openaimmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
123
evaluations/ar/Falcon3-7B-Instruct/acva_5_shot.json
Normal file
123
evaluations/ar/Falcon3-7B-Instruct/acva_5_shot.json
Normal file
@@ -0,0 +1,123 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.6045924225028703,
|
||||
"acc_stderr,none": 0.00523925695392083,
|
||||
"acc_norm,none": 0.5897818599311137,
|
||||
"acc_norm_stderr,none": 0.005270708411925859
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "5e10e017",
|
||||
"date": 1736889821.9957027,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.0",
|
||||
"upper_git_hash": "f64fe2f2a86055aaecced603b56097fd79201711",
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"acva": "f573ae5740e68711d257f2dc4a23db7c6b1c04895364f1af4b4eb64bfab793a4"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 600072.370318618,
|
||||
"end_time": 600217.222010416,
|
||||
"total_evaluation_time_seconds": "144.85169179795776"
|
||||
}
|
||||
142
evaluations/ar/Falcon3-7B-Instruct/ar_ifeval_0_shot.json
Normal file
142
evaluations/ar/Falcon3-7B-Instruct/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.08582089552238806,
|
||||
"prompt_level_strict_acc_stderr,none": 0.012109752724743699,
|
||||
"inst_level_strict_acc,none": 0.47918088737201364,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.13805970149253732,
|
||||
"prompt_level_loose_acc_stderr,none": 0.014914035308708435,
|
||||
"inst_level_loose_acc,none": 0.5276450511945392,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739621196.897086,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "ca837eed1e9f468712643d1fab81b7b48c88a8799239851476bdc889990e6b41"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
|
||||
"chat_template_sha": "914ccd80356f5822d1a50d97546e37f60c04ed831fe431aa40346574ec266901",
|
||||
"start_time": 1395880.012817552,
|
||||
"end_time": 1401371.318791154,
|
||||
"total_evaluation_time_seconds": "5491.305973601993"
|
||||
}
|
||||
126
evaluations/ar/Falcon3-7B-Instruct/araMath_v3_5_shot.json
Normal file
126
evaluations/ar/Falcon3-7B-Instruct/araMath_v3_5_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.5652892561983471,
|
||||
"acc_stderr,none": 0.020170519477736983,
|
||||
"acc_norm,none": 0.5652892561983471,
|
||||
"acc_norm_stderr,none": 0.020170519477736983
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u0645\u0646 \u0628\u064a\u0646 'A\u060c B\u060c C\u060c D' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739621084.921236,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "b7e29b20c532c7420cc659c6586d56642070560abff0925ed01ad8f200d8e72b"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
|
||||
"chat_template_sha": "914ccd80356f5822d1a50d97546e37f60c04ed831fe431aa40346574ec266901",
|
||||
"start_time": 1395768.116667791,
|
||||
"end_time": 1395816.745740765,
|
||||
"total_evaluation_time_seconds": "48.629072973970324"
|
||||
}
|
||||
130
evaluations/ar/Falcon3-7B-Instruct/araPro_0_shot.json
Normal file
130
evaluations/ar/Falcon3-7B-Instruct/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.41471705658868224,
|
||||
"acc_stderr,none": 0.006967450316480296,
|
||||
"acc_norm,none": 0.41471705658868224,
|
||||
"acc_norm_stderr,none": 0.006967450316480296
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617143.3614087,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"araPro": "063166ad2e52146b6a051c978bf54b1397281e222da633e81fa50357d2409ee9"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
|
||||
"chat_template_sha": "914ccd80356f5822d1a50d97546e37f60c04ed831fe431aa40346574ec266901",
|
||||
"start_time": 1391826.416201954,
|
||||
"end_time": 1394850.089034202,
|
||||
"total_evaluation_time_seconds": "3023.672832248034"
|
||||
}
|
||||
2090
evaluations/ar/Falcon3-7B-Instruct/arabicmmlu_0_shot.json
Normal file
2090
evaluations/ar/Falcon3-7B-Instruct/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/Falcon3-7B-Instruct/etec_v2_0_shot.json
Normal file
126
evaluations/ar/Falcon3-7B-Instruct/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.3751987281399046,
|
||||
"acc_stderr,none": 0.01114886834610489,
|
||||
"acc_norm,none": 0.3751987281399046,
|
||||
"acc_norm_stderr,none": 0.01114886834610489
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739620236.678696,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"etec_v2": "3a8dc6484af6c9538f122c1bbe5c6866dbe14df841fdf04ab7ff2b6437e8aeae"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
|
||||
"chat_template_sha": "914ccd80356f5822d1a50d97546e37f60c04ed831fe431aa40346574ec266901",
|
||||
"start_time": 1394919.684315533,
|
||||
"end_time": 1394995.42617788,
|
||||
"total_evaluation_time_seconds": "75.7418623471167"
|
||||
}
|
||||
125
evaluations/ar/Falcon3-7B-Instruct/exams_ar_5_shot.json
Normal file
125
evaluations/ar/Falcon3-7B-Instruct/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.31843575418994413,
|
||||
"acc_stderr,none": 0.020122499132803468,
|
||||
"acc_norm,none": 0.31843575418994413,
|
||||
"acc_norm_stderr,none": 0.020122499132803468
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "5e10e017",
|
||||
"date": 1736889028.6416683,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.0",
|
||||
"upper_git_hash": "f64fe2f2a86055aaecced603b56097fd79201711",
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"exams_ar": "f52ab3f14b240558420910fdb453ccb45c945cec187c0e60ea51cf6eff08973a"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 599279.04705073,
|
||||
"end_time": 599692.233103212,
|
||||
"total_evaluation_time_seconds": "413.1860524819931"
|
||||
}
|
||||
553
evaluations/ar/Falcon3-7B-Instruct/gat_0_shot.json
Normal file
553
evaluations/ar/Falcon3-7B-Instruct/gat_0_shot.json
Normal file
@@ -0,0 +1,553 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.27994481374639407,
|
||||
"acc_stderr,none": 0.003542796359675536,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.2571428571428571,
|
||||
"acc_stderr,none": 0.008420562208967575
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.24553734061930782,
|
||||
"acc_stderr,none": 0.008216476082874105
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.26573426573426573,
|
||||
"acc_stderr,none": 0.008475894211016492
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.24019138755980862,
|
||||
"acc_stderr,none": 0.013221495215360054
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.319672131147541,
|
||||
"acc_stderr,none": 0.013357022766710734
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.27520661157024795,
|
||||
"acc_stderr,none": 0.012844683062506254
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.26993865030674846,
|
||||
"acc_stderr,none": 0.01229815625441917
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.2876712328767123,
|
||||
"acc_stderr,none": 0.023726723391354485
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.3568998109640832,
|
||||
"acc_stderr,none": 0.009317121354774414
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.27994481374639407,
|
||||
"acc_stderr,none": 0.003542796359675536,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "5e10e017",
|
||||
"date": 1736891004.0192773,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.0",
|
||||
"upper_git_hash": "f64fe2f2a86055aaecced603b56097fd79201711",
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"gat_analogy": "04ac010c48ed039457058b512b7ac0586c7c76a628da7caaf9aeb8f3e99ae5e3",
|
||||
"gat_association": "2cbd868d220125bfcc54ae738592ad902191e4b7f804ce1772ae29e2d3bb3bf6",
|
||||
"gat_completion": "74cf159ef4a3455a6a0e984fed8e9e9a12f0dc21fde95c2058216c5a711a4d31",
|
||||
"gat_reading": "6f21934e536e7dca65361d01e5cafc27f8070c4f0dccf5a88c1fe071194b78a4",
|
||||
"gat_algebra": "20750c926608570eaf87d29981e5ab49b2b097bd52d7f749c44ab4e175d9fdd2",
|
||||
"gat_arithmetic": "c4b0c73c269d9eb3e8482fbda42e69191c28b95e75e1517d5f9142c6ef410204",
|
||||
"gat_comparisons": "88bc22db186a50cab28938ec1fc332366fa0bc886bc98edf810cc9ae938405db",
|
||||
"gat_contextual": "b8e88ff29b62b54eb834dca696304ca0fe1ce55d5cf7d0a9f0204456e3955be6",
|
||||
"gat_geometry": "229545188469d0512a3297737f4ec7afe88d8a30e7e04f87b4982548e83b1e56"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 601254.206185867,
|
||||
"end_time": 601373.470204397,
|
||||
"total_evaluation_time_seconds": "119.26401853002608"
|
||||
}
|
||||
127
evaluations/ar/Falcon3-7B-Instruct/moe_ien_mcq_0_shot.json
Normal file
127
evaluations/ar/Falcon3-7B-Instruct/moe_ien_mcq_0_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.5265265265265265,
|
||||
"acc_stderr,none": 0.004995706870392996,
|
||||
"acc_norm,none": 0.5265265265265265,
|
||||
"acc_norm_stderr,none": 0.004995706870392996
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739620378.768502,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "1ae93edb904d572143b5f36dd5dfcc4b901240916d4735ea328083598c912446"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
|
||||
"chat_template_sha": "914ccd80356f5822d1a50d97546e37f60c04ed831fe431aa40346574ec266901",
|
||||
"start_time": 1395061.894176973,
|
||||
"end_time": 1395336.684131379,
|
||||
"total_evaluation_time_seconds": "274.78995440597646"
|
||||
}
|
||||
129
evaluations/ar/Falcon3-7B-Instruct/moe_ien_tf_0_shot.json
Normal file
129
evaluations/ar/Falcon3-7B-Instruct/moe_ien_tf_0_shot.json
Normal file
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_tf": {
|
||||
"alias": "moe_ien_tf",
|
||||
"acc,none": 0.576335222393955,
|
||||
"acc_stderr,none": 0.006476086786980228,
|
||||
"acc_norm,none": 0.576335222393955,
|
||||
"acc_norm_stderr,none": 0.006476086786980228
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_tf": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_tf": {
|
||||
"task": "moe_ien_tf",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_tf/moe_ien_tf.py",
|
||||
"dataset_name": "moe_ien_tf",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n keys=[\"\u0635\u062d\u064a\u062d\u0629\",\n \"\u062e\u0627\u0637\u0626\u0629\"\n ]\n #keys =[\"\u0635\u0648\u0627\u0628\",\n # \"\u062e\u0637\u0623\"]\n target_key = int(doc[\"Answer\"])-1\n\n out_doc = {\n \"query\": \"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" +doc[\"Question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\", \n \"choices\": keys,\n \"gold\": target_key,\n }\n return out_doc\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{Subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d\u064a\u062d\u0629' \u0623\u0648 '\u062e\u0627\u0637\u0626\u0629' \u062f\u0648\u0646 \u0634\u0631\u062d ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_tf": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_tf": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_tf": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_tf": {
|
||||
"original": 5823,
|
||||
"effective": 5823
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=tiiuae/Falcon3-7B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7455550464,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "5563a370c1848366c7a095bde4bbff2cdb419cc6",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739620722.9521024,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|pad|>",
|
||||
"2023"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|endoftext|>",
|
||||
"11"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
null,
|
||||
"None"
|
||||
],
|
||||
"eot_token_id": 11,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"moe_ien_tf": "ed81617ccb178d095c9a81fef15f5ba8b655782b26d36117f53c38b0a84e62e5"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "tiiuae/Falcon3-7B-Instruct",
|
||||
"model_name_sanitized": "tiiuae__Falcon3-7B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if tools %}\n{{- '<|system|>\\n' }}\n{%- if messages[0]['role'] == 'system' %}\n{{- messages[0]['content'] }}\n{%- set remaining_messages = messages[1:] %}\n{%- else %}\n{%- set remaining_messages = messages %}\n{%- endif %}\n{{- 'You are a Falcon assistant skilled in function calling. You are helpful, respectful, and concise.\\n\\n# Tools\\n\\nYou have access to the following functions. You MUST use them to answer questions when needed. For each function call, you MUST return a JSON object inside <tool_call></tool_call> tags.\\n\\n<tools>' + tools|tojson(indent=2) + '</tools>\\n\\n# Output Format\\n\\nYour response MUST follow this format when making function calls:\\n<tool_call>\\n[\\n {\"name\": \"function_name\", \"arguments\": {\"arg1\": \"value1\", \"arg2\": \"value2\"}},\\n {\"name\": \"another_function\", \"arguments\": {\"arg\": \"value\"}}\\n]\\n</tool_call>\\nIf no function calls are needed, respond normally without the tool_call tags.\\n' }}\n{%- for message in remaining_messages %}\n{%- if message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if message.content %}\n{{- '<|assistant|>\\n' + message['content'] }}\n{%- endif %}\n{%- if message.tool_calls %}\n{{- '\\n<tool_call>\\n' }}\n{{- message.tool_calls|tojson(indent=2) }}\n{{- '\\n</tool_call>' }}\n{%- endif %}\n{{- eos_token + '\\n' }}\n{%- elif message['role'] == 'tool' %}\n{{- '<|assistant|>\\n<tool_response>\\n' + message['content'] + '\\n</tool_response>\\n' }}\n{%- endif %}\n{%- endfor %}\n{{- '<|assistant|>\\n' if add_generation_prompt }}\n{%- else %}\n{%- for message in messages %}\n{%- if message['role'] == 'system' %}\n{{- '<|system|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'user' %}\n{{- '<|user|>\\n' + message['content'] + '\\n' }}\n{%- elif message['role'] == 'assistant' %}\n{%- if not loop.last %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token + '\\n' }}\n{%- else %}\n{{- '<|assistant|>\\n' + message['content'] + eos_token }}\n{%- endif %}\n{%- endif %}\n{%- if loop.last and add_generation_prompt %}\n{{- '<|assistant|>\\n' }}\n{%- endif %}\n{%- endfor %}\n{%- endif %}",
|
||||
"chat_template_sha": "914ccd80356f5822d1a50d97546e37f60c04ed831fe431aa40346574ec266901",
|
||||
"start_time": 1395406.00589162,
|
||||
"end_time": 1395704.54657667,
|
||||
"total_evaluation_time_seconds": "298.54068504995666"
|
||||
}
|
||||
2711
evaluations/ar/Falcon3-7B-Instruct/openaimmlu_0_shot.json
Normal file
2711
evaluations/ar/Falcon3-7B-Instruct/openaimmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
125
evaluations/ar/Llama-3.3-70B-Instruct/acva_5_shot.json
Normal file
125
evaluations/ar/Llama-3.3-70B-Instruct/acva_5_shot.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.7847301951779564,
|
||||
"acc_stderr,none": 0.004404205705558861,
|
||||
"acc_norm,none": 0.769345579793341,
|
||||
"acc_norm_stderr,none": 0.004513957617295361
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1737861513.0031924,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.1",
|
||||
"upper_git_hash": "086919bd66f4e15fdcd4b792a7b27a698c1ba091",
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 822799.725415956,
|
||||
"end_time": 824041.525682158,
|
||||
"total_evaluation_time_seconds": "1241.8002662019571"
|
||||
}
|
||||
142
evaluations/ar/Llama-3.3-70B-Instruct/ar_ifeval_0_shot.json
Normal file
142
evaluations/ar/Llama-3.3-70B-Instruct/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.7089552238805971,
|
||||
"prompt_level_strict_acc_stderr,none": 0.019638685568678992,
|
||||
"inst_level_strict_acc,none": 0.8860068259385665,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.7947761194029851,
|
||||
"prompt_level_loose_acc_stderr,none": 0.017460611985170207,
|
||||
"inst_level_loose_acc,none": 0.9208191126279863,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738755018.193393,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "6bd5bfb26ee4f5909e16d66ee0e564fb2a5826815f16755272465c9e03f98a20"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 744977.123888747,
|
||||
"end_time": 758450.608805326,
|
||||
"total_evaluation_time_seconds": "13473.484916579095"
|
||||
}
|
||||
126
evaluations/ar/Llama-3.3-70B-Instruct/araMath_v3_5_shot.json
Normal file
126
evaluations/ar/Llama-3.3-70B-Instruct/araMath_v3_5_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.7090909090909091,
|
||||
"acc_stderr,none": 0.01848039016780232,
|
||||
"acc_norm,none": 0.7090909090909091,
|
||||
"acc_norm_stderr,none": 0.01848039016780232
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u0645\u0646 \u0628\u064a\u0646 'A\u060c B\u060c C\u060c D' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738750317.5038416,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "154ea94d6776e7d3980c98343cec49115ef3dc4dab8897fb4668f68494d55c76"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 740276.643313964,
|
||||
"end_time": 740434.169818474,
|
||||
"total_evaluation_time_seconds": "157.5265045099659"
|
||||
}
|
||||
130
evaluations/ar/Llama-3.3-70B-Instruct/araPro_0_shot.json
Normal file
130
evaluations/ar/Llama-3.3-70B-Instruct/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.7048590281943611,
|
||||
"acc_stderr,none": 0.006450314388729491,
|
||||
"acc_norm,none": 0.7048590281943611,
|
||||
"acc_norm_stderr,none": 0.006450314388729491
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738742514.712935,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"araPro": "ab4849e5668de72a27844a2a354787cbce92af5027f46a32300417b41913c5db"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 732473.787962617,
|
||||
"end_time": 736407.61692168,
|
||||
"total_evaluation_time_seconds": "3933.8289590630447"
|
||||
}
|
||||
2051
evaluations/ar/Llama-3.3-70B-Instruct/arabicmmlu_0_shot.json
Normal file
2051
evaluations/ar/Llama-3.3-70B-Instruct/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/Llama-3.3-70B-Instruct/etec_v2_0_shot.json
Normal file
126
evaluations/ar/Llama-3.3-70B-Instruct/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.6883942766295708,
|
||||
"acc_stderr,none": 0.010664745454850943,
|
||||
"acc_norm,none": 0.6883942766295708,
|
||||
"acc_norm_stderr,none": 0.010664745454850943
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738746708.9926562,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"etec_v2": "f9810ea40ab4721486631d02578e3b62811871d66f80ee350dc574ca63d72e12"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 736668.210182346,
|
||||
"end_time": 736927.122919428,
|
||||
"total_evaluation_time_seconds": "258.9127370819915"
|
||||
}
|
||||
127
evaluations/ar/Llama-3.3-70B-Instruct/exams_ar_5_shot.json
Normal file
127
evaluations/ar/Llama-3.3-70B-Instruct/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.6573556797020484,
|
||||
"acc_stderr,none": 0.02049932607490297,
|
||||
"acc_norm,none": 0.6573556797020484,
|
||||
"acc_norm_stderr,none": 0.02049932607490297
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1737862801.5409079,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.1",
|
||||
"upper_git_hash": "086919bd66f4e15fdcd4b792a7b27a698c1ba091",
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 824088.349963979,
|
||||
"end_time": 824352.47927673,
|
||||
"total_evaluation_time_seconds": "264.1293127509998"
|
||||
}
|
||||
545
evaluations/ar/Llama-3.3-70B-Instruct/gat_0_shot.json
Normal file
545
evaluations/ar/Llama-3.3-70B-Instruct/gat_0_shot.json
Normal file
@@ -0,0 +1,545 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.4412391822400602,
|
||||
"acc_stderr,none": 0.0038602448360070085,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.4148423005565863,
|
||||
"acc_stderr,none": 0.00949246890612482
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.3063752276867031,
|
||||
"acc_stderr,none": 0.008800291696618008
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.43908722856091276,
|
||||
"acc_stderr,none": 0.009522657932144745
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.3751196172248804,
|
||||
"acc_stderr,none": 0.014984183551431945
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.4131147540983607,
|
||||
"acc_stderr,none": 0.014102954212147805
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.5702479338842975,
|
||||
"acc_stderr,none": 0.014237301970481165
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.4148773006134969,
|
||||
"acc_stderr,none": 0.013649322722470929
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.473972602739726,
|
||||
"acc_stderr,none": 0.026171590093068544
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.5988657844990548,
|
||||
"acc_stderr,none": 0.00953188686023188
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.4412391822400602,
|
||||
"acc_stderr,none": 0.0038602448360070085,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
16
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1737819997.849324,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.1",
|
||||
"upper_git_hash": "086919bd66f4e15fdcd4b792a7b27a698c1ba091",
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 781284.750234253,
|
||||
"end_time": 782185.575911678,
|
||||
"total_evaluation_time_seconds": "900.8256774250185"
|
||||
}
|
||||
127
evaluations/ar/Llama-3.3-70B-Instruct/moe_ien_mcq_0_shot.json
Normal file
127
evaluations/ar/Llama-3.3-70B-Instruct/moe_ien_mcq_0_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.795995995995996,
|
||||
"acc_stderr,none": 0.004031937401121064,
|
||||
"acc_norm,none": 0.795995995995996,
|
||||
"acc_norm_stderr,none": 0.004031937401121064
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738747043.1224887,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "2f293909f445c6fdbe42ca2044dd07ac3eb752a7c1ea459602a8757356016dd9"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 737002.279803232,
|
||||
"end_time": 737981.71443428,
|
||||
"total_evaluation_time_seconds": "979.4346310478868"
|
||||
}
|
||||
129
evaluations/ar/Llama-3.3-70B-Instruct/moe_ien_tf_0_shot.json
Normal file
129
evaluations/ar/Llama-3.3-70B-Instruct/moe_ien_tf_0_shot.json
Normal file
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_tf": {
|
||||
"alias": "moe_ien_tf",
|
||||
"acc,none": 0.7880817448050833,
|
||||
"acc_stderr,none": 0.005355915518300743,
|
||||
"acc_norm,none": 0.7880817448050833,
|
||||
"acc_norm_stderr,none": 0.005355915518300743
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_tf": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_tf": {
|
||||
"task": "moe_ien_tf",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_tf/moe_ien_tf.py",
|
||||
"dataset_name": "moe_ien_tf",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n keys=[\"\u0635\u062d\u064a\u062d\u0629\",\n \"\u062e\u0627\u0637\u0626\u0629\"\n ]\n #keys =[\"\u0635\u0648\u0627\u0628\",\n # \"\u062e\u0637\u0623\"]\n target_key = int(doc[\"Answer\"])-1\n\n out_doc = {\n \"query\": \"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" +doc[\"Question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\", \n \"choices\": keys,\n \"gold\": target_key,\n }\n return out_doc\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{Subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d\u064a\u062d\u0629' \u0623\u0648 '\u062e\u0627\u0637\u0626\u0629' \u062f\u0648\u0646 \u0634\u0631\u062d ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_tf": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_tf": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_tf": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_tf": {
|
||||
"original": 5823,
|
||||
"effective": 5823
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Llama-3.3-70B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 70553706496,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "6f6073b423013f6a7d4d9f39144961bfbfbc386b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738748085.1630871,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|finetune_right_pad_id|>",
|
||||
"128004"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"moe_ien_tf": "ad47da488f5a4855855290b03172f21cc8709d26c8228bac708e4791056290c9"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Llama-3.3-70B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Llama-3.3-70B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 738044.375417544,
|
||||
"end_time": 739098.635400457,
|
||||
"total_evaluation_time_seconds": "1054.2599829129176"
|
||||
}
|
||||
2662
evaluations/ar/Llama-3.3-70B-Instruct/openaimmlu_0_shot.json
Normal file
2662
evaluations/ar/Llama-3.3-70B-Instruct/openaimmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
125
evaluations/ar/Meta-Llama-3.1-8B-Instruct/acva_5_shot.json
Normal file
125
evaluations/ar/Meta-Llama-3.1-8B-Instruct/acva_5_shot.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.7252583237657865,
|
||||
"acc_stderr,none": 0.004783265499715521,
|
||||
"acc_norm,none": 0.6993111366245695,
|
||||
"acc_norm_stderr,none": 0.004913712570670582
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "0e9e39f249a16976918f6564b8830bc894c89659",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1737779312.1802437,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.1",
|
||||
"upper_git_hash": "086919bd66f4e15fdcd4b792a7b27a698c1ba091",
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 26393.300114519,
|
||||
"end_time": 26592.595877222,
|
||||
"total_evaluation_time_seconds": "199.29576270300095"
|
||||
}
|
||||
138
evaluations/ar/Meta-Llama-3.1-8B-Instruct/ar_ifeval_0_shot.json
Normal file
138
evaluations/ar/Meta-Llama-3.1-8B-Instruct/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,138 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.5335820895522388,
|
||||
"prompt_level_strict_acc_stderr,none": 0.021568072772161277,
|
||||
"inst_level_strict_acc,none": 0.7931740614334472,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.6156716417910447,
|
||||
"prompt_level_loose_acc_stderr,none": 0.021030466164007045,
|
||||
"inst_level_loose_acc,none": 0.8327645051194539,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,tensor_parallel_size=4,data_parallel_size=2,download_dir=/tmp,enforce_eager=False",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738654504.3474658,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "7e137a94e1650273c7c8431db3a799d999471d4003bbb61e67fc4369b573a251"
|
||||
},
|
||||
"model_source": "vllm",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 437265.909736722,
|
||||
"end_time": 437523.975434726,
|
||||
"total_evaluation_time_seconds": "258.06569800397847"
|
||||
}
|
||||
122
evaluations/ar/Meta-Llama-3.1-8B-Instruct/araMath_v3_5_shot.json
Normal file
122
evaluations/ar/Meta-Llama-3.1-8B-Instruct/araMath_v3_5_shot.json
Normal file
@@ -0,0 +1,122 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.35702479338842974,
|
||||
"acc_stderr,none": 0.019495206164626543,
|
||||
"acc_norm,none": 0.35702479338842974,
|
||||
"acc_norm_stderr,none": 0.019495206164626543
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\nالسؤال: {question}\\n{choices}\\nالاجابة:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "من فضلك اختر إجابة واحدة من بين 'A، B، C، D' دون شرح",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,tensor_parallel_size=1,data_parallel_size=8,download_dir=/tmp,enforce_eager=False",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738675025.3226728,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.90\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "544990bff2e8bb7c1408ff006ba780ea68d8d7f78c633fb7035e71e43345d5a4"
|
||||
},
|
||||
"model_source": "vllm",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 1038640.023630069,
|
||||
"end_time": 1038724.32179284,
|
||||
"total_evaluation_time_seconds": "84.29816277103964"
|
||||
}
|
||||
130
evaluations/ar/Meta-Llama-3.1-8B-Instruct/araPro_0_shot.json
Normal file
130
evaluations/ar/Meta-Llama-3.1-8B-Instruct/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.5250949810037993,
|
||||
"acc_stderr,none": 0.007062156072028268,
|
||||
"acc_norm,none": 0.5250949810037993,
|
||||
"acc_norm_stderr,none": 0.007062156072028268
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "0e9e39f249a16976918f6564b8830bc894c89659",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617047.873544,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.90\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"araPro": "ab4849e5668de72a27844a2a354787cbce92af5027f46a32300417b41913c5db"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 1609822.907637183,
|
||||
"end_time": 1610372.150443636,
|
||||
"total_evaluation_time_seconds": "549.242806453025"
|
||||
}
|
||||
2051
evaluations/ar/Meta-Llama-3.1-8B-Instruct/arabicmmlu_0_shot.json
Normal file
2051
evaluations/ar/Meta-Llama-3.1-8B-Instruct/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/Meta-Llama-3.1-8B-Instruct/etec_v2_0_shot.json
Normal file
126
evaluations/ar/Meta-Llama-3.1-8B-Instruct/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.456809750927398,
|
||||
"acc_stderr,none": 0.01147024835105639,
|
||||
"acc_norm,none": 0.456809750927398,
|
||||
"acc_norm_stderr,none": 0.01147024835105639
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "0e9e39f249a16976918f6564b8830bc894c89659",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617646.5966089,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.90\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"etec_v2": "f9810ea40ab4721486631d02578e3b62811871d66f80ee350dc574ca63d72e12"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 1610421.453807966,
|
||||
"end_time": 1610498.158299866,
|
||||
"total_evaluation_time_seconds": "76.70449189981446"
|
||||
}
|
||||
127
evaluations/ar/Meta-Llama-3.1-8B-Instruct/exams_ar_5_shot.json
Normal file
127
evaluations/ar/Meta-Llama-3.1-8B-Instruct/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.5251396648044693,
|
||||
"acc_stderr,none": 0.02156939500417479,
|
||||
"acc_norm,none": 0.5251396648044693,
|
||||
"acc_norm_stderr,none": 0.02156939500417479
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "0e9e39f249a16976918f6564b8830bc894c89659",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1737779550.003421,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.1",
|
||||
"upper_git_hash": "086919bd66f4e15fdcd4b792a7b27a698c1ba091",
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 26630.902640257,
|
||||
"end_time": 26676.356655983,
|
||||
"total_evaluation_time_seconds": "45.45401572599803"
|
||||
}
|
||||
545
evaluations/ar/Meta-Llama-3.1-8B-Instruct/gat_0_shot.json
Normal file
545
evaluations/ar/Meta-Llama-3.1-8B-Instruct/gat_0_shot.json
Normal file
@@ -0,0 +1,545 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.3090430201931519,
|
||||
"acc_stderr,none": 0.003623669512802982,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.25120593692022264,
|
||||
"acc_stderr,none": 0.008355979196698268
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.2754098360655738,
|
||||
"acc_stderr,none": 0.008527935108212162
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.2929701877070298,
|
||||
"acc_stderr,none": 0.00873304494093164
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.26889952153110047,
|
||||
"acc_stderr,none": 0.013722501896040254
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.26721311475409837,
|
||||
"acc_stderr,none": 0.01267406341937153
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.37355371900826445,
|
||||
"acc_stderr,none": 0.013912503912467983
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.28450920245398775,
|
||||
"acc_stderr,none": 0.012499077975909817
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.2876712328767123,
|
||||
"acc_stderr,none": 0.023726723391354478
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.4400756143667297,
|
||||
"acc_stderr,none": 0.009653784894336059
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.3090430201931519,
|
||||
"acc_stderr,none": 0.003623669512802982,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "0e9e39f249a16976918f6564b8830bc894c89659",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
16
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1737768859.2760568,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.1",
|
||||
"upper_git_hash": "086919bd66f4e15fdcd4b792a7b27a698c1ba091",
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 15940.236927019,
|
||||
"end_time": 16130.776899079,
|
||||
"total_evaluation_time_seconds": "190.53997205999985"
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.5922922922922923,
|
||||
"acc_stderr,none": 0.004916788134998954,
|
||||
"acc_norm,none": 0.5922922922922923,
|
||||
"acc_norm_stderr,none": 0.004916788134998954
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 8030261248,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "0e9e39f249a16976918f6564b8830bc894c89659",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617794.6685781,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.90\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "2f293909f445c6fdbe42ca2044dd07ac3eb752a7c1ea459602a8757356016dd9"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 1610569.610297447,
|
||||
"end_time": 1610870.6725387,
|
||||
"total_evaluation_time_seconds": "301.0622412529774"
|
||||
}
|
||||
125
evaluations/ar/Meta-Llama-3.1-8B-Instruct/moe_ien_tf_0_shot.json
Normal file
125
evaluations/ar/Meta-Llama-3.1-8B-Instruct/moe_ien_tf_0_shot.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_tf": {
|
||||
"alias": "moe_ien_tf",
|
||||
"acc,none": 0.7173278378842521,
|
||||
"acc_stderr,none": 0.005901525152083598,
|
||||
"acc_norm,none": 0.7173278378842521,
|
||||
"acc_norm_stderr,none": 0.005901525152083598
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_tf": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_tf": {
|
||||
"task": "moe_ien_tf",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_tf/moe_ien_tf.py",
|
||||
"dataset_name": "moe_ien_tf",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n keys=[\"صحيحة\",\n \"خاطئة\"\n ]\n #keys =[\"صواب\",\n # \"خطأ\"]\n target_key = int(doc[\"Answer\"])-1\n\n out_doc = {\n \"query\": \"\\n\\nالسؤال:\" +doc[\"Question\"]+\"\\nإجابة:'\", \n \"choices\": keys,\n \"gold\": target_key,\n }\n return out_doc\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "فيما يلي عبارات إما صحيحة أو خاطئة حول {{Subject}}\n الرجاء تصنيف العبارة إلى 'صحيحة' أو 'خاطئة' دون شرح ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_tf": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_tf": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_tf": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_tf": {
|
||||
"original": 5823,
|
||||
"effective": 5823
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct,tensor_parallel_size=1,data_parallel_size=8,download_dir=/tmp,enforce_eager=False",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "788a3672",
|
||||
"date": 1738682459.4089465,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"<|eot_id|>",
|
||||
"128009"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<|begin_of_text|>",
|
||||
"128000"
|
||||
],
|
||||
"eot_token_id": 128009,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"moe_ien_tf": "8233395e832e9bd87361282558343c4a080c3ea607d00e045339d417c84f4e85"
|
||||
},
|
||||
"model_source": "vllm",
|
||||
"model_name": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"model_name_sanitized": "meta-llama__Meta-Llama-3.1-8B-Instruct",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
|
||||
"chat_template_sha": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65",
|
||||
"start_time": 1222389.067343241,
|
||||
"end_time": 1222491.234081002,
|
||||
"total_evaluation_time_seconds": "102.16673776088282"
|
||||
}
|
||||
2662
evaluations/ar/Meta-Llama-3.1-8B-Instruct/openaimmlu_0_shot.json
Normal file
2662
evaluations/ar/Meta-Llama-3.1-8B-Instruct/openaimmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
123
evaluations/ar/Mistral-7B-Instruct-v0.3/acva_5_shot.json
Normal file
123
evaluations/ar/Mistral-7B-Instruct-v0.3/acva_5_shot.json
Normal file
@@ -0,0 +1,123 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.6222732491389208,
|
||||
"acc_stderr,none": 0.005195116511309794,
|
||||
"acc_norm,none": 0.6025258323765786,
|
||||
"acc_norm_stderr,none": 0.005243945200841987
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "112b79143",
|
||||
"date": 1739212726.4606693,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 1323160.590628094,
|
||||
"end_time": 1324067.409366255,
|
||||
"total_evaluation_time_seconds": "906.8187381608877"
|
||||
}
|
||||
142
evaluations/ar/Mistral-7B-Instruct-v0.3/ar_ifeval_0_shot.json
Normal file
142
evaluations/ar/Mistral-7B-Instruct-v0.3/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.3041044776119403,
|
||||
"prompt_level_strict_acc_stderr,none": 0.019888706432720362,
|
||||
"inst_level_strict_acc,none": 0.6402730375426621,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.3656716417910448,
|
||||
"prompt_level_loose_acc_stderr,none": 0.020822161638297292,
|
||||
"inst_level_loose_acc,none": 0.6839590443686007,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618660.514274,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "b8aedf628540509f53512423803c97c0af76f913e1d9c5626e46aceefce168b2"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1583947.032479211,
|
||||
"end_time": 1595212.6691982,
|
||||
"total_evaluation_time_seconds": "11265.636718989117"
|
||||
}
|
||||
126
evaluations/ar/Mistral-7B-Instruct-v0.3/araMath_v3_5_shot.json
Normal file
126
evaluations/ar/Mistral-7B-Instruct-v0.3/araMath_v3_5_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.27107438016528923,
|
||||
"acc_stderr,none": 0.01808703482553977,
|
||||
"acc_norm,none": 0.27107438016528923,
|
||||
"acc_norm_stderr,none": 0.01808703482553977
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u0645\u0646 \u0628\u064a\u0646 'A\u060c B\u060c C\u060c D' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618557.9082067,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "fc6325d1e91d814a9212e7cd3d01a2ea0128526a5ff5a12b13029293c7b85a14"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1583844.288660905,
|
||||
"end_time": 1583895.209942275,
|
||||
"total_evaluation_time_seconds": "50.921281369868666"
|
||||
}
|
||||
130
evaluations/ar/Mistral-7B-Instruct-v0.3/araPro_0_shot.json
Normal file
130
evaluations/ar/Mistral-7B-Instruct-v0.3/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.4385122975404919,
|
||||
"acc_stderr,none": 0.007017396418135006,
|
||||
"acc_norm,none": 0.4385122975404919,
|
||||
"acc_norm_stderr,none": 0.007017396418135006
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617070.2494006,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"araPro": "199097343993a3034793f07adc5e21cca4b5d4e6175f4b73353037c1f92be7cc"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1582356.635046546,
|
||||
"end_time": 1582870.225454165,
|
||||
"total_evaluation_time_seconds": "513.5904076187871"
|
||||
}
|
||||
2045
evaluations/ar/Mistral-7B-Instruct-v0.3/arabicmmlu_0_shot.json
Normal file
2045
evaluations/ar/Mistral-7B-Instruct-v0.3/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/Mistral-7B-Instruct-v0.3/etec_v2_0_shot.json
Normal file
126
evaluations/ar/Mistral-7B-Instruct-v0.3/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.3566507684154743,
|
||||
"acc_stderr,none": 0.01102996491785656,
|
||||
"acc_norm,none": 0.3566507684154743,
|
||||
"acc_norm_stderr,none": 0.01102996491785656
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617648.4240222,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"etec_v2": "8dd4f73e94b492d082eebafc44fe527d605540255eaf869f23c7d51e4ffb37c4"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1582935.013239375,
|
||||
"end_time": 1583016.72552446,
|
||||
"total_evaluation_time_seconds": "81.71228508488275"
|
||||
}
|
||||
125
evaluations/ar/Mistral-7B-Instruct-v0.3/exams_ar_5_shot.json
Normal file
125
evaluations/ar/Mistral-7B-Instruct-v0.3/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.3407821229050279,
|
||||
"acc_stderr,none": 0.02047248187699896,
|
||||
"acc_norm,none": 0.3407821229050279,
|
||||
"acc_norm_stderr,none": 0.02047248187699896
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "112b79143",
|
||||
"date": 1739211970.5611851,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 1322404.630248276,
|
||||
"end_time": 1322480.6699447,
|
||||
"total_evaluation_time_seconds": "76.03969642403536"
|
||||
}
|
||||
543
evaluations/ar/Mistral-7B-Instruct-v0.3/gat_0_shot.json
Normal file
543
evaluations/ar/Mistral-7B-Instruct-v0.3/gat_0_shot.json
Normal file
@@ -0,0 +1,543 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.2664618086040386,
|
||||
"acc_stderr,none": 0.003495353970358859,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.24935064935064935,
|
||||
"acc_stderr,none": 0.008335372497778036
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.2983606557377049,
|
||||
"acc_stderr,none": 0.00873445255221157
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.25874125874125875,
|
||||
"acc_stderr,none": 0.008403358167147365
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.19138755980861244,
|
||||
"acc_stderr,none": 0.012175219862346352
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.30573770491803276,
|
||||
"acc_stderr,none": 0.013195760894549713
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.27603305785123966,
|
||||
"acc_stderr,none": 0.012856618756239491
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.2561349693251534,
|
||||
"acc_stderr,none": 0.012092310807729188
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.25205479452054796,
|
||||
"acc_stderr,none": 0.022757873597035808
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.2729678638941399,
|
||||
"acc_stderr,none": 0.008663668753419975
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.2664618086040386,
|
||||
"acc_stderr,none": 0.003495353970358859,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "112b79143",
|
||||
"date": 1739240499.1300695,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 1350933.020023772,
|
||||
"end_time": 1351628.555126437,
|
||||
"total_evaluation_time_seconds": "695.5351026649587"
|
||||
}
|
||||
127
evaluations/ar/Mistral-7B-Instruct-v0.3/moe_ien_mcq_0_shot.json
Normal file
127
evaluations/ar/Mistral-7B-Instruct-v0.3/moe_ien_mcq_0_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.5359359359359359,
|
||||
"acc_stderr,none": 0.004989814518061573,
|
||||
"acc_norm,none": 0.5359359359359359,
|
||||
"acc_norm_stderr,none": 0.004989814518061573
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617801.8553765,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "7b5b044e4260d8f2ccd928941529cc6f13c02303af5ed0b926cb22069d0a3368"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1583088.400197009,
|
||||
"end_time": 1583390.481922052,
|
||||
"total_evaluation_time_seconds": "302.08172504301183"
|
||||
}
|
||||
129
evaluations/ar/Mistral-7B-Instruct-v0.3/moe_ien_tf_0_shot.json
Normal file
129
evaluations/ar/Mistral-7B-Instruct-v0.3/moe_ien_tf_0_shot.json
Normal file
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_tf": {
|
||||
"alias": "moe_ien_tf",
|
||||
"acc,none": 0.6340374377468658,
|
||||
"acc_stderr,none": 0.0063130565613714554,
|
||||
"acc_norm,none": 0.6340374377468658,
|
||||
"acc_norm_stderr,none": 0.0063130565613714554
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_tf": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_tf": {
|
||||
"task": "moe_ien_tf",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_tf/moe_ien_tf.py",
|
||||
"dataset_name": "moe_ien_tf",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n keys=[\"\u0635\u062d\u064a\u062d\u0629\",\n \"\u062e\u0627\u0637\u0626\u0629\"\n ]\n #keys =[\"\u0635\u0648\u0627\u0628\",\n # \"\u062e\u0637\u0623\"]\n target_key = int(doc[\"Answer\"])-1\n\n out_doc = {\n \"query\": \"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" +doc[\"Question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\", \n \"choices\": keys,\n \"gold\": target_key,\n }\n return out_doc\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{Subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d\u064a\u062d\u0629' \u0623\u0648 '\u062e\u0627\u0637\u0626\u0629' \u062f\u0648\u0646 \u0634\u0631\u062d ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_tf": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_tf": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_tf": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_tf": {
|
||||
"original": 5823,
|
||||
"effective": 5823
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-7B-Instruct-v0.3,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 7248023552,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "e0bc86c23ce5aae1db576c8cca6f06f1f73af2db",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618159.7425826,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.88\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"moe_ien_tf": "87c1341e70cacc508279240f78ecd4d5d873569e238982ef3f15031c20f834da"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-7B-Instruct-v0.3",
|
||||
"model_name_sanitized": "mistralai__Mistral-7B-Instruct-v0.3",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1583446.306152082,
|
||||
"end_time": 1583776.933878196,
|
||||
"total_evaluation_time_seconds": "330.62772611388937"
|
||||
}
|
||||
2660
evaluations/ar/Mistral-7B-Instruct-v0.3/openaimmlu_0_shot.json
Normal file
2660
evaluations/ar/Mistral-7B-Instruct-v0.3/openaimmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
123
evaluations/ar/Mistral-Nemo-Instruct-2407/acva_5_shot.json
Normal file
123
evaluations/ar/Mistral-Nemo-Instruct-2407/acva_5_shot.json
Normal file
@@ -0,0 +1,123 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.7742824339839265,
|
||||
"acc_stderr,none": 0.004479692846303672,
|
||||
"acc_norm,none": 0.7692307692307693,
|
||||
"acc_norm_stderr,none": 0.004514744002858174
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 12247782400,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8aedd450f2583e9c67fae1929f6936b8fc5aef9c",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "5e10e017",
|
||||
"date": 1736969133.0360518,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.0",
|
||||
"upper_git_hash": "2e5cd5395faf76fea1afc96dd0f7161a9d3aa145",
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 4322.019001477,
|
||||
"end_time": 5037.885975796,
|
||||
"total_evaluation_time_seconds": "715.8669743190003"
|
||||
}
|
||||
142
evaluations/ar/Mistral-Nemo-Instruct-2407/ar_ifeval_0_shot.json
Normal file
142
evaluations/ar/Mistral-Nemo-Instruct-2407/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.3582089552238806,
|
||||
"prompt_level_strict_acc_stderr,none": 0.020729467924035978,
|
||||
"inst_level_strict_acc,none": 0.70580204778157,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.44402985074626866,
|
||||
"prompt_level_loose_acc_stderr,none": 0.021481021503779226,
|
||||
"inst_level_loose_acc,none": 0.7631399317406143,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 12247782400,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8aedd450f2583e9c67fae1929f6936b8fc5aef9c",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739619352.6594934,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "f326b8a98c506486038a589a169e687707c38c2ea33f7dd1189337e8bafb199b"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS][\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST]\" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST]\" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}\n {{- \"[TOOL_CALLS][\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- message[\"content\"] + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS]{\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e4676cb56dffea7782fd3e2b577cfaf1e123537e6ef49b3ec7caa6c095c62272",
|
||||
"start_time": 1394036.696567707,
|
||||
"end_time": 1403362.389299741,
|
||||
"total_evaluation_time_seconds": "9325.692732034018"
|
||||
}
|
||||
126
evaluations/ar/Mistral-Nemo-Instruct-2407/araMath_v3_5_shot.json
Normal file
126
evaluations/ar/Mistral-Nemo-Instruct-2407/araMath_v3_5_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.4,
|
||||
"acc_stderr,none": 0.01993366482555282,
|
||||
"acc_norm,none": 0.4,
|
||||
"acc_norm_stderr,none": 0.01993366482555282
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u0645\u0646 \u0628\u064a\u0646 'A\u060c B\u060c C\u060c D' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 12247782400,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8aedd450f2583e9c67fae1929f6936b8fc5aef9c",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739619227.1134682,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "3fff45213e85bf51326ed6c644cc5e49da5f0dc899148eedf05f142fb3a2e9d7"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS][\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST]\" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST]\" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}\n {{- \"[TOOL_CALLS][\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- message[\"content\"] + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS]{\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e4676cb56dffea7782fd3e2b577cfaf1e123537e6ef49b3ec7caa6c095c62272",
|
||||
"start_time": 1393911.575720478,
|
||||
"end_time": 1393979.700059605,
|
||||
"total_evaluation_time_seconds": "68.1243391269818"
|
||||
}
|
||||
130
evaluations/ar/Mistral-Nemo-Instruct-2407/araPro_0_shot.json
Normal file
130
evaluations/ar/Mistral-Nemo-Instruct-2407/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.5760847830433913,
|
||||
"acc_stderr,none": 0.006988720995850974,
|
||||
"acc_norm,none": 0.5760847830433913,
|
||||
"acc_norm_stderr,none": 0.006988720995850974
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 12247782400,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8aedd450f2583e9c67fae1929f6936b8fc5aef9c",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617064.9446375,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"araPro": "c501abfa12db371c0936f3cfe29510e3ea50fba562223331bd89379a5f2e9338"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS][\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST]\" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST]\" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}\n {{- \"[TOOL_CALLS][\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- message[\"content\"] + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS]{\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e4676cb56dffea7782fd3e2b577cfaf1e123537e6ef49b3ec7caa6c095c62272",
|
||||
"start_time": 1391749.239940259,
|
||||
"end_time": 1392584.762478395,
|
||||
"total_evaluation_time_seconds": "835.5225381359924"
|
||||
}
|
||||
2045
evaluations/ar/Mistral-Nemo-Instruct-2407/arabicmmlu_0_shot.json
Normal file
2045
evaluations/ar/Mistral-Nemo-Instruct-2407/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/Mistral-Nemo-Instruct-2407/etec_v2_0_shot.json
Normal file
126
evaluations/ar/Mistral-Nemo-Instruct-2407/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.492845786963434,
|
||||
"acc_stderr,none": 0.011512103852890532,
|
||||
"acc_norm,none": 0.492845786963434,
|
||||
"acc_norm_stderr,none": 0.011512103852890532
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 12247782400,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8aedd450f2583e9c67fae1929f6936b8fc5aef9c",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617957.7964923,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"etec_v2": "e06d601415c83f4efd3319516e349cd6cfb9329222e71456a9d89dce2525be0f"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS][\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST]\" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST]\" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}\n {{- \"[TOOL_CALLS][\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- message[\"content\"] + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS]{\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e4676cb56dffea7782fd3e2b577cfaf1e123537e6ef49b3ec7caa6c095c62272",
|
||||
"start_time": 1392642.204060316,
|
||||
"end_time": 1392751.762366377,
|
||||
"total_evaluation_time_seconds": "109.55830606096424"
|
||||
}
|
||||
127
evaluations/ar/Mistral-Nemo-Instruct-2407/exams_ar_5_shot.json
Normal file
127
evaluations/ar/Mistral-Nemo-Instruct-2407/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.4748603351955307,
|
||||
"acc_stderr,none": 0.0215693950041748,
|
||||
"acc_norm,none": 0.4748603351955307,
|
||||
"acc_norm_stderr,none": 0.0215693950041748
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 1.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 1.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 12247782400,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8aedd450f2583e9c67fae1929f6936b8fc5aef9c",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "150ae04f",
|
||||
"date": 1737022373.3396137,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.86\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.0",
|
||||
"upper_git_hash": "2e5cd5395faf76fea1afc96dd0f7161a9d3aa145",
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 4543.296069537,
|
||||
"end_time": 5184.026563092,
|
||||
"total_evaluation_time_seconds": "640.7304935550001"
|
||||
}
|
||||
539
evaluations/ar/Mistral-Nemo-Instruct-2407/gat_0_shot.json
Normal file
539
evaluations/ar/Mistral-Nemo-Instruct-2407/gat_0_shot.json
Normal file
@@ -0,0 +1,539 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.2544211714536561,
|
||||
"acc_stderr,none": 0.0034266849246390933,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.21818181818181817,
|
||||
"acc_stderr,none": 0.007957256646112694
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.22768670309653916,
|
||||
"acc_stderr,none": 0.008005224886568718
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.2388663967611336,
|
||||
"acc_stderr,none": 0.008181691396125238
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.19904306220095694,
|
||||
"acc_stderr,none": 0.012357421397385122
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.18934426229508197,
|
||||
"acc_stderr,none": 0.011221281369022177
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.3074380165289256,
|
||||
"acc_stderr,none": 0.01327073443676181
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.2837423312883436,
|
||||
"acc_stderr,none": 0.012488908992810271
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.2821917808219178,
|
||||
"acc_stderr,none": 0.02358987837397864
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.3444234404536862,
|
||||
"acc_stderr,none": 0.009241177951937967
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.2544211714536561,
|
||||
"acc_stderr,none": 0.0034266849246390933,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "vllm",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,tensor_parallel_size=1,data_parallel_size=2,gpu_memory_utilization=0.8,download_dir=/tmp",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "8e1bd48d",
|
||||
"date": 1735737160.254528,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.47.1",
|
||||
"upper_git_hash": "f64fe2f2a86055aaecced603b56097fd79201711",
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {},
|
||||
"model_source": "vllm",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 99835.105661851,
|
||||
"end_time": 100475.795755295,
|
||||
"total_evaluation_time_seconds": "640.6900934439909"
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.6842842842842843,
|
||||
"acc_stderr,none": 0.0046505613370222115,
|
||||
"acc_norm,none": 0.6842842842842843,
|
||||
"acc_norm_stderr,none": 0.0046505613370222115
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 12247782400,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8aedd450f2583e9c67fae1929f6936b8fc5aef9c",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618126.3322697,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "e05a3d8b5f495479981b5fde66a4e065b41dec7a24c3efeb64d267eaf3c23cbd"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS][\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST]\" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST]\" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}\n {{- \"[TOOL_CALLS][\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- message[\"content\"] + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS]{\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e4676cb56dffea7782fd3e2b577cfaf1e123537e6ef49b3ec7caa6c095c62272",
|
||||
"start_time": 1392810.603565529,
|
||||
"end_time": 1393281.038045333,
|
||||
"total_evaluation_time_seconds": "470.43447980401106"
|
||||
}
|
||||
129
evaluations/ar/Mistral-Nemo-Instruct-2407/moe_ien_tf_0_shot.json
Normal file
129
evaluations/ar/Mistral-Nemo-Instruct-2407/moe_ien_tf_0_shot.json
Normal file
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_tf": {
|
||||
"alias": "moe_ien_tf",
|
||||
"acc,none": 0.7178430362356174,
|
||||
"acc_stderr,none": 0.005898261619714902,
|
||||
"acc_norm,none": 0.7178430362356174,
|
||||
"acc_norm_stderr,none": 0.005898261619714902
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_tf": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_tf": {
|
||||
"task": "moe_ien_tf",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_tf/moe_ien_tf.py",
|
||||
"dataset_name": "moe_ien_tf",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n keys=[\"\u0635\u062d\u064a\u062d\u0629\",\n \"\u062e\u0627\u0637\u0626\u0629\"\n ]\n #keys =[\"\u0635\u0648\u0627\u0628\",\n # \"\u062e\u0637\u0623\"]\n target_key = int(doc[\"Answer\"])-1\n\n out_doc = {\n \"query\": \"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" +doc[\"Question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\", \n \"choices\": keys,\n \"gold\": target_key,\n }\n return out_doc\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{Subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d\u064a\u062d\u0629' \u0623\u0648 '\u062e\u0627\u0637\u0626\u0629' \u062f\u0648\u0646 \u0634\u0631\u062d ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_tf": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_tf": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_tf": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_tf": {
|
||||
"original": 5823,
|
||||
"effective": 5823
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Nemo-Instruct-2407,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 12247782400,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8aedd450f2583e9c67fae1929f6936b8fc5aef9c",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618656.899603,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"<unk>",
|
||||
"0"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 131072,
|
||||
"task_hashes": {
|
||||
"moe_ien_tf": "ff025e68710a3689e092aa2517e40514bb9f34f121dd37f9dcb54e7db60b2810"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
"model_name_sanitized": "mistralai__Mistral-Nemo-Instruct-2407",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS][\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST]\" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST]\" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif (message.tool_calls is defined and message.tool_calls is not none) %}\n {{- \"[TOOL_CALLS][\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- message[\"content\"] + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS]{\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e4676cb56dffea7782fd3e2b577cfaf1e123537e6ef49b3ec7caa6c095c62272",
|
||||
"start_time": 1393341.259869937,
|
||||
"end_time": 1393849.086177709,
|
||||
"total_evaluation_time_seconds": "507.8263077719603"
|
||||
}
|
||||
2662
evaluations/ar/Mistral-Nemo-Instruct-2407/openaimmlu_0_shot.json
Normal file
2662
evaluations/ar/Mistral-Nemo-Instruct-2407/openaimmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
123
evaluations/ar/Mistral-Small-Instruct-2409/acva_5_shot.json
Normal file
123
evaluations/ar/Mistral-Small-Instruct-2409/acva_5_shot.json
Normal file
@@ -0,0 +1,123 @@
|
||||
{
|
||||
"results": {
|
||||
"acva": {
|
||||
"alias": "acva",
|
||||
"acc,none": 0.7159586681974741,
|
||||
"acc_stderr,none": 0.004832263417483554,
|
||||
"acc_norm,none": 0.6893226176808266,
|
||||
"acc_norm_stderr,none": 0.004958861031051597
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"acva": []
|
||||
},
|
||||
"configs": {
|
||||
"acva": {
|
||||
"task": "acva",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "FreedomIntelligence/ACVA-Arabic-Cultural-Value-Alignment",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _format_subject(subject):\n \n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n \n def _generate_subject(doc):\n subject = _format_subject(doc[\"id\"].split(\"-\")[0])\n\n return subject\n \n def _process_docs(doc):\n keys = [\"\u0635\u062d\",\n \"\u062e\u0637\u0623\"]\n subject = _generate_subject(doc)\n gold = keys.index(doc['answer'])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644:\" + doc[\"question\"]+\"\\n\u0625\u062c\u0627\u0628\u0629:'\",\n \"choices\": keys,\n \"gold\": gold,\n \"subject\": subject,\n }\n \n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0639\u0628\u0627\u0631\u0627\u062a \u0625\u0645\u0627 \u0635\u062d\u064a\u062d\u0629 \u0623\u0648 \u062e\u0627\u0637\u0626\u0629 \u062d\u0648\u0644 {{subject}}\n \u0627\u0644\u0631\u062c\u0627\u0621 \u062a\u0635\u0646\u064a\u0641 \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0625\u0644\u0649 '\u0635\u062d' \u0623\u0648 '\u062e\u0637\u0623' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"acva": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"acva": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"acva": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"acva": {
|
||||
"original": 8710,
|
||||
"effective": 8710
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Small-Instruct-2409,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 22247282688,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8012044390bdc1c6d8ab162f5416220f43bf517b",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
64
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "5e10e017",
|
||||
"date": 1736969697.6002197,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.0",
|
||||
"upper_git_hash": "2e5cd5395faf76fea1afc96dd0f7161a9d3aa145",
|
||||
"tokenizer_pad_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Small-Instruct-2409",
|
||||
"model_name_sanitized": "mistralai__Mistral-Small-Instruct-2409",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 5310.719588598,
|
||||
"end_time": 7490.179107189,
|
||||
"total_evaluation_time_seconds": "2179.4595185910002"
|
||||
}
|
||||
142
evaluations/ar/Mistral-Small-Instruct-2409/ar_ifeval_0_shot.json
Normal file
142
evaluations/ar/Mistral-Small-Instruct-2409/ar_ifeval_0_shot.json
Normal file
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"results": {
|
||||
"ar_ifeval": {
|
||||
"alias": "ar_ifeval",
|
||||
"prompt_level_strict_acc,none": 0.5111940298507462,
|
||||
"prompt_level_strict_acc_stderr,none": 0.021611466915389024,
|
||||
"inst_level_strict_acc,none": 0.7815699658703071,
|
||||
"inst_level_strict_acc_stderr,none": "N/A",
|
||||
"prompt_level_loose_acc,none": 0.6436567164179104,
|
||||
"prompt_level_loose_acc_stderr,none": 0.020705444127112654,
|
||||
"inst_level_loose_acc,none": 0.8430034129692833,
|
||||
"inst_level_loose_acc_stderr,none": "N/A"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"ar_ifeval": []
|
||||
},
|
||||
"configs": {
|
||||
"ar_ifeval": {
|
||||
"task": "ar_ifeval",
|
||||
"dataset_path": "lm_eval/tasks/ar_ifeval/ar_ifeval.py",
|
||||
"dataset_name": "ar_ifeval",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"doc_to_text": "prompt",
|
||||
"doc_to_target": 0,
|
||||
"process_results": "def process_results(doc, results):\n\n response = results[0]\n out_strict = process_sample(doc, response, 'strict')\n out_loose = process_sample(doc, response, 'loose')\n\n\n return {\n \"prompt_level_strict_acc\": out_strict.follow_all_instructions,\n \"inst_level_strict_acc\": out_strict.follow_instruction_list,\n \"prompt_level_loose_acc\": out_loose.follow_all_instructions,\n \"inst_level_loose_acc\": out_loose.follow_instruction_list,\n }\n",
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "prompt_level_strict_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_strict_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "prompt_level_loose_acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "inst_level_loose_acc",
|
||||
"aggregation": "def agg_inst_level_acc(items):\n flat_items = [item for sublist in items for item in sublist]\n inst_level_acc = sum(flat_items) / len(flat_items)\n return inst_level_acc\n",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "generate_until",
|
||||
"generation_kwargs": {
|
||||
"until": [],
|
||||
"do_sample": false,
|
||||
"temperature": 0.0,
|
||||
"max_gen_toks": 1280
|
||||
},
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 4.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"ar_ifeval": 4.0
|
||||
},
|
||||
"n-shot": {
|
||||
"ar_ifeval": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"ar_ifeval": {
|
||||
"prompt_level_strict_acc": true,
|
||||
"inst_level_strict_acc": true,
|
||||
"prompt_level_loose_acc": true,
|
||||
"inst_level_loose_acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"ar_ifeval": {
|
||||
"original": 536,
|
||||
"effective": 536
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Small-Instruct-2409,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 22247282688,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8012044390bdc1c6d8ab162f5416220f43bf517b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739619509.695591,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"ar_ifeval": "04f79d36c1f856a7e0d2a4cc61bd745f1fdc633ccba1d094088f415f6471654b"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Small-Instruct-2409",
|
||||
"model_name_sanitized": "mistralai__Mistral-Small-Instruct-2409",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1461935.69256131,
|
||||
"end_time": 1471595.726226262,
|
||||
"total_evaluation_time_seconds": "9660.033664952032"
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"araMath_v3": {
|
||||
"alias": "araMath_v3",
|
||||
"acc,none": 0.4446280991735537,
|
||||
"acc_stderr,none": 0.020219570899233173,
|
||||
"acc_norm,none": 0.4446280991735537,
|
||||
"acc_norm_stderr,none": 0.020219570899233173
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araMath_v3": []
|
||||
},
|
||||
"configs": {
|
||||
"araMath_v3": {
|
||||
"task": "araMath_v3",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araMath_v3/araMath_v3.py",
|
||||
"dataset_name": "araMath_v3",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def remove_prefix(choice):\n prefixes = [\"(A)\", \"(B)\", \"(C)\", \"(D)\"]\n for prefix in prefixes:\n if choice.startswith(prefix + \" \"):\n return choice[len(prefix) + 1:] \n return choice \n\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"options\"])]\n )\n\n prompt = f\"\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": keys_en.index(doc[\"label\"]),\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u0645\u0646 \u0628\u064a\u0646 'A\u060c B\u060c C\u060c D' \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araMath_v3": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araMath_v3": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araMath_v3": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araMath_v3": {
|
||||
"original": 605,
|
||||
"effective": 605
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Small-Instruct-2409,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 22247282688,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8012044390bdc1c6d8ab162f5416220f43bf517b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739619380.3911364,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"araMath_v3": "8745758588621a4626b1d9dd0d3b59d90cdd106860afa2362c8e0cd8b77bd38a"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Small-Instruct-2409",
|
||||
"model_name_sanitized": "mistralai__Mistral-Small-Instruct-2409",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1461806.514496169,
|
||||
"end_time": 1461868.915775248,
|
||||
"total_evaluation_time_seconds": "62.40127907902934"
|
||||
}
|
||||
130
evaluations/ar/Mistral-Small-Instruct-2409/araPro_0_shot.json
Normal file
130
evaluations/ar/Mistral-Small-Instruct-2409/araPro_0_shot.json
Normal file
@@ -0,0 +1,130 @@
|
||||
{
|
||||
"results": {
|
||||
"araPro": {
|
||||
"alias": "araPro",
|
||||
"acc,none": 0.47730453909218157,
|
||||
"acc_stderr,none": 0.007063779668905028,
|
||||
"acc_norm,none": 0.47730453909218157,
|
||||
"acc_norm_stderr,none": 0.007063779668905028
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"araPro": []
|
||||
},
|
||||
"configs": {
|
||||
"araPro": {
|
||||
"task": "araPro",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/araPro/araPro.py",
|
||||
"dataset_name": "araPro",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.replace('.', '') if '.' in choice[:2] else choice\n \n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choice_num = ['choice1', 'choice2', 'choice3', 'choice4']\n choices = \"\".join(\n [f\"{key}. {remove_prefix(doc[choice_num[index]])}\\n\" for index, key in enumerate(keys)]\n )\n\n prompt = f\"\\n\\n\\n\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n #keys = [\"1\", \"2\", \"3\", \"4\"]\n keys = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys), \n \"choices\": keys,\n \"gold\": doc[\"answer\"]-1,\n } \n\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Question",
|
||||
"metadata": {
|
||||
"version": 2.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"araPro": 2.0
|
||||
},
|
||||
"n-shot": {
|
||||
"araPro": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"araPro": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"araPro": {
|
||||
"original": 5001,
|
||||
"effective": 5001
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Small-Instruct-2409,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 22247282688,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8012044390bdc1c6d8ab162f5416220f43bf517b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739617068.7956502,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"araPro": "7ae4350d99b977b9fbeea4421304e875323416c6b521abf45bd0eb9782f969b5"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Small-Instruct-2409",
|
||||
"model_name_sanitized": "mistralai__Mistral-Small-Instruct-2409",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1459495.184806751,
|
||||
"end_time": 1460928.893959109,
|
||||
"total_evaluation_time_seconds": "1433.7091523578856"
|
||||
}
|
||||
2051
evaluations/ar/Mistral-Small-Instruct-2409/arabicmmlu_0_shot.json
Normal file
2051
evaluations/ar/Mistral-Small-Instruct-2409/arabicmmlu_0_shot.json
Normal file
File diff suppressed because it is too large
Load Diff
126
evaluations/ar/Mistral-Small-Instruct-2409/etec_v2_0_shot.json
Normal file
126
evaluations/ar/Mistral-Small-Instruct-2409/etec_v2_0_shot.json
Normal file
@@ -0,0 +1,126 @@
|
||||
{
|
||||
"results": {
|
||||
"etec_v2": {
|
||||
"alias": "etec_v2",
|
||||
"acc,none": 0.40964493905670374,
|
||||
"acc_stderr,none": 0.011323732409166355,
|
||||
"acc_norm,none": 0.40964493905670374,
|
||||
"acc_norm_stderr,none": 0.011323732409166355
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"etec_v2": []
|
||||
},
|
||||
"configs": {
|
||||
"etec_v2": {
|
||||
"task": "etec_v2",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/etec_v2/etec.py",
|
||||
"dataset_name": "etec_v2",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc):\n def format_example(doc, keys):\n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices}\\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n print(doc[\"label\"])\n keys_ar = [\"\u0623\", \"\u0628\", \"\u062c\", \"\u062f\"]\n keys_en = [\"A\", \"B\", \"C\", \"D\"]\n out_doc = {\n \"query\": format_example(doc, keys_en),\n \"choices\": keys_en,\n \"gold\": int(doc[\"label\"])-1,\n }\n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "\u0641\u064a\u0645\u0627 \u064a\u0644\u064a \u0623\u0633\u0626\u0644\u0629 \u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631 \u0645\u0646 \u0645\u062a\u0639\u062f\u062f (\u0645\u0639 \u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a) \u0645\u0646 \u0641\u0636\u0644\u0643 \u0627\u062e\u062a\u0631 \u0625\u062c\u0627\u0628\u0629 \u0648\u0627\u062d\u062f\u0629 \u062f\u0648\u0646 \u0634\u0631\u062d\n ",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"etec_v2": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"etec_v2": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"etec_v2": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"etec_v2": {
|
||||
"original": 1887,
|
||||
"effective": 1887
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Small-Instruct-2409,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 22247282688,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8012044390bdc1c6d8ab162f5416220f43bf517b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618555.909214,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"etec_v2": "e77e8618d461a8245f026c3013170019168ca5e9431e9d9d1c176a55cdcf1552"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Small-Instruct-2409",
|
||||
"model_name_sanitized": "mistralai__Mistral-Small-Instruct-2409",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1460982.144801136,
|
||||
"end_time": 1461066.334385176,
|
||||
"total_evaluation_time_seconds": "84.18958403985016"
|
||||
}
|
||||
125
evaluations/ar/Mistral-Small-Instruct-2409/exams_ar_5_shot.json
Normal file
125
evaluations/ar/Mistral-Small-Instruct-2409/exams_ar_5_shot.json
Normal file
@@ -0,0 +1,125 @@
|
||||
{
|
||||
"results": {
|
||||
"exams_ar": {
|
||||
"alias": "exams_ar",
|
||||
"acc,none": 0.38733705772811916,
|
||||
"acc_stderr,none": 0.021041317803855382,
|
||||
"acc_norm,none": 0.38733705772811916,
|
||||
"acc_norm_stderr,none": 0.021041317803855382
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"exams_ar": []
|
||||
},
|
||||
"configs": {
|
||||
"exams_ar": {
|
||||
"task": "exams_ar",
|
||||
"tag": [
|
||||
"multiple_choice"
|
||||
],
|
||||
"dataset_path": "lm_eval/tasks/exams_ar",
|
||||
"dataset_name": "exams_ar",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n\n def _process_docs(doc):\n def format_example(doc, keys):\n \"\"\"\n <prompt>\n \u0633\u0624\u0627\u0644:\n A. <choice1>\n B. <choice2>\n C. <choice3>\n D. <choice4>\n \u0627\u062c\u0627\u0628\u0629:\n \"\"\"\n \n question = doc[\"question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt = f\"\u0627\u0644\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n def _format_subject(subject):\n arabic_words = subtasks_ar[subtasks.index(subject)]\n return arabic_words\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n \n subject = doc['id'].split(\"-\")[0]\n description = f\"\ufed2\ufef4\ufee3\ufe8d \ufef2\ufee0\ufef3 \ufe84\ufeb4\ufe8c\ufedf\ufe93 \ufe8d\ufefc\ufea8\ufe98\ufef3\ufe8d\ufead \ufee2\ufee7 \ufee2\ufe98\ufecb\ufea9\ufea9 (\ufee2\ufecb \ufe8d\ufefa\ufe9f\ufe8e\ufe91\ufe8e\ufe97) \ufea1\ufeee\ufedf {_format_subject(subject)} \\n\" #\ufee2\ufee7 \ufed2\ufec0\ufee0\ufedb \ufe8e\ufea8\ufe97\ufead \ufe88\ufe9f\ufe8e\ufe91\ufe93 \ufeed\ufe8e\ufea3\ufea9\ufe93 \ufee2\ufee7 \ufe90\ufef4\ufee7 'A\u060c B\u060c C\u060c D' \ufea9\ufeee\ufee7 \ufeb5\ufeae\ufea3\\n\"\n\n out_doc = {\n \"idx\": doc[\"idx\"],\n \"id\": doc[\"id\"],\n 'dsecription': description,\n \"query\": format_example(doc, keys), # \"Question: \" + doc[\"question\"]['stem'] + \"\\nAnswer:\",\n \"choices\": keys,\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"label\"]),\n }\n return out_doc\n\n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "choices",
|
||||
"description": "description",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 5,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"exams_ar": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"exams_ar": 5
|
||||
},
|
||||
"higher_is_better": {
|
||||
"exams_ar": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"exams_ar": {
|
||||
"original": 537,
|
||||
"effective": 537
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Small-Instruct-2409,trust_remote_code=True,cache_dir=/tmp,parallelize=False",
|
||||
"model_num_parameters": 22247282688,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8012044390bdc1c6d8ab162f5416220f43bf517b",
|
||||
"batch_size": "auto",
|
||||
"batch_sizes": [
|
||||
32
|
||||
],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "5e10e017",
|
||||
"date": 1736970120.592902,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100 80GB PCIe\nGPU 1: NVIDIA A100 80GB PCIe\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 48\nOn-line CPU(s) list: 0-47\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V13 64-Core Processor\nCPU family: 25\nModel: 1\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 1\nStepping: 1\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru arat umip vaes vpclmulqdq rdpid fsrm\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 1.5 MiB (48 instances)\nL1i cache: 1.5 MiB (48 instances)\nL2 cache: 24 MiB (48 instances)\nL3 cache: 192 MiB (6 instances)\nNUMA node(s): 2\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Not affected\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Vulnerable\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.0",
|
||||
"upper_git_hash": "2e5cd5395faf76fea1afc96dd0f7161a9d3aa145",
|
||||
"tokenizer_pad_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Small-Instruct-2409",
|
||||
"model_name_sanitized": "mistralai__Mistral-Small-Instruct-2409",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 11602.469319334,
|
||||
"end_time": 12824.398025607,
|
||||
"total_evaluation_time_seconds": "1221.928706273"
|
||||
}
|
||||
543
evaluations/ar/Mistral-Small-Instruct-2409/gat_0_shot.json
Normal file
543
evaluations/ar/Mistral-Small-Instruct-2409/gat_0_shot.json
Normal file
@@ -0,0 +1,543 @@
|
||||
{
|
||||
"results": {
|
||||
"gat": {
|
||||
"acc,none": 0.28816004013545715,
|
||||
"acc_stderr,none": 0.003569513517176158,
|
||||
"alias": "gat"
|
||||
},
|
||||
"gat_algebra": {
|
||||
"alias": " - gat_algebra",
|
||||
"acc,none": 0.2593692022263451,
|
||||
"acc_stderr,none": 0.008444254056089201
|
||||
},
|
||||
"gat_analogy": {
|
||||
"alias": " - gat_analogy",
|
||||
"acc,none": 0.26520947176684884,
|
||||
"acc_stderr,none": 0.008427218151737142
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"alias": " - gat_arithmetic",
|
||||
"acc,none": 0.27972027972027974,
|
||||
"acc_stderr,none": 0.008612865946138122
|
||||
},
|
||||
"gat_association": {
|
||||
"alias": " - gat_association",
|
||||
"acc,none": 0.27177033492822966,
|
||||
"acc_stderr,none": 0.01376844704683984
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"alias": " - gat_comparisons",
|
||||
"acc,none": 0.24508196721311476,
|
||||
"acc_stderr,none": 0.012319801935808129
|
||||
},
|
||||
"gat_completion": {
|
||||
"alias": " - gat_completion",
|
||||
"acc,none": 0.2983471074380165,
|
||||
"acc_stderr,none": 0.013158576974400435
|
||||
},
|
||||
"gat_contextual": {
|
||||
"alias": " - gat_contextual",
|
||||
"acc,none": 0.25766871165644173,
|
||||
"acc_stderr,none": 0.012115951274247083
|
||||
},
|
||||
"gat_geometry": {
|
||||
"alias": " - gat_geometry",
|
||||
"acc,none": 0.2958904109589041,
|
||||
"acc_stderr,none": 0.023924060011244693
|
||||
},
|
||||
"gat_reading": {
|
||||
"alias": " - gat_reading",
|
||||
"acc,none": 0.3856332703213611,
|
||||
"acc_stderr,none": 0.009466084278454174
|
||||
}
|
||||
},
|
||||
"groups": {
|
||||
"gat": {
|
||||
"acc,none": 0.28816004013545715,
|
||||
"acc_stderr,none": 0.003569513517176158,
|
||||
"alias": "gat"
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"gat": [
|
||||
"gat_analogy",
|
||||
"gat_association",
|
||||
"gat_completion",
|
||||
"gat_reading",
|
||||
"gat_algebra",
|
||||
"gat_arithmetic",
|
||||
"gat_comparisons",
|
||||
"gat_contextual",
|
||||
"gat_geometry"
|
||||
]
|
||||
},
|
||||
"configs": {
|
||||
"gat_algebra": {
|
||||
"task": "gat_algebra",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "algebra",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_analogy": {
|
||||
"task": "gat_analogy",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "analogy",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"task": "gat_arithmetic",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "arithmetic",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_association": {
|
||||
"task": "gat_association",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "association",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"task": "gat_comparisons",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "comparisons",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_completion": {
|
||||
"task": "gat_completion",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "completion",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_contextual": {
|
||||
"task": "gat_contextual",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "contextual",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "\u0627\u0648\u062c\u062f \u0627\u0644\u062e\u0637\u0623 \u0627\u0644\u0633\u064a\u0627\u0642\u064a \u0641\u064a \u0627\u0644\u0639\u0628\u0627\u0631\u0629 \u0627\u0644\u062a\u0627\u0644\u064a\u0629 \u0645\u0646 \u0628\u064a\u0646 \u0627\u0644\u062e\u064a\u0627\u0631\u0627\u062a:",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_geometry": {
|
||||
"task": "gat_geometry",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "geometry",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
},
|
||||
"gat_reading": {
|
||||
"task": "gat_reading",
|
||||
"dataset_path": "lm_eval/tasks/gat/gat_data/gat.py",
|
||||
"dataset_name": "reading",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n # def _process_doc(doc):\n \n # subject = doc['id'].split(\"-\")[0]\n # subject_ar = subtasks_ar[subtasks.index(subject)]\n # out_doc = {**doc, 'subject_ar': subject_ar}\n # print(subject_ar)\n # print(out_doc)\n # return out_doc\n\n return dataset\n",
|
||||
"doc_to_text": "{{question}}\n\u0623. {{choices[0]}}\n\u0628. {{choices[1]}}\n\u062c. {{choices[2]}}\n\u062f. {{choices[3]}}\n\u0627\u0644\u0627\u062c\u0627\u0628\u0629:",
|
||||
"doc_to_target": "{{label}}",
|
||||
"doc_to_choice": [
|
||||
"\u0623",
|
||||
"\u0628",
|
||||
"\u062c",
|
||||
"\u062f"
|
||||
],
|
||||
"description": "",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": false,
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"gat": 0,
|
||||
"gat_algebra": 0.0,
|
||||
"gat_analogy": 0.0,
|
||||
"gat_arithmetic": 0.0,
|
||||
"gat_association": 0.0,
|
||||
"gat_comparisons": 0.0,
|
||||
"gat_completion": 0.0,
|
||||
"gat_contextual": 0.0,
|
||||
"gat_geometry": 0.0,
|
||||
"gat_reading": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"gat_algebra": 0,
|
||||
"gat_analogy": 0,
|
||||
"gat_arithmetic": 0,
|
||||
"gat_association": 0,
|
||||
"gat_comparisons": 0,
|
||||
"gat_completion": 0,
|
||||
"gat_contextual": 0,
|
||||
"gat_geometry": 0,
|
||||
"gat_reading": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"gat": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_algebra": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_analogy": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_association": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_completion": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_contextual": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_geometry": {
|
||||
"acc": true
|
||||
},
|
||||
"gat_reading": {
|
||||
"acc": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"gat_analogy": {
|
||||
"original": 2745,
|
||||
"effective": 2745
|
||||
},
|
||||
"gat_association": {
|
||||
"original": 1045,
|
||||
"effective": 1045
|
||||
},
|
||||
"gat_completion": {
|
||||
"original": 1210,
|
||||
"effective": 1210
|
||||
},
|
||||
"gat_reading": {
|
||||
"original": 2645,
|
||||
"effective": 2645
|
||||
},
|
||||
"gat_algebra": {
|
||||
"original": 2695,
|
||||
"effective": 2695
|
||||
},
|
||||
"gat_arithmetic": {
|
||||
"original": 2717,
|
||||
"effective": 2717
|
||||
},
|
||||
"gat_comparisons": {
|
||||
"original": 1220,
|
||||
"effective": 1220
|
||||
},
|
||||
"gat_contextual": {
|
||||
"original": 1304,
|
||||
"effective": 1304
|
||||
},
|
||||
"gat_geometry": {
|
||||
"original": 365,
|
||||
"effective": 365
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "parallelize=False,pretrained=mistralai/Mistral-Small-Instruct-2409,trust_remote_code=True,mm=False",
|
||||
"model_num_parameters": 22247282688,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8012044390bdc1c6d8ab162f5416220f43bf517b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "3127d82f",
|
||||
"date": 1730953375.739498,
|
||||
"pretty_env_info": "PyTorch version: 2.1.0a0+29c30b1\nIs debug build: False\nCUDA used to build PyTorch: 12.2\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1073-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 550.90.07\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.87\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Reg file data sampling: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.22.2\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.1.0a0+29c30b1\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.16.0a0\n[pip3] triton==2.0.0.dev20221202\n[conda] Could not collect",
|
||||
"transformers_version": "4.46.2",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Small-Instruct-2409",
|
||||
"model_name_sanitized": "mistralai__Mistral-Small-Instruct-2409",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": null,
|
||||
"chat_template_sha": null,
|
||||
"start_time": 25487.850067782,
|
||||
"end_time": 28449.915428973,
|
||||
"total_evaluation_time_seconds": "2962.0653611909984"
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
{
|
||||
"results": {
|
||||
"moe_ien_mcq": {
|
||||
"alias": "moe_ien_mcq",
|
||||
"acc,none": 0.6064064064064064,
|
||||
"acc_stderr,none": 0.004888154163260656,
|
||||
"acc_norm,none": 0.6064064064064064,
|
||||
"acc_norm_stderr,none": 0.004888154163260656
|
||||
}
|
||||
},
|
||||
"group_subtasks": {
|
||||
"moe_ien_mcq": []
|
||||
},
|
||||
"configs": {
|
||||
"moe_ien_mcq": {
|
||||
"task": "moe_ien_mcq",
|
||||
"dataset_path": "lm_eval/tasks/moe_ien_mcq/ien_moe_mcq.py",
|
||||
"dataset_name": "moe_ien_mcq",
|
||||
"dataset_kwargs": {
|
||||
"trust_remote_code": true
|
||||
},
|
||||
"validation_split": "validation",
|
||||
"test_split": "test",
|
||||
"fewshot_split": "validation",
|
||||
"process_docs": "def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:\n def _process_docs(doc): \n def remove_prefix(choice):\n return choice.split(\". \", 1)[1] if \". \" in choice else choice\n\n def format_example(doc, keys):\n question = doc[\"Question\"].strip()\n \n choices = \"\".join(\n [f\"{key}. {remove_prefix(choice)}\\n\" for key, choice in zip(keys, doc[\"Choices\"])]\n \n )\n prompt = f\"\\n\\n\u0633\u0624\u0627\u0644: {question}\\n{choices} \\n\u0627\u062c\u0627\u0628\u0629:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"][0:len(doc[\"Choices\"])]\n out_doc = {\n \"Query\": format_example(doc, keys), \n \"Choices\": keys,\n \"gold\": int(doc[\"Answer\"])-1, ## \n } \n return out_doc\n \n return dataset.map(_process_docs)\n",
|
||||
"doc_to_text": "Query",
|
||||
"doc_to_target": "gold",
|
||||
"doc_to_choice": "{{Choices}}",
|
||||
"description": "\u0641\u064a\u0645\u0627\u202f\u064a\u0644\u064a\u202f\u0623\u0633\u0626\u0644\u0629\u202f\u0627\u0644\u0627\u062e\u062a\u064a\u0627\u0631\u202f\u0645\u0646\u202f\u0645\u062a\u0639\u062f\u062f\u202f(\u0645\u0639\u202f\u0627\u0644\u0625\u062c\u0627\u0628\u0627\u062a)\u202f\u0641\u064a\u202f{{Subject}}",
|
||||
"target_delimiter": " ",
|
||||
"fewshot_delimiter": "\n\n",
|
||||
"fewshot_config": {
|
||||
"sampler": "balanced_cat"
|
||||
},
|
||||
"num_fewshot": 0,
|
||||
"metric_list": [
|
||||
{
|
||||
"metric": "acc",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
},
|
||||
{
|
||||
"metric": "acc_norm",
|
||||
"aggregation": "mean",
|
||||
"higher_is_better": true
|
||||
}
|
||||
],
|
||||
"output_type": "multiple_choice",
|
||||
"repeats": 1,
|
||||
"should_decontaminate": true,
|
||||
"doc_to_decontamination_query": "Query",
|
||||
"metadata": {
|
||||
"version": 0.0
|
||||
}
|
||||
}
|
||||
},
|
||||
"versions": {
|
||||
"moe_ien_mcq": 0.0
|
||||
},
|
||||
"n-shot": {
|
||||
"moe_ien_mcq": 0
|
||||
},
|
||||
"higher_is_better": {
|
||||
"moe_ien_mcq": {
|
||||
"acc": true,
|
||||
"acc_norm": true
|
||||
}
|
||||
},
|
||||
"n-samples": {
|
||||
"moe_ien_mcq": {
|
||||
"original": 9990,
|
||||
"effective": 9990
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"model": "hf",
|
||||
"model_args": "pretrained=mistralai/Mistral-Small-Instruct-2409,trust_remote_code=True,cache_dir=/tmp,parallelize=True",
|
||||
"model_num_parameters": 22247282688,
|
||||
"model_dtype": "torch.bfloat16",
|
||||
"model_revision": "main",
|
||||
"model_sha": "8012044390bdc1c6d8ab162f5416220f43bf517b",
|
||||
"batch_size": 1,
|
||||
"batch_sizes": [],
|
||||
"device": null,
|
||||
"use_cache": null,
|
||||
"limit": null,
|
||||
"bootstrap_iters": 100000,
|
||||
"gen_kwargs": null,
|
||||
"random_seed": 0,
|
||||
"numpy_seed": 1234,
|
||||
"torch_seed": 1234,
|
||||
"fewshot_seed": 1234
|
||||
},
|
||||
"git_hash": "b955b2950",
|
||||
"date": 1739618710.0175338,
|
||||
"pretty_env_info": "PyTorch version: 2.4.0+cu121\nIs debug build: False\nCUDA used to build PyTorch: 12.1\nROCM used to build PyTorch: N/A\n\nOS: Ubuntu 22.04.3 LTS (x86_64)\nGCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\nClang version: Could not collect\nCMake version: version 3.27.1\nLibc version: glibc-2.35\n\nPython version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)\nPython platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.35\nIs CUDA available: True\nCUDA runtime version: 12.2.128\nCUDA_MODULE_LOADING set to: LAZY\nGPU models and configuration: \nGPU 0: NVIDIA A100-SXM4-80GB\nGPU 1: NVIDIA A100-SXM4-80GB\nGPU 2: NVIDIA A100-SXM4-80GB\nGPU 3: NVIDIA A100-SXM4-80GB\nGPU 4: NVIDIA A100-SXM4-80GB\nGPU 5: NVIDIA A100-SXM4-80GB\nGPU 6: NVIDIA A100-SXM4-80GB\nGPU 7: NVIDIA A100-SXM4-80GB\n\nNvidia driver version: 535.161.08\ncuDNN version: Probably one of the following:\n/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.4\n/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.4\nHIP runtime version: N/A\nMIOpen runtime version: N/A\nIs XNNPACK available: True\n\nCPU:\nArchitecture: x86_64\nCPU op-mode(s): 32-bit, 64-bit\nAddress sizes: 48 bits physical, 48 bits virtual\nByte Order: Little Endian\nCPU(s): 96\nOn-line CPU(s) list: 0-95\nVendor ID: AuthenticAMD\nModel name: AMD EPYC 7V12 64-Core Processor\nCPU family: 23\nModel: 49\nThread(s) per core: 1\nCore(s) per socket: 48\nSocket(s): 2\nStepping: 0\nBogoMIPS: 4890.89\nFlags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru arat umip rdpid\nHypervisor vendor: Microsoft\nVirtualization type: full\nL1d cache: 3 MiB (96 instances)\nL1i cache: 3 MiB (96 instances)\nL2 cache: 48 MiB (96 instances)\nL3 cache: 384 MiB (24 instances)\nNUMA node(s): 4\nNUMA node0 CPU(s): 0-23\nNUMA node1 CPU(s): 24-47\nNUMA node2 CPU(s): 48-71\nNUMA node3 CPU(s): 72-95\nVulnerability Gather data sampling: Not affected\nVulnerability Itlb multihit: Not affected\nVulnerability L1tf: Not affected\nVulnerability Mds: Not affected\nVulnerability Meltdown: Not affected\nVulnerability Mmio stale data: Not affected\nVulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled\nVulnerability Spec rstack overflow: Mitigation; safe RET, no microcode\nVulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp\nVulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization\nVulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected\nVulnerability Srbds: Not affected\nVulnerability Tsx async abort: Not affected\n\nVersions of relevant libraries:\n[pip3] numpy==1.26.4\n[pip3] onnx==1.14.0\n[pip3] pytorch-lightning==2.0.7\n[pip3] pytorch-quantization==2.1.2\n[pip3] torch==2.4.0\n[pip3] torch-tensorrt==2.0.0.dev0\n[pip3] torchaudio==2.1.0\n[pip3] torchdata==0.7.0a0\n[pip3] torchmetrics==1.2.0\n[pip3] torchvision==0.19.0\n[pip3] triton==3.0.0\n[conda] Could not collect",
|
||||
"transformers_version": "4.48.3",
|
||||
"upper_git_hash": null,
|
||||
"tokenizer_pad_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_eos_token": [
|
||||
"</s>",
|
||||
"2"
|
||||
],
|
||||
"tokenizer_bos_token": [
|
||||
"<s>",
|
||||
"1"
|
||||
],
|
||||
"eot_token_id": 2,
|
||||
"max_length": 32768,
|
||||
"task_hashes": {
|
||||
"moe_ien_mcq": "c2a20c63c9048b05e61ad12ca87f357a5e71433c713f9a22b7d537ed6bc7421d"
|
||||
},
|
||||
"model_source": "hf",
|
||||
"model_name": "mistralai/Mistral-Small-Instruct-2409",
|
||||
"model_name_sanitized": "mistralai__Mistral-Small-Instruct-2409",
|
||||
"system_instruction": null,
|
||||
"system_instruction_sha": null,
|
||||
"fewshot_as_multiturn": false,
|
||||
"chat_template": "{%- if messages[0][\"role\"] == \"system\" %}\n {%- set system_message = messages[0][\"content\"] %}\n {%- set loop_messages = messages[1:] %}\n{%- else %}\n {%- set loop_messages = messages %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n{%- set user_messages = loop_messages | selectattr(\"role\", \"equalto\", \"user\") | list %}\n\n{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}\n{%- set ns = namespace() %}\n{%- set ns.index = 0 %}\n{%- for message in loop_messages %}\n {%- if not (message.role == \"tool\" or message.role == \"tool_results\" or (message.tool_calls is defined and message.tool_calls is not none)) %}\n {%- if (message[\"role\"] == \"user\") != (ns.index % 2 == 0) %}\n {{- raise_exception(\"After the optional system message, conversation roles must alternate user/assistant/user/assistant/...\") }}\n {%- endif %}\n {%- set ns.index = ns.index + 1 %}\n {%- endif %}\n{%- endfor %}\n\n{{- bos_token }}\n{%- for message in loop_messages %}\n {%- if message[\"role\"] == \"user\" %}\n {%- if tools is not none and (message == user_messages[-1]) %}\n {{- \"[AVAILABLE_TOOLS] [\" }}\n {%- for tool in tools %}\n {%- set tool = tool.function %}\n {{- '{\"type\": \"function\", \"function\": {' }}\n {%- for key, val in tool.items() if key != \"return\" %}\n {%- if val is string %}\n {{- '\"' + key + '\": \"' + val + '\"' }}\n {%- else %}\n {{- '\"' + key + '\": ' + val|tojson }}\n {%- endif %}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \"}}\" }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" }}\n {%- endif %}\n {%- endfor %}\n {{- \"[/AVAILABLE_TOOLS]\" }}\n {%- endif %}\n {%- if loop.last and system_message is defined %}\n {{- \"[INST] \" + system_message + \"\\n\\n\" + message[\"content\"] + \"[/INST]\" }}\n {%- else %}\n {{- \"[INST] \" + message[\"content\"] + \"[/INST]\" }}\n {%- endif %}\n {%- elif message.tool_calls is defined and message.tool_calls is not none %}\n {{- \"[TOOL_CALLS] [\" }}\n {%- for tool_call in message.tool_calls %}\n {%- set out = tool_call.function|tojson %}\n {{- out[:-1] }}\n {%- if not tool_call.id is defined or tool_call.id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- ', \"id\": \"' + tool_call.id + '\"}' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- else %}\n {{- \"]\" + eos_token }}\n {%- endif %}\n {%- endfor %}\n {%- elif message[\"role\"] == \"assistant\" %}\n {{- \" \" + message[\"content\"]|trim + eos_token}}\n {%- elif message[\"role\"] == \"tool_results\" or message[\"role\"] == \"tool\" %}\n {%- if message.content is defined and message.content.content is defined %}\n {%- set content = message.content.content %}\n {%- else %}\n {%- set content = message.content %}\n {%- endif %}\n {{- '[TOOL_RESULTS] {\"content\": ' + content|string + \", \" }}\n {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}\n {{- raise_exception(\"Tool call IDs should be alphanumeric strings with length 9!\") }}\n {%- endif %}\n {{- '\"call_id\": \"' + message.tool_call_id + '\"}[/TOOL_RESULTS]' }}\n {%- else %}\n {{- raise_exception(\"Only user and assistant roles are supported, with the exception of an initial optional system message!\") }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"chat_template_sha": "e16746b40344d6c5b5265988e0328a0bf7277be86f1c335156eae07e29c82826",
|
||||
"start_time": 1461136.332656852,
|
||||
"end_time": 1461391.40888449,
|
||||
"total_evaluation_time_seconds": "255.07622763793916"
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user