nvidia/Minitron-8B-Base
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

Add evaluation preview

Browse Source
This commit is contained in:
Saurav Muralidharan
2024-07-23 10:36:45 -07:00
parent 66505a7a72
commit 019a9d026e
1 changed files with 23 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
README.md
View File

@@ -53,6 +53,29 @@ print(output_text)
Minitron is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf). Minitron is released under the [NVIDIA Open Model License Agreement](https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf).
## Evaluation Results
*5-shot performance.* Language Understanding evaluated using [Massive Multitask Language Understanding](https://arxiv.org/abs/2009.03300):
| Average |
| :---- |
| 63.8 |
*Zero-shot performance.* Evaluated using select datasets from the [LM Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) with additions:
HellaSwag | Winogrande | GSM8K| ARC-C | XLSum |
| :------------- | :------------- | :------------- | :------------- | :------------- |
| 80.7 | 79.0 | 51.3 | 52.6 | 31.2
*Code generation performance*. Evaluated using [HumanEval](https://github.com/openai/human-eval):
| p@1, 0-Shot |
| :------------- |
| 31.6 |
Please refer to our [paper](https://arxiv.org/abs/2407.14679) for the full set of results.
## Citation ## Citation
If you find our work helpful, please consider citing our paper: If you find our work helpful, please consider citing our paper: