207 lines
7.3 KiB
Markdown
207 lines
7.3 KiB
Markdown
---
|
|
license: other
|
|
library_name: transformers
|
|
base_model: meta-llama/Meta-Llama-3-8B
|
|
datasets:
|
|
- mlabonne/orpo-dpo-mix-40k
|
|
- Open-Orca/SlimOrca-Dedup
|
|
- jondurbin/airoboros-3.2
|
|
- microsoft/orca-math-word-problems-200k
|
|
- m-a-p/Code-Feedback
|
|
- MaziyarPanahi/WizardLM_evol_instruct_V2_196k
|
|
model-index:
|
|
- name: llama-3-neural-chat-v1-8b
|
|
results:
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: AI2 Reasoning Challenge (25-Shot)
|
|
type: ai2_arc
|
|
config: ARC-Challenge
|
|
split: test
|
|
args:
|
|
num_few_shot: 25
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 60.84
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: HellaSwag (10-Shot)
|
|
type: hellaswag
|
|
split: validation
|
|
args:
|
|
num_few_shot: 10
|
|
metrics:
|
|
- type: acc_norm
|
|
value: 84.13
|
|
name: normalized accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: MMLU (5-Shot)
|
|
type: cais/mmlu
|
|
config: all
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 64.69
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: TruthfulQA (0-shot)
|
|
type: truthful_qa
|
|
config: multiple_choice
|
|
split: validation
|
|
args:
|
|
num_few_shot: 0
|
|
metrics:
|
|
- type: mc2
|
|
value: 56.34
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: Winogrande (5-shot)
|
|
type: winogrande
|
|
config: winogrande_xl
|
|
split: validation
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 78.22
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
|
|
name: Open LLM Leaderboard
|
|
- task:
|
|
type: text-generation
|
|
name: Text Generation
|
|
dataset:
|
|
name: GSM8k (5-shot)
|
|
type: gsm8k
|
|
config: main
|
|
split: test
|
|
args:
|
|
num_few_shot: 5
|
|
metrics:
|
|
- type: acc
|
|
value: 54.81
|
|
name: accuracy
|
|
source:
|
|
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
|
|
name: Open LLM Leaderboard
|
|
---
|
|
|
|
# llama-3-neural-chat-v1-8b
|
|
|
|
<!-- Provide a quick summary of what the model is/does. -->
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Model Details
|
|
|
|
### Model Description
|
|
|
|
<!-- Provide a longer summary of what this model is. -->
|
|
|
|
I fine-tuned llama-3 8B on an approach similar to Intel's neural chat language model. I have slightly modified the data sources so it is stronger in coding, math, and writing. I use both SFT and DPO.
|
|
|
|
- **Developed by:** Locutusque
|
|
- **Model type:** Built with Meta Llama 3
|
|
- **Language(s) (NLP):** Many?
|
|
- **License:** Llama 3 license https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/LICENSE
|
|
|
|
## Quants
|
|
|
|
### EXL2 [@bartowski](https://huggingface.co/bartowski/)
|
|
|
|
- https://huggingface.co/bartowski/llama-3-neural-chat-v1-8b-exl2
|
|
|
|
### GGUF [@bartowski](https://huggingface.co/bartowski/)
|
|
|
|
- https://huggingface.co/bartowski/llama-3-neural-chat-v1-8b-GGUF
|
|
|
|
## Uses
|
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
|
|
|
This model has great performance in writing and coding.
|
|
|
|
## Training Data
|
|
- Open-Orca/SlimOrca-Dedup
|
|
- jondurbin/airoboros-3.2
|
|
- microsoft/orca-math-word-problems-200k
|
|
- m-a-p/Code-Feedback
|
|
- MaziyarPanahi/WizardLM_evol_instruct_V2_196k
|
|
- mlabonne/orpo-dpo-mix-40k
|
|
|
|
### Direct Use
|
|
|
|
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
|
|
|
Conversational AI.
|
|
|
|
## Evaluations
|
|
|
|
| Tasks |Version| Filter |n-shot| Metric |Value | |Stderr|
|
|
|---------------------------------|-------|----------------|-----:|-----------|-----:|---|-----:|
|
|
|truthfulqa_mc2 | 2|none | 0|acc |0.5627|± |0.0154|
|
|
|gsm8k | 3|strict-match | 5|exact_match|0.5481|± |0.0137|
|
|
| | |flexible-extract| 5|exact_match|0.5557|± |0.0137|
|
|
|agieval_nous |N/A |none | 0|acc |0.3763|± |0.0093|
|
|
| | |none | 0|acc_norm |0.3665|± |0.0093|
|
|
| - agieval_aqua_rat | 1|none | 0|acc |0.2087|± |0.0255|
|
|
| | |none | 0|acc_norm |0.2047|± |0.0254|
|
|
| - agieval_logiqa_en | 1|none | 0|acc |0.3456|± |0.0187|
|
|
| | |none | 0|acc_norm |0.3594|± |0.0188|
|
|
| - agieval_lsat_ar | 1|none | 0|acc |0.1826|± |0.0255|
|
|
| | |none | 0|acc_norm |0.1783|± |0.0253|
|
|
| - agieval_lsat_lr | 1|none | 0|acc |0.3549|± |0.0212|
|
|
| | |none | 0|acc_norm |0.3451|± |0.0211|
|
|
| - agieval_lsat_rc | 1|none | 0|acc |0.5242|± |0.0305|
|
|
| | |none | 0|acc_norm |0.5130|± |0.0305|
|
|
| - agieval_sat_en | 1|none | 0|acc |0.6650|± |0.0330|
|
|
| | |none | 0|acc_norm |0.6505|± |0.0333|
|
|
| - agieval_sat_en_without_passage| 1|none | 0|acc |0.4175|± |0.0344|
|
|
| | |none | 0|acc_norm |0.3738|± |0.0338|
|
|
| - agieval_sat_math | 1|none | 0|acc |0.4227|± |0.0334|
|
|
| | |none | 0|acc_norm |0.3682|± |0.0326|
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Locutusque__llama-3-neural-chat-v1-8b)
|
|
|
|
| Metric |Value|
|
|
|---------------------------------|----:|
|
|
|Avg. |66.50|
|
|
|AI2 Reasoning Challenge (25-Shot)|60.84|
|
|
|HellaSwag (10-Shot) |84.13|
|
|
|MMLU (5-Shot) |64.69|
|
|
|TruthfulQA (0-shot) |56.34|
|
|
|Winogrande (5-shot) |78.22|
|
|
|GSM8k (5-shot) |54.81|
|
|
|