初始化项目，由ModelHub XC社区提供模型
Model: sethuiyer/SynthIQ-7b Source: Original Platform
This commit is contained in:
ModelHub XC
230
README.md
Normal file
230
README.md
Normal file
@@ -0,0 +1,230 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: llama2
|
||||
library_name: transformers
|
||||
tags:
|
||||
- mistral
|
||||
- merge
|
||||
datasets:
|
||||
- stingning/ultrachat
|
||||
- garage-bAInd/Open-Platypus
|
||||
- Open-Orca/OpenOrca
|
||||
- TIGER-Lab/MathInstruct
|
||||
- OpenAssistant/oasst_top1_2023-08-25
|
||||
- teknium/openhermes
|
||||
- meta-math/MetaMathQA
|
||||
- Open-Orca/SlimOrca
|
||||
pipeline_tag: text-generation
|
||||
base_model:
|
||||
- Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp
|
||||
- ehartford/dolphin-2.1-mistral-7b
|
||||
- Open-Orca/Mistral-7B-OpenOrca
|
||||
- bhenrym14/mistral-7b-platypus-fp16
|
||||
- ehartford/samantha-1.2-mistral-7b
|
||||
- iteknium/CollectiveCognition-v1.1-Mistral-7B
|
||||
- HuggingFaceH4/zephyr-7b-alpha
|
||||
model-index:
|
||||
- name: sethuiyer/SynthIQ-7b
|
||||
results:
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: AI2 Reasoning Challenge (25-Shot)
|
||||
type: ai2_arc
|
||||
config: ARC-Challenge
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 25
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 65.87
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: HellaSwag (10-Shot)
|
||||
type: hellaswag
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 10
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 85.82
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: MMLU (5-Shot)
|
||||
type: cais/mmlu
|
||||
config: all
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 64.75
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: TruthfulQA (0-shot)
|
||||
type: truthful_qa
|
||||
config: multiple_choice
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 0
|
||||
metrics:
|
||||
- type: mc2
|
||||
value: 57
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: Winogrande (5-shot)
|
||||
type: winogrande
|
||||
config: winogrande_xl
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 78.69
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: GSM8k (5-shot)
|
||||
type: gsm8k
|
||||
config: main
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 64.06
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/SynthIQ-7b
|
||||
name: Open LLM Leaderboard
|
||||
---
|
||||
|
||||
<p align="center">
|
||||
<img src="https://codeberg.org/aninokuma/DeydooAssistant/raw/branch/main/logo.webp" height="256px" alt="SynthIQ">
|
||||
</p>
|
||||
|
||||
# SynthIQ
|
||||
|
||||
This is SynthIQ, rated **92.23/100** by GPT-4 across varied complex prompts. I used [mergekit](https://github.com/cg123/mergekit) to merge models.
|
||||
|
||||
| Benchmark Name | Score |
|
||||
| ---------------- | ----- |
|
||||
| ARC | 65.87 |
|
||||
| HellaSwag | 85.82 |
|
||||
| MMLU | 64.75 |
|
||||
| TruthfulQA | 57.00 |
|
||||
| Winogrande | 78.69 |
|
||||
| GSM8K | 64.06 |
|
||||
| AGIEval | 42.67 |
|
||||
| GPT4All | 73.71 |
|
||||
| Bigbench | 44.59 |
|
||||
|
||||
|
||||
## Update - 19/01/2024
|
||||
Tested to work well with autogen and CrewAI
|
||||
|
||||
GGUF Files
|
||||
|
||||
[Q4_K_M](https://huggingface.co/sethuiyer/SynthIQ_GGUF/blob/main/synthiq.Q4_K_M.gguf) - medium, balanced quality - recommended
|
||||
|
||||
[Q_6_K](https://huggingface.co/sethuiyer/SynthIQ_GGUF/blob/main/synthiq.Q6_K.gguf) - very large, extremely low quality loss
|
||||
|
||||
[Q8_0](https://huggingface.co/sethuiyer/SynthIQ_GGUF/blob/main/synthiq.Q8.gguf) - very large, extremely low quality loss - not recommended
|
||||
|
||||
**Important Update**: SynthIQ is now available on Ollama. You can use it by running the command ```ollama run stuehieyr/synthiq``` in your
|
||||
terminal. If you have limited computing resources, check out this [video](https://www.youtube.com/watch?v=Qa1h7ygwQq8) to learn how to run it on
|
||||
a Google Colab backend.
|
||||
|
||||
|
||||
# Yaml Config
|
||||
|
||||
```yaml
|
||||
|
||||
slices:
|
||||
- sources:
|
||||
- model: Weyaxi/OpenHermes-2.5-neural-chat-v3-3-openchat-3.5-1210-Slerp
|
||||
layer_range: [0, 32]
|
||||
- model: uukuguy/speechless-mistral-six-in-one-7b
|
||||
layer_range: [0, 32]
|
||||
|
||||
merge_method: slerp
|
||||
base_model: mistralai/Mistral-7B-v0.1
|
||||
|
||||
parameters:
|
||||
t:
|
||||
- filter: self_attn
|
||||
value: [0, 0.5, 0.3, 0.7, 1]
|
||||
- filter: mlp
|
||||
value: [1, 0.5, 0.7, 0.3, 0]
|
||||
- value: 0.5 # fallback for rest of tensors
|
||||
tokenizer_source: union
|
||||
|
||||
dtype: bfloat16
|
||||
|
||||
```
|
||||
|
||||
<!-- prompt-template start -->
|
||||
## Prompt template: ChatML
|
||||
|
||||
```
|
||||
<|im_start|>system
|
||||
{system_message}<|im_end|>
|
||||
<|im_start|>user
|
||||
{prompt}<|im_end|>
|
||||
<|im_start|>assistant
|
||||
|
||||
```
|
||||
|
||||
<!-- prompt-template end -->
|
||||
|
||||
|
||||
License is LLama2 license as uukuguy/speechless-mistral-six-in-one-7b is llama2 license.
|
||||
|
||||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__SynthIQ-7b)
|
||||
|
||||
# [Nous Benchmark Evalation Results](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard)
|
||||
Detailed results can be found [here](https://gist.github.com/sethuiyer/f47dee388a4e95d46181c98d37d66a58)
|
||||
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__SynthIQ-7b)
|
||||
|
||||
| Metric |Value|
|
||||
|---------------------------------|----:|
|
||||
|Avg. |69.37|
|
||||
|AI2 Reasoning Challenge (25-Shot)|65.87|
|
||||
|HellaSwag (10-Shot) |85.82|
|
||||
|MMLU (5-Shot) |64.75|
|
||||
|TruthfulQA (0-shot) |57.00|
|
||||
|Winogrande (5-shot) |78.69|
|
||||
|GSM8k (5-shot) |64.06|
|
||||
|
||||
Reference in New Issue
Block a user