初始化项目,由ModelHub XC社区提供模型
Model: QuantFactory/Einstein-v6.1-Llama3-8B-GGUF Source: Original Platform
This commit is contained in:
49
.gitattributes
vendored
Normal file
49
.gitattributes
vendored
Normal file
@@ -0,0 +1,49 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
Einstein-v6.1-Llama3-8B.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
||||
3
Einstein-v6.1-Llama3-8B.Q2_K.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q2_K.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:eeab94efb262e4b2ef5ed23c623b462baf553b077dabd80e2fa7afb95b6214d2
|
||||
size 3179152000
|
||||
3
Einstein-v6.1-Llama3-8B.Q3_K_L.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q3_K_L.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:06fc7e3e7e8fcfa87a7e21876ed48799978633a9a7b59fab483ad5649145923f
|
||||
size 4321978624
|
||||
3
Einstein-v6.1-Llama3-8B.Q3_K_M.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q3_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:981bc5f8362457fecebc55dd9a439885ce171ee1892ebff1d926ef1be4b925eb
|
||||
size 4018940160
|
||||
3
Einstein-v6.1-Llama3-8B.Q3_K_S.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q3_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:36d126615e811458648f3aa13b9134e76721b4fd936b0d5b459ccdcd7c5049fc
|
||||
size 3664521472
|
||||
3
Einstein-v6.1-Llama3-8B.Q4_0.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q4_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3481488571ac890043b374da08d1428cb2c576606b45e87a89e5a84feda19db4
|
||||
size 4661236096
|
||||
3
Einstein-v6.1-Llama3-8B.Q4_1.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q4_1.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d0252176a00ba50b719477f71c0adf12fe5b952f781924a82356ae1dae3fbbc4
|
||||
size 5130278272
|
||||
3
Einstein-v6.1-Llama3-8B.Q4_K_M.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q4_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:c4191ae5628582e46fca1a1faf35b199243cf656b2e60b4ddc15e408a4b49e3f
|
||||
size 4920758656
|
||||
3
Einstein-v6.1-Llama3-8B.Q4_K_S.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q4_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2f541a702ed9434e10f4720994342000c34d9e2edfb091ce6e464ba20f128d20
|
||||
size 4692693376
|
||||
3
Einstein-v6.1-Llama3-8B.Q5_0.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q5_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d30a82ba14e24be5818fe1e54a40ad8ce2bfaab90203ebe85f3f4083713bfb0b
|
||||
size 5599320448
|
||||
3
Einstein-v6.1-Llama3-8B.Q5_1.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q5_1.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a5fdc9fb1ba3abae4a9200bacd6f09c15b49c0a28b458f6859f475ac18ffb440
|
||||
size 6068362624
|
||||
3
Einstein-v6.1-Llama3-8B.Q5_K_M.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q5_K_M.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:2fa1f74c3082e7b31039e64476d36c6b54058141e70fc5f043fc478c27d36908
|
||||
size 5733013888
|
||||
3
Einstein-v6.1-Llama3-8B.Q5_K_S.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q5_K_S.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:add41a4e7e6f4cceced3f67c230b8ae6a6c5e108078e12578f1ed9ac47e10aef
|
||||
size 5599320448
|
||||
3
Einstein-v6.1-Llama3-8B.Q6_K.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q6_K.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:75725574670315f765a8b4b029fa82c5adedb823e78bac24f7790b1fa0fccb19
|
||||
size 6596035072
|
||||
3
Einstein-v6.1-Llama3-8B.Q8_0.gguf
Normal file
3
Einstein-v6.1-Llama3-8B.Q8_0.gguf
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:490308051f32bd5262cbcb65e0594c30ac12459efa3fb72f315800587f825f84
|
||||
size 8540807424
|
||||
554
README.md
Normal file
554
README.md
Normal file
@@ -0,0 +1,554 @@
|
||||
|
||||
---
|
||||
|
||||
language:
|
||||
- en
|
||||
license: other
|
||||
tags:
|
||||
- axolotl
|
||||
- generated_from_trainer
|
||||
- instruct
|
||||
- finetune
|
||||
- chatml
|
||||
- gpt4
|
||||
- synthetic data
|
||||
- science
|
||||
- physics
|
||||
- chemistry
|
||||
- biology
|
||||
- math
|
||||
- llama
|
||||
- llama3
|
||||
base_model: meta-llama/Meta-Llama-3-8B
|
||||
datasets:
|
||||
- allenai/ai2_arc
|
||||
- camel-ai/physics
|
||||
- camel-ai/chemistry
|
||||
- camel-ai/biology
|
||||
- camel-ai/math
|
||||
- metaeval/reclor
|
||||
- openbookqa
|
||||
- mandyyyyii/scibench
|
||||
- derek-thomas/ScienceQA
|
||||
- TIGER-Lab/ScienceEval
|
||||
- jondurbin/airoboros-3.2
|
||||
- LDJnr/Capybara
|
||||
- Cot-Alpaca-GPT4-From-OpenHermes-2.5
|
||||
- STEM-AI-mtl/Electrical-engineering
|
||||
- knowrohit07/saraswati-stem
|
||||
- sablo/oasst2_curated
|
||||
- lmsys/lmsys-chat-1m
|
||||
- TIGER-Lab/MathInstruct
|
||||
- bigbio/med_qa
|
||||
- meta-math/MetaMathQA-40K
|
||||
- openbookqa
|
||||
- piqa
|
||||
- metaeval/reclor
|
||||
- derek-thomas/ScienceQA
|
||||
- scibench
|
||||
- sciq
|
||||
- Open-Orca/SlimOrca
|
||||
- migtissera/Synthia-v1.3
|
||||
- TIGER-Lab/ScienceEval
|
||||
- allenai/WildChat
|
||||
- microsoft/orca-math-word-problems-200k
|
||||
- openchat/openchat_sharegpt4_dataset
|
||||
- teknium/GPTeacher-General-Instruct
|
||||
- m-a-p/CodeFeedback-Filtered-Instruction
|
||||
- totally-not-an-llm/EverythingLM-data-V3
|
||||
- HuggingFaceH4/no_robots
|
||||
- OpenAssistant/oasst_top1_2023-08-25
|
||||
- WizardLM/WizardLM_evol_instruct_70k
|
||||
model-index:
|
||||
- name: Einstein-v6.1-Llama3-8B
|
||||
results:
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: AI2 Reasoning Challenge (25-Shot)
|
||||
type: ai2_arc
|
||||
config: ARC-Challenge
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 25
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 62.46
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: HellaSwag (10-Shot)
|
||||
type: hellaswag
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 10
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 82.41
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: MMLU (5-Shot)
|
||||
type: cais/mmlu
|
||||
config: all
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 66.19
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: TruthfulQA (0-shot)
|
||||
type: truthful_qa
|
||||
config: multiple_choice
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 0
|
||||
metrics:
|
||||
- type: mc2
|
||||
value: 55.1
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: Winogrande (5-shot)
|
||||
type: winogrande
|
||||
config: winogrande_xl
|
||||
split: validation
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 79.32
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: GSM8k (5-shot)
|
||||
type: gsm8k
|
||||
config: main
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 66.11
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: IFEval (0-Shot)
|
||||
type: HuggingFaceH4/ifeval
|
||||
args:
|
||||
num_few_shot: 0
|
||||
metrics:
|
||||
- type: inst_level_strict_acc and prompt_level_strict_acc
|
||||
value: 45.68
|
||||
name: strict accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: BBH (3-Shot)
|
||||
type: BBH
|
||||
args:
|
||||
num_few_shot: 3
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 29.38
|
||||
name: normalized accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: MATH Lvl 5 (4-Shot)
|
||||
type: hendrycks/competition_math
|
||||
args:
|
||||
num_few_shot: 4
|
||||
metrics:
|
||||
- type: exact_match
|
||||
value: 5.74
|
||||
name: exact match
|
||||
source:
|
||||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: GPQA (0-shot)
|
||||
type: Idavidrein/gpqa
|
||||
args:
|
||||
num_few_shot: 0
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 4.25
|
||||
name: acc_norm
|
||||
source:
|
||||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: MuSR (0-shot)
|
||||
type: TAUR-Lab/MuSR
|
||||
args:
|
||||
num_few_shot: 0
|
||||
metrics:
|
||||
- type: acc_norm
|
||||
value: 11.23
|
||||
name: acc_norm
|
||||
source:
|
||||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
- task:
|
||||
type: text-generation
|
||||
name: Text Generation
|
||||
dataset:
|
||||
name: MMLU-PRO (5-shot)
|
||||
type: TIGER-Lab/MMLU-Pro
|
||||
config: main
|
||||
split: test
|
||||
args:
|
||||
num_few_shot: 5
|
||||
metrics:
|
||||
- type: acc
|
||||
value: 23.68
|
||||
name: accuracy
|
||||
source:
|
||||
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
name: Open LLM Leaderboard
|
||||
|
||||
---
|
||||
|
||||
[](https://hf.co/QuantFactory)
|
||||
|
||||
|
||||
# QuantFactory/Einstein-v6.1-Llama3-8B-GGUF
|
||||
This is quantized version of [Weyaxi/Einstein-v6.1-Llama3-8B](https://huggingface.co/Weyaxi/Einstein-v6.1-Llama3-8B) created using llama.cpp
|
||||
|
||||
# Original Model Card
|
||||
|
||||

|
||||
|
||||
# 🔬 Einstein-v6.1-Llama3-8B
|
||||
|
||||
This model is a full fine-tuned version of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on diverse datasets.
|
||||
|
||||
This model is finetuned using `8xRTX3090` + `1xRTXA6000` using [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl).
|
||||
|
||||
This model's training was sponsored by [sablo.ai](https://sablo.ai).
|
||||
|
||||
<details><summary>See axolotl config</summary>
|
||||
|
||||
axolotl version: `0.4.0`
|
||||
```yaml
|
||||
base_model: meta-llama/Meta-Llama-3-8B
|
||||
model_type: LlamaForCausalLM
|
||||
tokenizer_type: AutoTokenizer
|
||||
|
||||
load_in_8bit: false
|
||||
load_in_4bit: false
|
||||
strict: false
|
||||
|
||||
chat_template: chatml
|
||||
datasets:
|
||||
- path: data/merged_all.json
|
||||
ds_type: json
|
||||
type: alpaca
|
||||
conversation: chatml
|
||||
|
||||
- path: data/gpteacher-instruct-special-alpaca.json
|
||||
ds_type: json
|
||||
type: gpteacher
|
||||
conversation: chatml
|
||||
|
||||
- path: data/wizardlm_evol_instruct_70k_random_half.json
|
||||
ds_type: json
|
||||
type: alpaca
|
||||
conversation: chatml
|
||||
|
||||
- path: data/capybara_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
- path: data/synthia-v1.3_sharegpt_12500.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
- path: data/cot_alpaca_gpt4_extracted_openhermes_2.5_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
- path: data/slimorca_dedup_filtered_95k_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
- path: data/airoboros_3.2_without_contextual_slimorca_orca_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
- path: data/allenai_wild_chat_gpt4_english_toxic_random_half_4k_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
strict: false
|
||||
conversation: chatml
|
||||
|
||||
- path: data/pippa_bagel_repo_3k_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
- path: data/gpt4_data_lmys_1m_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
- path: data/sharegpt_gpt4_english.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
conversation: chatml
|
||||
|
||||
- path: data/no_robots_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
strict: false
|
||||
conversation: chatml
|
||||
|
||||
- path: data/oasst_top1_from_fusechatmixture_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
strict: false
|
||||
conversation: chatml
|
||||
|
||||
- path: data/everythinglm-data-v3_sharegpt.json
|
||||
ds_type: json
|
||||
type: sharegpt
|
||||
strict: false
|
||||
conversation: chatml
|
||||
|
||||
dataset_prepared_path: last_run_prepared
|
||||
val_set_size: 0.002
|
||||
|
||||
output_dir: ./Einstein-v6.1-Llama3-8B-model
|
||||
|
||||
sequence_len: 8192
|
||||
sample_packing: true
|
||||
pad_to_sequence_len: true
|
||||
eval_sample_packing: false
|
||||
|
||||
wandb_project: Einstein
|
||||
wandb_entity:
|
||||
wandb_watch:
|
||||
wandb_name: Einstein-v6.1-Llama3-2-epoch
|
||||
wandb_log_model:
|
||||
hub_model_id: Weyaxi/Einstein-v6.1-Llama3-8B
|
||||
|
||||
save_safetensors: true
|
||||
|
||||
gradient_accumulation_steps: 4
|
||||
micro_batch_size: 1
|
||||
num_epochs: 2
|
||||
optimizer: adamw_bnb_8bit # look
|
||||
lr_scheduler: cosine
|
||||
learning_rate: 0.000005 # look
|
||||
|
||||
train_on_inputs: false
|
||||
group_by_length: false
|
||||
bf16: true
|
||||
fp16: false
|
||||
tf32: false
|
||||
|
||||
gradient_checkpointing: true
|
||||
early_stopping_patience:
|
||||
resume_from_checkpoint:
|
||||
local_rank:
|
||||
logging_steps: 1
|
||||
xformers_attention:
|
||||
flash_attention: true
|
||||
|
||||
warmup_steps: 10
|
||||
evals_per_epoch: 2
|
||||
eval_table_size:
|
||||
eval_table_max_new_tokens: 128
|
||||
saves_per_epoch: 2
|
||||
debug:
|
||||
|
||||
deepspeed: zero3_bf16_cpuoffload_params.json
|
||||
weight_decay: 0.0
|
||||
fsdp:
|
||||
fsdp_config:
|
||||
special_tokens:
|
||||
bos_token: "<s>"
|
||||
eos_token: "<|im_end|>"
|
||||
unk_token: "<unk>"
|
||||
pad_token: <|end_of_text|> # changed
|
||||
tokens:
|
||||
- "<|im_start|>"
|
||||
```
|
||||
</details><br>
|
||||
|
||||
# 💬 Prompt Template
|
||||
|
||||
You can use ChatML prompt template while using the model:
|
||||
|
||||
### ChatML
|
||||
|
||||
```
|
||||
<|im_start|>system
|
||||
{system}<|im_end|>
|
||||
<|im_start|>user
|
||||
{user}<|im_end|>
|
||||
<|im_start|>assistant
|
||||
{asistant}<|im_end|>
|
||||
```
|
||||
|
||||
This prompt template is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating), which means you can format messages using the
|
||||
`tokenizer.apply_chat_template()` method:
|
||||
|
||||
```python
|
||||
messages = [
|
||||
{"role": "system", "content": "You are helpful AI asistant."},
|
||||
{"role": "user", "content": "Hello!"}
|
||||
]
|
||||
gen_input = tokenizer.apply_chat_template(message, return_tensors="pt")
|
||||
model.generate(**gen_input)
|
||||
```
|
||||
|
||||
# 📊 Datasets used in this model
|
||||
|
||||
The datasets used to train this model are listed in the metadata section of the model card.
|
||||
|
||||
Please note that certain datasets mentioned in the metadata may have undergone filtering based on various criteria.
|
||||
|
||||
The results of this filtering process and its outcomes are in the data folder of this repository:
|
||||
|
||||
[Weyaxi/Einstein-v6.1-Llama3-8B/data](https://huggingface.co/Weyaxi/Einstein-v6.1-Llama3-8B/tree/main/data)
|
||||
|
||||
# 🔄 Quantizationed versions
|
||||
|
||||
## GGUF [@bartowski](https://huggingface.co/bartowski)
|
||||
|
||||
- https://huggingface.co/bartowski/Einstein-v6.1-Llama3-8B-GGUF
|
||||
|
||||
## ExLlamaV2 [@bartowski](https://huggingface.co/bartowski)
|
||||
|
||||
- https://huggingface.co/bartowski/Einstein-v6.1-Llama3-8B-exl2
|
||||
|
||||
## AWQ [@solidrust](https://huggingface.co/solidrust)
|
||||
|
||||
- https://huggingface.co/solidrust/Einstein-v6.1-Llama3-8B-AWQ
|
||||
|
||||
# 🎯 [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
||||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Weyaxi__Einstein-v6.1-Llama3-8B)
|
||||
|
||||
| Metric |Value|
|
||||
|---------------------------------|----:|
|
||||
|Avg. |68.60|
|
||||
|AI2 Reasoning Challenge (25-Shot)|62.46|
|
||||
|HellaSwag (10-Shot) |82.41|
|
||||
|MMLU (5-Shot) |66.19|
|
||||
|TruthfulQA (0-shot) |55.10|
|
||||
|Winogrande (5-shot) |79.32|
|
||||
|GSM8k (5-shot) |66.11|
|
||||
|
||||
# 🎯 [Open LLM Leaderboard v2 Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
|
||||
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Weyaxi__Einstein-v6.1-Llama3-8B)
|
||||
|
||||
| Metric |Value|
|
||||
|-------------------|----:|
|
||||
|Avg. |19.99|
|
||||
|IFEval (0-Shot) |45.68|
|
||||
|BBH (3-Shot) |29.38|
|
||||
|MATH Lvl 5 (4-Shot)| 5.74|
|
||||
|GPQA (0-shot) | 4.25|
|
||||
|MuSR (0-shot) |11.23|
|
||||
|MMLU-PRO (5-shot) |23.68|
|
||||
|
||||
|
||||
# 📚 Some resources, discussions and reviews aboout this model
|
||||
|
||||
#### 🐦 Announcement tweet:
|
||||
|
||||
- https://twitter.com/Weyaxi/status/1783050724659675627
|
||||
|
||||
#### 🔍 Reddit post in r/LocalLLaMA:
|
||||
|
||||
- https://www.reddit.com/r/LocalLLaMA/comments/1cdlym1/introducing_einstein_v61_based_on_the_new_llama3/
|
||||
|
||||
#### ▶️ Youtube Video(s)
|
||||
|
||||
- [Install Einstein v6.1 Llama3-8B Locally on Windows](https://www.youtube.com/watch?v=VePvv6OM0JY)
|
||||
|
||||
#### 📱 Octopus-V4-3B
|
||||
|
||||
- [Octopus-V4-3B](https://huggingface.co/NexaAIDev/Octopus-v4) leverages the incredible physics capabilities of [Einstein-v6.1-Llama3-8B](https://huggingface.co/Weyaxi/Einstein-v6.1-Llama3-8B) in their model.
|
||||
|
||||
# 🤖 Additional information about training
|
||||
|
||||
This model is full fine-tuned for 2 epoch.
|
||||
|
||||
Total number of steps was 2026.
|
||||
|
||||
<details><summary>Loss graph</summary>
|
||||
|
||||

|
||||
|
||||
</details><br>
|
||||
|
||||
# 🤝 Acknowledgments
|
||||
|
||||
Thanks to [sablo.ai](https://sablo.ai) for sponsoring this model.
|
||||
|
||||
Thanks to all the dataset authors mentioned in the datasets section.
|
||||
|
||||
Thanks to [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) for making the repository I used to make this model.
|
||||
|
||||
Thanks to all open source AI community.
|
||||
|
||||
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
||||
|
||||
If you would like to support me:
|
||||
|
||||
[☕ Buy Me a Coffee](https://www.buymeacoffee.com/weyaxi)
|
||||
1
configuration.json
Normal file
1
configuration.json
Normal file
@@ -0,0 +1 @@
|
||||
{"framework": "pytorch", "task": "text-generation", "allow_remote": true}
|
||||
Reference in New Issue
Block a user