初始化项目，由ModelHub XC社区提供模型
Model: rombodawg/Everyone-Coder-4x7b-Base Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
104
README.md
Normal file
104
README.md
Normal file
@@ -0,0 +1,104 @@
|
||||
---
|
||||
license: cc-by-4.0
|
||||
tags:
|
||||
- merge
|
||||
- moe
|
||||
---
|
||||
Everyone-Coder-4x7b-Base
|
||||
|
||||
|
||||
|
||||
|
||||
EveryoneLLM series of models are a new Mixtral type model created using experts that were finetuned by the community, for the community. This is the first model to release in the series and it is a coding specific model. EveryoneLLM, which will be a more generalized model, will be released in the near future after more work is done to fine tune the process of merging Mistral models into a larger Mixtral models with greater success.
|
||||
|
||||
The goal of the EveryoneLLM series of models is to be a replacement or an alternative to Mixtral-8x7b that is more suitable for general and specific use, as well as easier to fine tune. Since Mistralai is being secretive about the "secret sause" that makes Mixtral-Instruct such an effective fine tune of the Mixtral-base model, I've decided its time for the community to directly compete with Mistralai on our own.
|
||||
|
||||
The models that were used in this merger were as follow:
|
||||
|
||||
- https://huggingface.co/fblgit/UNA-TheBeagle-7b-v1
|
||||
|
||||
- https://huggingface.co/LucciAI/openchat-3.5-0106-function-calling
|
||||
|
||||
- https://huggingface.co/WizardLM/WizardMath-7B-V1.1
|
||||
|
||||
- https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser
|
||||
|
||||
|
||||
Thank you to the creators of the above ai models, they have full credit for the EveryoneLLM series of models. Without their hard work we wouldnt be able to achieve the great success we have in the open source community. 💗
|
||||
|
||||
You can find the write up for this model here:
|
||||
|
||||
https://docs.google.com/document/d/1_vOftBnrk9NRk5h10UqrfJ5CDih9KBKL61yvrZtVWPE/edit?usp=sharing
|
||||
|
||||
Config for the merger can be found bellow:
|
||||
|
||||
|
||||
```yaml
|
||||
base_model: mistralai_Mistral-7B-v0.1
|
||||
gate_mode: hidden
|
||||
dtype: float16
|
||||
experts:
|
||||
- source_model: cognitivecomputations_dolphin-2.6-mistral-7b-dpo-laser
|
||||
positive_prompts:
|
||||
- "Help me debug this code."
|
||||
- "Rewrite this function in Python."
|
||||
- "Optimize this C# script."
|
||||
- "Implement this feature using JavaScript."
|
||||
- "Convert this HTML structure into a more efficient design."
|
||||
- "Assist me with writing a program that"
|
||||
- source_model: fblgit_UNA-TheBeagle-7b-v1
|
||||
positive_prompts:
|
||||
- "How do you"
|
||||
- "Explain the concept of"
|
||||
- "Give an overview of"
|
||||
- "Compare and contrast between"
|
||||
- "Provide information about"
|
||||
- "Help me understand"
|
||||
- "Summarize"
|
||||
- "Make a recommendation on"
|
||||
- "Answer this question"
|
||||
- source_model: LucciAI_openchat-3.5-0106-function-calling
|
||||
positive_prompts:
|
||||
- "Write a program to solve this problem"
|
||||
- "Modify this function to improve its performance"
|
||||
- "Refactor this code to enhance readability"
|
||||
- "Create a custom function for this specific use case"
|
||||
- "Optimize this algorithm to reduce computational complexity"
|
||||
- "Implement this feature by extending existing codebase"
|
||||
- "Integrate this API call into the application"
|
||||
- "Help me troubleshoot and fix this bug"
|
||||
- "Review and test this code snippet before deployment"
|
||||
- "Analyze this error log to identify potential issues"
|
||||
- "Generate a set of unit tests for this module"
|
||||
- "Evaluate different approaches to solving this problem"
|
||||
- "Do a web search for"
|
||||
- "Use the plugin to"
|
||||
- source_model: WizardLM_WizardMath-7B-V1.1
|
||||
positive_prompts:
|
||||
- "add these numbers"
|
||||
- "whats 2+2"
|
||||
- "subtraction"
|
||||
- "division"
|
||||
- "multiplication"
|
||||
- "addition"
|
||||
- "I need help with a math problem"
|
||||
- "Solve for x"
|
||||
- "Add these two numbers together: 4 + 3 = 7"
|
||||
- "Multiply 5 by 6: 5 * 6 = 30"
|
||||
- "Divide 8 by 2: 8 / 2 = 4"
|
||||
- "Find the remainder when 9 is divided by 3: 9 % 3 = 0"
|
||||
- "Calculate the square root of 16: sqrt(16) = 4"
|
||||
- "Simplify the expression (a+b)/(c-d): (a+b)/(c-d)"
|
||||
- "Factor out the common factor of 2 from 4x + 6y: 2(2x + 3y)"
|
||||
- "Solve for x in the equation 3x - 7 = 2x + 5: x = 12"
|
||||
- "Graph the line y = 2x + 3"
|
||||
- "Approximate pi to three decimal places: 3.142"
|
||||
- "Find the derivative of f(x) = sin(x): f'(x) = cos(x)"
|
||||
- "Integrate g(x) = x^2 over the interval [0, 1]: g(1) - g(0) = 1/3"
|
||||
- "Calculate the determinant of the matrix A = [[2, 3], [4, 5]]: det(A) = 2*5 - 3*4 = -2"
|
||||
- "Solve the system of equations Ax = b: x = [-5, 10]"
|
||||
- "Calculate the sum of the first n natural numbers using the formula Sn = n*(n+1)/2: sum(n=1 to 5) = 15"
|
||||
```
|
||||
|
||||
|
||||
|
||||
30
config.json
Normal file
30
config.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"_name_or_path": "Everyone-Coder-4x7b-Base",
|
||||
"architectures": [
|
||||
"MixtralForCausalLM"
|
||||
],
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 14336,
|
||||
"max_position_embeddings": 32768,
|
||||
"model_type": "mixtral",
|
||||
"num_attention_heads": 32,
|
||||
"num_experts_per_tok": 2,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 8,
|
||||
"num_local_experts": 4,
|
||||
"output_router_logits": false,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_theta": 10000.0,
|
||||
"router_aux_loss_coef": 0.001,
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.36.2",
|
||||
"use_cache": true,
|
||||
"vocab_size": 32000
|
||||
}
|
||||
64
mergekit_moe_config.yml
Normal file
64
mergekit_moe_config.yml
Normal file
@@ -0,0 +1,64 @@
|
||||
base_model: D:\Merging_models_on_D_drive\Frankenmerge\mistralai_Mistral-7B-v0.1
|
||||
gate_mode: hidden
|
||||
dtype: float16
|
||||
experts:
|
||||
- source_model: D:\Merging_models_on_D_drive\Frankenmerge\cognitivecomputations_dolphin-2.6-mistral-7b-dpo-laser
|
||||
positive_prompts:
|
||||
- "Help me debug this code."
|
||||
- "Rewrite this function in Python."
|
||||
- "Optimize this C# script."
|
||||
- "Implement this feature using JavaScript."
|
||||
- "Convert this HTML structure into a more efficient design."
|
||||
- "Assist me with writing a program that"
|
||||
- source_model: D:\Merging_models_on_D_drive\Frankenmerge\fblgit_UNA-TheBeagle-7b-v1
|
||||
positive_prompts:
|
||||
- "How do you"
|
||||
- "Explain the concept of"
|
||||
- "Give an overview of"
|
||||
- "Compare and contrast between"
|
||||
- "Provide information about"
|
||||
- "Help me understand"
|
||||
- "Summarize"
|
||||
- "Make a recommendation on"
|
||||
- "Answer this question"
|
||||
- source_model: D:\Merging_models_on_D_drive\Frankenmerge\LucciAI_openchat-3.5-0106-function-calling
|
||||
positive_prompts:
|
||||
- "Write a program to solve this problem"
|
||||
- "Modify this function to improve its performance"
|
||||
- "Refactor this code to enhance readability"
|
||||
- "Create a custom function for this specific use case"
|
||||
- "Optimize this algorithm to reduce computational complexity"
|
||||
- "Implement this feature by extending existing codebase"
|
||||
- "Integrate this API call into the application"
|
||||
- "Help me troubleshoot and fix this bug"
|
||||
- "Review and test this code snippet before deployment"
|
||||
- "Analyze this error log to identify potential issues"
|
||||
- "Generate a set of unit tests for this module"
|
||||
- "Evaluate different approaches to solving this problem"
|
||||
- "Do a web search for"
|
||||
- "Use the plugin to"
|
||||
- source_model: D:\Merging_models_on_D_drive\Frankenmerge\WizardLM_WizardMath-7B-V1.1
|
||||
positive_prompts:
|
||||
- "add these numbers"
|
||||
- "whats 2+2"
|
||||
- "subtraction"
|
||||
- "division"
|
||||
- "multiplication"
|
||||
- "addition"
|
||||
- "I need help with a math problem"
|
||||
- "Solve for x"
|
||||
- "Add these two numbers together: 4 + 3 = 7"
|
||||
- "Multiply 5 by 6: 5 * 6 = 30"
|
||||
- "Divide 8 by 2: 8 / 2 = 4"
|
||||
- "Find the remainder when 9 is divided by 3: 9 % 3 = 0"
|
||||
- "Calculate the square root of 16: sqrt(16) = 4"
|
||||
- "Simplify the expression (a+b)/(c-d): (a+b)/(c-d)"
|
||||
- "Factor out the common factor of 2 from 4x + 6y: 2(2x + 3y)"
|
||||
- "Solve for x in the equation 3x - 7 = 2x + 5: x = 12"
|
||||
- "Graph the line y = 2x + 3"
|
||||
- "Approximate pi to three decimal places: 3.142"
|
||||
- "Find the derivative of f(x) = sin(x): f'(x) = cos(x)"
|
||||
- "Integrate g(x) = x^2 over the interval [0, 1]: g(1) - g(0) = 1/3"
|
||||
- "Calculate the determinant of the matrix A = [[2, 3], [4, 5]]: det(A) = 2*5 - 3*4 = -2"
|
||||
- "Solve the system of equations Ax = b: x = [-5, 10]"
|
||||
- "Calculate the sum of the first n natural numbers using the formula Sn = n*(n+1)/2: sum(n=1 to 5) = 15"
|
||||
3
model-00001-of-00005.safetensors
Normal file
3
model-00001-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:59bf76a81ae23edbc1845d6dfc20725a7f7167f482e9367b5a4c61be0dbf823a
|
||||
size 9919813584
|
||||
3
model-00002-of-00005.safetensors
Normal file
3
model-00002-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7d2d08806f1b6bdaca2e2bed5edbae3d41943acf9bedbad2fc70a7ac1a086fed
|
||||
size 9982454640
|
||||
3
model-00003-of-00005.safetensors
Normal file
3
model-00003-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:9383eac3802c898df97259418b0be990628fe5965221056f11641b41834139bc
|
||||
size 9982454672
|
||||
3
model-00004-of-00005.safetensors
Normal file
3
model-00004-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a5028f632c65b0d9a3c05fafaf9108fe62ae88e6a24530878e2cdd2990d8fc24
|
||||
size 9982454632
|
||||
3
model-00005-of-00005.safetensors
Normal file
3
model-00005-of-00005.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:24ff58a528a4c455813dfb9baa03eae4a4c7ae8f50b6b44318f96999fe495015
|
||||
size 8440279224
|
||||
1
model.safetensors.index.json
Normal file
1
model.safetensors.index.json
Normal file
File diff suppressed because one or more lines are too long
24
special_tokens_map.json
Normal file
24
special_tokens_map.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": "<s>",
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
91129
tokenizer.json
Normal file
91129
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
Binary file not shown.
42
tokenizer_config.json
Normal file
42
tokenizer_config.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"add_bos_token": true,
|
||||
"add_eos_token": false,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [],
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"legacy": true,
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": "<s>",
|
||||
"sp_model_kwargs": {},
|
||||
"spaces_between_special_tokens": false,
|
||||
"tokenizer_class": "LlamaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user