macadeliccc/laser-dolphin-mixtral-4x7b-dpo
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: macadeliccc/laser-dolphin-mixtral-4x7b-dpo
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-05-26 11:10:41 +08:00
commit 979a39e75a
17 changed files with 91583 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
dolphin_moe.png filter=lfs diff=lfs merge=lfs -text

229
README.md Normal file
View File

@@ -0,0 +1,229 @@
---
license: apache-2.0
library_name: transformers
model-index:
- name: laser-dolphin-mixtral-4x7b-dpo
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 64.93
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-dolphin-mixtral-4x7b-dpo
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 85.81
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-dolphin-mixtral-4x7b-dpo
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 63.04
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-dolphin-mixtral-4x7b-dpo
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 63.77
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-dolphin-mixtral-4x7b-dpo
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 77.82
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-dolphin-mixtral-4x7b-dpo
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 44.88
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-dolphin-mixtral-4x7b-dpo
name: Open LLM Leaderboard
---
# Laser-Dolphin-Mixtral-4x7b-dpo
![laser_dolphin_image](./dolphin_moe.png)
Credit to Fernando Fernandes and Eric Hartford for their project [laserRMT](https://github.com/cognitivecomputations/laserRMT)
This model is a medium-sized MoE implementation based on [cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser)
The process is outlined in this [notebook](https://github.com/cognitivecomputations/laserRMT/blob/main/examples/laser-dolphin-mixtral-2x7b.ipynb)
## Code Example
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
def generate_response(prompt):
"""
Generate a response from the model based on the input prompt.
Args:
prompt (str): Prompt for the model.
Returns:
str: The generated response from the model.
"""
# Tokenize the input prompt
inputs = tokenizer(prompt, return_tensors="pt")
# Generate output tokens
outputs = model.generate(**inputs, max_new_tokens=256, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id)
# Decode the generated tokens to a string
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
return response
# Load the model and tokenizer
model_id = "macadeliccc/laser-dolphin-mixtral-4x7b-dpo"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True)
prompt = "Write a quicksort algorithm in python"
# Generate and print responses for each language
print("Response:")
print(generate_response(prompt), "\n")
```
## Example output
can you write me a quicksort algorithm in python?
Sure, here's a quicksort algorithm implemented in Python:
```python
def quicksort(arr):
if len(arr) <= 1:
return arr
pivot = arr[len(arr) // 2]
left = [x for x in arr if x < pivot]
middle = [x for x in arr if x == pivot]
right = [x for x in arr if x > pivot]
return quicksort(left) + middle + quicksort(right)
```
This implementation uses the median of the array as the pivot.
It first checks if the array has one or fewer elements, in which case it is already sorted and can be returned as is.
Otherwise, it selects the pivot as the middle element of the array. Then, it partitions the array into three sub-arrays: elements less than the pivot, elements equal to the pivot, and elements greater than the pivot.
It recursively sorts the left and right sub-arrays and concatenates the results with the middle sub-array to obtain the final sorted array.
## Quantization
[4-bit AWQ](https://huggingface.co/macadeliccc/laser-dolphin-mixtral-4x7b-dpo-AWQ)
## Eval
**Model evaluated in 4bit**
----Benchmark Complete----
+ 2024-01-24 15:03:08
+ Time taken: 37.4 mins
+ Prompt Format: Mistral
+ Model: macadeliccc/laser-dolphin-mixtral-4x7b-dpo
+ Score (v2): 71.04
+ Parseable: 169.0
---------------
## Citations
Fernando Fernandes Neto and Eric Hartford. "Optimizing Large Language Models Using Layer-Selective Rank Reduction and Random Matrix Theory." 2024.
```bibtex
@article{sharma2023truth,
title={The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction},
author={Sharma, Pratyusha and Ash, Jordan T and Misra, Dipendra},
journal={arXiv preprint arXiv:2312.13558},
year={2023} }
```
```bibtex
@article{gao2021framework,
title={A framework for few-shot language model evaluation},
author={Gao, Leo and Tow, Jonathan and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and McDonell, Kyle and Muennighoff, Niklas and others},
journal={Version v0. 0.1. Sept},
year={2021}
}
```
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_macadeliccc__laser-dolphin-mixtral-4x7b-dpo)
| Metric |Value|
|---------------------------------|----:|
|Avg. |66.71|
|AI2 Reasoning Challenge (25-Shot)|64.93|
|HellaSwag (10-Shot) |85.81|
|MMLU (5-Shot) |63.04|
|TruthfulQA (0-shot) |63.77|
|Winogrande (5-shot) |77.82|
|GSM8k (5-shot) |44.88|

3
added_tokens.json Normal file
View File

@@ -0,0 +1,3 @@
{
"<|im_start|>": 32000
}

30
config.json Normal file
View File

@@ -0,0 +1,30 @@
{
"_name_or_path": "mlabonne/Marcoro14-7B-slerp",
"architectures": [
"MixtralForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 32768,
"model_type": "mixtral",
"num_attention_heads": 32,
"num_experts_per_tok": 2,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"num_local_experts": 4,
"output_router_logits": false,
"rms_norm_eps": 1e-05,
"rope_theta": 10000.0,
"router_aux_loss_coef": 0.001,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.37.0.dev0",
"use_cache": true,
"vocab_size": 32000
}

BIN
dolphin-cpp-2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 117 KiB

3
dolphin_moe.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5f82457da1aa82007718e010a67cd0d47308741efe70e61555e74ed4cbc9e34d
size 3385329

59
mergekit_moe_config.yml Normal file
View File

@@ -0,0 +1,59 @@
base_model: mlabonne/Marcoro14-7B-slerp
gate_mode: hidden
dtype: bfloat16
experts:
- source_model: mlabonne/NeuralBeagle14-7B
positive_prompts:
- "Help me debug this code."
- "Optimize this C# script."
- "Implement this feature using JavaScript."
- "Convert this HTML structure into a more efficient design."
- "Design me a theoretical software idea"
- "Use creativity to make a unique idea."
- source_model: WizardLM/WizardMath-7B-V1.1
positive_prompts:
- "add these numbers"
- "whats 2+2"
- "subtraction"
- "division"
- "multiplication"
- "addition"
- "I need help with a math problem"
- "Solve for x"
- "Add these two numbers together: 4 + 3 = 7"
- "Multiply 5 by 6: 5 * 6 = 30"
- "Divide 8 by 2: 8 / 2 = 4"
- "Find the remainder when 9 is divided by 3: 9 % 3 = 0"
- "Calculate the square root of 16: sqrt(16) = 4"
- "Simplify the expression (a+b)/(c-d): (a+b)/(c-d)"
- "Factor out the common factor of 2 from 4x + 6y: 2(2x + 3y)"
- "Solve for x in the equation 3x - 7 = 2x + 5: x = 12"
- "Graph the line y = 2x + 3"
- "Approximate pi to three decimal places: 3.142"
- "Find the derivative of f(x) = sin(x): f'(x) = cos(x)"
- "Integrate g(x) = x^2 over the interval [0, 1]: g(1) - g(0) = 1/3"
- "Calculate the determinant of the matrix A = [[2, 3], [4, 5]]: det(A) = 2*5 - 3*4 = -2"
- "Solve the system of equations Ax = b: x = [-5, 10]"
- "Calculate the sum of the first n natural numbers using the formula Sn = n*(n+1)/2: sum(n=1 to 5) = 15"
- source_model: cognitivecomputations/dolphin-2.6-mistral-7b-dpo
positive_prompts:
- "How do you"
- "Explain the concept of"
- "Give an overview of"
- "Compare and contrast between"
- "Provide information about"
- "Help me understand"
- "Summarize"
- "Make a recommendation on"
- "Answer this question"
- source_model: Open-Orca/Mistral-7B-OpenOrca
positive_prompts:
- "logical deduction"
- "critical reasoning skills"

3
model-00001-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:28a0b788f690df2a24e0c8a956ccc7598d9c6567127ce4f70373a0758f6a08a4
size 9919813704

3
model-00002-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a459521de5f6d889c4579415e32c49cad5057cba103cb8cd7a2ea668f8dd619e
size 9982454720

3
model-00003-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:655f097973b2681037ce018bebc81f4ee02e8d2e92a589b9907ec8c0ac105311
size 9982454752

3
model-00004-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2ea025be3f86b889e7e55e9db0e170b9be510172208ff0e987cb399451df5d9d
size 9982454720

3
model-00005-of-00005.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1bebe4c5d09e932279364c82b8345f3c0e362fd7ecda9dd1e1e554bab976ea82
size 8440279464

1
model.safetensors.index.json Normal file
View File

File diff suppressed because one or more lines are too long

29
special_tokens_map.json Normal file
View File

@@ -0,0 +1,29 @@
{
"additional_special_tokens": [
"<unk>",
"<s>",
"</s>"
],
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "<s>",
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

91129
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
tokenizer.model (Stored with Git LFS) Normal file
View File

Binary file not shown.

46
tokenizer_config.json Normal file
View File

@@ -0,0 +1,46 @@
{
"add_bos_token": true,
"add_eos_token": false,
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<unk>",
"<s>",
"</s>"
],
"bos_token": "<s>",
"clean_up_tokenization_spaces": false,
"eos_token": "</s>",
"legacy": true,
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<s>",
"sp_model_kwargs": {},
"spaces_between_special_tokens": false,
"tokenizer_class": "LlamaTokenizer",
"unk_token": "<unk>",
"use_default_system_prompt": true
}