初始化项目，由ModelHub XC社区提供模型
Model: Technoculture/Medorca-2x7b Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
94
README.md
Normal file
94
README.md
Normal file
@@ -0,0 +1,94 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- moe
|
||||
- merge
|
||||
- epfl-llm/meditron-7b
|
||||
- microsoft/Orca-2-7b
|
||||
---
|
||||
|
||||
|
||||
|
||||
# Medorca-2x7b
|
||||
|
||||
Medorca-2x7b is a Mixure of Experts (MoE) made with the following models:
|
||||
* [epfl-llm/meditron-7b](https://huggingface.co/epfl-llm/meditron-7b)
|
||||
* [microsoft/Orca-2-7b](https://huggingface.co/microsoft/Orca-2-7b)
|
||||
|
||||
## Evaluations
|
||||
|
||||
| Benchmark | Medorca-2x7b | Orca-2-7b | llama-2-7b | meditron-7b | meditron-70b |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| MedMCQA | | | | | |
|
||||
| ClosedPubMedQA | | | | | |
|
||||
| PubMedQA | | | | | |
|
||||
| MedQA | | | | | |
|
||||
| MedQA4 | | | | | |
|
||||
| MedicationQA | | | | | |
|
||||
| MMLU Medical | | | | | |
|
||||
| MMLU | 53.3 | **56.37** | | | |
|
||||
| TruthfulQA | 48.04 | **52.45** | | | |
|
||||
| GSM8K | 20.64 | **47.2** | | | |
|
||||
| ARC | 54.1 | 54.1 | | | |
|
||||
| HellaSwag | 76.04 | **76.19** | | | |
|
||||
| Winogrande | **74.51** | 73.48 | | | |
|
||||
|
||||
More details on the Open LLM Leaderboard evaluation results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Technoculture__Medorca-2x7b.)
|
||||
|
||||
## 🧩 Configuration
|
||||
|
||||
```yaml
|
||||
base_model: microsoft/Orca-2-7b
|
||||
gate_mode: hidden
|
||||
dtype: bfloat16
|
||||
experts:
|
||||
- source_model: epfl-llm/meditron-7b
|
||||
positive_prompts:
|
||||
- "How does sleep affect cardiovascular health?"
|
||||
- "Could a plant-based diet improve arthritis symptoms?"
|
||||
- "A patient comes in with symptoms of dizziness and nausea..."
|
||||
- "When discussing diabetes management, the key factors to consider are..."
|
||||
- "The differential diagnosis for a headache with visual aura could include..."
|
||||
negative_prompts:
|
||||
- "Recommend a good recipe for a vegetarian lasagna."
|
||||
- "Give an overview of the French Revolution."
|
||||
- "Explain how a digital camera captures an image."
|
||||
- "What are the environmental impacts of deforestation?"
|
||||
- "The recent advancements in artificial intelligence have led to developments in..."
|
||||
- "The fundamental concepts in economics include ideas like supply and demand, which explain..."
|
||||
- source_model: microsoft/Orca-2-7b
|
||||
positive_prompts:
|
||||
- "Here is a funny joke for you -"
|
||||
- "When considering the ethical implications of artificial intelligence, one must take into account..."
|
||||
- "In strategic planning, a company must analyze its strengths and weaknesses, which involves..."
|
||||
- "Understanding consumer behavior in marketing requires considering factors like..."
|
||||
- "The debate on climate change solutions hinges on arguments that..."
|
||||
negative_prompts:
|
||||
- "In discussing dietary adjustments for managing hypertension, it's crucial to emphasize..."
|
||||
- "For early detection of melanoma, dermatologists recommend that patients regularly check their skin for..."
|
||||
- "Explaining the importance of vaccination, a healthcare professional should highlight..."
|
||||
```
|
||||
|
||||
## 💻 Usage
|
||||
|
||||
```python
|
||||
!pip install -qU transformers bitsandbytes accelerate
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
import transformers
|
||||
import torch
|
||||
|
||||
model = "Technoculture/Medorca-2x7b"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model)
|
||||
pipeline = transformers.pipeline(
|
||||
"text-generation",
|
||||
model=model,
|
||||
model_kwargs={"torch_dtype": torch.float16},
|
||||
)
|
||||
|
||||
messages = [{"role": "user", "content": "Why am i feeling so tired this month?"}]
|
||||
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
|
||||
print(outputs[0]["generated_text"])
|
||||
```
|
||||
5
added_tokens.json
Normal file
5
added_tokens.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"<|im_end|>": 32002,
|
||||
"<|im_start|>": 32001,
|
||||
"[PAD]": 32000
|
||||
}
|
||||
33
config.json
Normal file
33
config.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"_name_or_path": "microsoft/Orca-2-7b",
|
||||
"architectures": [
|
||||
"MixtralForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 4096,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 11008,
|
||||
"max_position_embeddings": 4096,
|
||||
"model_type": "mixtral",
|
||||
"num_attention_heads": 32,
|
||||
"num_experts_per_tok": 2,
|
||||
"num_hidden_layers": 32,
|
||||
"num_key_value_heads": 32,
|
||||
"num_local_experts": 2,
|
||||
"output_router_logits": false,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 10000.0,
|
||||
"router_aux_loss_coef": 0.001,
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.36.2",
|
||||
"use_cache": true,
|
||||
"vocab_size": 32003
|
||||
}
|
||||
17
mergekit_config.yml
Normal file
17
mergekit_config.yml
Normal file
@@ -0,0 +1,17 @@
|
||||
|
||||
slices:
|
||||
- sources:
|
||||
- model: epfl-llm/meditron-7b
|
||||
layer_range: [0, 32]
|
||||
- model: microsoft/Orca-2-7b
|
||||
layer_range: [0, 32]
|
||||
merge_method: slerp
|
||||
base_model: epfl-llm/meditron-7b
|
||||
parameters:
|
||||
t:
|
||||
- filter: self_attn
|
||||
value: [0, 0.5, 0.3, 0.7, 1]
|
||||
- filter: mlp
|
||||
value: [1, 0.5, 0.7, 0.3, 0]
|
||||
- value: 0.5 # fallback for rest of tensors
|
||||
dtype: float16
|
||||
30
mergekit_moe_config.yml
Normal file
30
mergekit_moe_config.yml
Normal file
@@ -0,0 +1,30 @@
|
||||
|
||||
base_model: microsoft/Orca-2-7b
|
||||
gate_mode: hidden
|
||||
dtype: bfloat16
|
||||
experts:
|
||||
- source_model: epfl-llm/meditron-7b
|
||||
positive_prompts:
|
||||
- "How does sleep affect cardiovascular health?"
|
||||
- "Could a plant-based diet improve arthritis symptoms?"
|
||||
- "A patient comes in with symptoms of dizziness and nausea..."
|
||||
- "When discussing diabetes management, the key factors to consider are..."
|
||||
- "The differential diagnosis for a headache with visual aura could include..."
|
||||
negative_prompts:
|
||||
- "Recommend a good recipe for a vegetarian lasagna."
|
||||
- "Give an overview of the French Revolution."
|
||||
- "Explain how a digital camera captures an image."
|
||||
- "What are the environmental impacts of deforestation?"
|
||||
- "The recent advancements in artificial intelligence have led to developments in..."
|
||||
- "The fundamental concepts in economics include ideas like supply and demand, which explain..."
|
||||
- source_model: microsoft/Orca-2-7b
|
||||
positive_prompts:
|
||||
- "Here is a funny joke for you -"
|
||||
- "When considering the ethical implications of artificial intelligence, one must take into account..."
|
||||
- "In strategic planning, a company must analyze its strengths and weaknesses, which involves..."
|
||||
- "Understanding consumer behavior in marketing requires considering factors like..."
|
||||
- "The debate on climate change solutions hinges on arguments that..."
|
||||
negative_prompts:
|
||||
- "In discussing dietary adjustments for managing hypertension, it's crucial to emphasize..."
|
||||
- "For early detection of melanoma, dermatologists recommend that patients regularly check their skin for..."
|
||||
- "Explaining the importance of vaccination, a healthcare professional should highlight..."
|
||||
3
model-00001-of-00002.safetensors
Normal file
3
model-00001-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b7312033024738baff24be5c914396422b027380cc3cbc44cb6c0eb62b74cf94
|
||||
size 9976603184
|
||||
3
model-00001-of-00003.safetensors
Normal file
3
model-00001-of-00003.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:f1c840e1ed0c60c73ab21953cf450dee5362a2412db3411fdae858d649d08261
|
||||
size 9993266472
|
||||
3
model-00002-of-00002.safetensors
Normal file
3
model-00002-of-00002.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:89ae0a8853bb7c1d3029f8a3bb295d8ba70835180159f5edd840fd28d24d947a
|
||||
size 3500310752
|
||||
3
model-00002-of-00003.safetensors
Normal file
3
model-00002-of-00003.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ae2432a8a64091cc18c79124794cdebdef9b2d0dfe6a6d466398f53e4d77e57e
|
||||
size 9993214272
|
||||
3
model-00003-of-00003.safetensors
Normal file
3
model-00003-of-00003.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:24f8c33dd495ac7f6bb3210049edf02737817427591f78ce81257a78abad30bc
|
||||
size 2148019232
|
||||
1
model.safetensors.index.json
Normal file
1
model.safetensors.index.json
Normal file
File diff suppressed because one or more lines are too long
24
special_tokens_map.json
Normal file
24
special_tokens_map.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": "<s>",
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
93425
tokenizer.json
Normal file
93425
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
BIN
tokenizer.model
(Stored with Git LFS)
Normal file
Binary file not shown.
66
tokenizer_config.json
Normal file
66
tokenizer_config.json
Normal file
@@ -0,0 +1,66 @@
|
||||
{
|
||||
"add_bos_token": true,
|
||||
"add_eos_token": false,
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "</s>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32000": {
|
||||
"content": "[PAD]",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"32001": {
|
||||
"content": "<|im_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"32002": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": true,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
}
|
||||
},
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"legacy": false,
|
||||
"model_max_length": 4096,
|
||||
"pad_token": "<s>",
|
||||
"padding_side": "left",
|
||||
"sp_model_kwargs": {},
|
||||
"spaces_between_special_tokens": false,
|
||||
"tokenizer_class": "LlamaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": true
|
||||
}
|
||||
Reference in New Issue
Block a user