Technoculture/Medtulu-2x7b
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: Technoculture/Medtulu-2x7b
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-09-15 13:06:31 +08:00
commit 91a5723a45
13 changed files with 94012 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

84
README.md Normal file
View File

@@ -0,0 +1,84 @@
---
license: apache-2.0
tags:
- moe
- merge
- Technoculture/MT7Bi-dpo
- allenai/tulu-2-dpo-7b
---
# Medtulu-2x7b
Medtulu-2x7b is a Mixure of Experts (MoE) made with the following models:
* [Technoculture/MT7Bi-dpo](https://huggingface.co/Technoculture/MT7Bi-dpo)
* [allenai/tulu-2-dpo-7b](https://huggingface.co/allenai/tulu-2-dpo-7b)
## 🧩 Configuration
```yaml
base_model: Technoculture/MT7Bi-dpo
tokenizer_source: union
gate_mode: hidden
dtype: bfloat16
experts:
- source_model: Technoculture/MT7Bi-dpo
positive_prompts:
- "Are elevated serum levels of interleukin 21 associated with disease severity in patients with psoriasis?"
- "Which one of the following does NOT present antigens?"
- "A 25-year-old male patient presents to your clinic in significant distress. He states he has excruciating, stabbing pain around the left side of his head, and his left eye will not stop tearing. These types of headaches have been occurring for the past week every morning when he awakens and last around 60 minutes. He denies any aura, nausea, or vomiting. He denies any other past medical history. What is this patient's diagnosis?"
- "When using an inhaler, when should a patient be asked to rinse their mouth?"
- "What is the embryological origin of the hyoid bone?"
- "After what period of time does maximal dynamic exercise become predominantly aerobic?"
- source_model: allenai/tulu-2-dpo-7b
positive_prompts:
- "Who composed the tune of 'Twinkle, Twinkle, Little Star'?"
- "Gem went to get new supplies for her hamster and she found snacks and exercise balls She chose the _ because her hamster was fat."
- "John orders food for a massive restaurant. He orders 1000 pounds of beef for $8 per pound. He also orders twice that much chicken at $3 per pound. How much did everything cost?"
- "The gravitational force of the Sun affects the planets in our solar system. Which of these is influenced the most by this force?"
- "2sin(x) + yz ="
- "Hobbies and Crafts"
```
## Evaluations
| Benchmark | Medtulu-2x7b | Orca-2-7b | llama-2-7b | meditron-7b | meditron-70b |
| --- | --- | --- | --- | --- | --- |
| MedMCQA | | | | | |
| ClosedPubMedQA | | | | | |
| PubMedQA | | | | | |
| MedQA | | | | | |
| MedQA4 | | | | | |
| MedicationQA | | | | | |
| MMLU Medical | | | | | |
| MMLU | | | | | |
| TruthfulQA | | | | | |
| GSM8K | | | | | |
| ARC | | | | | |
| HellaSwag | | | | | |
| Winogrande | | | | | |
More details on the Open LLM Leaderboard evaluation results can be found here.
## 💻 Usage
```python
!pip install -qU transformers bitsandbytes accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "Technoculture/Medtulu-2x7b"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
)
messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```

19
added_tokens.json Normal file
View File

@@ -0,0 +1,19 @@
{
"<CLS>": 32000,
"<EOD>": 32002,
"<MASK>": 32003,
"<PAD>": 32004,
"<SEP>": 32001,
"[/bib]": 32010,
"[/bib_ref]": 32006,
"[/fig]": 32012,
"[/fig_ref]": 32008,
"[/formula]": 32016,
"[/table]": 32014,
"[bib]": 32009,
"[bib_ref]": 32005,
"[fig]": 32011,
"[fig_ref]": 32007,
"[formula]": 32015,
"[table]": 32013
}

33
config.json Normal file
View File

@@ -0,0 +1,33 @@
{
"_name_or_path": "Technoculture/MT7Bi-dpo",
"architectures": [
"MixtralForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 11008,
"max_position_embeddings": 2048,
"model_type": "mixtral",
"num_attention_heads": 32,
"num_experts_per_tok": 2,
"num_hidden_layers": 32,
"num_key_value_heads": 32,
"num_local_experts": 2,
"output_router_logits": false,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 10000.0,
"router_aux_loss_coef": 0.001,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.37.2",
"use_cache": false,
"vocab_size": 32017
}

22
mergekit_moe_config.yml Normal file
View File

@@ -0,0 +1,22 @@
base_model: Technoculture/MT7Bi-dpo
tokenizer_source: union
gate_mode: hidden
dtype: bfloat16
experts:
- source_model: Technoculture/MT7Bi-dpo
positive_prompts:
- "Are elevated serum levels of interleukin 21 associated with disease severity in patients with psoriasis?"
- "Which one of the following does NOT present antigens?"
- "A 25-year-old male patient presents to your clinic in significant distress. He states he has excruciating, stabbing pain around the left side of his head, and his left eye will not stop tearing. These types of headaches have been occurring for the past week every morning when he awakens and last around 60 minutes. He denies any aura, nausea, or vomiting. He denies any other past medical history. What is this patient's diagnosis?"
- "When using an inhaler, when should a patient be asked to rinse their mouth?"
- "What is the embryological origin of the hyoid bone?"
- "After what period of time does maximal dynamic exercise become predominantly aerobic?"
- source_model: allenai/tulu-2-dpo-7b
positive_prompts:
- "Who composed the tune of 'Twinkle, Twinkle, Little Star'?"
- "Gem went to get new supplies for her hamster and she found snacks and exercise balls She chose the _ because her hamster was fat."
- "John orders food for a massive restaurant. He orders 1000 pounds of beef for $8 per pound. He also orders twice that much chicken at $3 per pound. How much did everything cost?"
- "The gravitational force of the Sun affects the planets in our solar system. Which of these is influenced the most by this force?"
- "2sin(x) + yz ="
- "Hobbies and Crafts"

3
model-00001-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3ebff2ca05c7c24483906196e43a4c71666e2ee69314ea8cc19df202c238b938
size 9993495848

3
model-00002-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b823fe87b739a8c8bbfbf240fbcd3e7ce772afeb0f5492856eb71846c8c9f66e
size 9993214272

3
model-00003-of-00003.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:4ee08f47977c83a4fd9b4ba13a8170a792a2ee6e5b72e22dbb83f754871c6170
size 2148019232

1
model.safetensors.index.json Normal file
View File

File diff suppressed because one or more lines are too long

59
special_tokens_map.json Normal file
View File

@@ -0,0 +1,59 @@
{
"additional_special_tokens": [
"[bib_ref]",
"[/bib_ref]",
"[fig_ref]",
"[/fig_ref]",
"[bib]",
"[/bib]",
"[fig]",
"[/fig]",
"[table]",
"[/table]",
"[formula]",
"[/formula]"
],
"bos_token": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"cls_token": {
"content": "<CLS>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"mask_token": {
"content": "<MASK>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "<s>",
"sep_token": {
"content": "<SEP>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"unk_token": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

93551
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
tokenizer.model (Stored with Git LFS) Normal file
View File

Binary file not shown.

196
tokenizer_config.json Normal file
View File

@@ -0,0 +1,196 @@
{
"add_bos_token": true,
"add_eos_token": false,
"added_tokens_decoder": {
"0": {
"content": "<unk>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"1": {
"content": "<s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"2": {
"content": "</s>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32000": {
"content": "<CLS>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32001": {
"content": "<SEP>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32002": {
"content": "<EOD>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32003": {
"content": "<MASK>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32004": {
"content": "<PAD>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32005": {
"content": "[bib_ref]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32006": {
"content": "[/bib_ref]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32007": {
"content": "[fig_ref]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32008": {
"content": "[/fig_ref]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32009": {
"content": "[bib]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32010": {
"content": "[/bib]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32011": {
"content": "[fig]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32012": {
"content": "[/fig]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32013": {
"content": "[table]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32014": {
"content": "[/table]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32015": {
"content": "[formula]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"32016": {
"content": "[/formula]",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"[bib_ref]",
"[/bib_ref]",
"[fig_ref]",
"[/fig_ref]",
"[bib]",
"[/bib]",
"[fig]",
"[/fig]",
"[table]",
"[/table]",
"[formula]",
"[/formula]"
],
"bos_token": "<s>",
"clean_up_tokenization_spaces": false,
"cls_token": "<CLS>",
"eos_token": "</s>",
"legacy": true,
"mask_token": "<MASK>",
"model_max_length": 1000000000000000019884624838656,
"pad_token": "<s>",
"sep_token": "<SEP>",
"sp_model_kwargs": {},
"spaces_between_special_tokens": false,
"tokenizer_class": "LlamaTokenizer",
"trust_remote_code": false,
"unk_token": "<unk>",
"use_default_system_prompt": false,
"use_fast": true
}