初始化项目，由ModelHub XC社区提供模型

Model: abideen/NexoNimbus-MoE-2x7B Source: Original Platform
2026-10-18 07:14:17 +08:00
commit 23e869ca74
23 changed files with 91507 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,120 @@
+---
+license: apache-2.0
+tags:
+- moe
+- merge
+- abideen/NexoNimbus-7B
+- mlabonne/NeuralMarcoro14-7B
+language:
+- en
+library_name: transformers
+---
+
+# NexoNimbus-MoE-2x7B
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/64e380b2e12618b261fa6ba0/_bzC6xkVIHW0tSigBxUI3.png)
+
+NexoNimbus-MoE-2x7B is a Mixure of Experts (MoE) made with the following models:
+* [abideen/NexoNimbus-7B](https://huggingface.co/abideen/NexoNimbus-7B)
+* [mlabonne/NeuralMarcoro14-7B](https://huggingface.co/mlabonne/NeuralMarcoro14-7B)
+
+🏆 Evaluation
+NexoNimbus-MoE-2x7B is the 10th best-performing 13B LLM on the Open LLM Leaderboard:
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/64e380b2e12618b261fa6ba0/z8E728H5fJqVtKNeGuwjX.png)
+
+
+|    Task     |Version| Metric |Value|   |Stderr|
+|-------------|------:|--------|----:|---|-----:|
+|arc_challenge|      0|acc     |62.28|±  |  1.41|
+|             |       |acc_norm|66.80|±  |  1.37|
+|hellaswag    |      0|acc     |66.83|±  |  0.46|
+|             |       |acc_norm|85.66|±  |  0.34|
+|gsm8k        |      0|acc     |53.52|±  |  1.37|
+|winogrande   |      0|acc     |81.53|±  |  1.09|
+|mmlu         |      0|acc     |64.51|±  |  1.00|
+
+Average: 67.51%
+
+### TruthfulQA
+|    Task     |Version|Metric|Value|   |Stderr|
+|-------------|------:|------|----:|---|-----:|
+|truthfulqa_mc|      1|mc1   |35.98|±  |  1.68|
+|             |       |mc2   |53.05|±  |  1.53|
+
+
+## 🧩 Configuration
+
+```yaml
+base_model: teknium/OpenHermes-2.5-Mistral-7B
+gate_mode: hidden
+dtype: bfloat16
+experts:
+  - source_model: abideen/NexoNimbus-7B
+    positive_prompts:
+    - "Mathematics"
+    - "Physics"
+    - "Chemistry"
+    - "Biology"
+    - "Medicine"
+    - "Engineering"
+    - "Computer Science"
+
+    negative_prompts:
+    - "History"
+    - "Philosophy"
+    - "Linguistics"
+    - "Literature"
+    - "Art and Art History"
+    - "Music Theory and Composition"
+    - "Performing Arts (Theater, Dance)"
+
+  - source_model: mlabonne/NeuralMarcoro14-7B 
+    positive_prompts:
+    - "Earth Sciences (Geology, Meteorology, Oceanography)"
+    - "Environmental Science"
+    - "Astronomy and Space Science"
+    - "Psychology"
+    - "Sociology"
+    - "Anthropology"
+    - "Political Science"
+    - "Economics"
+    negative_prompts:
+    - "Education"
+    - "Law"
+    - "Theology and Religious Studies"
+    - "Communication Studies"
+    - "Business and Management"
+    - "Agricultural Sciences"
+    - "Nutrition and Food Science"
+    - "Sports Science"
+```
+
+## 💻 Usage
+
+Here's a [Colab notebook](https://colab.research.google.com/drive/1B1Q7vO95cDkEJbKIPhOWr6exB9-Q_lr-?usp=sharing) to run NexoNimbus-MoE-2x7B in 4-bit precision on a free T4 GPU.
+
+```python
+!pip install -qU transformers bitsandbytes accelerate
+
+from transformers import AutoTokenizer
+import transformers
+import torch
+
+model = "abideen/NexoNimbus-MoE-2x7B"
+
+tokenizer = AutoTokenizer.from_pretrained(model)
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model,
+    model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
+)
+
+messages = [{"role": "user", "content": "Explain what is data science."}]
+prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+print(outputs[0]["generated_text"])
+```
+
+"Data science is an interdisciplinary field that combines mathematics, statistics, computer science, and domain expertise in order to extract meaningful insights and knowledge from structured and unstructured data. It involves the process of collecting, cleaning, transforming, analyzing, and visualizing data in order to identify patterns, trends, and relationships that can inform decision-making and drive business strategies. Data scientists use various tools and techniques, such as machine learning, deep learning, and natural language processing, to develop predictive models, optimize processes, and automate decision-making. The field of data science is rapidly evolving as more and more data is generated and the demand for data-driven insights continues to grow."
--- a/added_tokens.json
+++ b/added_tokens.json
@@ -0,0 +1,4 @@
+{
+  "<|im_end|>": 32000,
+  "<|im_start|>": 32001
+}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "teknium/OpenHermes-2.5-Mistral-7B",
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 32000,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mixtral",
+  "num_attention_heads": 32,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "num_local_experts": 2,
+  "output_router_logits": false,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "router_aux_loss_coef": 0.001,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.36.2",
+  "use_cache": false,
+  "vocab_size": 32002
+}
--- a/mergekit_moe_config.yml
+++ b/mergekit_moe_config.yml
@@ -0,0 +1,43 @@
+
+base_model: teknium/OpenHermes-2.5-Mistral-7B
+gate_mode: hidden
+dtype: bfloat16
+experts:
+  - source_model: abideen/NexoNimbus-7B
+    positive_prompts:
+    - "Mathematics"
+    - "Physics"
+    - "Chemistry"
+    - "Biology"
+    - "Medicine"
+    - "Engineering"
+    - "Computer Science"
+
+    negative_prompts:
+    - "History"
+    - "Philosophy"
+    - "Linguistics"
+    - "Literature"
+    - "Art and Art History"
+    - "Music Theory and Composition"
+    - "Performing Arts (Theater, Dance)"
+
+  - source_model: mlabonne/NeuralMarcoro14-7B 
+    positive_prompts:
+    - "Earth Sciences (Geology, Meteorology, Oceanography)"
+    - "Environmental Science"
+    - "Astronomy and Space Science"
+    - "Psychology"
+    - "Sociology"
+    - "Anthropology"
+    - "Political Science"
+    - "Economics"
+    negative_prompts:
+    - "Education"
+    - "Law"
+    - "Theology and Religious Studies"
+    - "Communication Studies"
+    - "Business and Management"
+    - "Agricultural Sciences"
+    - "Nutrition and Food Science"
+    - "Sports Science"
--- a/model-00001-of-00013.safetensors
+++ b/model-00001-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:228396042caec524cb9a8d977125d891567269cee62398be60c76dcd42a2fd6a
+size 1933882680
--- a/model-00002-of-00013.safetensors
+++ b/model-00002-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5b5fc4f7677994e99775128466e242702bdf386da063abe9d702253f6a03224
+size 1996490960
--- a/model-00003-of-00013.safetensors
+++ b/model-00003-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69cb7edeeb8f513db672035c45022c027342d1e7bea7ca5408072105d5e2bc0a
+size 1996490968
--- a/model-00004-of-00013.safetensors
+++ b/model-00004-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c86c56959986d6ad96279fcc7909d41fc3bc2d57e376b72195ca4be203e2749b
+size 1996490968
--- a/model-00005-of-00013.safetensors
+++ b/model-00005-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b61584e421014bfee9675b8faf0ac9a76a428253644215ca551fb591cfa47a31
+size 1996490952
--- a/model-00006-of-00013.safetensors
+++ b/model-00006-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f65552884c3aacb22be1f5c14c26e47eb7fd2adc0cb1ea83ea4d03bde3cb029c
+size 1996490960
--- a/model-00007-of-00013.safetensors
+++ b/model-00007-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4168cd0ed9b2c738c7228bb95e85810aef52768917c1c78f1deee50955eb02d
+size 1996490968
--- a/model-00008-of-00013.safetensors
+++ b/model-00008-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e7ab8143cb55c579b686470e9ddc96f2c2ded92c2030acac6e8ec3f71017f49
+size 1996490960
--- a/model-00009-of-00013.safetensors
+++ b/model-00009-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50d3260eb3b6b46fa7efad2fca87be2388fe0046b2b9bef5a1901ab5e32eaed4
+size 1996490952
--- a/model-00010-of-00013.safetensors
+++ b/model-00010-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68a30025f88f4752d2a37eec6baf8c6dfa7d4f4e23c03feaa640b010d53932c8
+size 1996490968
--- a/model-00011-of-00013.safetensors
+++ b/model-00011-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bffc5751540340f4bdc253876c511d44061778561dbe40730b1e735cb9f4f395
+size 1996490968
--- a/model-00012-of-00013.safetensors
+++ b/model-00012-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd6f4b2739b9ac422249f8f1212cb97acffae273a96ff74de605d29e03662e75
+size 1979981568
--- a/model-00013-of-00013.safetensors
+++ b/model-00013-of-00013.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ce7bcebfb9bf3e6fd501a27252699851ef4b91efbc9945b04ab18c96b2d0b3b
+size 1879588416
--- a/model.safetensors.index.json
+++ b/model.safetensors.index.json
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer.model
+++ b/tokenizer.model
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,61 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32000": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32001": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "trust_remote_code": false,
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true,
+  "use_fast": true
+}