初始化项目，由ModelHub XC社区提供模型

Model: macadeliccc/laser-polyglot-4x7b
Source: Original Platform

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+polyglot.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,273 @@
+---
+language:
+- ja
+- en
+- zh
+license: apache-2.0
+model-index:
+- name: laser-polyglot-4x7b
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: AI2 Reasoning Challenge (25-Shot)
+      type: ai2_arc
+      config: ARC-Challenge
+      split: test
+      args:
+        num_few_shot: 25
+    metrics:
+    - type: acc_norm
+      value: 64.16
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: HellaSwag (10-Shot)
+      type: hellaswag
+      split: validation
+      args:
+        num_few_shot: 10
+    metrics:
+    - type: acc_norm
+      value: 84.98
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MMLU (5-Shot)
+      type: cais/mmlu
+      config: all
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 63.88
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: TruthfulQA (0-shot)
+      type: truthful_qa
+      config: multiple_choice
+      split: validation
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: mc2
+      value: 55.47
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: Winogrande (5-shot)
+      type: winogrande
+      config: winogrande_xl
+      split: validation
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 77.82
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: GSM8k (5-shot)
+      type: gsm8k
+      config: main
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 48.45
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=macadeliccc/laser-polyglot-4x7b
+      name: Open LLM Leaderboard
+---
+# Polyglot-4x7b-24b
+
+![polyglot](polyglot.png)
+
+Polyglot-4x7b is a Mixture of Experts approach to a multilingual model.
+
+This project is an experiment to see if each expert can be of a different language. The answer is yes.
+
+The model is a merge of models that are capable of Chinese and Japanese output.
+
++ teknium/OpenHermes-2.5-Mistral-7B
++ oshizo/japanese-e5-mistral-7b_slerp
++ cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser
++ s3nh/Mistral-7B-Evol-Instruct-Chinese
+
+TODO:
+1. [] polyglot tokenizer
+
+## Other polyglot models
+
++ [macadeliccc/Polyglot-8x7b-v0.1](https://huggingface.co/macadeliccc/Polyglot-8x7b-v0.1) (adds 3 more languages)
+# Code Example 
+
+Inference [Colab](https://colab.research.google.com/drive/1tYSb63IKZDsiQ5BIJU8Oc92phxugAmB3?usp=sharing)
+Live demo available on [Spaces](https://huggingface.co/spaces/macadeliccc/polyglot-4x7b-chat?logs=build)
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+def generate_response(prompt):
+    """
+    Generate a response from the model based on the input prompt.
+
+    Args:
+    prompt (str): Prompt for the model.
+
+    Returns:
+    str: The generated response from the model.
+    """
+    # Tokenize the input prompt
+    inputs = tokenizer(prompt, return_tensors="pt")
+
+    # Generate output tokens
+    outputs = model.generate(**inputs, max_new_tokens=256, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id)
+
+    # Decode the generated tokens to a string
+    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+    return response
+
+# Load the model and tokenizer
+model_id = "macadeliccc/laser-polyglot-4x7b"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True)
+
+# Example prompts in different languages
+english_prompt = "Write a quicksort algorithm in python"
+chinese_prompt = "用Python写一个快速排序算法"
+japanese_prompt = "Pythonでクイックソートアルゴリズムを書いてください"
+
+# Generate and print responses for each language
+print("English Response:")
+print(generate_response(english_prompt), "\n")
+
+print("Chinese Response:")
+print(generate_response(chinese_prompt), "\n")
+
+print("Japanese Response:")
+print(generate_response(japanese_prompt), "\n")
+
+```
+
+## Example Output
+English: 
+
+  Write a quicksort algorithm in python.
+  
+  ```python
+  def quicksort(arr):
+      if len(arr) <= 1:
+          return arr
+      else:
+          pivot = arr[0]
+          less = [i for i in arr[1:] if i <= pivot]
+          greater = [i for i in arr[1:] if i > pivot]
+          return quicksort(less) + [pivot] + quicksort(greater)
+  
+  arr = [5, 2, 9, 1, 5, 7, 4, 8, 6, 3]
+  print(quicksort(arr))
+  ```
+  
+  This is a simple implementation of the quicksort algorithm in python. The function `quicksort` takes an array as input and returns a sorted array. The algorithm works by selecting a pivot element from the array and partitioning the other elements into two sub-arrays, according to whether they are less than or greater than the pivot. The process is then repeated recursively on the sub-arrays until the entire array is sorted. 
+
+Chinese Response:
+
+  用Python写一个快速排序算法
+  
+  ```python
+  def quick_sort(arr):
+      if len(arr) <= 1:
+          return arr
+      else:
+          pivot = arr[0]
+          less = [i for i in arr[1:] if i <= pivot]
+          greater = [i for i in arr[1:] if i > pivot]
+          return quick_sort(less) + [pivot] + quick_sort(greater)
+  
+  arr = [3, 5, 2, 1, 4, 6, 8, 7]
+  print(quick_sort(arr))
+  ```
+  这个程序的时间复杂度为O(nlogn)，空间复杂度为O(n)。 
+
+Japanese Response: 
+
+  Pythonでクイックソートアルゴリズムを書いてください。
+
+  ```python
+  def quicksort(arr):
+      if len(arr) <= 1:
+          return arr
+      pivot = arr[0]
+      left = [x for x in arr[1:] if x < pivot]
+      right = [x for x in arr[1:] if x >= pivot]
+      return quicksort(left) + [pivot] + quicksort(right)
+  
+  print(quicksort([3,6,8,10,1,5,9,2,4,7]))
+  ```
+  
+  このコードはクイックソートアルゴリズムを実装しています。クイックソートは一種の分割と conquers アルゴリズムで、配列を分割し、それぞれの部分配列を再帰的にソートします。
+  
+  この実装では、配列の最初の要素をピボットとして使用します。そして、配列を2つの 
+
+
+
+# Evaluations
+
+|    Tasks    |Version|Filter|n-shot| Metric |Value |   |Stderr|
+|-------------|-------|------|-----:|--------|-----:|---|-----:|
+|arc_challenge|Yaml   |none  |     0|acc     |0.5495|±  |0.0145|
+|             |       |none  |     0|acc_norm|0.5794|±  |0.0144|
+|arc_easy     |Yaml   |none  |     0|acc     |0.8304|±  |0.0077|
+|             |       |none  |     0|acc_norm|0.8068|±  |0.0081|
+|boolq        |Yaml   |none  |     0|acc     |0.8749|±  |0.0058|
+|hellaswag    |Yaml   |none  |     0|acc     |0.6276|±  |0.0048|
+|             |       |none  |     0|acc_norm|0.8157|±  |0.0039|
+|openbookqa   |Yaml   |none  |     0|acc     |0.3180|±  |0.0208|
+|             |       |none  |     0|acc_norm|0.4460|±  |0.0223|
+|piqa         |Yaml   |none  |     0|acc     |0.8139|±  |0.0091|
+|             |       |none  |     0|acc_norm|0.8237|±  |0.0089|
+|winogrande   |Yaml   |none  |     0|acc     |0.7419|±  |0.0123|
+# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
+Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_macadeliccc__laser-polyglot-4x7b)
+
+|             Metric              |Value|
+|---------------------------------|----:|
+|Avg.                             |65.79|
+|AI2 Reasoning Challenge (25-Shot)|64.16|
+|HellaSwag (10-Shot)              |84.98|
+|MMLU (5-Shot)                    |63.88|
+|TruthfulQA (0-shot)              |55.47|
+|Winogrande (5-shot)              |77.82|
+|GSM8k (5-shot)                   |48.45|
+

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<|im_start|>": 32000
+}

config.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser",
+  "architectures": [
+    "MixtralForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mixtral",
+  "num_attention_heads": 32,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "num_local_experts": 4,
+  "output_router_logits": false,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "router_aux_loss_coef": 0.001,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.36.2",
+  "use_cache": false,
+  "vocab_size": 32001
+}

mergekit_moe_config.yml

@@ -0,0 +1,28 @@
+base_model: cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser
+gate_mode: hidden
+dtype: bfloat16
+experts:
+  - source_model: teknium/OpenHermes-2.5-Mistral-7B
+    positive_prompts:
+      - "Science"
+      - "Philosophy"
+      - "Reasoning"
+      - "Summarization" 
+
+  - source_model: oshizo/japanese-e5-mistral-7b_slerp
+    positive_prompts:
+      - "Japanese"
+
+
+  - source_model: cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser
+    positive_prompts:
+      - "Guidance"
+      - "Management"
+
+  - source_model: s3nh/Mistral-7B-Evol-Instruct-Chinese
+    positive_prompts:
+      - "Chinese"
+ 
+     
+    
+

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:272d4262e3141adeb42c4e74cd7be91ba9bdffd20758e4f3bbf091490dd6e4ea
+size 9919830088

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39501a627c420d4133738779e4a860c129b228c03386783325d0fbc15170d49d
+size 9982454720

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0daaf6571289f3b14cc3279e3b230263bf2a9eff3ff3ebe0ce638c2066136d5e
+size 9982454752

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26d94c42781a6d53ec72c7c992898c70fe6430da729069bd98d1de7b180e04c0
+size 9982454720

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d161895595fdc637bbc669ca4e4bd92982cb307045e9f50e9d023b24d63abe0
+size 8440279464

polyglot.png

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18a4c2be3655dde57dc6dafe9d34bfc02b3ac396d45bc134074616107d4f457c
+size 2541970

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}