232 lines
9.1 KiB
Markdown
232 lines
9.1 KiB
Markdown
---
|
|
license: apache-2.0
|
|
base_model:
|
|
- Qwen/Qwen3-0.6B
|
|
- suayptalha/Qwen3-0.6B-Code-Expert
|
|
- suayptalha/Qwen3-0.6B-Math-Expert
|
|
- suayptalha/Qwen3-0.6B-Medical-Expert
|
|
library_name: transformers
|
|
license_link: https://huggingface.co/Qwen/Qwen3-0.6B/blob/main/LICENSE
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- moe
|
|
---
|
|
# huihui-ai/Huihui-MoE-1B-A0.6B
|
|
|
|
## Model Overview
|
|
Huihui-MoE-1B-A0.6B is a **Mixture of Experts (MoE)** language model developed by **huihui.ai**, built upon the **[Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)** base model. It enhances the standard Transformer architecture by replacing MLP layers with MoE layers, each containing 3 experts, to achieve high performance with efficient inference. The model is designed for natural language processing tasks, including text generation, question answering, and conversational applications.
|
|
|
|
This version does not support ollama because tie_word_embeddings=True results in the absence of lm_head parameters being saved; therefore, ollama cannot be used. If ollama support is required, please choose the latest version [huihui-ai/Huihui-MoE-1.2B-A0.6B](https://huggingface.co/huihui-ai/Huihui-MoE-1.2B-A0.6B).
|
|
|
|
- **Architecture**: Qwen3MoeForCausalLM model with 3 experts per layer (num_experts=3), activating 1 expert per token (num_experts_per_tok=1).
|
|
- **Total Parameters**: ~1.1 billion (1B)
|
|
- **Activated Parameters**: ~0.62 billion (0.6B) during inference, comparable to Qwen3-0.6B
|
|
- **Developer**: huihui.ai
|
|
- **Release Date**: June 2025
|
|
- **License**: Inherits the license of the Qwen3 base model (apache-2.0)
|
|
|
|
## Expert Models:
|
|
|
|
### Coding:
|
|
[suayptalha/Qwen3-0.6B-Code-Expert](https://huggingface.co/suayptalha/Qwen3-0.6B-Code-Expert)
|
|
|
|
This model was fully fine-tuned with BF16 on first 20k rows of `nvidia/OpenCodeReasoning` dataset for 1 epoch.
|
|
|
|
### Math:
|
|
[suayptalha/Qwen3-0.6B-Math-Expert](https://huggingface.co/suayptalha/Qwen3-0.6B-Math-Expert)
|
|
|
|
This model was fully fine-tuned with BF16 on entire `unsloth/OpenMathReasoning-mini` dataset for 1 epoch.
|
|
|
|
### Medical:
|
|
[suayptalha/Qwen3-0.6B-Medical-Expert](https://huggingface.co/suayptalha/Qwen3-0.6B-Medical-Expert)
|
|
|
|
This model was fully fine-tuned with BF16 on first 20k rows of `FreedomIntelligence/medical-o1-reasoning-SFT` dataset for 1 epoch.
|
|
|
|
### Instruction Following:
|
|
[Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B)
|
|
|
|
`Qwen/Qwen3-0.6B` model was directly used for this expert, no fine-tune was applied.
|
|
|
|
## Training
|
|
|
|
- **Base Model**: Qwen3-0.6B, pre-trained by the Qwen team, Experts, pre-trained by the Suayptalha team.
|
|
- **Conversion**: The model copies embeddings, self-attention, and normalization weights from Qwen3-0.6B, replacing MLP layers with MoE layers (3 experts). Gating weights are randomly initialized.
|
|
- **Fine-Tuning**: Not fine-tuned; users are recommended to fine-tune for specific tasks to optimize expert routing. The fine-tuned version is already available and can be referred to as [huihui-ai/Huihui-MoE-1B-A0.6B-SFT](https://huggingface.co/huihui-ai/Huihui-MoE-1B-A0.6B-SFT).
|
|
|
|
## Usage
|
|
|
|
```
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TextStreamer
|
|
import torch
|
|
import os
|
|
import signal
|
|
|
|
cpu_count = os.cpu_count()
|
|
print(f"Number of CPU cores in the system: {cpu_count}")
|
|
half_cpu_count = cpu_count // 2
|
|
os.environ["MKL_NUM_THREADS"] = str(half_cpu_count)
|
|
os.environ["OMP_NUM_THREADS"] = str(half_cpu_count)
|
|
torch.set_num_threads(half_cpu_count)
|
|
|
|
print(f"PyTorch threads: {torch.get_num_threads()}")
|
|
print(f"MKL threads: {os.getenv('MKL_NUM_THREADS')}")
|
|
print(f"OMP threads: {os.getenv('OMP_NUM_THREADS')}")
|
|
|
|
# Load the model and tokenizer
|
|
NEW_MODEL_ID = "huihui-ai/Huihui-MoE-1B-A0.6B"
|
|
print(f"Load Model {NEW_MODEL_ID} ... ")
|
|
quant_config_4 = BitsAndBytesConfig(
|
|
load_in_4bit=True,
|
|
bnb_4bit_compute_dtype=torch.bfloat16,
|
|
bnb_4bit_use_double_quant=True,
|
|
llm_int8_enable_fp32_cpu_offload=True,
|
|
)
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
NEW_MODEL_ID,
|
|
device_map="auto",
|
|
trust_remote_code=True,
|
|
#quantization_config=quant_config_4,
|
|
torch_dtype=torch.bfloat16
|
|
)
|
|
tokenizer = AutoTokenizer.from_pretrained(NEW_MODEL_ID, trust_remote_code=True)
|
|
if tokenizer.pad_token is None:
|
|
tokenizer.pad_token = tokenizer.eos_token
|
|
tokenizer.pad_token_id = tokenizer.eos_token_id
|
|
|
|
initial_messages = [{"role": "system", "content": "You are a helpful assistant."}]
|
|
messages = initial_messages.copy()
|
|
enable_thinking = True
|
|
skip_prompt=True
|
|
skip_special_tokens=True
|
|
|
|
class CustomTextStreamer(TextStreamer):
|
|
def __init__(self, tokenizer, skip_prompt=True, skip_special_tokens=True):
|
|
super().__init__(tokenizer, skip_prompt=skip_prompt, skip_special_tokens=skip_special_tokens)
|
|
self.generated_text = ""
|
|
self.stop_flag = False
|
|
|
|
def on_finalized_text(self, text: str, stream_end: bool = False):
|
|
self.generated_text += text
|
|
print(text, end="", flush=True)
|
|
if self.stop_flag:
|
|
raise StopIteration
|
|
|
|
def stop_generation(self):
|
|
self.stop_flag = True
|
|
|
|
def generate_stream(model, tokenizer, messages, enable_thinking, skip_prompt, skip_special_tokens, max_new_tokens):
|
|
input_ids = tokenizer.apply_chat_template(
|
|
messages,
|
|
tokenize=True,
|
|
enable_thinking = enable_thinking,
|
|
add_generation_prompt=True,
|
|
return_tensors="pt"
|
|
)
|
|
attention_mask = torch.ones_like(input_ids, dtype=torch.long)
|
|
tokens = input_ids.to(model.device)
|
|
attention_mask = attention_mask.to(model.device)
|
|
|
|
streamer = CustomTextStreamer(tokenizer, skip_prompt=skip_prompt, skip_special_tokens=skip_special_tokens)
|
|
|
|
def signal_handler(sig, frame):
|
|
streamer.stop_generation()
|
|
print("\n[Generation stopped by user with Ctrl+C]")
|
|
|
|
signal.signal(signal.SIGINT, signal_handler)
|
|
|
|
print("Response: ", end="", flush=True)
|
|
try:
|
|
generated_ids = model.generate(
|
|
tokens,
|
|
attention_mask=attention_mask,
|
|
#use_cache=False,
|
|
max_new_tokens=max_new_tokens,
|
|
do_sample=True,
|
|
pad_token_id=tokenizer.pad_token_id,
|
|
streamer=streamer
|
|
)
|
|
del generated_ids
|
|
except StopIteration:
|
|
print("\n[Stopped by user]")
|
|
|
|
del input_ids, attention_mask
|
|
torch.cuda.empty_cache()
|
|
signal.signal(signal.SIGINT, signal.SIG_DFL)
|
|
|
|
return streamer.generated_text, streamer.stop_flag
|
|
|
|
while True:
|
|
user_input = input("User: ").strip()
|
|
if user_input.lower() == "/exit":
|
|
print("Exiting chat.")
|
|
break
|
|
if user_input.lower() == "/clear":
|
|
messages = initial_messages.copy()
|
|
print("Chat history cleared. Starting a new conversation.")
|
|
continue
|
|
if user_input.lower() == "/nothink":
|
|
if enable_thinking:
|
|
enable_thinking = False
|
|
print("Thinking = False.")
|
|
else:
|
|
enable_thinking = True
|
|
print("Thinking = True.")
|
|
continue
|
|
if user_input.lower() == "/skip_prompt":
|
|
if skip_prompt:
|
|
skip_prompt = False
|
|
print("skip_prompt = False.")
|
|
else:
|
|
skip_prompt = True
|
|
print("skip_prompt = True.")
|
|
continue
|
|
if user_input.lower() == "/skip_special_tokens":
|
|
if skip_special_tokens:
|
|
skip_special_tokens = False
|
|
print("skip_special_tokens = False.")
|
|
else:
|
|
skip_special_tokens = True
|
|
print("skip_special_tokens = True.")
|
|
continue
|
|
if not user_input:
|
|
print("Input cannot be empty. Please enter something.")
|
|
continue
|
|
messages.append({"role": "user", "content": user_input})
|
|
response, stop_flag = generate_stream(model, tokenizer, messages, enable_thinking, skip_prompt, skip_special_tokens, 14192)
|
|
print("", flush=True)
|
|
if stop_flag:
|
|
continue
|
|
messages.append({"role": "assistant", "content": response})
|
|
|
|
```
|
|
|
|
## Applications
|
|
|
|
- **Text Generation: Articles**, dialogues, and creative writing.
|
|
- **Question Answering**: Information retrieval and query resolution.
|
|
- **Conversational AI**: Multi-turn dialogues for chatbots.
|
|
- **Research**: Exploration of MoE architectures and efficient model scaling.
|
|
|
|
## Limitations
|
|
|
|
- **Fine-Tuning Required**: Randomly initialized gating weights may lead to suboptimal expert utilization without fine-tuning.
|
|
- **Compatibility**: Developed with transformers 4.52.4; ensure matching versions to avoid loading issues.
|
|
- **Inference Speed**: While efficient for an MoE model, performance depends on hardware (GPU recommended).
|
|
|
|
## Ethical Considerations
|
|
|
|
- **Bias**: Inherits potential biases from the Qwen3-0.6B base model; users should evaluate outputs for fairness.
|
|
- **Usage**: Intended for research and responsible applications; avoid generating harmful or misleading content.
|
|
|
|
## Contact
|
|
|
|
- **Developer**: huihui.ai
|
|
- **Repository**: huihui-ai/Huihui-MoE-1B-A0.6B (available locally or on Hugging Face)
|
|
- **Issues**: Report bugs or request features via the repository or please send an email to support@huihui.ai
|
|
|
|
## Acknowledgments
|
|
|
|
- Built upon the Qwen3-0.6B model by the Qwen team.
|
|
- Built upon the Experts model by the Suayptalha team.
|
|
- Powered by the Hugging Face transformers library. |