license, language, library_name, pipeline_tag, tags, base_model

license	language	library_name	pipeline_tag	tags	base_model
apache-2.0	zh en	transformers	text-generation	qwen moe causal-lm text-generation	Qwen/Qwen3-4B-Instruct-2507

Qwen3-3B-A0.9B

This repository contains the current best checkpoint from a local Qwen3-style MoE architecture exploration focused on a lightweight conversational baseline.

Overview

• Model name: Qwen3-Lite-3B-0.9B
• Model type: causal language model
• Architecture family: Qwen3-style MoE
• Intended use: lightweight experimentation, architecture recovery, simple short-form dialogue and QA smoke testing
• Training status: research checkpoint, not a fully aligned production assistant

Files

• Model weights in Hugging Face format
• Architecture config: qwen3_3p1b_a0p85b_moe_30biso_4l.json
• Recovery finetune config: recover_dialogue_qwen3_3p1b_30biso_recovery_cn_v1.yaml
• Smoke evaluation snapshot: candidate_v1_smoke_suite.json

Architecture Summary

• Base family: Qwen3 MoE
• Hidden size: 2048
• Layers: 4
• Attention heads: 32
• KV heads: 4
• Experts: 128
• Active experts per token: 8
• MoE intermediate size: 768
• Dense intermediate size: 6144
• Dtype: bfloat16

This checkpoint keeps the official Qwen-style export layout so it can be loaded with standard Hugging Face workflows.

Current Best Local Status

This upload corresponds to the checkpoint currently documented as the best working local baseline in:

• README.md
• docs/stage1/qwen3_moe_4layer_recovery.md

Its practical status is:

• It can handle simple QA and part of short Chinese dialogue.
• It is not yet a fully repaired dialogue model.
• Later recovery branches did not consistently outperform this baseline.

Limitations

• The model is still a recovery-oriented research checkpoint rather than a finished instruct model.
• Dialogue stability is limited on longer turns and emotionally nuanced prompts.
• Benchmark coverage is incomplete relative to official large-scale release evaluation.
• Safety alignment and refusal behavior should not be assumed to match official Qwen releases.

Tokenizer

The tokenizer used during local experiments is the official Qwen tokenizer from the Qwen3-4B-Instruct release.

Tokenizer files are not re-exported in this checkpoint bundle because local training followed the same save style as the official weight export workflow.

Example

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "refinefuture-ai/Qwen3-Lite-3B-0.9B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

prompt = "请用中文做一个简短的自我介绍。"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))