---
license: apache-2.0
language:
  - zh
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - qwen
  - moe
  - causal-lm
  - text-generation
base_model:
  - Qwen/Qwen3-4B-Instruct-2507
---

# Qwen3-3B-A0.9B

This repository contains the current best checkpoint from a local Qwen3-style MoE architecture exploration focused on a lightweight conversational baseline.

## Overview

- Model name: `Qwen3-Lite-3B-0.9B`
- Model type: causal language model
- Architecture family: Qwen3-style MoE
- Intended use: lightweight experimentation, architecture recovery, simple short-form dialogue and QA smoke testing
- Training status: research checkpoint, not a fully aligned production assistant

## Files

- Model weights in Hugging Face format
- Architecture config: `qwen3_3p1b_a0p85b_moe_30biso_4l.json`
- Recovery finetune config: `recover_dialogue_qwen3_3p1b_30biso_recovery_cn_v1.yaml`
- Smoke evaluation snapshot: `candidate_v1_smoke_suite.json`

## Architecture Summary

- Base family: Qwen3 MoE
- Hidden size: `2048`
- Layers: `4`
- Attention heads: `32`
- KV heads: `4`
- Experts: `128`
- Active experts per token: `8`
- MoE intermediate size: `768`
- Dense intermediate size: `6144`
- Dtype: `bfloat16`

This checkpoint keeps the official Qwen-style export layout so it can be loaded with standard Hugging Face workflows.

## Current Best Local Status

This upload corresponds to the checkpoint currently documented as the best working local baseline in:

- `README.md`
- `docs/stage1/qwen3_moe_4layer_recovery.md`

Its practical status is:

- It can handle simple QA and part of short Chinese dialogue.
- It is not yet a fully repaired dialogue model.
- Later recovery branches did not consistently outperform this baseline.

## Limitations

- The model is still a recovery-oriented research checkpoint rather than a finished instruct model.
- Dialogue stability is limited on longer turns and emotionally nuanced prompts.
- Benchmark coverage is incomplete relative to official large-scale release evaluation.
- Safety alignment and refusal behavior should not be assumed to match official Qwen releases.

## Tokenizer

The tokenizer used during local experiments is the official Qwen tokenizer from the Qwen3-4B-Instruct release.

Tokenizer files are not re-exported in this checkpoint bundle because local training followed the same save style as the official weight export workflow.

## Example

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "refinefuture-ai/Qwen3-Lite-3B-0.9B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

prompt = "请用中文做一个简短的自我介绍。"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```