96 lines
2.9 KiB
Markdown
96 lines
2.9 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
language:
|
||
|
|
- zh
|
||
|
|
- en
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- qwen
|
||
|
|
- moe
|
||
|
|
- causal-lm
|
||
|
|
- text-generation
|
||
|
|
base_model:
|
||
|
|
- Qwen/Qwen3-4B-Instruct-2507
|
||
|
|
---
|
||
|
|
|
||
|
|
# Qwen3-3B-A0.9B
|
||
|
|
|
||
|
|
This repository contains the current best checkpoint from a local Qwen3-style MoE architecture exploration focused on a lightweight conversational baseline.
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
- Model name: `Qwen3-Lite-3B-0.9B`
|
||
|
|
- Model type: causal language model
|
||
|
|
- Architecture family: Qwen3-style MoE
|
||
|
|
- Intended use: lightweight experimentation, architecture recovery, simple short-form dialogue and QA smoke testing
|
||
|
|
- Training status: research checkpoint, not a fully aligned production assistant
|
||
|
|
|
||
|
|
## Files
|
||
|
|
|
||
|
|
- Model weights in Hugging Face format
|
||
|
|
- Architecture config: `qwen3_3p1b_a0p85b_moe_30biso_4l.json`
|
||
|
|
- Recovery finetune config: `recover_dialogue_qwen3_3p1b_30biso_recovery_cn_v1.yaml`
|
||
|
|
- Smoke evaluation snapshot: `candidate_v1_smoke_suite.json`
|
||
|
|
|
||
|
|
## Architecture Summary
|
||
|
|
|
||
|
|
- Base family: Qwen3 MoE
|
||
|
|
- Hidden size: `2048`
|
||
|
|
- Layers: `4`
|
||
|
|
- Attention heads: `32`
|
||
|
|
- KV heads: `4`
|
||
|
|
- Experts: `128`
|
||
|
|
- Active experts per token: `8`
|
||
|
|
- MoE intermediate size: `768`
|
||
|
|
- Dense intermediate size: `6144`
|
||
|
|
- Dtype: `bfloat16`
|
||
|
|
|
||
|
|
This checkpoint keeps the official Qwen-style export layout so it can be loaded with standard Hugging Face workflows.
|
||
|
|
|
||
|
|
## Current Best Local Status
|
||
|
|
|
||
|
|
This upload corresponds to the checkpoint currently documented as the best working local baseline in:
|
||
|
|
|
||
|
|
- `README.md`
|
||
|
|
- `docs/stage1/qwen3_moe_4layer_recovery.md`
|
||
|
|
|
||
|
|
Its practical status is:
|
||
|
|
|
||
|
|
- It can handle simple QA and part of short Chinese dialogue.
|
||
|
|
- It is not yet a fully repaired dialogue model.
|
||
|
|
- Later recovery branches did not consistently outperform this baseline.
|
||
|
|
|
||
|
|
## Limitations
|
||
|
|
|
||
|
|
- The model is still a recovery-oriented research checkpoint rather than a finished instruct model.
|
||
|
|
- Dialogue stability is limited on longer turns and emotionally nuanced prompts.
|
||
|
|
- Benchmark coverage is incomplete relative to official large-scale release evaluation.
|
||
|
|
- Safety alignment and refusal behavior should not be assumed to match official Qwen releases.
|
||
|
|
|
||
|
|
## Tokenizer
|
||
|
|
|
||
|
|
The tokenizer used during local experiments is the official Qwen tokenizer from the Qwen3-4B-Instruct release.
|
||
|
|
|
||
|
|
Tokenizer files are not re-exported in this checkpoint bundle because local training followed the same save style as the official weight export workflow.
|
||
|
|
|
||
|
|
## Example
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
|
|
||
|
|
model_id = "refinefuture-ai/Qwen3-Lite-3B-0.9B"
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(
|
||
|
|
model_id,
|
||
|
|
torch_dtype="auto",
|
||
|
|
device_map="auto",
|
||
|
|
trust_remote_code=True,
|
||
|
|
)
|
||
|
|
|
||
|
|
prompt = "请用中文做一个简短的自我介绍。"
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||
|
|
outputs = model.generate(**inputs, max_new_tokens=64)
|
||
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||
|
|
```
|