100 lines
2.9 KiB
Markdown
100 lines
2.9 KiB
Markdown
---
|
|
library_name: transformers
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- qwen3
|
|
- causal-lm
|
|
- safetensors
|
|
- bf16
|
|
---
|
|
|
|
# Qwen3 Causal Language Model (BF16)
|
|
|
|
This repository contains a decoder-only causal language model using the Qwen3
|
|
architecture. The weights are stored in sharded Safetensors format using BF16
|
|
precision.
|
|
|
|
## Model details
|
|
|
|
| Item | Value |
|
|
| --- | --- |
|
|
| Architecture | `Qwen3ForCausalLM` |
|
|
| Model type | `qwen3` |
|
|
| Parameters | 8,190,735,360 |
|
|
| Weight precision | BF16 |
|
|
| Hidden size | 4096 |
|
|
| Layers | 36 |
|
|
| Attention heads | 32 |
|
|
| Key/value heads | 8 |
|
|
| Vocabulary size | 151,936 |
|
|
| Configured context length | 40,960 tokens |
|
|
| Weight format | 7 sharded Safetensors files (about 5 GB each) |
|
|
|
|
The tokenizer configuration currently declares a maximum length of 131,072,
|
|
while the model configuration declares 40,960 positions. Until the longer
|
|
context length is independently validated, use 40,960 tokens as the supported
|
|
maximum.
|
|
|
|
## Usage
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
model_id = "YOUR_USERNAME/YOUR_MODEL_NAME"
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
model_id,
|
|
torch_dtype=torch.bfloat16,
|
|
device_map="auto",
|
|
attn_implementation="sdpa",
|
|
)
|
|
|
|
messages = [{"role": "user", "content": "你好，请介绍一下你自己。"}]
|
|
text = tokenizer.apply_chat_template(
|
|
messages,
|
|
tokenize=False,
|
|
add_generation_prompt=True,
|
|
)
|
|
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
|
|
|
with torch.inference_mode():
|
|
output = model.generate(**inputs, max_new_tokens=256)
|
|
|
|
answer = output[0, inputs.input_ids.shape[1]:]
|
|
print(tokenizer.decode(answer, skip_special_tokens=True))
|
|
```
|
|
|
|
BF16 weights require roughly 16 GB just for the model parameters, plus
|
|
additional memory for inference state and the KV cache. Use a machine with
|
|
sufficient RAM or accelerator memory. The example selects SDPA so that
|
|
FlashAttention 2 is not required; installations with FlashAttention 2 may
|
|
choose `attn_implementation="flash_attention_2"` instead.
|
|
|
|
## Training and provenance
|
|
|
|
The files supplied with this model do not identify the following information:
|
|
|
|
- Base model or checkpoint
|
|
- Training or fine-tuning method
|
|
- Training datasets
|
|
- Evaluation results
|
|
- Model author or organization
|
|
|
|
The uploader should add these details before publishing so users can assess the
|
|
model's capabilities, limitations, and provenance.
|
|
|
|
## License
|
|
|
|
No license information was included with the supplied model files. The uploader
|
|
must verify the licenses of the base model, training data, and resulting weights
|
|
and add the correct license before making the repository public. Absence of a
|
|
license does not grant permission to use, modify, or redistribute the model.
|
|
|
|
## Limitations
|
|
|
|
No benchmark, safety, bias, or long-context evaluation results were supplied.
|
|
Do not rely on this model for high-impact decisions without appropriate testing
|
|
and human review.
|