---
license: mit
language:
- zh
library_name: transformers
pipeline_tag: text-generation
datasets:
- opencsg/Fineweb-Edu-Chinese-V2.2
metrics:
- perplexity
- accuracy
tags:
- pytorch
- transformers
- causal-lm
- qwen3
- chinese
- from-scratch
- attention-residuals
- baseline
- standard-residual
---

# Attention Residuals 100M Baseline

This is the 100M baseline checkpoint for the attention-residuals-reproduction project.
It uses a standard Qwen3-style decoder-only Transformer with standard residual connections, trained from scratch on Chinese data.

## Model Details

- Mode: `baseline`
- Architecture: Qwen3-style causal language model
- Residual type: standard residual connection
- Hidden size: 512
- Layers: 12
- Attention heads: 8
- KV heads: 4
- FFN intermediate size: 1536
- Sequence length: 2048
- Training steps: 20,000
- Training data: `opencsg/Fineweb-Edu-Chinese-V2.2`

## Intended Use

This checkpoint is mainly intended for research comparison with Attention Residuals variants.
It is not instruction-tuned and should not be used as a chat model.

## Evaluation

| Metric | Result |
|--------|--------|
| Chinese Held-out PPL | 128.58 |
| C-Eval Acc | 0.2664 |
| CMMLU Acc | 0.2594 |

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "你的用户名/attention-residuals-100M-baseline"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "人工智能的发展"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))