73 lines
2.0 KiB
Markdown
73 lines
2.0 KiB
Markdown
---
|
|
library_name: transformers
|
|
license: apache-2.0
|
|
base_model: Qwen/Qwen3-4B-Base
|
|
tags:
|
|
- llama-factory
|
|
- full-finetuning
|
|
- qwen3
|
|
- qwen3-4b
|
|
- think
|
|
- ocr
|
|
- sft
|
|
model-index:
|
|
- name: think_s1_ep23
|
|
results: []
|
|
---
|
|
|
|
# Qwen3-4B Think S1 Ep23 (Full SFT)
|
|
|
|
Full-parameter supervised fine-tuning (SFT) of [Qwen/Qwen3-4B-Base](https://huggingface.co/Qwen/Qwen3-4B-Base) on the **ocr_think_50k** dataset with the Qwen3 chat template (think-style reasoning).
|
|
|
|
This checkpoint is **stage 1 episode 23**: training continued from an internal `think_s1` run at **checkpoint-302** (same base architecture), then fine-tuned for two additional epochs on ocr_think_50k.
|
|
|
|
## Model description
|
|
|
|
- **Method**: full SFT (all weights trainable), DeepSpeed ZeRO-3, 4 GPUs
|
|
- **Dataset**: ocr_think_50k
|
|
- **Template**: qwen3
|
|
- **Not LoRA / not QLoRA**: entire 4B model was updated
|
|
|
|
## Training details
|
|
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| Epochs | 2 |
|
|
| Seed | 42 |
|
|
| cutoff_len | 24576 |
|
|
| packing | true |
|
|
| neat_packing | false |
|
|
| per_device_train_batch_size | 1 |
|
|
| gradient_accumulation_steps | 16 |
|
|
| effective_batch_size | 64 |
|
|
| learning_rate | 5e-5 |
|
|
| train_loss | 0.5416 |
|
|
| train_steps | 604 |
|
|
| finished_at | 2026-06-10 05:23 CST |
|
|
|
|
Optimizer: AdamW (fused), cosine schedule, warmup ratio 0.1. Framework: Transformers 5.6.0, PyTorch 2.8.0+cu128.
|
|
|
|
## Related models
|
|
|
|
- **No-think SFT (same project)**: [modrill/qwen3-4b-nothink-s1-full-sft](https://huggingface.co/modrill/qwen3-4b-nothink-s1-full-sft) - OpenCodeInstruct, qwen3_nothink template
|
|
- **Base**: [Qwen/Qwen3-4B-Base](https://huggingface.co/Qwen/Qwen3-4B-Base)
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
repo_id = "modrill/qwen3-4b-think-s1-ep23-full-sft"
|
|
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
repo_id,
|
|
torch_dtype="auto",
|
|
device_map="auto",
|
|
trust_remote_code=True,
|
|
)
|
|
```
|
|
|
|
## License
|
|
|
|
Released under Apache 2.0 (see `LICENSE` in the upstream Qwen model card if not bundled here).
|