---
base_model: unsloth/Qwen3-4B-Instruct-2507
datasets:
- u-10bei/structured_data_with_cot_dataset_512_v2
- u-10bei/structured_data_with_cot_dataset_512_v4
- u-10bei/structured_data_with_cot_dataset_512_v5
- u-10bei/structured_data_with_cot_dataset_512
- u-10bei/structured_data_with_cot_dataset_v2
- u-10bei/structured_data_with_cot_dataset
- u-10bei/dpo-dataset-qwen-cot
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- sft
- dpo
- unsloth
- qwen
- reasoning
- structured-data
---

# Qwen3-4B-StructEval-Iterative-Alignment

This model is a highly optimized version of **Qwen3-4B-Instruct-2507**, specifically developed for the **StructEval-T** task. It features a sophisticated iterative alignment process to maximize performance in structured data reasoning and Chain-of-Thought (CoT) generation.

This repository contains **full-merged 16-bit weights**. No adapter loading is required.

## Training Pipeline
Unlike standard fine-tuning, this model has undergone a **four-stage iterative training process** to ensure precise alignment and deep reasoning capabilities:

1. **Stage 1: SFT** - Foundation building with structured CoT trajectories.
2. **Stage 2: DPO** - First alignment to preference reasoning paths.
3. **Stage 3: SFT** - Knowledge reinforcement and format refinement.
4. **Stage 4: DPO** - Final preference optimization for high-fidelity structured outputs.

## Training Objective
The model is engineered to excel in:
- **Complex Reasoning**: Enhanced Chain-of-Thought processing for structural evaluation.
- **Structural Integrity**: Strict adherence to complex data formats (JSON, Tables, etc.).
- **Consistency**: High-reliability outputs across iterative multi-turn interactions.

## Training Configuration (Final Stage)
- **Method**: Iterative DPO (Direct Preference Optimization)
- **Base model**: unsloth/Qwen3-4B-Instruct-2507
- **Epochs**: 1
- **Learning rate**: 3e-06
- **Beta**: 0.05
- **Max sequence length**: 2560
- **Platform**: Trained with Unsloth

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "moushi21/dpo-qwen-cot-merged20"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
```

## Sources & Terms (IMPORTANT)

Training data:
- u-10bei/structured_data_with_cot_dataset_512_v2
- u-10bei/structured_data_with_cot_dataset_512_v4
- u-10bei/structured_data_with_cot_dataset_512_v5
- u-10bei/structured_data_with_cot_dataset_512
- u-10bei/structured_data_with_cot_dataset_v2
- u-10bei/structured_data_with_cot_dataset
- u-10bei/dpo-dataset-qwen-cot

Dataset License: MIT License. This dataset is used and distributed under the terms of the MIT License.
Compliance: Users must comply with the MIT license (including copyright notice) and the base model's original terms of use.