初始化项目，由ModelHub XC社区提供模型

Model: kenzrx/dpo-qwen-cot-merged
Source: Original Platform

README.md

@@ -0,0 +1,85 @@
+---
+base_model: kenzrx/qwen3-4b-sft-merged
+datasets:
+- structured_data_with_cot_dataset_v2
+- structured_data_with_cot_dataset_v2
+language:
+- en
+license: apache-2.0
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- qwen
+- unsloth
+- transformers
+- text-generation
+- lora
+- merged
+- dpo
+- alignment
+- sft
+---
+
+# qwen3-4b-instruct-2507-sft-dpo-qwen-cot-merged
+
+This repository provides **full-merged 16-bit weights** (no adapter loading required).
+
+## What this model is
+This model was trained in **two stages**:
+
+1) **SFT (Supervised Fine-Tuning)** to learn high-quality reference answers / formatting  
+2) **DPO (Direct Preference Optimization)** to align outputs toward preferred responses
+
+### Lineage
+- **Original base**: Qwen/Qwen3-4B-Instruct-2507
+- **Stage 1 (SFT) output (merged)**: kenzrx/qwen3-4b-sft-merged
+- **Stage 2 (DPO) output (this repo)**: merged 16-bit weights
+
+## Training Objective (DPO)
+The DPO stage optimizes the model to prefer **chosen** outputs over **rejected** outputs
+given the same prompt, improving response alignment and structured quality.
+
+## Training Configuration (DPO)
+- **Start model (SFT merged)**: kenzrx/qwen3-4b-sft-merged
+- **Method**: DPO (Direct Preference Optimization)
+- **Epochs**: 1
+- **Learning rate**: 1e-07
+- **Beta**: 0.1
+- **Max sequence length**: 1024
+- **LoRA Config (during training)**: r=8, alpha=16 (merged into final 16-bit weights)
+
+## Datasets
+- **SFT dataset**: structured_data_with_cot_dataset_v2  
+- **DPO preference dataset**: structured_data_with_cot_dataset_v2
+
+## Usage
+You can use this model directly with `transformers`.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "your_id/your-repo-name"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.float16,
+    device_map="auto",
+)
+
+prompt = "Your question here"
+messages = [
+    {"role": "user", "content": prompt},
+]
+
+inputs = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_tensors="pt",
+).to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=512)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))