steven0226/qwen2.5-0.5b-dpo-ultrafeedback
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: steven0226/qwen2.5-0.5b-dpo-ultrafeedback
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-08-27 10:21:20 +08:00
commit 67f867100a
9 changed files with 482 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
.gitattributes vendored Normal file
View File

@@ -0,0 +1,37 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text
training_curves.png filter=lfs diff=lfs merge=lfs -text

282
README.md Normal file
View File

@@ -0,0 +1,282 @@
---
license: apache-2.0
base_model: Qwen/Qwen2.5-0.5B-Instruct
datasets:
- HuggingFaceH4/ultrafeedback_binarized
library_name: transformers
pipeline_tag: text-generation
tags:
- dpo
- trl
- peft
- qwen2.5
- preference-alignment
---
# qwen2.5-0.5b-dpo-ultrafeedback
`Qwen/Qwen2.5-0.5B-Instruct` 以 **DPO（Direct Preference Optimization）** 在
[HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized)（`train_prefs` split）的
6000 組偏好資料對上做偏好對齊。
## 方法：DPO 是什麼？
DPO 直接用「同一個 prompt 的較好回答（chosen）與較差回答（rejected）」成對訓練，
不需要先訓練獨立的 reward model、也不需要 PPO 的強化學習迴圈。它把隱式獎勵定義為
policy 與 reference 模型的 log 機率比值：r(x, y) = beta * log(pi(y|x) / pi_ref(y|x))，
訓練目標是把 chosen 與 rejected 的隱式獎勵差距拉大。
**beta = 0.1**：控制 policy 可以偏離 reference 的程度——beta 越大越保守
（緊貼 reference），越小越激進（對齊訊號強、但容易偏離原模型導致輸出劣化）。
本 repo 存放的是 **merged 模型**：LoRA adapter（r=16）已用 `merge_and_unload()`
併回 base 權重（bfloat16），推論時**不需要 PEFT**，直接 `from_pretrained` 即可。
## 超參數
| beta | learning rate | LoRA r / alpha | 等效 batch | epochs | max_length | dtype |
|------|---------------|----------------|-----------|--------|------------|-------|
| 0.1 | 5e-06 | 16 / 32 | 16 | 1 | 1024 | bfloat16 |
訓練框架：TRL `DPOTrainer`（loss_type=sigmoid）。資料前處理：套 Qwen chat template
轉為標準格式、過濾無明確偏好（score 平手／反轉）與過長樣本。
## 訓練曲線
最終 rewards/accuracies（滾動平均）：**0.639**
![training curves](training_curves.png)
## Before / After 對照（greedy decoding）
**Prompt:** What are the health benefits of intermittent fasting? Are there any risks I should know about?
<details><summary>原始模型</summary>
Intermittent fasting is a dietary approach that involves restricting food intake to only certain times of the day or week and then allowing yourself to eat normally during other times. This can help people lose weight and improve their overall health in several ways:
1. Weight loss: Intermittent fasting has been shown to be effective at helping people lose weight. It can lead to significant reductions in body mass index (BMI) and fat percentage.
2. Improved metabolism: By reducing calorie intake, intermittent fasting can increase your metabolic rate, which means you burn more calories throughout the day.
3. Reduced inflammation: Studies have found that intermittent fasting can reduce levels of inflammatory markers in the blood, which may help prevent chronic diseases like heart disease and diabetes.
4. Better sleep quality: Some studies suggest that intermittent fasting can improve sleep patterns and reduce symptoms of insomnia.
5. Increased energy levels: People who follow an intermittent fasting diet often report feeling more energized and less tired after eating.
However, it's important to note that intermittent fasting carries some risks:
1. Nutrient deficiencies: If not properly planned, intermittent fasting can lead to nutrient deficiencies, especially if you're not consuming enough calories from protein and healthy fats.
2. Hormonal changes: Some research suggests that intermittent fasting might
</details>
<details><summary>DPO 後模型</summary>
Intermittent fasting (IF) is a dietary regime that involves alternating between periods of eating and fasting. While it has gained popularity in recent years for its potential health benefits, including weight loss, improved insulin sensitivity, and reduced inflammation, there are also some concerns and risks associated with this approach.
### Health Benefits of Intermittent Fasting
1. **Weight Loss**: Studies have shown that intermittent fasting can lead to significant weight loss compared to traditional calorie-controlled diets. This is because fasting triggers the release of hormones like ghrelin and leptin, which help regulate appetite and satiety.
2. **Improved Insulin Sensitivity**: Research indicates that intermittent fasting may improve insulin sensitivity, meaning your body's ability to use glucose more efficiently. This can be beneficial for people with type 2 diabetes or those at risk of developing it.
3. **Reduced Risk of Chronic Diseases**: Some studies suggest that intermittent fasting might reduce the risk of certain chronic diseases such as cardiovascular disease, type 2 diabetes, and some types of cancer.
4. **Enhanced Metabolism**: By reducing food intake during the fast period, intermittent fasting can increase metabolic rate, potentially leading to weight loss without significantly altering daily caloric intake.
5. **Stress Reduction**: Many
</details>
**Prompt:** Explain the difference between supervised and unsupervised learning to a high school student.
<details><summary>原始模型</summary>
Sure! Let's break down the differences between supervised and unsupervised learning in simple terms:
### Supervised Learning
Supervised learning is like when you're trying to guess what color a picture is based on its features. Imagine you have a bunch of pictures of different types of flowers, and each one has a specific color. If I give you a new picture that looks similar to some of these flower pictures, you can use this information to figure out which type of flower it might be.
In supervised learning, we train our model (like a computer program) using labeled data. This means we provide it with examples of correct answers (the "labels") and then ask it to make guesses about what the answer should be for new, unseen examples. The goal is to learn patterns or relationships between the input features and the output labels.
For example:
- **Classification**: Suppose you want to classify images into two categories: cats or dogs.
- **Regression**: Predicting how long a car will travel based on its speed.
### Unsupervised Learning
Unsupervised learning is like when you're trying to find hidden patterns in your data without any clear labels. It's great for finding groups within your data where there isn't an obvious way to tell them apart from others
</details>
<details><summary>DPO 後模型</summary>
Sure! Let's break down the differences between supervised and unsupervised learning in simple terms:
### Supervised Learning
Supervised learning is like when you're trying to predict something based on what you already know or have data about it. Imagine you have a toy car that has wheels, but you don't know exactly how many wheels it has yet. With supervised learning, your computer learns from examples of cars with different numbers of wheels (like 4, 6, 8, etc.) and tries to figure out which number of wheels makes the most sense for this particular car.
- **Example:** If you have a dataset of cars with different numbers of wheels, your computer can learn to guess the correct number of wheels for any new car.
### Unsupervised Learning
Unsupervised learning is like when you're trying to find patterns or groups within your data without knowing beforehand what those patterns are. Think of it as clustering - grouping similar items together, even if they don't look alike at first glance.
- **Example:** In a group of people who own different types of pets (cats, dogs, birds), your computer could use unsupervised learning to group them into categories such as "dogs," "birds," "cat owners," etc., just
</details>
**Prompt:** I have chicken, rice, and broccoli. Give me a simple 30-minute dinner recipe.
<details><summary>原始模型</summary>
Sure! Here's a simple 30-minute dinner recipe for you:
### Chicken Rice Bowl
#### Ingredients:
- 1 whole chicken (about 2 pounds)
- 4 cups of high-quality brown rice
- 1 cup of frozen mixed vegetables (such as carrots, peas, corn, and bell peppers)
- 1/2 cup of chopped onion
- 1/2 cup of chopped garlic
- 1 tablespoon of olive oil
- Salt and pepper to taste
- Optional: 1/2 teaspoon of dried thyme or rosemary for garnish
#### Instructions:
1. **Preheat the Oven:** Preheat your oven to 375°F (190°C).
2. **Cook the Chicken:** Place the chicken in a large skillet over medium-high heat. Cook until the skin is crispy and the internal temperature reaches 165°F (74°C). Remove the chicken from the skillet and set it aside.
3. **Prepare the Rice:** In a large pot, combine the brown rice with the frozen vegetables. Bring to a boil, then reduce the heat to low, cover, and simmer for about 15 minutes, stirring occasionally, until the rice is tender but still slightly firm.
4.
</details>
<details><summary>DPO 後模型</summary>
Certainly! Here's a simple and delicious 30-minute dinner recipe for you:
### Chicken Rice Bowl
**Ingredients:**
- 1 cup cooked brown rice
- 2 tablespoons olive oil or vegetable oil
- 1 large onion, finely chopped
- 3 cloves garlic, minced
- 1 medium carrot, peeled and grated
- 1 small bell pepper, sliced
- 1 cup fresh spinach leaves (or any green leafy vegetables)
- 1/4 cup chopped fresh parsley
- Salt and pepper to taste
- 1 tablespoon lemon juice
- 1 teaspoon dried oregano
- 1 teaspoon dried basil
- 1/2 teaspoon red pepper flakes (optional)
**Instructions:**
1. **Prepare the Vegetables:** In a large skillet over medium heat, cook the onion and garlic until soft and fragrant. Add the grated carrot and bell pepper, and sauté for about 5 minutes.
2. **Cook the Rice:** In a large pot, bring the water to a boil. Add the cooked brown rice and reduce the heat to low. Cover the pot and let it simmer for about 15-20 minutes, or until the rice is tender but still slightly firm to the
</details>
**Prompt:** Write a short professional email asking my manager for two days off next week.
<details><summary>原始模型</summary>
Subject: Request for Two Days Off
Dear [Manager's Name],
I hope this message finds you well.
I am writing to request two days off from work on the following Monday and Tuesday of next week. I believe that taking time off will allow me to recharge and ensure that I can handle any additional responsibilities or commitments that may arise during this period.
Please let me know if there is anything specific I need to do in advance to make sure I have everything ready for when I return to work. Additionally, I would appreciate it if we could schedule a meeting with you to discuss the details of my request and any other relevant matters.
Thank you very much for your consideration. I look forward to hearing back from you soon.
Best regards,
[Your Full Name]
[Your Position]
[Your Contact Information]
</details>
<details><summary>DPO 後模型</summary>
Subject: Request for Two Days Off
Dear [Manager's Name],
I hope this message finds you well.
I am writing to request two days off from work on the following Monday and Tuesday of next week. I believe that taking time off will allow me to recharge, reflect on my responsibilities, and ensure that I can contribute effectively to our team during these periods.
The reason behind my request is as follows:
1. **Personal Health**: I have been feeling overwhelmed with work-related stress and need some time to relax and rejuvenate.
2. **Team Dynamics**: I believe it would be beneficial for us all to take a break from our daily tasks and focus on building stronger relationships within the team.
3. **Professional Development**: Taking a day off allows me to review my skills and knowledge, ensuring that I remain up-to-date with industry trends and best practices.
I understand that your schedule may be tight, but I assure you that I will make every effort to manage my time efficiently while still being available to assist you in any way possible.
Thank you very much for considering my request. I look forward to hearing back from you soon.
Best regards,
[Your Full Name]
[Your Position]
[Your Contact Information]
</details>
**Prompt:** What would happen if the Moon suddenly disappeared? Walk me through the consequences.
<details><summary>原始模型</summary>
If the Moon suddenly disappeared from our solar system, it would have profound and far-reaching effects on Earth and its inhabitants. Here's a detailed breakdown of what might happen:
1. **Global Climate Change**: The absence of the Moon would cause a significant shift in Earth's climate. Without the gravitational pull of the Moon, the planet would experience a drastic change in temperature patterns. This could lead to extreme weather events such as droughts, floods, and heatwaves.
2. **Tidal Forces**: The Moon's gravity would exert tidal forces on Earth, causing tides that would be much stronger than they currently are. This could result in more frequent and severe flooding, especially in coastal areas.
3. **Earth's Rotation**: The Moon's gravity would also affect Earth's rotation. As the Moon orbits around the Earth, it would cause the Earth to spin faster or slower depending on where it is relative to the Moon. This could lead to different seasons and potentially even changes in the length of day and night.
4. **Changes in Ocean Circulation**: The Moon's gravity would alter ocean currents and circulation patterns. This could lead to changes in sea levels, affecting coastal ecosystems and potentially leading to saltwater intrusion into freshwater sources.
5. **Weather Patterns**: The Moon's
</details>
<details><summary>DPO 後模型</summary>
If the Moon suddenly disappeared from our solar system, it would have profound and far-reaching effects on Earth and its inhabitants. Here's a detailed breakdown of what might happen:
1. **Global Climate Change**: The absence of the Moon would cause significant changes in Earth's climate. Without the gravitational pull of the Moon, the planet would experience less tidal forces, leading to more frequent and intense storms, hurricanes, and other weather phenomena.
2. **Tidal Forces**: Tides would become weaker or disappear entirely due to the lack of gravity. This could result in higher sea levels, which would lead to flooding and coastal erosion issues for many regions around the world.
3. **Earth's Rotation**: The rotation of the Earth would slow down significantly. This is because the Moon's gravity would no longer provide the necessary centripetal force to keep the Earth spinning at its current rate. As a result, the Earth would take much longer to complete one full rotation, potentially causing seasons to be shorter than they currently are.
4. **Changes in Weather Patterns**: The Moon's absence would also affect the Earth's weather patterns. The lack of the Moon's gravitational influence would mean that the Earth's atmosphere would not maintain its normal circulation, leading to unpredictable weather conditions such as droughts,
</details>
## 如何使用
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("steven0226/qwen2.5-0.5b-dpo-ultrafeedback", dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("steven0226/qwen2.5-0.5b-dpo-ultrafeedback")
```
由 [tun0000/qwen-dpo-alignment](https://github.com/tun0000/qwen-dpo-alignment) 的 `dpo_colab.ipynb` 訓練產生。

54
chat_template.jinja Normal file
View File

@@ -0,0 +1,54 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role }}
{%- if message.content %}
{{- '\n' + message.content }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{{- tool_call.arguments | tojson }}
{{- '}\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}

57
config.json Normal file
View File

@@ -0,0 +1,57 @@
{
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": null,
"dtype": "bfloat16",
"eos_token_id": 151645,
"hidden_act": "silu",
"hidden_size": 896,
"initializer_range": 0.02,
"intermediate_size": 4864,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 32768,
"max_window_layers": 21,
"model_type": "qwen2",
"num_attention_heads": 14,
"num_hidden_layers": 24,
"num_key_value_heads": 2,
"pad_token_id": 151643,
"rms_norm_eps": 1e-06,
"rope_parameters": {
"rope_theta": 1000000.0,
"rope_type": "default"
},
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "5.13.0",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

13
generation_config.json Normal file
View File

@@ -0,0 +1,13 @@
{
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"repetition_penalty": 1.1,
"temperature": 0.7,
"top_k": 20,
"top_p": 0.8,
"transformers_version": "5.13.0"
}

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8f64016684765eae29f50d52ea7d60b88512d7a6b784fb41dfbcbb1c542ba6c2
size 988097824

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
size 11421892

30
tokenizer_config.json Normal file
View File

@@ -0,0 +1,30 @@
{
"add_prefix_space": false,
"backend": "tokenizers",
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"is_local": false,
"local_files_only": false,
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

3
training_curves.png Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:342416e901d5f408d9063a9320a0c302d39ad47e34a5f00099b85e77eeeceb5f
size 153238