初始化项目，由ModelHub XC社区提供模型

Model: jackf857/Llama-3.2-1B-Instruct-DPO-HH Source: Original Platform
2026-09-07 01:31:14 +08:00
commit 8d27ef5d7c
8 changed files with 357 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,156 @@
+---
+license: llama3.2
+base_model: meta-llama/Llama-3.2-1B-Instruct
+datasets:
+  - Anthropic/hh-rlhf
+language:
+  - en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - dpo
+  - preference-optimization
+  - rlhf
+  - llama-3.2
+  - from-scratch
+---
+
+# Llama-3.2-1B-Instruct-DPO-HH
+
+`meta-llama/Llama-3.2-1B-Instruct` aligned with **Direct Preference Optimization** on Anthropic HH-RLHF,
+using a from-scratch DPO implementation — no TRL, no `DPOTrainer`, no Axolotl, no
+Lightning. The loss, sequence scoring, completion masking, reference handling and
+training loop are all explicit PyTorch.
+
+This is research and learning code. Read the limitations before using it.
+
+## Training
+
+| | |
+|---|---|
+| Base model | `meta-llama/Llama-3.2-1B-Instruct` |
+| Objective | DPO (Rafailov et al., 2023), summed completion log-probabilities |
+| Reference | Frozen copy of the base model, live (not cached) |
+| Dataset | `Anthropic/hh-rlhf`, `train` split |
+| Pairs after parsing | 159,384 |
+| Pairs seen | **79,360** (50% of one epoch, 1,240 steps) |
+| Hardware | 1× NVIDIA H200 |
+
+### Hyperparameters
+
+| | |
+|---|---|
+| β | 0.1 |
+| Learning rate | 5e-07 |
+| Schedule | cosine to 0, 10% warmup |
+| Optimizer | AdamW, β₁ 0.9, β₂ 0.95, wd 0.0 |
+| Effective batch | 64 pairs (8 per device × 8 accumulation) |
+| Max grad norm | 1.0 |
+| Parameter dtype | **float32** (BF16 autocast for compute) |
+| Max length | 1024 (prompt 640, completion 384) |
+| Seed | 42 |
+
+**On precision.** Parameters are stored in FP32 and BF16 is used only for
+forward/backward compute. Storing *trainable* parameters in BF16 silently breaks
+preference tuning: at a weight of 0.01 the gap between representable BF16 values is
+6.1e-5, while an AdamW update at these learning rates is ~1e-6, so updates round to a
+no-op while the loss curve still looks plausible. This pipeline rejects BF16 parameter
+storage for training outright.
+
+## Results
+
+Training metrics only — **no held-out evaluation was run.** These are in-training
+statistics on the optimized data, not a measure of generalization.
+
+| steps | loss | reward margin | reward accuracy |
+|---|---|---|---|
+| 0–248 | 0.6809 | +0.0420 | 0.548 |
+| 248–496 | 0.6513 | +0.1620 | 0.596 |
+| 496–744 | 0.6453 | +0.1894 | 0.631 |
+| 744–992 | 0.6437 | +0.2152 | 0.618 |
+| 992–1240 | 0.6435 | +0.2027 | 0.627 |
+| **final** | **0.6455** | **+0.2002** | **0.615** |
+
+DPO's loss is exactly `log 2 = 0.693147` when policy and reference are identical. This
+run began at `0.690420` with implicit rewards of `-0.0112` /
+`-0.0176`, confirming the frozen reference and the completion masking were
+correct at step 0.
+
+**Where the margin comes from.** `chosen_reward` moved +0.1242 → +0.0333
+and `rejected_reward` +0.0595 → -0.1797. The separation is
+produced mainly by pushing the *rejected* responses below the reference, not by making the
+chosen ones more likely — the likelihood-displacement behaviour DPO is known for. Read the
+margin as "less likely to produce the dispreferred response", which is not the same claim
+as "more likely to produce the preferred one".
+
+## How large is the effect, really?
+
+Stated plainly, because a reward margin alone does not tell you this:
+
+- Relative weight change from the base model: **0.000354**
+- Greedy generations that are byte-identical to the base model: **0 of 4** spot-check prompts
+
+This is a *modest* perturbation of the base model, which is what half an epoch at lr 5e-7 with β=0.1 should produce — β exists precisely to keep the policy near its reference. On general prompts the outputs are recognisably the base model's, with differences in phrasing and formatting. Do not expect a dramatically different assistant.
+
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "jackf857/Llama-3.2-1B-Instruct-DPO-HH"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
+
+messages = [{"role": "user", "content": "Explain why the sky is blue, briefly."}]
+ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
+out = model.generate(ids.to(model.device), max_new_tokens=128)
+print(tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
+```
+
+The Llama 3 chat template injects the current date unless `date_string` is pinned.
+Training used `date_string="26 Jul 2024"`.
+
+## Data processing
+
+HH transcripts were parsed into canonical `(messages, chosen, rejected)` triples with no
+chat markup, then rendered through the official Llama 3 chat template. Only the final
+assistant response is scored; the completion mask is exactly
+`[0]*prompt_len + [1]*completion_len`, enforced by the prompt-prefix invariant.
+
+Verified against the real Llama 3 tokenizer: exactly one BOS per sequence, 0% BPE merges
+across the prompt/completion boundary, every completion ending in `<|eot_id|>`.
+
+
+
+## Limitations
+
+- **No held-out evaluation.** Every number above is a training metric. There is no
+  evidence here that this model is better than its base — only that DPO optimized what it
+  was asked to optimize.
+- **50% of one epoch on a 1B model.** A short run on a small model.
+- **HH-RLHF is noisy.** Preference labels are known to be inconsistent, and many pairs
+  have no clear quality difference. Some labels prefer epistemic humility ("I don't know")
+  over confident answers, so the model may become more hedging.
+- **Safety is not established.** No safety evaluation was performed. Do not deploy where
+  harmful output matters. Inherits all limitations of the base model.
+- English only; 1B models hallucinate readily.
+
+## License
+
+Governed by the [Llama 3.2 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/LICENSE),
+inherited from the base model. That license requires derivative model names to begin with
+"Llama". Anthropic HH-RLHF is MIT licensed.
+
+## Citation
+
+```bibtex
+@inproceedings{rafailov2023direct,
+  title     = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
+  author    = {Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and
+               Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea},
+  booktitle = {Advances in Neural Information Processing Systems},
+  year      = {2023}
+}
+```
--- a/chat_template.jinja
+++ b/chat_template.jinja
@@ -0,0 +1,93 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+    {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+    {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+    {%- if strftime_now is defined %}
+        {%- set date_string = strftime_now("%d %b %Y") %}
+    {%- else %}
+        {%- set date_string = "26 Jul 2024" %}
+    {%- endif %}
+{%- endif %}
+{%- if not tools is defined %}
+    {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+    {%- set system_message = messages[0]['content']|trim %}
+    {%- set messages = messages[1:] %}
+{%- else %}
+    {%- set system_message = "" %}
+{%- endif %}
+
+{#- System message #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if tools is not none %}
+    {{- "Environment: ipython\n" }}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+    {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+    {#- Extract the first user message so we can plug it in here #}
+    {%- if messages | length != 0 %}
+        {%- set first_user_message = messages[0]['content']|trim %}
+        {%- set messages = messages[1:] %}
+    {%- else %}
+        {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+{%- endif %}
+    {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+    {{- "Given the following functions, please respond with a JSON for a function call " }}
+    {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+    {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
+    {{- "Do not use variables.\n\n" }}
+    {%- for t in tools %}
+        {{- t | tojson(indent=4) }}
+        {{- "\n\n" }}
+    {%- endfor %}
+    {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+    {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+        {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+    {%- elif 'tool_calls' in message %}
+        {%- if not message.tool_calls|length == 1 %}
+            {{- raise_exception("This model only supports single tool-calls at once!") }}
+        {%- endif %}
+        {%- set tool_call = message.tool_calls[0].function %}
+        {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+        {{- '{"name": "' + tool_call.name + '", ' }}
+        {{- '"parameters": ' }}
+        {{- tool_call.arguments | tojson }}
+        {{- "}" }}
+        {{- "<|eot_id|>" }}
+    {%- elif message.role == "tool" or message.role == "ipython" %}
+        {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+        {%- if message.content is mapping or message.content is iterable %}
+            {{- message.content | tojson }}
+        {%- else %}
+            {{- message.content }}
+        {%- endif %}
+        {{- "<|eot_id|>" }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}
--- a/config.json
+++ b/config.json
@@ -0,0 +1,40 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 128000,
+  "dtype": "float32",
+  "eos_token_id": [
+    128001,
+    128008,
+    128009
+  ],
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 8192,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 16,
+  "num_key_value_heads": 8,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "factor": 32.0,
+    "high_freq_factor": 4.0,
+    "low_freq_factor": 1.0,
+    "original_max_position_embeddings": 8192,
+    "rope_theta": 500000.0,
+    "rope_type": "llama3"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "vocab_size": 128256
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,12 @@
+{
+  "bos_token_id": 128000,
+  "do_sample": true,
+  "eos_token_id": [
+    128001,
+    128008,
+    128009
+  ],
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "5.14.1"
+}
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6a7864675a5083cd8256abb667850286d9ab823bd0f9d8dc059c6b4317b6695
+size 4943274328
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|begin_of_text|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|eot_id|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 131072,
+  "tokenizer_class": "TokenizersBackend"
+}