commit 2c13ce153c5f57248506591a615e77a84b5b379e
Author: ModelHub XC <noreply@modelhub.org.cn>
Date:   Tue Jul 21 15:26:10 2026 +0800

    初始化项目，由ModelHub XC社区提供模型
    
    Model: MohdNihal03/qwen2.5-coder-1.5b-CodeSLM-Nihal
    Source: Original Platform

diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000..f82056b
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+loss_curve.png filter=lfs diff=lfs merge=lfs -text
+adapter/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..27c556e
--- /dev/null
+++ b/README.md
@@ -0,0 +1,106 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
+tags:
+  - code
+  - python
+  - qlora
+  - lora
+  - peft
+  - qwen2
+  - text-generation
+datasets:
+  - sahil2801/CodeAlpaca-20k
+language:
+  - en
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+# qwen2.5-coder-1.5b-CodeSLM-Nihal
+
+A **QLoRA fine-tune of Qwen2.5-Coder-1.5B-Instruct** into a **terse, code-first Python assistant**,
+built as a hands-on, phase-by-phase fine-tuning learning project on a single 8 GB consumer GPU
+(NVIDIA RTX 5060).
+
+The base model is a capable coder but chronically verbose — every answer trails a prose essay. This
+fine-tune trains it to reply with **correct, minimal code and nothing else.**
+
+## Headline result
+
+![Output length: 78% shorter after fine-tuning](./token_reduction.png)
+
+On 18 held-out prompts (greedy decoding), average output length dropped **from 272 to 60 tokens
+(−78%)** while core algorithm correctness was preserved.
+
+## Training curve
+
+![Training vs. validation loss](./loss_curve.png)
+
+| Checkpoint (epoch) | Validation loss |
+|---|---|
+| 0.13 | 0.5157 |
+| 0.38 | 0.5020 |
+| 0.88 | 0.4927 |
+| 1.00 | 0.4928 |
+
+Train and validation loss tracked each other the entire run — clean convergence, **no overfitting.**
+Final train loss **0.4906**; validation mean token-accuracy **~85.4%**.
+
+> These are training/next-token metrics plus a qualitative 18-prompt comparison. No standardized code
+> benchmark (HumanEval/MBPP) was run.
+
+## Training details
+
+| | |
+|---|---|
+| Base | `Qwen/Qwen2.5-Coder-1.5B-Instruct` |
+| Method | QLoRA (4-bit NF4 base + LoRA), completion-only loss |
+| Dataset | `sahil2801/CodeAlpaca-20k` → Qwen ChatML (19,020 train / 1,002 val) |
+| LoRA | r=16, α=32, dropout=0.05, targets q/k/v/o/gate/up/down |
+| Trainable params | 18.46M (~1.18%) |
+| Schedule | 1 epoch, 1,189 steps, LR 2e-4 cosine + 3% warmup |
+| Batch | 2 × 8 grad-accum = effective 16 |
+| Optimizer | paged_adamw_8bit, bf16, gradient checkpointing, max_len 1024 |
+| Hardware | 1× RTX 5060 (8 GB), ~50 min, peak VRAM 3.32 GB |
+
+## Files in this repo
+
+- **Root** — merged fp16 model (Transformers format); load directly with `from_pretrained`.
+- **`adapter/`** — the standalone LoRA adapter (~36 MB) to apply onto the base yourself.
+- **`gguf/`** — `…-f16.gguf` (full precision) and `…-Q4_K_M.gguf` (~986 MB) for llama.cpp / Ollama.
+
+## Usage
+
+### Transformers
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+m = "MohdNihal03/qwen2.5-coder-1.5b-CodeSLM-Nihal"
+tok = AutoTokenizer.from_pretrained(m)
+model = AutoModelForCausalLM.from_pretrained(m, device_map="auto")
+msgs = [{"role": "user", "content": "Write a Python function that reverses a string without slicing."}]
+ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
+print(tok.decode(model.generate(ids, max_new_tokens=128)[0][ids.shape[1]:], skip_special_tokens=True))
+```
+
+### Ollama
+```
+ollama run mohdnihalll03/qwen2.5-coder-1.5b-codeslm-nihal
+```
+
+Prompt format: Qwen2.5 ChatML. Suggested: `temperature 0.2`, stop on `<|im_start|>` / `<|im_end|>`.
+
+## Limitations
+
+- **Style over correctness:** fine-tuning changed formatting far more than correctness. A few subtle
+  base-model bugs persist (an email-regex character-class quirk; a `@timer` decorator missing
+  `functools.wraps`), and one bracket-matching answer regressed to a logic bug on empty-stack input.
+  **Review generated code before use.**
+- Python-focused; 1.5B params + 4-bit quantization — not a substitute for a large frontier model.
+- Occasional instruction drift (`print` vs `return`, tabulation vs memoization).
+
+## Attribution
+
+- Base: Qwen2.5-Coder-1.5B-Instruct (© Alibaba Cloud, Apache-2.0)
+- Data: `sahil2801/CodeAlpaca-20k`
+- Fine-tuned by **Nihal** as a QLoRA learning project (TRL / PEFT / bitsandbytes).
diff --git a/adapter/README.md b/adapter/README.md
new file mode 100644
index 0000000..6baa0c5
--- /dev/null
+++ b/adapter/README.md
@@ -0,0 +1,62 @@
+---
+base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
+library_name: peft
+model_name: lora-adapter
+tags:
+- base_model:adapter:Qwen/Qwen2.5-Coder-1.5B-Instruct
+- lora
+- sft
+- transformers
+- trl
+licence: license
+pipeline_tag: text-generation
+---
+
+# Model Card for lora-adapter
+
+This model is a fine-tuned version of [Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="None", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+ 
+
+
+
+This model was trained with SFT.
+
+### Framework versions
+
+- PEFT 0.19.1
+- TRL: 1.7.0
+- Transformers: 5.12.1
+- Pytorch: 2.11.0+cu128
+- Datasets: 5.0.0
+- Tokenizers: 0.22.2
+
+## Citations
+
+
+
+Cite TRL as:
+    
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```
\ No newline at end of file
diff --git a/adapter/adapter_config.json b/adapter/adapter_config.json
new file mode 100644
index 0000000..cb2cb51
--- /dev/null
+++ b/adapter/adapter_config.json
@@ -0,0 +1,48 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen2.5-Coder-1.5B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "v_proj",
+    "gate_proj",
+    "down_proj",
+    "q_proj",
+    "o_proj",
+    "k_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}
\ No newline at end of file
diff --git a/adapter/adapter_model.safetensors b/adapter/adapter_model.safetensors
new file mode 100644
index 0000000..804abd6
--- /dev/null
+++ b/adapter/adapter_model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc1c086d050c1087e0f503123de03e7ba81afca44876da8596a5abb91ec7a29b
+size 36981856
diff --git a/adapter/chat_template.jinja b/adapter/chat_template.jinja
new file mode 100644
index 0000000..64e3784
--- /dev/null
+++ b/adapter/chat_template.jinja
@@ -0,0 +1,59 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- generation %}
+        {{- message.content + '<|im_end|>' + '\n' }}
+        {%- endgeneration %}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}
diff --git a/adapter/tokenizer.json b/adapter/tokenizer.json
new file mode 100644
index 0000000..34510ff
--- /dev/null
+++ b/adapter/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
+size 11421892
diff --git a/adapter/tokenizer_config.json b/adapter/tokenizer_config.json
new file mode 100644
index 0000000..6028fd5
--- /dev/null
+++ b/adapter/tokenizer_config.json
@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}
diff --git a/adapter/training_args.bin b/adapter/training_args.bin
new file mode 100644
index 0000000..861f852
--- /dev/null
+++ b/adapter/training_args.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ce4ce8b4e89d28f67ace5f5e759fdd2a53979ebbb5a30ad399c82f6526bcca6
+size 5777
diff --git a/adapter/training_log.json b/adapter/training_log.json
new file mode 100644
index 0000000..ccf108f
--- /dev/null
+++ b/adapter/training_log.json
@@ -0,0 +1,1279 @@
+[
+  {
+    "loss": 0.6644901275634766,
+    "grad_norm": 0.69140625,
+    "learning_rate": 5e-05,
+    "entropy": 0.41775863207876685,
+    "num_tokens": 17886.0,
+    "mean_token_accuracy": 0.8308710962533951,
+    "epoch": 0.008412197686645636,
+    "step": 10
+  },
+  {
+    "loss": 0.5578639030456543,
+    "grad_norm": 0.6328125,
+    "learning_rate": 0.00010555555555555557,
+    "entropy": 0.5046202793717385,
+    "num_tokens": 35672.0,
+    "mean_token_accuracy": 0.8467379540205002,
+    "epoch": 0.016824395373291272,
+    "step": 20
+  },
+  {
+    "loss": 0.49953713417053225,
+    "grad_norm": 0.42578125,
+    "learning_rate": 0.0001611111111111111,
+    "entropy": 0.5429252408444881,
+    "num_tokens": 52882.0,
+    "mean_token_accuracy": 0.848870538175106,
+    "epoch": 0.025236593059936908,
+    "step": 30
+  },
+  {
+    "loss": 0.4875448703765869,
+    "grad_norm": 0.388671875,
+    "learning_rate": 0.00019999665919423514,
+    "entropy": 0.49813074097037313,
+    "num_tokens": 70413.0,
+    "mean_token_accuracy": 0.8602197140455246,
+    "epoch": 0.033648790746582544,
+    "step": 40
+  },
+  {
+    "loss": 0.5065551280975342,
+    "grad_norm": 0.396484375,
+    "learning_rate": 0.0001999372733012762,
+    "entropy": 0.5205996319651603,
+    "num_tokens": 87742.0,
+    "mean_token_accuracy": 0.8523474678397178,
+    "epoch": 0.04206098843322818,
+    "step": 50
+  },
+  {
+    "loss": 0.5336020946502685,
+    "grad_norm": 0.58984375,
+    "learning_rate": 0.0001998036980255298,
+    "entropy": 0.5422930240631103,
+    "num_tokens": 106667.0,
+    "mean_token_accuracy": 0.8395245373249054,
+    "epoch": 0.050473186119873815,
+    "step": 60
+  },
+  {
+    "loss": 0.584792947769165,
+    "grad_norm": 0.4453125,
+    "learning_rate": 0.00019959603252786895,
+    "entropy": 0.5817637398838997,
+    "num_tokens": 123772.0,
+    "mean_token_accuracy": 0.8319423466920852,
+    "epoch": 0.058885383806519455,
+    "step": 70
+  },
+  {
+    "loss": 0.5139331817626953,
+    "grad_norm": 0.314453125,
+    "learning_rate": 0.00019931443097074569,
+    "entropy": 0.5499239668250084,
+    "num_tokens": 142732.0,
+    "mean_token_accuracy": 0.8438791155815124,
+    "epoch": 0.06729758149316509,
+    "step": 80
+  },
+  {
+    "loss": 0.48215503692626954,
+    "grad_norm": 0.353515625,
+    "learning_rate": 0.000198959102403747,
+    "entropy": 0.5112259805202484,
+    "num_tokens": 161271.0,
+    "mean_token_accuracy": 0.8517454102635383,
+    "epoch": 0.07570977917981073,
+    "step": 90
+  },
+  {
+    "loss": 0.5477577209472656,
+    "grad_norm": 0.423828125,
+    "learning_rate": 0.00019853031060840508,
+    "entropy": 0.5490782242268324,
+    "num_tokens": 179202.0,
+    "mean_token_accuracy": 0.835928975045681,
+    "epoch": 0.08412197686645637,
+    "step": 100
+  },
+  {
+    "loss": 0.5046371936798095,
+    "grad_norm": 0.37890625,
+    "learning_rate": 0.0001980283739023765,
+    "entropy": 0.49583916887640955,
+    "num_tokens": 197597.0,
+    "mean_token_accuracy": 0.8535919323563576,
+    "epoch": 0.09253417455310199,
+    "step": 110
+  },
+  {
+    "loss": 0.4952244758605957,
+    "grad_norm": 0.421875,
+    "learning_rate": 0.00019745366490313607,
+    "entropy": 0.5152403030544519,
+    "num_tokens": 214819.0,
+    "mean_token_accuracy": 0.8491240337491035,
+    "epoch": 0.10094637223974763,
+    "step": 120
+  },
+  {
+    "loss": 0.5076200485229492,
+    "grad_norm": 0.390625,
+    "learning_rate": 0.00019680661025136078,
+    "entropy": 0.49193753488361835,
+    "num_tokens": 232053.0,
+    "mean_token_accuracy": 0.8580134496092796,
+    "epoch": 0.10935856992639327,
+    "step": 130
+  },
+  {
+    "loss": 0.5707052230834961,
+    "grad_norm": 0.462890625,
+    "learning_rate": 0.00019608769029420895,
+    "entropy": 0.5594139508903027,
+    "num_tokens": 248970.0,
+    "mean_token_accuracy": 0.8354306548833847,
+    "epoch": 0.11777076761303891,
+    "step": 140
+  },
+  {
+    "loss": 0.5284650325775146,
+    "grad_norm": 0.48828125,
+    "learning_rate": 0.00019529743872873014,
+    "entropy": 0.5149690546095371,
+    "num_tokens": 266885.0,
+    "mean_token_accuracy": 0.8453154772520065,
+    "epoch": 0.12618296529968454,
+    "step": 150
+  },
+  {
+    "eval_loss": 0.5157255530357361,
+    "eval_runtime": 43.5089,
+    "eval_samples_per_second": 23.03,
+    "eval_steps_per_second": 5.769,
+    "eval_entropy": 0.5205607345973353,
+    "eval_num_tokens": 266885.0,
+    "eval_mean_token_accuracy": 0.8498107833691327,
+    "epoch": 0.12618296529968454,
+    "step": 150
+  },
+  {
+    "loss": 0.5166601181030274,
+    "grad_norm": 0.3984375,
+    "learning_rate": 0.00019443644220566985,
+    "entropy": 0.5193028658628464,
+    "num_tokens": 285094.0,
+    "mean_token_accuracy": 0.8514553621411324,
+    "epoch": 0.13459516298633017,
+    "step": 160
+  },
+  {
+    "loss": 0.4895184993743896,
+    "grad_norm": 0.396484375,
+    "learning_rate": 0.00019350533989396395,
+    "entropy": 0.5498913679271936,
+    "num_tokens": 302878.0,
+    "mean_token_accuracy": 0.8499076500535011,
+    "epoch": 0.14300736067297581,
+    "step": 170
+  },
+  {
+    "loss": 0.489929723739624,
+    "grad_norm": 0.34765625,
+    "learning_rate": 0.00019250482300624543,
+    "entropy": 0.4781280752271414,
+    "num_tokens": 321038.0,
+    "mean_token_accuracy": 0.8518652722239495,
+    "epoch": 0.15141955835962145,
+    "step": 180
+  },
+  {
+    "loss": 0.5677974700927735,
+    "grad_norm": 0.392578125,
+    "learning_rate": 0.00019143563428571634,
+    "entropy": 0.560076616331935,
+    "num_tokens": 338613.0,
+    "mean_token_accuracy": 0.8355910241603851,
+    "epoch": 0.1598317560462671,
+    "step": 190
+  },
+  {
+    "loss": 0.47907423973083496,
+    "grad_norm": 0.380859375,
+    "learning_rate": 0.00019029856745476508,
+    "entropy": 0.486137630045414,
+    "num_tokens": 355794.0,
+    "mean_token_accuracy": 0.8546374544501305,
+    "epoch": 0.16824395373291273,
+    "step": 200
+  },
+  {
+    "loss": 0.5049150466918946,
+    "grad_norm": 0.515625,
+    "learning_rate": 0.0001890944666257392,
+    "entropy": 0.5356668088585138,
+    "num_tokens": 374388.0,
+    "mean_token_accuracy": 0.8522498905658722,
+    "epoch": 0.17665615141955837,
+    "step": 210
+  },
+  {
+    "loss": 0.5084699153900146,
+    "grad_norm": 0.474609375,
+    "learning_rate": 0.00018782422567431063,
+    "entropy": 0.48354478403925893,
+    "num_tokens": 393466.0,
+    "mean_token_accuracy": 0.8485782667994499,
+    "epoch": 0.18506834910620398,
+    "step": 220
+  },
+  {
+    "loss": 0.5076002597808837,
+    "grad_norm": 0.357421875,
+    "learning_rate": 0.0001864887875758985,
+    "entropy": 0.5002748906612396,
+    "num_tokens": 411947.0,
+    "mean_token_accuracy": 0.8519161671400071,
+    "epoch": 0.19348054679284962,
+    "step": 230
+  },
+  {
+    "loss": 0.5514575004577636,
+    "grad_norm": 0.60546875,
+    "learning_rate": 0.00018508914370564243,
+    "entropy": 0.5646139703691005,
+    "num_tokens": 429099.0,
+    "mean_token_accuracy": 0.8357668131589889,
+    "epoch": 0.20189274447949526,
+    "step": 240
+  },
+  {
+    "loss": 0.49529438018798827,
+    "grad_norm": 0.58984375,
+    "learning_rate": 0.00018362633310244574,
+    "entropy": 0.518617831915617,
+    "num_tokens": 446977.0,
+    "mean_token_accuracy": 0.8446412399411202,
+    "epoch": 0.2103049421661409,
+    "step": 250
+  },
+  {
+    "loss": 0.5090048313140869,
+    "grad_norm": 0.408203125,
+    "learning_rate": 0.00018210144169763522,
+    "entropy": 0.502267974242568,
+    "num_tokens": 465009.0,
+    "mean_token_accuracy": 0.851601280272007,
+    "epoch": 0.21871713985278654,
+    "step": 260
+  },
+  {
+    "loss": 0.5145918369293213,
+    "grad_norm": 0.41796875,
+    "learning_rate": 0.00018051560150880952,
+    "entropy": 0.5074335142970086,
+    "num_tokens": 481925.0,
+    "mean_token_accuracy": 0.8518189460039138,
+    "epoch": 0.22712933753943218,
+    "step": 270
+  },
+  {
+    "loss": 0.47620458602905275,
+    "grad_norm": 0.29296875,
+    "learning_rate": 0.00017886998979947522,
+    "entropy": 0.5017431110143662,
+    "num_tokens": 499792.0,
+    "mean_token_accuracy": 0.8548987463116646,
+    "epoch": 0.23554153522607782,
+    "step": 280
+  },
+  {
+    "loss": 0.507429552078247,
+    "grad_norm": 0.39453125,
+    "learning_rate": 0.00017716582820509423,
+    "entropy": 0.5222229663282633,
+    "num_tokens": 517055.0,
+    "mean_token_accuracy": 0.8444660127162933,
+    "epoch": 0.24395373291272346,
+    "step": 290
+  },
+  {
+    "loss": 0.5420027732849121,
+    "grad_norm": 0.39453125,
+    "learning_rate": 0.00017540438182619083,
+    "entropy": 0.5492073997855187,
+    "num_tokens": 536355.0,
+    "mean_token_accuracy": 0.8354690104722977,
+    "epoch": 0.25236593059936907,
+    "step": 300
+  },
+  {
+    "eval_loss": 0.5092939138412476,
+    "eval_runtime": 43.2398,
+    "eval_samples_per_second": 23.173,
+    "eval_steps_per_second": 5.805,
+    "eval_entropy": 0.4988216485872687,
+    "eval_num_tokens": 536355.0,
+    "eval_mean_token_accuracy": 0.849776790911458,
+    "epoch": 0.25236593059936907,
+    "step": 300
+  },
+  {
+    "loss": 0.536237096786499,
+    "grad_norm": 0.91015625,
+    "learning_rate": 0.00017358695828919264,
+    "entropy": 0.5142859980463982,
+    "num_tokens": 553515.0,
+    "mean_token_accuracy": 0.8491675749421119,
+    "epoch": 0.2607781282860147,
+    "step": 310
+  },
+  {
+    "loss": 0.5729623317718506,
+    "grad_norm": 0.44921875,
+    "learning_rate": 0.00017171490677570138,
+    "entropy": 0.558933037891984,
+    "num_tokens": 571228.0,
+    "mean_token_accuracy": 0.8405497163534165,
+    "epoch": 0.26919032597266035,
+    "step": 320
+  },
+  {
+    "loss": 0.5896058559417725,
+    "grad_norm": 0.5703125,
+    "learning_rate": 0.00016978961702091548,
+    "entropy": 0.572591795027256,
+    "num_tokens": 589110.0,
+    "mean_token_accuracy": 0.8290358498692513,
+    "epoch": 0.277602523659306,
+    "step": 330
+  },
+  {
+    "loss": 0.5112470626831055,
+    "grad_norm": 0.400390625,
+    "learning_rate": 0.0001678125182819466,
+    "entropy": 0.5400734946131707,
+    "num_tokens": 606112.0,
+    "mean_token_accuracy": 0.8408979564905167,
+    "epoch": 0.28601472134595163,
+    "step": 340
+  },
+  {
+    "loss": 0.4602771282196045,
+    "grad_norm": 0.4296875,
+    "learning_rate": 0.0001657850782767974,
+    "entropy": 0.4790476318448782,
+    "num_tokens": 623646.0,
+    "mean_token_accuracy": 0.8584060147404671,
+    "epoch": 0.29442691903259727,
+    "step": 350
+  },
+  {
+    "loss": 0.5069724559783936,
+    "grad_norm": 0.48046875,
+    "learning_rate": 0.00016370880209478684,
+    "entropy": 0.5200437907129526,
+    "num_tokens": 641902.0,
+    "mean_token_accuracy": 0.8489004954695701,
+    "epoch": 0.3028391167192429,
+    "step": 360
+  },
+  {
+    "loss": 0.5578237533569336,
+    "grad_norm": 0.369140625,
+    "learning_rate": 0.00016158523107923317,
+    "entropy": 0.5649974066764116,
+    "num_tokens": 659952.0,
+    "mean_token_accuracy": 0.8369685173034668,
+    "epoch": 0.31125131440588855,
+    "step": 370
+  },
+  {
+    "loss": 0.47066588401794435,
+    "grad_norm": 0.359375,
+    "learning_rate": 0.00015941594168322282,
+    "entropy": 0.47163805216550825,
+    "num_tokens": 676446.0,
+    "mean_token_accuracy": 0.8630838245153427,
+    "epoch": 0.3196635120925342,
+    "step": 380
+  },
+  {
+    "loss": 0.4772009372711182,
+    "grad_norm": 0.400390625,
+    "learning_rate": 0.0001572025442993157,
+    "entropy": 0.4667525518685579,
+    "num_tokens": 694154.0,
+    "mean_token_accuracy": 0.8570062339305877,
+    "epoch": 0.3280757097791798,
+    "step": 390
+  },
+  {
+    "loss": 0.5249220848083496,
+    "grad_norm": 0.375,
+    "learning_rate": 0.00015494668206405483,
+    "entropy": 0.5329773716628552,
+    "num_tokens": 711669.0,
+    "mean_token_accuracy": 0.8415302649140358,
+    "epoch": 0.33648790746582546,
+    "step": 400
+  },
+  {
+    "loss": 0.4875462055206299,
+    "grad_norm": 0.5234375,
+    "learning_rate": 0.00015265002963816846,
+    "entropy": 0.4823704816401005,
+    "num_tokens": 728983.0,
+    "mean_token_accuracy": 0.858619874715805,
+    "epoch": 0.3449001051524711,
+    "step": 410
+  },
+  {
+    "loss": 0.44158411026000977,
+    "grad_norm": 0.37890625,
+    "learning_rate": 0.00015031429196336982,
+    "entropy": 0.4349179297685623,
+    "num_tokens": 745793.0,
+    "mean_token_accuracy": 0.8672094404697418,
+    "epoch": 0.35331230283911674,
+    "step": 420
+  },
+  {
+    "loss": 0.44835238456726073,
+    "grad_norm": 0.390625,
+    "learning_rate": 0.00014794120299667737,
+    "entropy": 0.4709930635988712,
+    "num_tokens": 763754.0,
+    "mean_token_accuracy": 0.8656966030597687,
+    "epoch": 0.3617245005257624,
+    "step": 430
+  },
+  {
+    "loss": 0.46076221466064454,
+    "grad_norm": 0.375,
+    "learning_rate": 0.00014553252442319538,
+    "entropy": 0.45636184960603715,
+    "num_tokens": 782350.0,
+    "mean_token_accuracy": 0.8606420174241066,
+    "epoch": 0.37013669821240797,
+    "step": 440
+  },
+  {
+    "loss": 0.41022934913635256,
+    "grad_norm": 0.32421875,
+    "learning_rate": 0.00014309004434831036,
+    "entropy": 0.41649786606431005,
+    "num_tokens": 801737.0,
+    "mean_token_accuracy": 0.8772262930870056,
+    "epoch": 0.3785488958990536,
+    "step": 450
+  },
+  {
+    "eval_loss": 0.5019798278808594,
+    "eval_runtime": 43.2448,
+    "eval_samples_per_second": 23.17,
+    "eval_steps_per_second": 5.804,
+    "eval_entropy": 0.49474223777354953,
+    "eval_num_tokens": 801737.0,
+    "eval_mean_token_accuracy": 0.8509889845829086,
+    "epoch": 0.3785488958990536,
+    "step": 450
+  },
+  {
+    "loss": 0.4886730670928955,
+    "grad_norm": 0.40625,
+    "learning_rate": 0.0001406155759702739,
+    "entropy": 0.4626791954040527,
+    "num_tokens": 818998.0,
+    "mean_token_accuracy": 0.8578041106462478,
+    "epoch": 0.38696109358569925,
+    "step": 460
+  },
+  {
+    "loss": 0.43723235130310056,
+    "grad_norm": 0.36328125,
+    "learning_rate": 0.00013811095623415804,
+    "entropy": 0.4580305341631174,
+    "num_tokens": 837490.0,
+    "mean_token_accuracy": 0.8644093602895737,
+    "epoch": 0.3953732912723449,
+    "step": 470
+  },
+  {
+    "loss": 0.5131179332733155,
+    "grad_norm": 0.44921875,
+    "learning_rate": 0.00013557804446818133,
+    "entropy": 0.5031312361359597,
+    "num_tokens": 853819.0,
+    "mean_token_accuracy": 0.8500203013420105,
+    "epoch": 0.4037854889589905,
+    "step": 480
+  },
+  {
+    "loss": 0.4966330051422119,
+    "grad_norm": 0.46875,
+    "learning_rate": 0.00013301872100341912,
+    "entropy": 0.5123794697225094,
+    "num_tokens": 870402.0,
+    "mean_token_accuracy": 0.8551265612244606,
+    "epoch": 0.41219768664563616,
+    "step": 490
+  },
+  {
+    "loss": 0.5083879947662353,
+    "grad_norm": 0.408203125,
+    "learning_rate": 0.00013043488577792184,
+    "entropy": 0.5123440526425839,
+    "num_tokens": 887321.0,
+    "mean_token_accuracy": 0.8488352715969085,
+    "epoch": 0.4206098843322818,
+    "step": 500
+  },
+  {
+    "loss": 0.4652432918548584,
+    "grad_norm": 0.447265625,
+    "learning_rate": 0.0001278284569262781,
+    "entropy": 0.4698960553854704,
+    "num_tokens": 905696.0,
+    "mean_token_accuracy": 0.8606894016265869,
+    "epoch": 0.42902208201892744,
+    "step": 510
+  },
+  {
+    "loss": 0.4380183219909668,
+    "grad_norm": 0.408203125,
+    "learning_rate": 0.00012520136935566904,
+    "entropy": 0.43917837329208853,
+    "num_tokens": 924737.0,
+    "mean_token_accuracy": 0.8651972591876984,
+    "epoch": 0.4374342797055731,
+    "step": 520
+  },
+  {
+    "loss": 0.46619858741760256,
+    "grad_norm": 0.51171875,
+    "learning_rate": 0.0001225555733094719,
+    "entropy": 0.4861061923205853,
+    "num_tokens": 941252.0,
+    "mean_token_accuracy": 0.8588027134537697,
+    "epoch": 0.4458464773922187,
+    "step": 530
+  },
+  {
+    "loss": 0.4590827465057373,
+    "grad_norm": 0.32421875,
+    "learning_rate": 0.00011989303291947826,
+    "entropy": 0.4769519090652466,
+    "num_tokens": 959549.0,
+    "mean_token_accuracy": 0.8566611051559448,
+    "epoch": 0.45425867507886436,
+    "step": 540
+  },
+  {
+    "loss": 0.5028058528900147,
+    "grad_norm": 0.390625,
+    "learning_rate": 0.00011721572474780205,
+    "entropy": 0.49060247242450716,
+    "num_tokens": 977934.0,
+    "mean_token_accuracy": 0.8463574901223183,
+    "epoch": 0.46267087276551,
+    "step": 550
+  },
+  {
+    "loss": 0.5106627941131592,
+    "grad_norm": 0.353515625,
+    "learning_rate": 0.00011452563631955997,
+    "entropy": 0.5243487633764744,
+    "num_tokens": 995534.0,
+    "mean_token_accuracy": 0.8489840433001519,
+    "epoch": 0.47108307045215564,
+    "step": 560
+  },
+  {
+    "loss": 0.5040833473205566,
+    "grad_norm": 0.431640625,
+    "learning_rate": 0.00011182476464741359,
+    "entropy": 0.511018768697977,
+    "num_tokens": 1012521.0,
+    "mean_token_accuracy": 0.8465926513075829,
+    "epoch": 0.4794952681388013,
+    "step": 570
+  },
+  {
+    "loss": 0.520852518081665,
+    "grad_norm": 0.380859375,
+    "learning_rate": 0.0001091151147490677,
+    "entropy": 0.5150163471698761,
+    "num_tokens": 1029532.0,
+    "mean_token_accuracy": 0.8415179714560509,
+    "epoch": 0.4879074658254469,
+    "step": 580
+  },
+  {
+    "loss": 0.3955108404159546,
+    "grad_norm": 0.404296875,
+    "learning_rate": 0.00010639869815882668,
+    "entropy": 0.42485857382416725,
+    "num_tokens": 1047599.0,
+    "mean_token_accuracy": 0.8708899393677711,
+    "epoch": 0.49631966351209256,
+    "step": 590
+  },
+  {
+    "loss": 0.5156550884246827,
+    "grad_norm": 0.404296875,
+    "learning_rate": 0.00010367753143431278,
+    "entropy": 0.4836477965116501,
+    "num_tokens": 1066584.0,
+    "mean_token_accuracy": 0.8515340134501457,
+    "epoch": 0.5047318611987381,
+    "step": 600
+  },
+  {
+    "eval_loss": 0.5016828179359436,
+    "eval_runtime": 44.4349,
+    "eval_samples_per_second": 22.55,
+    "eval_steps_per_second": 5.649,
+    "eval_entropy": 0.5061054474922765,
+    "eval_num_tokens": 1066584.0,
+    "eval_mean_token_accuracy": 0.8509582224120182,
+    "epoch": 0.5047318611987381,
+    "step": 600
+  },
+  {
+    "loss": 0.5101516246795654,
+    "grad_norm": 0.412109375,
+    "learning_rate": 0.00010095363465945549,
+    "entropy": 0.5069868132472038,
+    "num_tokens": 1084792.0,
+    "mean_token_accuracy": 0.8516661524772644,
+    "epoch": 0.5131440588853838,
+    "step": 610
+  },
+  {
+    "loss": 0.5655738353729248,
+    "grad_norm": 0.43359375,
+    "learning_rate": 9.822902994486313e-05,
+    "entropy": 0.5374718602746725,
+    "num_tokens": 1102585.0,
+    "mean_token_accuracy": 0.8460393868386745,
+    "epoch": 0.5215562565720294,
+    "step": 620
+  },
+  {
+    "loss": 0.5003692626953125,
+    "grad_norm": 0.380859375,
+    "learning_rate": 9.550573992668964e-05,
+    "entropy": 0.4945683263242245,
+    "num_tokens": 1118590.0,
+    "mean_token_accuracy": 0.8570186719298363,
+    "epoch": 0.5299684542586751,
+    "step": 630
+  },
+  {
+    "loss": 0.4934247970581055,
+    "grad_norm": 0.35546875,
+    "learning_rate": 9.278578626511173e-05,
+    "entropy": 0.49867704808712005,
+    "num_tokens": 1136773.0,
+    "mean_token_accuracy": 0.8551020383834839,
+    "epoch": 0.5383806519453207,
+    "step": 640
+  },
+  {
+    "loss": 0.45653066635131834,
+    "grad_norm": 0.361328125,
+    "learning_rate": 9.007118814353019e-05,
+    "entropy": 0.4587596397846937,
+    "num_tokens": 1154768.0,
+    "mean_token_accuracy": 0.8645354241132737,
+    "epoch": 0.5467928496319664,
+    "step": 650
+  },
+  {
+    "loss": 0.43801026344299315,
+    "grad_norm": 0.3515625,
+    "learning_rate": 8.736396076960973e-05,
+    "entropy": 0.436482329107821,
+    "num_tokens": 1171141.0,
+    "mean_token_accuracy": 0.8680778846144677,
+    "epoch": 0.555205047318612,
+    "step": 660
+  },
+  {
+    "loss": 0.48572382926940916,
+    "grad_norm": 0.33984375,
+    "learning_rate": 8.466611387927075e-05,
+    "entropy": 0.4897559147328138,
+    "num_tokens": 1188848.0,
+    "mean_token_accuracy": 0.8528486609458923,
+    "epoch": 0.5636172450052577,
+    "step": 670
+  },
+  {
+    "loss": 0.4595578670501709,
+    "grad_norm": 0.423828125,
+    "learning_rate": 8.197965024474269e-05,
+    "entropy": 0.48105588629841806,
+    "num_tokens": 1205462.0,
+    "mean_token_accuracy": 0.8587836340069771,
+    "epoch": 0.5720294426919033,
+    "step": 680
+  },
+  {
+    "loss": 0.5170315265655517,
+    "grad_norm": 0.296875,
+    "learning_rate": 7.930656418778692e-05,
+    "entropy": 0.5295548569411039,
+    "num_tokens": 1223225.0,
+    "mean_token_accuracy": 0.8485119223594666,
+    "epoch": 0.580441640378549,
+    "step": 690
+  },
+  {
+    "loss": 0.49773259162902833,
+    "grad_norm": 0.439453125,
+    "learning_rate": 7.664884009919337e-05,
+    "entropy": 0.4959330677986145,
+    "num_tokens": 1241341.0,
+    "mean_token_accuracy": 0.8523301362991333,
+    "epoch": 0.5888538380651945,
+    "step": 700
+  },
+  {
+    "loss": 0.45866031646728517,
+    "grad_norm": 0.333984375,
+    "learning_rate": 7.400845096564895e-05,
+    "entropy": 0.4629350196570158,
+    "num_tokens": 1260070.0,
+    "mean_token_accuracy": 0.8623251214623451,
+    "epoch": 0.5972660357518401,
+    "step": 710
+  },
+  {
+    "loss": 0.45479936599731446,
+    "grad_norm": 0.466796875,
+    "learning_rate": 7.138735690507232e-05,
+    "entropy": 0.45315413922071457,
+    "num_tokens": 1277817.0,
+    "mean_token_accuracy": 0.8606215000152588,
+    "epoch": 0.6056782334384858,
+    "step": 720
+  },
+  {
+    "loss": 0.4651636600494385,
+    "grad_norm": 0.50390625,
+    "learning_rate": 6.87875037115016e-05,
+    "entropy": 0.4955283936113119,
+    "num_tokens": 1296319.0,
+    "mean_token_accuracy": 0.8529913663864136,
+    "epoch": 0.6140904311251314,
+    "step": 730
+  },
+  {
+    "loss": 0.458605432510376,
+    "grad_norm": 0.375,
+    "learning_rate": 6.621082141061594e-05,
+    "entropy": 0.46719060391187667,
+    "num_tokens": 1314222.0,
+    "mean_token_accuracy": 0.8618179276585579,
+    "epoch": 0.6225026288117771,
+    "step": 740
+  },
+  {
+    "loss": 0.4238747119903564,
+    "grad_norm": 0.37890625,
+    "learning_rate": 6.365922282696246e-05,
+    "entropy": 0.46433731317520144,
+    "num_tokens": 1332578.0,
+    "mean_token_accuracy": 0.8674190878868103,
+    "epoch": 0.6309148264984227,
+    "step": 750
+  },
+  {
+    "eval_loss": 0.49650052189826965,
+    "eval_runtime": 44.1675,
+    "eval_samples_per_second": 22.686,
+    "eval_steps_per_second": 5.683,
+    "eval_entropy": 0.49466295991523335,
+    "eval_num_tokens": 1332578.0,
+    "eval_mean_token_accuracy": 0.8537447065471178,
+    "epoch": 0.6309148264984227,
+    "step": 750
+  },
+  {
+    "loss": 0.4777510166168213,
+    "grad_norm": 0.296875,
+    "learning_rate": 6.113460216395276e-05,
+    "entropy": 0.46779244504868983,
+    "num_tokens": 1350642.0,
+    "mean_token_accuracy": 0.8585006296634674,
+    "epoch": 0.6393270241850684,
+    "step": 760
+  },
+  {
+    "loss": 0.43058485984802247,
+    "grad_norm": 0.390625,
+    "learning_rate": 5.8638833597683295e-05,
+    "entropy": 0.4450514350086451,
+    "num_tokens": 1368256.0,
+    "mean_token_accuracy": 0.8651673465967178,
+    "epoch": 0.647739221871714,
+    "step": 770
+  },
+  {
+    "loss": 0.4754780292510986,
+    "grad_norm": 0.2294921875,
+    "learning_rate": 5.6173769885622816e-05,
+    "entropy": 0.48611559383571146,
+    "num_tokens": 1385497.0,
+    "mean_token_accuracy": 0.8574813604354858,
+    "epoch": 0.6561514195583596,
+    "step": 780
+  },
+  {
+    "loss": 0.48357625007629396,
+    "grad_norm": 0.380859375,
+    "learning_rate": 5.3741240991200214e-05,
+    "entropy": 0.47957778461277484,
+    "num_tokens": 1404550.0,
+    "mean_token_accuracy": 0.8487858697772026,
+    "epoch": 0.6645636172450052,
+    "step": 790
+  },
+  {
+    "loss": 0.5497960567474365,
+    "grad_norm": 0.54296875,
+    "learning_rate": 5.134305272531394e-05,
+    "entropy": 0.5210029605776072,
+    "num_tokens": 1421268.0,
+    "mean_token_accuracy": 0.840529902279377,
+    "epoch": 0.6729758149316509,
+    "step": 800
+  },
+  {
+    "loss": 0.4972085952758789,
+    "grad_norm": 0.443359375,
+    "learning_rate": 4.898098540577122e-05,
+    "entropy": 0.5074868854135275,
+    "num_tokens": 1438077.0,
+    "mean_token_accuracy": 0.8451508775353431,
+    "epoch": 0.6813880126182965,
+    "step": 810
+  },
+  {
+    "loss": 0.42497959136962893,
+    "grad_norm": 0.578125,
+    "learning_rate": 4.665679253565191e-05,
+    "entropy": 0.4287653032690287,
+    "num_tokens": 1455329.0,
+    "mean_token_accuracy": 0.870225128531456,
+    "epoch": 0.6898002103049422,
+    "step": 820
+  },
+  {
+    "loss": 0.49147672653198243,
+    "grad_norm": 0.302734375,
+    "learning_rate": 4.437219950157927e-05,
+    "entropy": 0.5095056083053351,
+    "num_tokens": 1472973.0,
+    "mean_token_accuracy": 0.8493282213807106,
+    "epoch": 0.6982124079915878,
+    "step": 830
+  },
+  {
+    "loss": 0.49974665641784666,
+    "grad_norm": 0.400390625,
+    "learning_rate": 4.2128902292862625e-05,
+    "entropy": 0.5031959179788827,
+    "num_tokens": 1491004.0,
+    "mean_token_accuracy": 0.8519662648439408,
+    "epoch": 0.7066246056782335,
+    "step": 840
+  },
+  {
+    "loss": 0.4568824768066406,
+    "grad_norm": 0.369140625,
+    "learning_rate": 3.9928566242463516e-05,
+    "entropy": 0.46861373744904994,
+    "num_tokens": 1507930.0,
+    "mean_token_accuracy": 0.8673135682940483,
+    "epoch": 0.7150368033648791,
+    "step": 850
+  },
+  {
+    "loss": 0.4371347427368164,
+    "grad_norm": 0.3515625,
+    "learning_rate": 3.777282479072023e-05,
+    "entropy": 0.4369157709181309,
+    "num_tokens": 1525250.0,
+    "mean_token_accuracy": 0.8657293424010277,
+    "epoch": 0.7234490010515248,
+    "step": 860
+  },
+  {
+    "loss": 0.48940362930297854,
+    "grad_norm": 0.54296875,
+    "learning_rate": 3.56632782727477e-05,
+    "entropy": 0.48742267247289417,
+    "num_tokens": 1542980.0,
+    "mean_token_accuracy": 0.8546120315790177,
+    "epoch": 0.7318611987381703,
+    "step": 870
+  },
+  {
+    "loss": 0.4359768867492676,
+    "grad_norm": 0.51171875,
+    "learning_rate": 3.360149273041358e-05,
+    "entropy": 0.4555686455219984,
+    "num_tokens": 1560118.0,
+    "mean_token_accuracy": 0.8625203266739845,
+    "epoch": 0.7402733964248159,
+    "step": 880
+  },
+  {
+    "loss": 0.47966947555541994,
+    "grad_norm": 0.326171875,
+    "learning_rate": 3.158899874977221e-05,
+    "entropy": 0.5010597914457321,
+    "num_tokens": 1577819.0,
+    "mean_token_accuracy": 0.8502609938383102,
+    "epoch": 0.7486855941114616,
+    "step": 890
+  },
+  {
+    "loss": 0.48172292709350584,
+    "grad_norm": 0.353515625,
+    "learning_rate": 2.9627290324819635e-05,
+    "entropy": 0.47771124467253684,
+    "num_tokens": 1595465.0,
+    "mean_token_accuracy": 0.8525931060314178,
+    "epoch": 0.7570977917981072,
+    "step": 900
+  },
+  {
+    "eval_loss": 0.49366605281829834,
+    "eval_runtime": 44.2235,
+    "eval_samples_per_second": 22.658,
+    "eval_steps_per_second": 5.676,
+    "eval_entropy": 0.49088804530669966,
+    "eval_num_tokens": 1595465.0,
+    "eval_mean_token_accuracy": 0.8535285991026585,
+    "epoch": 0.7570977917981072,
+    "step": 900
+  },
+  {
+    "loss": 0.4764974117279053,
+    "grad_norm": 0.318359375,
+    "learning_rate": 2.7717823748412786e-05,
+    "entropy": 0.4569669846445322,
+    "num_tokens": 1612331.0,
+    "mean_token_accuracy": 0.859618866443634,
+    "epoch": 0.7655099894847529,
+    "step": 910
+  },
+  {
+    "loss": 0.5832089900970459,
+    "grad_norm": 0.5078125,
+    "learning_rate": 2.586201653117687e-05,
+    "entropy": 0.5448153771460056,
+    "num_tokens": 1629141.0,
+    "mean_token_accuracy": 0.8399338677525521,
+    "epoch": 0.7739221871713985,
+    "step": 920
+  },
+  {
+    "loss": 0.4664586067199707,
+    "grad_norm": 0.5234375,
+    "learning_rate": 2.4061246349202836e-05,
+    "entropy": 0.468376400321722,
+    "num_tokens": 1646892.0,
+    "mean_token_accuracy": 0.8635450974106789,
+    "epoch": 0.7823343848580442,
+    "step": 930
+  },
+  {
+    "loss": 0.515719985961914,
+    "grad_norm": 0.390625,
+    "learning_rate": 2.2316850021316383e-05,
+    "entropy": 0.4714161459356546,
+    "num_tokens": 1664928.0,
+    "mean_token_accuracy": 0.8522258847951889,
+    "epoch": 0.7907465825446898,
+    "step": 940
+  },
+  {
+    "loss": 0.46754937171936034,
+    "grad_norm": 0.361328125,
+    "learning_rate": 2.0630122516677952e-05,
+    "entropy": 0.4566764455288649,
+    "num_tokens": 1683193.0,
+    "mean_token_accuracy": 0.860313655436039,
+    "epoch": 0.7991587802313355,
+    "step": 950
+  },
+  {
+    "loss": 0.45092215538024905,
+    "grad_norm": 0.53515625,
+    "learning_rate": 1.900231599345006e-05,
+    "entropy": 0.49635875867679713,
+    "num_tokens": 1700570.0,
+    "mean_token_accuracy": 0.8524207696318626,
+    "epoch": 0.807570977917981,
+    "step": 960
+  },
+  {
+    "loss": 0.5068153381347656,
+    "grad_norm": 0.375,
+    "learning_rate": 1.743463886924569e-05,
+    "entropy": 0.49028092734515666,
+    "num_tokens": 1718365.0,
+    "mean_token_accuracy": 0.8554800346493721,
+    "epoch": 0.8159831756046267,
+    "step": 970
+  },
+  {
+    "loss": 0.4151425838470459,
+    "grad_norm": 0.341796875,
+    "learning_rate": 1.592825492404816e-05,
+    "entropy": 0.453617250174284,
+    "num_tokens": 1736733.0,
+    "mean_token_accuracy": 0.8672206476330757,
+    "epoch": 0.8243953732912723,
+    "step": 980
+  },
+  {
+    "loss": 0.47887310981750486,
+    "grad_norm": 0.439453125,
+    "learning_rate": 1.448428243626796e-05,
+    "entropy": 0.48988366313278675,
+    "num_tokens": 1756048.0,
+    "mean_token_accuracy": 0.8519709303975105,
+    "epoch": 0.832807570977918,
+    "step": 990
+  },
+  {
+    "loss": 0.41326084136962893,
+    "grad_norm": 0.37109375,
+    "learning_rate": 1.3103793352578153e-05,
+    "entropy": 0.425517388433218,
+    "num_tokens": 1773583.0,
+    "mean_token_accuracy": 0.8752173602581024,
+    "epoch": 0.8412197686645636,
+    "step": 1000
+  },
+  {
+    "loss": 0.44759268760681153,
+    "grad_norm": 0.283203125,
+    "learning_rate": 1.1787812492144611e-05,
+    "entropy": 0.4636589106172323,
+    "num_tokens": 1791572.0,
+    "mean_token_accuracy": 0.8631374597549438,
+    "epoch": 0.8496319663512093,
+    "step": 1010
+  },
+  {
+    "loss": 0.48625845909118653,
+    "grad_norm": 0.423828125,
+    "learning_rate": 1.05373167858418e-05,
+    "entropy": 0.49628445636481044,
+    "num_tokens": 1809222.0,
+    "mean_token_accuracy": 0.8517171651124954,
+    "epoch": 0.8580441640378549,
+    "step": 1020
+  },
+  {
+    "loss": 0.4556222438812256,
+    "grad_norm": 0.3828125,
+    "learning_rate": 9.353234551018787e-06,
+    "entropy": 0.45326445177197455,
+    "num_tokens": 1827208.0,
+    "mean_token_accuracy": 0.8658925801515579,
+    "epoch": 0.8664563617245006,
+    "step": 1030
+  },
+  {
+    "loss": 0.49567365646362305,
+    "grad_norm": 0.466796875,
+    "learning_rate": 8.236444802354194e-06,
+    "entropy": 0.4830797161906958,
+    "num_tokens": 1844866.0,
+    "mean_token_accuracy": 0.8538633629679679,
+    "epoch": 0.8748685594111462,
+    "step": 1040
+  },
+  {
+    "loss": 0.49363341331481936,
+    "grad_norm": 0.365234375,
+    "learning_rate": 7.187776599311225e-06,
+    "entropy": 0.49590680450201036,
+    "num_tokens": 1863508.0,
+    "mean_token_accuracy": 0.8481405600905418,
+    "epoch": 0.8832807570977917,
+    "step": 1050
+  },
+  {
+    "eval_loss": 0.492710679769516,
+    "eval_runtime": 44.3795,
+    "eval_samples_per_second": 22.578,
+    "eval_steps_per_second": 5.656,
+    "eval_entropy": 0.4946926224279214,
+    "eval_num_tokens": 1863508.0,
+    "eval_mean_token_accuracy": 0.8535925515619407,
+    "epoch": 0.8832807570977917,
+    "step": 1050
+  },
+  {
+    "loss": 0.46772065162658694,
+    "grad_norm": 0.330078125,
+    "learning_rate": 6.2080084306775255e-06,
+    "entropy": 0.48166475519537927,
+    "num_tokens": 1882313.0,
+    "mean_token_accuracy": 0.859215559065342,
+    "epoch": 0.8916929547844374,
+    "step": 1060
+  },
+  {
+    "loss": 0.4697853088378906,
+    "grad_norm": 0.419921875,
+    "learning_rate": 5.297867636646625e-06,
+    "entropy": 0.4738619901239872,
+    "num_tokens": 1899903.0,
+    "mean_token_accuracy": 0.8621121436357498,
+    "epoch": 0.900105152471083,
+    "step": 1070
+  },
+  {
+    "loss": 0.48142337799072266,
+    "grad_norm": 0.369140625,
+    "learning_rate": 4.458029868870106e-06,
+    "entropy": 0.4744772035628557,
+    "num_tokens": 1917878.0,
+    "mean_token_accuracy": 0.8594217240810395,
+    "epoch": 0.9085173501577287,
+    "step": 1080
+  },
+  {
+    "loss": 0.40869989395141604,
+    "grad_norm": 0.4296875,
+    "learning_rate": 3.6891185888811354e-06,
+    "entropy": 0.42849237024784087,
+    "num_tokens": 1935443.0,
+    "mean_token_accuracy": 0.8725451439619064,
+    "epoch": 0.9169295478443743,
+    "step": 1090
+  },
+  {
+    "loss": 0.4973776817321777,
+    "grad_norm": 0.29296875,
+    "learning_rate": 2.9917046052618913e-06,
+    "entropy": 0.4887580588459969,
+    "num_tokens": 1953376.0,
+    "mean_token_accuracy": 0.8516494154930114,
+    "epoch": 0.92534174553102,
+    "step": 1100
+  },
+  {
+    "loss": 0.5040076732635498,
+    "grad_norm": 0.333984375,
+    "learning_rate": 2.3663056498984035e-06,
+    "entropy": 0.5119045183062554,
+    "num_tokens": 1970989.0,
+    "mean_token_accuracy": 0.8522839710116387,
+    "epoch": 0.9337539432176656,
+    "step": 1110
+  },
+  {
+    "loss": 0.5222736358642578,
+    "grad_norm": 0.458984375,
+    "learning_rate": 1.8133859936374065e-06,
+    "entropy": 0.5279249057173729,
+    "num_tokens": 1987794.0,
+    "mean_token_accuracy": 0.8435431227087975,
+    "epoch": 0.9421661409043113,
+    "step": 1120
+  },
+  {
+    "loss": 0.5045926570892334,
+    "grad_norm": 0.494140625,
+    "learning_rate": 1.333356101630434e-06,
+    "entropy": 0.5016367115080357,
+    "num_tokens": 2006103.0,
+    "mean_token_accuracy": 0.8468659013509751,
+    "epoch": 0.9505783385909569,
+    "step": 1130
+  },
+  {
+    "loss": 0.4517922878265381,
+    "grad_norm": 0.41015625,
+    "learning_rate": 9.265723286211114e-07,
+    "entropy": 0.4635258123278618,
+    "num_tokens": 2024653.0,
+    "mean_token_accuracy": 0.858749409019947,
+    "epoch": 0.9589905362776026,
+    "step": 1140
+  },
+  {
+    "loss": 0.4429300308227539,
+    "grad_norm": 0.333984375,
+    "learning_rate": 5.933366544018881e-07,
+    "entropy": 0.46379459239542487,
+    "num_tokens": 2042110.0,
+    "mean_token_accuracy": 0.8633494853973389,
+    "epoch": 0.9674027339642481,
+    "step": 1150
+  },
+  {
+    "loss": 0.48291401863098143,
+    "grad_norm": 0.404296875,
+    "learning_rate": 3.338964596363958e-07,
+    "entropy": 0.4801469299942255,
+    "num_tokens": 2059221.0,
+    "mean_token_accuracy": 0.8571181014180184,
+    "epoch": 0.9758149316508938,
+    "step": 1160
+  },
+  {
+    "loss": 0.4716052055358887,
+    "grad_norm": 0.3828125,
+    "learning_rate": 1.4844434221414904e-07,
+    "entropy": 0.4888222087174654,
+    "num_tokens": 2076725.0,
+    "mean_token_accuracy": 0.8568236321210861,
+    "epoch": 0.9842271293375394,
+    "step": 1170
+  },
+  {
+    "loss": 0.4568983554840088,
+    "grad_norm": 0.4453125,
+    "learning_rate": 3.7117974273603865e-08,
+    "entropy": 0.4492193166166544,
+    "num_tokens": 2093837.0,
+    "mean_token_accuracy": 0.8625048115849495,
+    "epoch": 0.9926393270241851,
+    "step": 1180
+  },
+  {
+    "eval_loss": 0.49275869131088257,
+    "eval_runtime": 44.373,
+    "eval_samples_per_second": 22.581,
+    "eval_steps_per_second": 5.657,
+    "eval_entropy": 0.4945634504950854,
+    "eval_num_tokens": 2109469.0,
+    "eval_mean_token_accuracy": 0.8536091024182232,
+    "epoch": 1.0,
+    "step": 1189
+  },
+  {
+    "train_runtime": 2984.9918,
+    "train_samples_per_second": 6.372,
+    "train_steps_per_second": 0.398,
+    "total_flos": 2.585949872638771e+16,
+    "train_loss": 0.49064892395090115,
+    "epoch": 1.0,
+    "step": 1189
+  }
+]
\ No newline at end of file
diff --git a/chat_template.jinja b/chat_template.jinja
new file mode 100644
index 0000000..9840d40
--- /dev/null
+++ b/chat_template.jinja
@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}
diff --git a/config.json b/config.json
new file mode 100644
index 0000000..8227b12
--- /dev/null
+++ b/config.json
@@ -0,0 +1,61 @@
+{
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "dtype": "float16",
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 8960,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen2",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 2,
+  "pad_token_id": null,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.1",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 151936
+}
diff --git a/generation_config.json b/generation_config.json
new file mode 100644
index 0000000..89d0ec9
--- /dev/null
+++ b/generation_config.json
@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "repetition_penalty": 1.1,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "5.12.1"
+}
diff --git a/loss_curve.png b/loss_curve.png
new file mode 100644
index 0000000..868fb0d
--- /dev/null
+++ b/loss_curve.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daf98241987116a07a4cf7c39e2040bb0d4500fffc479f66966eaddf8957b6ce
+size 146432
diff --git a/model.safetensors b/model.safetensors
new file mode 100644
index 0000000..6999d6c
--- /dev/null
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49101962d7ca1d1056ba5c063e3618649e8cd8f878d6717d7b47b1d6e895ec41
+size 3087466808
diff --git a/token_reduction.png b/token_reduction.png
new file mode 100644
index 0000000..3f27a96
Binary files /dev/null and b/token_reduction.png differ
diff --git a/tokenizer.json b/tokenizer.json
new file mode 100644
index 0000000..34510ff
--- /dev/null
+++ b/tokenizer.json
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
+size 11421892
diff --git a/tokenizer_config.json b/tokenizer_config.json
new file mode 100644
index 0000000..6028fd5
--- /dev/null
+++ b/tokenizer_config.json
@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}