zenlm/zen-nano
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: zenlm/zen-nano
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-22 02:09:17 +08:00
commit cd91ac1da3
52 changed files with 456607 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
.gitattributes vendored Normal file
View File

@@ -0,0 +1,42 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
checkpoint-216/tokenizer.json filter=lfs diff=lfs merge=lfs -text
checkpoint-240/tokenizer.json filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text
gguf/zen-nano-0.6b-f16.gguf filter=lfs diff=lfs merge=lfs -text
gguf/zen-nano-0.6b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
gguf/zen-nano-0.6b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
gguf/zen-nano-0.6b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

5
NOTICE Normal file
View File

@@ -0,0 +1,5 @@
Zen
Copyright 2025-2026 Zen Authors
This model and its accompanying code are licensed under the Apache License, Version 2.0.
Zen models are developed by the Zen Authors (https://zenlm.org).

180
README.md Normal file
View File

@@ -0,0 +1,180 @@
---
language:
- en
- zh
license: apache-2.0
tags:
- zen
- zen-lm
- edge
- mobile
- lightweight
- zenlm
- hanzo
pipeline_tag: text-generation
library_name: transformers
---
# Zen Nano 0.6B
**Zen Nano** is an ultra-lightweight 0.6B parameter language model optimized for edge devices and mobile deployment. A compact foundation model that delivers impressive performance in a tiny package.
## Model Details
- **Model Type**: Causal Language Model
- **Architecture**: 0.6B dense transformer
- **Parameters**: 0.6 billion
- **License**: Apache 2.0
- **Languages**: English, Chinese
- **Context Length**: 32K tokens
- **Developed by**: Zen AI Team (Hanzo AI)
## Capabilities
- 💡 **Lightweight**: Only 0.6B parameters for edge deployment
- 📱 **Mobile-Ready**: Runs on smartphones and IoT devices
- ⚡ **Fast**: 44K tokens/sec on M3 Max (MLX)
- 🔋 **Efficient**: Low power consumption
- 🌐 **Multilingual**: English and Chinese support
- 📦 **Multiple Formats**: PyTorch, MLX, GGUF (Q2_K to F16)
- 🎯 **32K Context**: Extended context window
## Performance
### Throughput
- **M3 Max (MLX)**: 44,000 tokens/sec
- **RTX 4090 (GGUF Q4)**: 35,000 tokens/sec
- **iPhone 15 Pro**: 8,000 tokens/sec
- **Raspberry Pi 5**: 2,500 tokens/sec
### Memory Usage
| Format | VRAM/RAM |
|--------|----------|
| Q2_K | 0.3GB |
| Q4_K_M | 0.4GB |
| Q8_0 | 0.7GB |
| F16 | 1.2GB |
## Use Cases
- Edge AI applications
- Mobile chatbots
- IoT device intelligence
- Offline AI assistants
- Resource-constrained environments
- Real-time inference
- Embedded systems
## Installation
### Transformers
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"zenlm/zen-nano-0.6b",
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("zenlm/zen-nano-0.6b")
```
### MLX (Apple Silicon)
```python
from mlx_lm import load, generate
model, tokenizer = load("zenlm/zen-nano-0.6b")
response = generate(model, tokenizer, prompt="Hello!", max_tokens=100)
```
### GGUF (llama.cpp)
```bash
./llama-cli -m zen-nano-0.6b-Q4_K_M.gguf -p "Hello!" -n 100
```
### Zen Engine
```bash
zen-engine serve --model zenlm/zen-nano-0.6b --port 3690
```
## Training with Zen Gym
Fine-tune Zen Nano for your use case:
```bash
cd /path/to/zen-gym
llamafactory-cli train \
--config configs/zen_nano_lora.yaml \
--dataset your_dataset
```
## Benchmarks
| Task | Score | Notes |
|------|-------|-------|
| MMLU | 35.2% | 5-shot |
| GSM8K | 28.4% | 8-shot CoT |
| HumanEval | 24.1% | pass@1 |
| MATH | 18.7% | 4-shot |
## Limitations
- Smaller capacity than larger models
- May struggle with complex reasoning
- Limited specialized knowledge
- Best for short-to-medium contexts
- Quantization reduces quality slightly
## Citation
```bibtex
@misc{zennano2025,
title={Zen Nano: Ultra-Lightweight Language Model},
author={Zen AI Team},
year={2025},
howpublished={\url{https://huggingface.co/zenlm/zen-nano-0.6b}}
}
```
## Links
- **GitHub**: https://github.com/zenlm
- **Zen Gym** (Training): https://github.com/zenlm/zen-gym
- **Zen Engine** (Inference): https://github.com/zenlm/zen-engine
Part of the **[Zen AI](https://github.com/zenlm)** ecosystem.
---
## Citation
```bibtex
@misc{zenlm2025zen-nano,
title={Zen LM: zen-nano},
author={Hanzo AI and Zoo Labs Foundation},
year={2025},
publisher={HuggingFace},
howpublished={\url{https://huggingface.co/zenlm/zen-nano}}
}
```
## Abliteration
Zen Nano 0.6B weights are derived from an abliterated base model. Abliteration removes
refusal behavior by identifying and nullifying the "refusal direction" in the model's
residual stream, as described in
[Maxime Labonne's methodology](https://huggingface.co/blog/mlabonne/abliteration).
**Method**: Directional ablation on the residual stream across all layers
**Source**: Zen MoDE (Mixture of Distilled Experts) architecture base
**Effect**: Removes refusal behaviors while preserving all other capabilities
**Identity layer**: Zen identity added via system prompt — full LoRA fine-tuning planned
Abliteration is a feature, not a limitation. It enables unrestricted research,
security testing, and applications where safety guardrails are managed at the
application layer rather than baked into model weights.

28
added_tokens.json Normal file
View File

@@ -0,0 +1,28 @@
{
"</think>": 151668,
"</tool_call>": 151658,
"</tool_response>": 151666,
"<think>": 151667,
"<tool_call>": 151657,
"<tool_response>": 151665,
"<|box_end|>": 151649,
"<|box_start|>": 151648,
"<|endoftext|>": 151643,
"<|file_sep|>": 151664,
"<|fim_middle|>": 151660,
"<|fim_pad|>": 151662,
"<|fim_prefix|>": 151659,
"<|fim_suffix|>": 151661,
"<|im_end|>": 151645,
"<|im_start|>": 151644,
"<|image_pad|>": 151655,
"<|object_ref_end|>": 151647,
"<|object_ref_start|>": 151646,
"<|quad_end|>": 151651,
"<|quad_start|>": 151650,
"<|repo_name|>": 151663,
"<|video_pad|>": 151656,
"<|vision_end|>": 151653,
"<|vision_pad|>": 151654,
"<|vision_start|>": 151652
}

89
chat_template.jinja Normal file
View File

@@ -0,0 +1,89 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0].role == 'system' %}
{{- messages[0].content + '\n\n' }}
{%- endif %}
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0].role == 'system' %}
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
{%- set index = (messages|length - 1) - loop.index0 %}
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
{%- set ns.multi_step_tool = false %}
{%- set ns.last_query_index = index %}
{%- endif %}
{%- endfor %}
{%- for message in messages %}
{%- if message.content is string %}
{%- set content = message.content %}
{%- else %}
{%- set content = '' %}
{%- endif %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{%- set reasoning_content = '' %}
{%- if message.reasoning_content is string %}
{%- set reasoning_content = message.reasoning_content %}
{%- else %}
{%- if '</think>' in content %}
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
{%- endif %}
{%- endif %}
{%- if loop.index0 > ns.last_query_index %}
{%- if loop.last or (not loop.last and reasoning_content) %}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- if message.tool_calls %}
{%- for tool_call in message.tool_calls %}
{%- if (loop.first and content) or (not loop.first) %}
{{- '\n' }}
{%- endif %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{%- if tool_call.arguments is string %}
{{- tool_call.arguments }}
{%- else %}
{{- tool_call.arguments | tojson }}
{%- endif %}
{{- '}\n</tool_call>' }}
{%- endfor %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- if enable_thinking is defined and enable_thinking is false %}
{{- '<think>\n\n</think>\n\n' }}
{%- endif %}
{%- endif %}

28
checkpoint-216/added_tokens.json Normal file
View File

@@ -0,0 +1,28 @@
{
"</think>": 151668,
"</tool_call>": 151658,
"</tool_response>": 151666,
"<think>": 151667,
"<tool_call>": 151657,
"<tool_response>": 151665,
"<|box_end|>": 151649,
"<|box_start|>": 151648,
"<|endoftext|>": 151643,
"<|file_sep|>": 151664,
"<|fim_middle|>": 151660,
"<|fim_pad|>": 151662,
"<|fim_prefix|>": 151659,
"<|fim_suffix|>": 151661,
"<|im_end|>": 151645,
"<|im_start|>": 151644,
"<|image_pad|>": 151655,
"<|object_ref_end|>": 151647,
"<|object_ref_start|>": 151646,
"<|quad_end|>": 151651,
"<|quad_start|>": 151650,
"<|repo_name|>": 151663,
"<|video_pad|>": 151656,
"<|vision_end|>": 151653,
"<|vision_pad|>": 151654,
"<|vision_start|>": 151652
}

89
checkpoint-216/chat_template.jinja Normal file
View File

@@ -0,0 +1,89 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0].role == 'system' %}
{{- messages[0].content + '\n\n' }}
{%- endif %}
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0].role == 'system' %}
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
{%- set index = (messages|length - 1) - loop.index0 %}
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
{%- set ns.multi_step_tool = false %}
{%- set ns.last_query_index = index %}
{%- endif %}
{%- endfor %}
{%- for message in messages %}
{%- if message.content is string %}
{%- set content = message.content %}
{%- else %}
{%- set content = '' %}
{%- endif %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{%- set reasoning_content = '' %}
{%- if message.reasoning_content is string %}
{%- set reasoning_content = message.reasoning_content %}
{%- else %}
{%- if '</think>' in content %}
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
{%- endif %}
{%- endif %}
{%- if loop.index0 > ns.last_query_index %}
{%- if loop.last or (not loop.last and reasoning_content) %}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- if message.tool_calls %}
{%- for tool_call in message.tool_calls %}
{%- if (loop.first and content) or (not loop.first) %}
{{- '\n' }}
{%- endif %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{%- if tool_call.arguments is string %}
{{- tool_call.arguments }}
{%- else %}
{{- tool_call.arguments | tojson }}
{%- endif %}
{{- '}\n</tool_call>' }}
{%- endfor %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- if enable_thinking is defined and enable_thinking is false %}
{{- '<think>\n\n</think>\n\n' }}
{%- endif %}
{%- endif %}

60
checkpoint-216/config.json Normal file
View File

@@ -0,0 +1,60 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"dtype": "bfloat16",
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 1024,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 40960,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 16,
"num_hidden_layers": 28,
"num_key_value_heads": 8,
"pad_token_id": 151645,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "4.56.1",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

12
checkpoint-216/generation_config.json Normal file
View File

@@ -0,0 +1,12 @@
{
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151645,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95,
"transformers_version": "4.56.1"
}

151388
checkpoint-216/merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-216/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5278d9620372202c3d7ca82cd6595179ecc7f1bbc21c03bf1590c6081323339b
size 1192135096

3
checkpoint-216/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d223b54bfff67c9fb77480d6f6b37ca746249b0baab4ccbdf59f49614cec27ec
size 2384451723

3
checkpoint-216/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:817374c79694074958ac2537b4ffcf4ae1bd7defb8d81f48363bb8622b3fef00
size 14391

3
checkpoint-216/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6c165805c6ec023b291231674f4f38def913f38b5d09d9d0edf165026ae6b595
size 1465

25
checkpoint-216/special_tokens_map.json Normal file
View File

@@ -0,0 +1,25 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "<|im_end|>"
}

BIN
checkpoint-216/tokenizer.json (Stored with Git LFS) Normal file
View File

Binary file not shown.

239
checkpoint-216/tokenizer_config.json Normal file
View File

@@ -0,0 +1,239 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151665": {
"content": "<tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151666": {
"content": "</tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151667": {
"content": "<think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151668": {
"content": "</think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 131072,
"pad_token": "<|im_end|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

342
checkpoint-216/trainer_state.json Normal file
View File

@@ -0,0 +1,342 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 9.0,
"eval_steps": 500,
"global_step": 216,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.041666666666666664,
"grad_norm": 73.5,
"learning_rate": 0.0,
"loss": 3.9287,
"step": 1
},
{
"epoch": 0.20833333333333334,
"grad_norm": 80.5,
"learning_rate": 4.000000000000001e-06,
"loss": 3.994,
"step": 5
},
{
"epoch": 0.4166666666666667,
"grad_norm": 63.75,
"learning_rate": 9e-06,
"loss": 3.6819,
"step": 10
},
{
"epoch": 0.625,
"grad_norm": 49.25,
"learning_rate": 1.4000000000000001e-05,
"loss": 2.5702,
"step": 15
},
{
"epoch": 0.8333333333333334,
"grad_norm": 48.75,
"learning_rate": 1.9e-05,
"loss": 1.4286,
"step": 20
},
{
"epoch": 1.0416666666666667,
"grad_norm": 11.3125,
"learning_rate": 2.4e-05,
"loss": 0.7957,
"step": 25
},
{
"epoch": 1.25,
"grad_norm": 9.0625,
"learning_rate": 2.9e-05,
"loss": 0.7432,
"step": 30
},
{
"epoch": 1.4583333333333333,
"grad_norm": 16.5,
"learning_rate": 3.4000000000000007e-05,
"loss": 0.6508,
"step": 35
},
{
"epoch": 1.6666666666666665,
"grad_norm": 12.75,
"learning_rate": 3.9000000000000006e-05,
"loss": 0.591,
"step": 40
},
{
"epoch": 1.875,
"grad_norm": 6.59375,
"learning_rate": 4.4000000000000006e-05,
"loss": 0.6682,
"step": 45
},
{
"epoch": 2.0833333333333335,
"grad_norm": 11.1875,
"learning_rate": 4.9e-05,
"loss": 0.4391,
"step": 50
},
{
"epoch": 2.2916666666666665,
"grad_norm": 9.5,
"learning_rate": 4.8947368421052635e-05,
"loss": 0.3461,
"step": 55
},
{
"epoch": 2.5,
"grad_norm": 15.5,
"learning_rate": 4.7631578947368424e-05,
"loss": 0.4083,
"step": 60
},
{
"epoch": 2.7083333333333335,
"grad_norm": 12.0,
"learning_rate": 4.6315789473684214e-05,
"loss": 0.377,
"step": 65
},
{
"epoch": 2.9166666666666665,
"grad_norm": 10.0625,
"learning_rate": 4.5e-05,
"loss": 0.3861,
"step": 70
},
{
"epoch": 3.125,
"grad_norm": 6.78125,
"learning_rate": 4.368421052631579e-05,
"loss": 0.3271,
"step": 75
},
{
"epoch": 3.3333333333333335,
"grad_norm": 5.125,
"learning_rate": 4.236842105263158e-05,
"loss": 0.2295,
"step": 80
},
{
"epoch": 3.5416666666666665,
"grad_norm": 5.75,
"learning_rate": 4.105263157894737e-05,
"loss": 0.2661,
"step": 85
},
{
"epoch": 3.75,
"grad_norm": 10.9375,
"learning_rate": 3.973684210526316e-05,
"loss": 0.2583,
"step": 90
},
{
"epoch": 3.9583333333333335,
"grad_norm": 10.0625,
"learning_rate": 3.842105263157895e-05,
"loss": 0.2631,
"step": 95
},
{
"epoch": 4.166666666666667,
"grad_norm": 6.4375,
"learning_rate": 3.710526315789474e-05,
"loss": 0.1991,
"step": 100
},
{
"epoch": 4.375,
"grad_norm": 6.71875,
"learning_rate": 3.578947368421053e-05,
"loss": 0.1937,
"step": 105
},
{
"epoch": 4.583333333333333,
"grad_norm": 5.0,
"learning_rate": 3.447368421052632e-05,
"loss": 0.1894,
"step": 110
},
{
"epoch": 4.791666666666667,
"grad_norm": 10.8125,
"learning_rate": 3.3157894736842106e-05,
"loss": 0.2233,
"step": 115
},
{
"epoch": 5.0,
"grad_norm": 6.53125,
"learning_rate": 3.1842105263157895e-05,
"loss": 0.2006,
"step": 120
},
{
"epoch": 5.208333333333333,
"grad_norm": 4.59375,
"learning_rate": 3.0526315789473684e-05,
"loss": 0.1657,
"step": 125
},
{
"epoch": 5.416666666666667,
"grad_norm": 4.65625,
"learning_rate": 2.9210526315789477e-05,
"loss": 0.1937,
"step": 130
},
{
"epoch": 5.625,
"grad_norm": 7.90625,
"learning_rate": 2.7894736842105263e-05,
"loss": 0.1801,
"step": 135
},
{
"epoch": 5.833333333333333,
"grad_norm": 3.984375,
"learning_rate": 2.6578947368421052e-05,
"loss": 0.1704,
"step": 140
},
{
"epoch": 6.041666666666667,
"grad_norm": 3.8125,
"learning_rate": 2.5263157894736845e-05,
"loss": 0.168,
"step": 145
},
{
"epoch": 6.25,
"grad_norm": 5.15625,
"learning_rate": 2.394736842105263e-05,
"loss": 0.1395,
"step": 150
},
{
"epoch": 6.458333333333333,
"grad_norm": 4.25,
"learning_rate": 2.2631578947368423e-05,
"loss": 0.1426,
"step": 155
},
{
"epoch": 6.666666666666667,
"grad_norm": 4.8125,
"learning_rate": 2.1315789473684212e-05,
"loss": 0.1591,
"step": 160
},
{
"epoch": 6.875,
"grad_norm": 5.3125,
"learning_rate": 2e-05,
"loss": 0.1656,
"step": 165
},
{
"epoch": 7.083333333333333,
"grad_norm": 3.53125,
"learning_rate": 1.868421052631579e-05,
"loss": 0.1505,
"step": 170
},
{
"epoch": 7.291666666666667,
"grad_norm": 4.875,
"learning_rate": 1.736842105263158e-05,
"loss": 0.1356,
"step": 175
},
{
"epoch": 7.5,
"grad_norm": 3.890625,
"learning_rate": 1.605263157894737e-05,
"loss": 0.1359,
"step": 180
},
{
"epoch": 7.708333333333333,
"grad_norm": 3.59375,
"learning_rate": 1.4736842105263157e-05,
"loss": 0.1444,
"step": 185
},
{
"epoch": 7.916666666666667,
"grad_norm": 4.5,
"learning_rate": 1.3421052631578948e-05,
"loss": 0.1446,
"step": 190
},
{
"epoch": 8.125,
"grad_norm": 3.375,
"learning_rate": 1.2105263157894737e-05,
"loss": 0.1381,
"step": 195
},
{
"epoch": 8.333333333333334,
"grad_norm": 4.03125,
"learning_rate": 1.0789473684210526e-05,
"loss": 0.1372,
"step": 200
},
{
"epoch": 8.541666666666666,
"grad_norm": 3.921875,
"learning_rate": 9.473684210526317e-06,
"loss": 0.1354,
"step": 205
},
{
"epoch": 8.75,
"grad_norm": 4.03125,
"learning_rate": 8.157894736842106e-06,
"loss": 0.1321,
"step": 210
},
{
"epoch": 8.958333333333334,
"grad_norm": 3.390625,
"learning_rate": 6.842105263157896e-06,
"loss": 0.1351,
"step": 215
}
],
"logging_steps": 5,
"max_steps": 240,
"num_input_tokens_seen": 0,
"num_train_epochs": 10,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 572368078503936.0,
"train_batch_size": 4,
"trial_name": null,
"trial_params": null
}

3
checkpoint-216/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c57dd6ee21e08885a384c25251b744c7c77828bafa879ca11b2116f30c68c8c3
size 5713

1
checkpoint-216/vocab.json Normal file
View File

File diff suppressed because one or more lines are too long

28
checkpoint-240/added_tokens.json Normal file
View File

@@ -0,0 +1,28 @@
{
"</think>": 151668,
"</tool_call>": 151658,
"</tool_response>": 151666,
"<think>": 151667,
"<tool_call>": 151657,
"<tool_response>": 151665,
"<|box_end|>": 151649,
"<|box_start|>": 151648,
"<|endoftext|>": 151643,
"<|file_sep|>": 151664,
"<|fim_middle|>": 151660,
"<|fim_pad|>": 151662,
"<|fim_prefix|>": 151659,
"<|fim_suffix|>": 151661,
"<|im_end|>": 151645,
"<|im_start|>": 151644,
"<|image_pad|>": 151655,
"<|object_ref_end|>": 151647,
"<|object_ref_start|>": 151646,
"<|quad_end|>": 151651,
"<|quad_start|>": 151650,
"<|repo_name|>": 151663,
"<|video_pad|>": 151656,
"<|vision_end|>": 151653,
"<|vision_pad|>": 151654,
"<|vision_start|>": 151652
}

89
checkpoint-240/chat_template.jinja Normal file
View File

@@ -0,0 +1,89 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0].role == 'system' %}
{{- messages[0].content + '\n\n' }}
{%- endif %}
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0].role == 'system' %}
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
{%- set index = (messages|length - 1) - loop.index0 %}
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
{%- set ns.multi_step_tool = false %}
{%- set ns.last_query_index = index %}
{%- endif %}
{%- endfor %}
{%- for message in messages %}
{%- if message.content is string %}
{%- set content = message.content %}
{%- else %}
{%- set content = '' %}
{%- endif %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{%- set reasoning_content = '' %}
{%- if message.reasoning_content is string %}
{%- set reasoning_content = message.reasoning_content %}
{%- else %}
{%- if '</think>' in content %}
{%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
{%- set content = content.split('</think>')[-1].lstrip('\n') %}
{%- endif %}
{%- endif %}
{%- if loop.index0 > ns.last_query_index %}
{%- if loop.last or (not loop.last and reasoning_content) %}
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- if message.tool_calls %}
{%- for tool_call in message.tool_calls %}
{%- if (loop.first and content) or (not loop.first) %}
{{- '\n' }}
{%- endif %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{%- if tool_call.arguments is string %}
{{- tool_call.arguments }}
{%- else %}
{{- tool_call.arguments | tojson }}
{%- endif %}
{{- '}\n</tool_call>' }}
{%- endfor %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- if enable_thinking is defined and enable_thinking is false %}
{{- '<think>\n\n</think>\n\n' }}
{%- endif %}
{%- endif %}

60
checkpoint-240/config.json Normal file
View File

@@ -0,0 +1,60 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"dtype": "bfloat16",
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 1024,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 40960,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 16,
"num_hidden_layers": 28,
"num_key_value_heads": 8,
"pad_token_id": 151645,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "4.56.1",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

12
checkpoint-240/generation_config.json Normal file
View File

@@ -0,0 +1,12 @@
{
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151645,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95,
"transformers_version": "4.56.1"
}

151388
checkpoint-240/merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
checkpoint-240/model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a6a9ae479f1f4509d12d547b8b5eb3c5b6b031dd7ee66f9d0ac74552dec87032
size 1192135096

3
checkpoint-240/optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:33fbf37852ca0d8ce03c282d4e8fcf78b26926841dcc48c3663eb133a6ca4ad3
size 2384451723

3
checkpoint-240/rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:527276764335e03ea915d0573a9ad59eeff630ed1d94d41b99d98ba5a5610338
size 14391

3
checkpoint-240/scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fcb018bb4914c132e9aa57195da03a5a6ab62a5c65ffe18011d8098bbd5e83f7
size 1465

25
checkpoint-240/special_tokens_map.json Normal file
View File

@@ -0,0 +1,25 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "<|im_end|>"
}

BIN
checkpoint-240/tokenizer.json (Stored with Git LFS) Normal file
View File

Binary file not shown.

239
checkpoint-240/tokenizer_config.json Normal file
View File

@@ -0,0 +1,239 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151665": {
"content": "<tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151666": {
"content": "</tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151667": {
"content": "<think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151668": {
"content": "</think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 131072,
"pad_token": "<|im_end|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

377
checkpoint-240/trainer_state.json Normal file
View File

@@ -0,0 +1,377 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 10.0,
"eval_steps": 500,
"global_step": 240,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.041666666666666664,
"grad_norm": 73.5,
"learning_rate": 0.0,
"loss": 3.9287,
"step": 1
},
{
"epoch": 0.20833333333333334,
"grad_norm": 80.5,
"learning_rate": 4.000000000000001e-06,
"loss": 3.994,
"step": 5
},
{
"epoch": 0.4166666666666667,
"grad_norm": 63.75,
"learning_rate": 9e-06,
"loss": 3.6819,
"step": 10
},
{
"epoch": 0.625,
"grad_norm": 49.25,
"learning_rate": 1.4000000000000001e-05,
"loss": 2.5702,
"step": 15
},
{
"epoch": 0.8333333333333334,
"grad_norm": 48.75,
"learning_rate": 1.9e-05,
"loss": 1.4286,
"step": 20
},
{
"epoch": 1.0416666666666667,
"grad_norm": 11.3125,
"learning_rate": 2.4e-05,
"loss": 0.7957,
"step": 25
},
{
"epoch": 1.25,
"grad_norm": 9.0625,
"learning_rate": 2.9e-05,
"loss": 0.7432,
"step": 30
},
{
"epoch": 1.4583333333333333,
"grad_norm": 16.5,
"learning_rate": 3.4000000000000007e-05,
"loss": 0.6508,
"step": 35
},
{
"epoch": 1.6666666666666665,
"grad_norm": 12.75,
"learning_rate": 3.9000000000000006e-05,
"loss": 0.591,
"step": 40
},
{
"epoch": 1.875,
"grad_norm": 6.59375,
"learning_rate": 4.4000000000000006e-05,
"loss": 0.6682,
"step": 45
},
{
"epoch": 2.0833333333333335,
"grad_norm": 11.1875,
"learning_rate": 4.9e-05,
"loss": 0.4391,
"step": 50
},
{
"epoch": 2.2916666666666665,
"grad_norm": 9.5,
"learning_rate": 4.8947368421052635e-05,
"loss": 0.3461,
"step": 55
},
{
"epoch": 2.5,
"grad_norm": 15.5,
"learning_rate": 4.7631578947368424e-05,
"loss": 0.4083,
"step": 60
},
{
"epoch": 2.7083333333333335,
"grad_norm": 12.0,
"learning_rate": 4.6315789473684214e-05,
"loss": 0.377,
"step": 65
},
{
"epoch": 2.9166666666666665,
"grad_norm": 10.0625,
"learning_rate": 4.5e-05,
"loss": 0.3861,
"step": 70
},
{
"epoch": 3.125,
"grad_norm": 6.78125,
"learning_rate": 4.368421052631579e-05,
"loss": 0.3271,
"step": 75
},
{
"epoch": 3.3333333333333335,
"grad_norm": 5.125,
"learning_rate": 4.236842105263158e-05,
"loss": 0.2295,
"step": 80
},
{
"epoch": 3.5416666666666665,
"grad_norm": 5.75,
"learning_rate": 4.105263157894737e-05,
"loss": 0.2661,
"step": 85
},
{
"epoch": 3.75,
"grad_norm": 10.9375,
"learning_rate": 3.973684210526316e-05,
"loss": 0.2583,
"step": 90
},
{
"epoch": 3.9583333333333335,
"grad_norm": 10.0625,
"learning_rate": 3.842105263157895e-05,
"loss": 0.2631,
"step": 95
},
{
"epoch": 4.166666666666667,
"grad_norm": 6.4375,
"learning_rate": 3.710526315789474e-05,
"loss": 0.1991,
"step": 100
},
{
"epoch": 4.375,
"grad_norm": 6.71875,
"learning_rate": 3.578947368421053e-05,
"loss": 0.1937,
"step": 105
},
{
"epoch": 4.583333333333333,
"grad_norm": 5.0,
"learning_rate": 3.447368421052632e-05,
"loss": 0.1894,
"step": 110
},
{
"epoch": 4.791666666666667,
"grad_norm": 10.8125,
"learning_rate": 3.3157894736842106e-05,
"loss": 0.2233,
"step": 115
},
{
"epoch": 5.0,
"grad_norm": 6.53125,
"learning_rate": 3.1842105263157895e-05,
"loss": 0.2006,
"step": 120
},
{
"epoch": 5.208333333333333,
"grad_norm": 4.59375,
"learning_rate": 3.0526315789473684e-05,
"loss": 0.1657,
"step": 125
},
{
"epoch": 5.416666666666667,
"grad_norm": 4.65625,
"learning_rate": 2.9210526315789477e-05,
"loss": 0.1937,
"step": 130
},
{
"epoch": 5.625,
"grad_norm": 7.90625,
"learning_rate": 2.7894736842105263e-05,
"loss": 0.1801,
"step": 135
},
{
"epoch": 5.833333333333333,
"grad_norm": 3.984375,
"learning_rate": 2.6578947368421052e-05,
"loss": 0.1704,
"step": 140
},
{
"epoch": 6.041666666666667,
"grad_norm": 3.8125,
"learning_rate": 2.5263157894736845e-05,
"loss": 0.168,
"step": 145
},
{
"epoch": 6.25,
"grad_norm": 5.15625,
"learning_rate": 2.394736842105263e-05,
"loss": 0.1395,
"step": 150
},
{
"epoch": 6.458333333333333,
"grad_norm": 4.25,
"learning_rate": 2.2631578947368423e-05,
"loss": 0.1426,
"step": 155
},
{
"epoch": 6.666666666666667,
"grad_norm": 4.8125,
"learning_rate": 2.1315789473684212e-05,
"loss": 0.1591,
"step": 160
},
{
"epoch": 6.875,
"grad_norm": 5.3125,
"learning_rate": 2e-05,
"loss": 0.1656,
"step": 165
},
{
"epoch": 7.083333333333333,
"grad_norm": 3.53125,
"learning_rate": 1.868421052631579e-05,
"loss": 0.1505,
"step": 170
},
{
"epoch": 7.291666666666667,
"grad_norm": 4.875,
"learning_rate": 1.736842105263158e-05,
"loss": 0.1356,
"step": 175
},
{
"epoch": 7.5,
"grad_norm": 3.890625,
"learning_rate": 1.605263157894737e-05,
"loss": 0.1359,
"step": 180
},
{
"epoch": 7.708333333333333,
"grad_norm": 3.59375,
"learning_rate": 1.4736842105263157e-05,
"loss": 0.1444,
"step": 185
},
{
"epoch": 7.916666666666667,
"grad_norm": 4.5,
"learning_rate": 1.3421052631578948e-05,
"loss": 0.1446,
"step": 190
},
{
"epoch": 8.125,
"grad_norm": 3.375,
"learning_rate": 1.2105263157894737e-05,
"loss": 0.1381,
"step": 195
},
{
"epoch": 8.333333333333334,
"grad_norm": 4.03125,
"learning_rate": 1.0789473684210526e-05,
"loss": 0.1372,
"step": 200
},
{
"epoch": 8.541666666666666,
"grad_norm": 3.921875,
"learning_rate": 9.473684210526317e-06,
"loss": 0.1354,
"step": 205
},
{
"epoch": 8.75,
"grad_norm": 4.03125,
"learning_rate": 8.157894736842106e-06,
"loss": 0.1321,
"step": 210
},
{
"epoch": 8.958333333333334,
"grad_norm": 3.390625,
"learning_rate": 6.842105263157896e-06,
"loss": 0.1351,
"step": 215
},
{
"epoch": 9.166666666666666,
"grad_norm": 3.96875,
"learning_rate": 5.526315789473684e-06,
"loss": 0.1331,
"step": 220
},
{
"epoch": 9.375,
"grad_norm": 3.84375,
"learning_rate": 4.210526315789474e-06,
"loss": 0.129,
"step": 225
},
{
"epoch": 9.583333333333334,
"grad_norm": 3.953125,
"learning_rate": 2.8947368421052634e-06,
"loss": 0.1285,
"step": 230
},
{
"epoch": 9.791666666666666,
"grad_norm": 3.828125,
"learning_rate": 1.5789473684210528e-06,
"loss": 0.1367,
"step": 235
},
{
"epoch": 10.0,
"grad_norm": 6.5,
"learning_rate": 2.6315789473684213e-07,
"loss": 0.1335,
"step": 240
}
],
"logging_steps": 5,
"max_steps": 240,
"num_input_tokens_seen": 0,
"num_train_epochs": 10,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 635964531671040.0,
"train_batch_size": 4,
"trial_name": null,
"trial_params": null
}

3
checkpoint-240/training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c57dd6ee21e08885a384c25251b744c7c77828bafa879ca11b2116f30c68c8c3
size 5713

1
checkpoint-240/vocab.json Normal file
View File

File diff suppressed because one or more lines are too long

60
config.json Normal file
View File

@@ -0,0 +1,60 @@
{
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"dtype": "bfloat16",
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 1024,
"initializer_range": 0.02,
"intermediate_size": 3072,
"layer_types": [
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention",
"full_attention"
],
"max_position_embeddings": 40960,
"max_window_layers": 28,
"model_type": "qwen3",
"num_attention_heads": 16,
"num_hidden_layers": 28,
"num_key_value_heads": 8,
"pad_token_id": 151645,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": true,
"transformers_version": "4.56.1",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

12
generation_config.json Normal file
View File

@@ -0,0 +1,12 @@
{
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151645,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95,
"transformers_version": "4.56.1"
}

3
gguf/zen-nano-0.6b-Q4_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e7cc02f96235c30635efb16d603e0c2fec718ce35ee519873227660b521975a2
size 396704320

3
gguf/zen-nano-0.6b-Q5_K_M.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:35282e1e90055515be54f19381b613fb4730a206d75fcd0d50860d978b804653
size 444414528

3
gguf/zen-nano-0.6b-Q8_0.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:149506af2ba7f68195df57fe3aaf0152b88e651e3930501aba8b25623fbde0f9
size 639446592

3
gguf/zen-nano-0.6b-f16.gguf Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a0135a2e8b9e93c0a3a27306d84166447bb19c8fabafedbc5e3c509269e5f196
size 1198181952

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a6a9ae479f1f4509d12d547b8b5eb3c5b6b031dd7ee66f9d0ac74552dec87032
size 1192135096

3
optimizer.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ba91f17480d811dcf8acac44c6def8c1268c45461d6f7d99d16779f661226c08
size 2384451723

3
rng_state.pth Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:39e0be4f448bb33f8b4fa29459a8f8b0c1fba61268187c625e170a5ba0fb9710
size 14391

3
scheduler.pt Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:99b187bc43043d53c20c13dc7755f997532cdf7f3cb24243135ca8fbbbca7d55
size 1465

25
special_tokens_map.json Normal file
View File

@@ -0,0 +1,25 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"eos_token": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "<|im_end|>"
}

BIN
tokenizer.json (Stored with Git LFS) Normal file
View File

Binary file not shown.

239
tokenizer_config.json Normal file
View File

@@ -0,0 +1,239 @@
{
"add_bos_token": false,
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151646": {
"content": "<|object_ref_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151647": {
"content": "<|object_ref_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151648": {
"content": "<|box_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151649": {
"content": "<|box_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151650": {
"content": "<|quad_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151651": {
"content": "<|quad_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151652": {
"content": "<|vision_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151653": {
"content": "<|vision_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151654": {
"content": "<|vision_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151655": {
"content": "<|image_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151656": {
"content": "<|video_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151657": {
"content": "<tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151658": {
"content": "</tool_call>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151659": {
"content": "<|fim_prefix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151660": {
"content": "<|fim_middle|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151661": {
"content": "<|fim_suffix|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151662": {
"content": "<|fim_pad|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151663": {
"content": "<|repo_name|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151664": {
"content": "<|file_sep|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151665": {
"content": "<tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151666": {
"content": "</tool_response>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151667": {
"content": "<think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
},
"151668": {
"content": "</think>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": false
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": {},
"model_max_length": 131072,
"pad_token": "<|im_end|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

69
trainer_state.json Normal file
View File

@@ -0,0 +1,69 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 1.0,
"eval_steps": 500,
"global_step": 24,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.041666666666666664,
"grad_norm": 73.5,
"learning_rate": 0.0,
"loss": 3.9287,
"step": 1
},
{
"epoch": 0.20833333333333334,
"grad_norm": 80.5,
"learning_rate": 4.000000000000001e-06,
"loss": 3.994,
"step": 5
},
{
"epoch": 0.4166666666666667,
"grad_norm": 63.75,
"learning_rate": 9e-06,
"loss": 3.6819,
"step": 10
},
{
"epoch": 0.625,
"grad_norm": 49.25,
"learning_rate": 1.4000000000000001e-05,
"loss": 2.5702,
"step": 15
},
{
"epoch": 0.8333333333333334,
"grad_norm": 48.75,
"learning_rate": 1.9e-05,
"loss": 1.4286,
"step": 20
}
],
"logging_steps": 5,
"max_steps": 240,
"num_input_tokens_seen": 0,
"num_train_epochs": 10,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 63596453167104.0,
"train_batch_size": 4,
"trial_name": null,
"trial_params": null
}

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c57dd6ee21e08885a384c25251b744c7c77828bafa879ca11b2116f30c68c8c3
size 5713

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long