jayzou3773/Qwen1.5-MOE-sft-ESFT-law
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: jayzou3773/Qwen1.5-MOE-sft-ESFT-law
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-06-05 02:12:02 +08:00
commit 4d41752c94
22 changed files with 157506 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
.gitattributes vendored Normal file
View File

@@ -0,0 +1,36 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
tokenizer.json filter=lfs diff=lfs merge=lfs -text

57
README.md Normal file
View File

@@ -0,0 +1,57 @@
---
base_model: Qwen/Qwen1.5-MoE-A2.7B
datasets: RoxanneWsyw/ESFT-law
library_name: transformers
tags:
- generated_from_trainer
- open-r1
licence: license
---
# Model Card for None
This model is a fine-tuned version of [Qwen/Qwen1.5-MoE-A2.7B](https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B) on the [RoxanneWsyw/ESFT-law](https://huggingface.co/datasets/RoxanneWsyw/ESFT-law) dataset.
It has been trained using [TRL](https://github.com/huggingface/trl).
## Quick start
```python
from transformers import pipeline
question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
generator = pipeline("text-generation", model="None", device="cuda")
output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
print(output["generated_text"])
```
## Training procedure
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/jayzxinkai-uc-san-diego/moe-honing/runs/0eqhj6hc)
This model was trained with SFT.
### Framework versions
- TRL: 0.16.0.dev0
- Transformers: 4.49.0
- Pytorch: 2.6.0
- Datasets: 4.6.1
- Tokenizers: 0.21.4
## Citations
Cite TRL as:
```bibtex
@misc{vonwerra2022trl,
title = {{TRL: Transformer Reinforcement Learning}},
author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
year = 2020,
journal = {GitHub repository},
publisher = {GitHub},
howpublished = {\url{https://github.com/huggingface/trl}}
}
```

5
added_tokens.json Normal file
View File

@@ -0,0 +1,5 @@
{
"<|endoftext|>": 151643,
"<|im_end|>": 151645,
"<|im_start|>": 151644
}

8
all_results.json Normal file
View File

@@ -0,0 +1,8 @@
{
"total_flos": 1.2005510253379584e+17,
"train_loss": 0.40030854754149914,
"train_runtime": 469.3197,
"train_samples": 927,
"train_samples_per_second": 7.901,
"train_steps_per_second": 0.247
}

38
config.json Normal file
View File

@@ -0,0 +1,38 @@
{
"_name_or_path": "Qwen/Qwen1.5-MoE-A2.7B",
"architectures": [
"Qwen2MoeForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"decoder_sparse_step": 1,
"eos_token_id": 151643,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 5632,
"max_position_embeddings": 8192,
"max_window_layers": 21,
"mlp_only_layers": [],
"model_type": "qwen2_moe",
"moe_intermediate_size": 1408,
"norm_topk_prob": false,
"num_attention_heads": 16,
"num_experts": 60,
"num_experts_per_tok": 4,
"num_hidden_layers": 24,
"num_key_value_heads": 16,
"output_router_logits": false,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000.0,
"router_aux_loss_coef": 0.001,
"shared_expert_intermediate_size": 5632,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.49.0",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

9
generation_config.json Normal file
View File

@@ -0,0 +1,9 @@
{
"bos_token_id": 151643,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"transformers_version": "4.49.0"
}

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model-00001-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:344914f814bc9a1437226686b91bddefaf219e9546567f105a40570dc28deef3
size 4996577736

3
model-00002-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:25bfa2f383eb196bfadc91fa3a4a6c5b82e1179c70186f6283377e12f506ed2c
size 4996347752

3
model-00003-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1cc27200c2922f80528c8a144f9ed50e7aeadf75caf913c389aee87652b1b3e0
size 4997127120

3
model-00004-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:dccf6462a594bc5c4cdb82ef1ad2b5dd21733781bc8ad5de1aeeb547510ebfa3
size 4985592520

3
model-00005-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:db446901b5fcd7a7d9ed236830f0780f7c2d7fe9b796b398b6505e201c792bac
size 4996348976

3
model-00006-of-00006.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:405e31e09a8fb02627f254fde165cacbe1c5706631e061704ddaade183060287
size 3660151400

4666
model.safetensors.index.json Normal file
View File

File diff suppressed because it is too large Load Diff

14
special_tokens_map.json Normal file
View File

@@ -0,0 +1,14 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "<|endoftext|>"
}

3
tokenizer.json Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6599ce66e598632db3f472a89775688f2817760c59927d4298c0e496df39a21b
size 11418364

45
tokenizer_config.json Normal file
View File

@@ -0,0 +1,45 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"bos_token": null,
"chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
"clean_up_tokenization_spaces": false,
"eos_token": "<|endoftext|>",
"errors": "replace",
"extra_special_tokens": {},
"fast_tokenizer": true,
"model_max_length": 32768,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}

8
train_results.json Normal file
View File

@@ -0,0 +1,8 @@
{
"total_flos": 1.2005510253379584e+17,
"train_loss": 0.40030854754149914,
"train_runtime": 469.3197,
"train_samples": 927,
"train_samples_per_second": 7.901,
"train_steps_per_second": 0.247
}

970
trainer_state.json Normal file
View File

@@ -0,0 +1,970 @@
{
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 4.0,
"eval_steps": 500,
"global_step": 116,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.034482758620689655,
"grad_norm": 40.85477828979492,
"learning_rate": 8.333333333333333e-07,
"loss": 1.3774,
"mean_token_accuracy": 0.740469753742218,
"step": 1
},
{
"epoch": 0.06896551724137931,
"grad_norm": 31.841533660888672,
"learning_rate": 1.6666666666666667e-06,
"loss": 1.1835,
"mean_token_accuracy": 0.7526056170463562,
"step": 2
},
{
"epoch": 0.10344827586206896,
"grad_norm": 35.83158493041992,
"learning_rate": 2.5e-06,
"loss": 1.1499,
"mean_token_accuracy": 0.7815449237823486,
"step": 3
},
{
"epoch": 0.13793103448275862,
"grad_norm": 32.397979736328125,
"learning_rate": 3.3333333333333333e-06,
"loss": 1.2419,
"mean_token_accuracy": 0.7577303051948547,
"step": 4
},
{
"epoch": 0.1724137931034483,
"grad_norm": 31.138023376464844,
"learning_rate": 4.166666666666667e-06,
"loss": 1.244,
"mean_token_accuracy": 0.746874988079071,
"step": 5
},
{
"epoch": 0.20689655172413793,
"grad_norm": 18.774599075317383,
"learning_rate": 5e-06,
"loss": 1.007,
"mean_token_accuracy": 0.7697392702102661,
"step": 6
},
{
"epoch": 0.2413793103448276,
"grad_norm": 17.945812225341797,
"learning_rate": 5.833333333333334e-06,
"loss": 1.0226,
"mean_token_accuracy": 0.7652671933174133,
"step": 7
},
{
"epoch": 0.27586206896551724,
"grad_norm": 8.929885864257812,
"learning_rate": 6.666666666666667e-06,
"loss": 1.0524,
"mean_token_accuracy": 0.7645803689956665,
"step": 8
},
{
"epoch": 0.3103448275862069,
"grad_norm": 7.886545181274414,
"learning_rate": 7.500000000000001e-06,
"loss": 0.7061,
"mean_token_accuracy": 0.8247748017311096,
"step": 9
},
{
"epoch": 0.3448275862068966,
"grad_norm": 6.431151390075684,
"learning_rate": 8.333333333333334e-06,
"loss": 0.7137,
"mean_token_accuracy": 0.8260869383811951,
"step": 10
},
{
"epoch": 0.3793103448275862,
"grad_norm": 6.6982502937316895,
"learning_rate": 9.166666666666666e-06,
"loss": 0.9198,
"mean_token_accuracy": 0.7834821343421936,
"step": 11
},
{
"epoch": 0.41379310344827586,
"grad_norm": 6.049954414367676,
"learning_rate": 1e-05,
"loss": 0.853,
"mean_token_accuracy": 0.7753646969795227,
"step": 12
},
{
"epoch": 0.4482758620689655,
"grad_norm": 6.920111656188965,
"learning_rate": 9.997947030202901e-06,
"loss": 0.7117,
"mean_token_accuracy": 0.8271827101707458,
"step": 13
},
{
"epoch": 0.4827586206896552,
"grad_norm": 6.141343593597412,
"learning_rate": 9.991789994004929e-06,
"loss": 0.7161,
"mean_token_accuracy": 0.8164419531822205,
"step": 14
},
{
"epoch": 0.5172413793103449,
"grad_norm": 5.594657897949219,
"learning_rate": 9.98153450927691e-06,
"loss": 0.623,
"mean_token_accuracy": 0.842275083065033,
"step": 15
},
{
"epoch": 0.5517241379310345,
"grad_norm": 5.3693742752075195,
"learning_rate": 9.967189933441243e-06,
"loss": 0.7262,
"mean_token_accuracy": 0.8148788809776306,
"step": 16
},
{
"epoch": 0.5862068965517241,
"grad_norm": 5.857855796813965,
"learning_rate": 9.948769354933905e-06,
"loss": 0.7161,
"mean_token_accuracy": 0.7958254218101501,
"step": 17
},
{
"epoch": 0.6206896551724138,
"grad_norm": 4.9951581954956055,
"learning_rate": 9.926289581262147e-06,
"loss": 0.7478,
"mean_token_accuracy": 0.7893890738487244,
"step": 18
},
{
"epoch": 0.6551724137931034,
"grad_norm": 5.202838897705078,
"learning_rate": 9.899771123668811e-06,
"loss": 0.6994,
"mean_token_accuracy": 0.8181459307670593,
"step": 19
},
{
"epoch": 0.6896551724137931,
"grad_norm": 5.341423988342285,
"learning_rate": 9.869238178417235e-06,
"loss": 0.618,
"mean_token_accuracy": 0.8386844396591187,
"step": 20
},
{
"epoch": 0.7241379310344828,
"grad_norm": 5.327294826507568,
"learning_rate": 9.834718604713825e-06,
"loss": 0.7653,
"mean_token_accuracy": 0.8060739040374756,
"step": 21
},
{
"epoch": 0.7586206896551724,
"grad_norm": 5.64478874206543,
"learning_rate": 9.796243899288455e-06,
"loss": 0.7214,
"mean_token_accuracy": 0.8227323889732361,
"step": 22
},
{
"epoch": 0.7931034482758621,
"grad_norm": 5.176536560058594,
"learning_rate": 9.753849167655881e-06,
"loss": 0.8185,
"mean_token_accuracy": 0.8079403042793274,
"step": 23
},
{
"epoch": 0.8275862068965517,
"grad_norm": 4.750691890716553,
"learning_rate": 9.707573092084368e-06,
"loss": 0.7444,
"mean_token_accuracy": 0.8039276003837585,
"step": 24
},
{
"epoch": 0.8620689655172413,
"grad_norm": 5.0928473472595215,
"learning_rate": 9.65745789630079e-06,
"loss": 0.5925,
"mean_token_accuracy": 0.8463665246963501,
"step": 25
},
{
"epoch": 0.896551724137931,
"grad_norm": 4.8972086906433105,
"learning_rate": 9.603549306964408e-06,
"loss": 0.6491,
"mean_token_accuracy": 0.8238112926483154,
"step": 26
},
{
"epoch": 0.9310344827586207,
"grad_norm": 5.463573932647705,
"learning_rate": 9.545896511944416e-06,
"loss": 0.7093,
"mean_token_accuracy": 0.8327044248580933,
"step": 27
},
{
"epoch": 0.9655172413793104,
"grad_norm": 4.660519599914551,
"learning_rate": 9.484552115439445e-06,
"loss": 0.6958,
"mean_token_accuracy": 0.8255624175071716,
"step": 28
},
{
"epoch": 1.0,
"grad_norm": 5.265268325805664,
"learning_rate": 9.419572089979821e-06,
"loss": 0.8103,
"mean_token_accuracy": 0.8134814500808716,
"step": 29
},
{
"epoch": 1.0344827586206897,
"grad_norm": 6.0437774658203125,
"learning_rate": 9.351015725356515e-06,
"loss": 0.4465,
"mean_token_accuracy": 0.896746814250946,
"step": 30
},
{
"epoch": 1.0689655172413792,
"grad_norm": 4.806868553161621,
"learning_rate": 9.278945574523293e-06,
"loss": 0.5036,
"mean_token_accuracy": 0.8651062250137329,
"step": 31
},
{
"epoch": 1.103448275862069,
"grad_norm": 5.691929817199707,
"learning_rate": 9.203427396521454e-06,
"loss": 0.523,
"mean_token_accuracy": 0.8620548248291016,
"step": 32
},
{
"epoch": 1.1379310344827587,
"grad_norm": 4.686606407165527,
"learning_rate": 9.124530096479258e-06,
"loss": 0.5221,
"mean_token_accuracy": 0.8660033345222473,
"step": 33
},
{
"epoch": 1.1724137931034484,
"grad_norm": 3.97752046585083,
"learning_rate": 9.042325662740726e-06,
"loss": 0.5354,
"mean_token_accuracy": 0.8449214100837708,
"step": 34
},
{
"epoch": 1.206896551724138,
"grad_norm": 5.035070896148682,
"learning_rate": 8.956889101181262e-06,
"loss": 0.4946,
"mean_token_accuracy": 0.8720434308052063,
"step": 35
},
{
"epoch": 1.2413793103448276,
"grad_norm": 4.9860100746154785,
"learning_rate": 8.868298366769956e-06,
"loss": 0.6021,
"mean_token_accuracy": 0.8342487812042236,
"step": 36
},
{
"epoch": 1.2758620689655173,
"grad_norm": 4.826323509216309,
"learning_rate": 8.776634292441049e-06,
"loss": 0.4321,
"mean_token_accuracy": 0.884772002696991,
"step": 37
},
{
"epoch": 1.3103448275862069,
"grad_norm": 5.355347633361816,
"learning_rate": 8.681980515339464e-06,
"loss": 0.4768,
"mean_token_accuracy": 0.8703784346580505,
"step": 38
},
{
"epoch": 1.3448275862068966,
"grad_norm": 5.1744842529296875,
"learning_rate": 8.584423400507679e-06,
"loss": 0.4668,
"mean_token_accuracy": 0.8853046298027039,
"step": 39
},
{
"epoch": 1.3793103448275863,
"grad_norm": 5.17063045501709,
"learning_rate": 8.484051962083579e-06,
"loss": 0.4847,
"mean_token_accuracy": 0.8667626976966858,
"step": 40
},
{
"epoch": 1.4137931034482758,
"grad_norm": 4.590514183044434,
"learning_rate": 8.380957782081198e-06,
"loss": 0.42,
"mean_token_accuracy": 0.8825867176055908,
"step": 41
},
{
"epoch": 1.4482758620689655,
"grad_norm": 5.383031845092773,
"learning_rate": 8.275234926828446e-06,
"loss": 0.3628,
"mean_token_accuracy": 0.9045745134353638,
"step": 42
},
{
"epoch": 1.4827586206896552,
"grad_norm": 4.623243808746338,
"learning_rate": 8.166979861138076e-06,
"loss": 0.4927,
"mean_token_accuracy": 0.8554299473762512,
"step": 43
},
{
"epoch": 1.5172413793103448,
"grad_norm": 4.59343147277832,
"learning_rate": 8.056291360290202e-06,
"loss": 0.339,
"mean_token_accuracy": 0.9112599492073059,
"step": 44
},
{
"epoch": 1.5517241379310345,
"grad_norm": 4.420445442199707,
"learning_rate": 7.943270419906655e-06,
"loss": 0.4281,
"mean_token_accuracy": 0.8751758337020874,
"step": 45
},
{
"epoch": 1.5862068965517242,
"grad_norm": 4.380794048309326,
"learning_rate": 7.828020163799455e-06,
"loss": 0.4444,
"mean_token_accuracy": 0.8713410496711731,
"step": 46
},
{
"epoch": 1.6206896551724137,
"grad_norm": 4.382615566253662,
"learning_rate": 7.710645749877448e-06,
"loss": 0.3932,
"mean_token_accuracy": 0.8761938214302063,
"step": 47
},
{
"epoch": 1.6551724137931034,
"grad_norm": 4.465359210968018,
"learning_rate": 7.5912542741969585e-06,
"loss": 0.4346,
"mean_token_accuracy": 0.8759894371032715,
"step": 48
},
{
"epoch": 1.6896551724137931,
"grad_norm": 4.440632343292236,
"learning_rate": 7.469954673244032e-06,
"loss": 0.38,
"mean_token_accuracy": 0.8933333158493042,
"step": 49
},
{
"epoch": 1.7241379310344827,
"grad_norm": 4.296090126037598,
"learning_rate": 7.346857624537407e-06,
"loss": 0.4955,
"mean_token_accuracy": 0.861520528793335,
"step": 50
},
{
"epoch": 1.7586206896551724,
"grad_norm": 4.688018798828125,
"learning_rate": 7.222075445642904e-06,
"loss": 0.3928,
"mean_token_accuracy": 0.8793990015983582,
"step": 51
},
{
"epoch": 1.793103448275862,
"grad_norm": 4.597250461578369,
"learning_rate": 7.095721991691411e-06,
"loss": 0.4669,
"mean_token_accuracy": 0.8728665709495544,
"step": 52
},
{
"epoch": 1.8275862068965516,
"grad_norm": 4.70115852355957,
"learning_rate": 6.967912551493913e-06,
"loss": 0.3774,
"mean_token_accuracy": 0.904330313205719,
"step": 53
},
{
"epoch": 1.8620689655172413,
"grad_norm": 3.8735404014587402,
"learning_rate": 6.838763742348414e-06,
"loss": 0.4539,
"mean_token_accuracy": 0.8638014793395996,
"step": 54
},
{
"epoch": 1.896551724137931,
"grad_norm": 3.819071054458618,
"learning_rate": 6.708393403634696e-06,
"loss": 0.3833,
"mean_token_accuracy": 0.8801261782646179,
"step": 55
},
{
"epoch": 1.9310344827586206,
"grad_norm": 3.6699230670928955,
"learning_rate": 6.576920489294011e-06,
"loss": 0.4962,
"mean_token_accuracy": 0.8549042344093323,
"step": 56
},
{
"epoch": 1.9655172413793105,
"grad_norm": 3.6867268085479736,
"learning_rate": 6.444464959291814e-06,
"loss": 0.4416,
"mean_token_accuracy": 0.8705803751945496,
"step": 57
},
{
"epoch": 2.0,
"grad_norm": 3.870262861251831,
"learning_rate": 6.311147670162576e-06,
"loss": 0.4248,
"mean_token_accuracy": 0.8728753328323364,
"step": 58
},
{
"epoch": 2.0344827586206895,
"grad_norm": 3.8891749382019043,
"learning_rate": 6.177090264736525e-06,
"loss": 0.2321,
"mean_token_accuracy": 0.929407000541687,
"step": 59
},
{
"epoch": 2.0689655172413794,
"grad_norm": 3.4715583324432373,
"learning_rate": 6.042415061148954e-06,
"loss": 0.2797,
"mean_token_accuracy": 0.9197627305984497,
"step": 60
},
{
"epoch": 2.103448275862069,
"grad_norm": 4.050957679748535,
"learning_rate": 5.907244941233371e-06,
"loss": 0.2365,
"mean_token_accuracy": 0.9324618577957153,
"step": 61
},
{
"epoch": 2.1379310344827585,
"grad_norm": 3.6912150382995605,
"learning_rate": 5.771703238400288e-06,
"loss": 0.209,
"mean_token_accuracy": 0.9365726709365845,
"step": 62
},
{
"epoch": 2.1724137931034484,
"grad_norm": 3.456387758255005,
"learning_rate": 5.635913625104001e-06,
"loss": 0.2006,
"mean_token_accuracy": 0.9414021372795105,
"step": 63
},
{
"epoch": 2.206896551724138,
"grad_norm": 3.8323400020599365,
"learning_rate": 5.500000000000001e-06,
"loss": 0.1921,
"mean_token_accuracy": 0.9395722150802612,
"step": 64
},
{
"epoch": 2.2413793103448274,
"grad_norm": 3.9986572265625,
"learning_rate": 5.3640863748960016e-06,
"loss": 0.1956,
"mean_token_accuracy": 0.9349231719970703,
"step": 65
},
{
"epoch": 2.2758620689655173,
"grad_norm": 4.387889862060547,
"learning_rate": 5.228296761599713e-06,
"loss": 0.1881,
"mean_token_accuracy": 0.9423368573188782,
"step": 66
},
{
"epoch": 2.310344827586207,
"grad_norm": 3.891286849975586,
"learning_rate": 5.092755058766631e-06,
"loss": 0.1887,
"mean_token_accuracy": 0.9313790798187256,
"step": 67
},
{
"epoch": 2.344827586206897,
"grad_norm": 4.44399356842041,
"learning_rate": 4.957584938851048e-06,
"loss": 0.2098,
"mean_token_accuracy": 0.9372294545173645,
"step": 68
},
{
"epoch": 2.3793103448275863,
"grad_norm": 4.2808051109313965,
"learning_rate": 4.822909735263477e-06,
"loss": 0.2379,
"mean_token_accuracy": 0.923653244972229,
"step": 69
},
{
"epoch": 2.413793103448276,
"grad_norm": 4.003400802612305,
"learning_rate": 4.6888523298374245e-06,
"loss": 0.1771,
"mean_token_accuracy": 0.9492968320846558,
"step": 70
},
{
"epoch": 2.4482758620689653,
"grad_norm": 3.573258399963379,
"learning_rate": 4.555535040708186e-06,
"loss": 0.1843,
"mean_token_accuracy": 0.9385930895805359,
"step": 71
},
{
"epoch": 2.4827586206896552,
"grad_norm": 3.7771646976470947,
"learning_rate": 4.423079510705992e-06,
"loss": 0.2245,
"mean_token_accuracy": 0.9214980006217957,
"step": 72
},
{
"epoch": 2.5172413793103448,
"grad_norm": 3.48586106300354,
"learning_rate": 4.2916065963653045e-06,
"loss": 0.2004,
"mean_token_accuracy": 0.9363507628440857,
"step": 73
},
{
"epoch": 2.5517241379310347,
"grad_norm": 3.424337863922119,
"learning_rate": 4.1612362576515875e-06,
"loss": 0.2171,
"mean_token_accuracy": 0.9295426607131958,
"step": 74
},
{
"epoch": 2.586206896551724,
"grad_norm": 3.287777900695801,
"learning_rate": 4.032087448506089e-06,
"loss": 0.2347,
"mean_token_accuracy": 0.9248573184013367,
"step": 75
},
{
"epoch": 2.6206896551724137,
"grad_norm": 3.127174139022827,
"learning_rate": 3.904278008308589e-06,
"loss": 0.2315,
"mean_token_accuracy": 0.919683575630188,
"step": 76
},
{
"epoch": 2.655172413793103,
"grad_norm": 2.9863779544830322,
"learning_rate": 3.777924554357096e-06,
"loss": 0.2087,
"mean_token_accuracy": 0.9256926774978638,
"step": 77
},
{
"epoch": 2.689655172413793,
"grad_norm": 3.1884548664093018,
"learning_rate": 3.653142375462596e-06,
"loss": 0.2092,
"mean_token_accuracy": 0.9343122243881226,
"step": 78
},
{
"epoch": 2.7241379310344827,
"grad_norm": 2.532046318054199,
"learning_rate": 3.5300453267559676e-06,
"loss": 0.2545,
"mean_token_accuracy": 0.9118472933769226,
"step": 79
},
{
"epoch": 2.7586206896551726,
"grad_norm": 2.7691566944122314,
"learning_rate": 3.408745725803042e-06,
"loss": 0.2153,
"mean_token_accuracy": 0.9324372410774231,
"step": 80
},
{
"epoch": 2.793103448275862,
"grad_norm": 2.883653402328491,
"learning_rate": 3.2893542501225535e-06,
"loss": 0.177,
"mean_token_accuracy": 0.946940541267395,
"step": 81
},
{
"epoch": 2.8275862068965516,
"grad_norm": 3.113863706588745,
"learning_rate": 3.1719798362005444e-06,
"loss": 0.2499,
"mean_token_accuracy": 0.9145299196243286,
"step": 82
},
{
"epoch": 2.862068965517241,
"grad_norm": 2.6909308433532715,
"learning_rate": 3.056729580093346e-06,
"loss": 0.2271,
"mean_token_accuracy": 0.9211804270744324,
"step": 83
},
{
"epoch": 2.896551724137931,
"grad_norm": 2.7156546115875244,
"learning_rate": 2.9437086397097996e-06,
"loss": 0.2328,
"mean_token_accuracy": 0.9269275069236755,
"step": 84
},
{
"epoch": 2.9310344827586206,
"grad_norm": 2.657344102859497,
"learning_rate": 2.8330201388619257e-06,
"loss": 0.2214,
"mean_token_accuracy": 0.9271735548973083,
"step": 85
},
{
"epoch": 2.9655172413793105,
"grad_norm": 2.594271183013916,
"learning_rate": 2.7247650731715563e-06,
"loss": 0.1721,
"mean_token_accuracy": 0.9469696879386902,
"step": 86
},
{
"epoch": 3.0,
"grad_norm": 3.0993478298187256,
"learning_rate": 2.6190422179188046e-06,
"loss": 0.2366,
"mean_token_accuracy": 0.9250257015228271,
"step": 87
},
{
"epoch": 3.0344827586206895,
"grad_norm": 2.1011862754821777,
"learning_rate": 2.515948037916423e-06,
"loss": 0.1081,
"mean_token_accuracy": 0.9683519601821899,
"step": 88
},
{
"epoch": 3.0689655172413794,
"grad_norm": 2.291515588760376,
"learning_rate": 2.415576599492321e-06,
"loss": 0.1186,
"mean_token_accuracy": 0.961465060710907,
"step": 89
},
{
"epoch": 3.103448275862069,
"grad_norm": 2.1782500743865967,
"learning_rate": 2.3180194846605367e-06,
"loss": 0.0928,
"mean_token_accuracy": 0.9697393178939819,
"step": 90
},
{
"epoch": 3.1379310344827585,
"grad_norm": 2.3693504333496094,
"learning_rate": 2.223365707558953e-06,
"loss": 0.0922,
"mean_token_accuracy": 0.9721619486808777,
"step": 91
},
{
"epoch": 3.1724137931034484,
"grad_norm": 2.1274962425231934,
"learning_rate": 2.131701633230045e-06,
"loss": 0.0829,
"mean_token_accuracy": 0.9756795167922974,
"step": 92
},
{
"epoch": 3.206896551724138,
"grad_norm": 1.9486541748046875,
"learning_rate": 2.043110898818738e-06,
"loss": 0.0728,
"mean_token_accuracy": 0.9797441363334656,
"step": 93
},
{
"epoch": 3.2413793103448274,
"grad_norm": 2.245460033416748,
"learning_rate": 1.957674337259275e-06,
"loss": 0.1177,
"mean_token_accuracy": 0.9649968147277832,
"step": 94
},
{
"epoch": 3.2758620689655173,
"grad_norm": 2.6302707195281982,
"learning_rate": 1.875469903520743e-06,
"loss": 0.1082,
"mean_token_accuracy": 0.9706632494926453,
"step": 95
},
{
"epoch": 3.310344827586207,
"grad_norm": 2.351165533065796,
"learning_rate": 1.7965726034785466e-06,
"loss": 0.0791,
"mean_token_accuracy": 0.9741272926330566,
"step": 96
},
{
"epoch": 3.344827586206897,
"grad_norm": 2.1048247814178467,
"learning_rate": 1.7210544254767098e-06,
"loss": 0.0611,
"mean_token_accuracy": 0.977523684501648,
"step": 97
},
{
"epoch": 3.3793103448275863,
"grad_norm": 2.681243419647217,
"learning_rate": 1.648984274643487e-06,
"loss": 0.0869,
"mean_token_accuracy": 0.9723304510116577,
"step": 98
},
{
"epoch": 3.413793103448276,
"grad_norm": 2.781121015548706,
"learning_rate": 1.5804279100201799e-06,
"loss": 0.0826,
"mean_token_accuracy": 0.969332754611969,
"step": 99
},
{
"epoch": 3.4482758620689653,
"grad_norm": 3.5514438152313232,
"learning_rate": 1.515447884560556e-06,
"loss": 0.0885,
"mean_token_accuracy": 0.9686388373374939,
"step": 100
},
{
"epoch": 3.4827586206896552,
"grad_norm": 2.2951314449310303,
"learning_rate": 1.4541034880555837e-06,
"loss": 0.077,
"mean_token_accuracy": 0.9765592217445374,
"step": 101
},
{
"epoch": 3.5172413793103448,
"grad_norm": 2.6437036991119385,
"learning_rate": 1.3964506930355947e-06,
"loss": 0.0775,
"mean_token_accuracy": 0.9763970971107483,
"step": 102
},
{
"epoch": 3.5517241379310347,
"grad_norm": 3.129058599472046,
"learning_rate": 1.3425421036992098e-06,
"loss": 0.0979,
"mean_token_accuracy": 0.9674403667449951,
"step": 103
},
{
"epoch": 3.586206896551724,
"grad_norm": 2.5986857414245605,
"learning_rate": 1.292426907915634e-06,
"loss": 0.0914,
"mean_token_accuracy": 0.9685618877410889,
"step": 104
},
{
"epoch": 3.6206896551724137,
"grad_norm": 2.418269634246826,
"learning_rate": 1.2461508323441185e-06,
"loss": 0.0777,
"mean_token_accuracy": 0.9750581979751587,
"step": 105
},
{
"epoch": 3.655172413793103,
"grad_norm": 2.7729599475860596,
"learning_rate": 1.203756100711545e-06,
"loss": 0.0733,
"mean_token_accuracy": 0.9765625,
"step": 106
},
{
"epoch": 3.689655172413793,
"grad_norm": 2.909587860107422,
"learning_rate": 1.165281395286177e-06,
"loss": 0.0986,
"mean_token_accuracy": 0.9670926332473755,
"step": 107
},
{
"epoch": 3.7241379310344827,
"grad_norm": 2.416217803955078,
"learning_rate": 1.130761821582766e-06,
"loss": 0.0695,
"mean_token_accuracy": 0.9779465198516846,
"step": 108
},
{
"epoch": 3.7586206896551726,
"grad_norm": 2.8723764419555664,
"learning_rate": 1.1002288763311892e-06,
"loss": 0.0989,
"mean_token_accuracy": 0.968372642993927,
"step": 109
},
{
"epoch": 3.793103448275862,
"grad_norm": 2.604686975479126,
"learning_rate": 1.0737104187378543e-06,
"loss": 0.0971,
"mean_token_accuracy": 0.9691616892814636,
"step": 110
},
{
"epoch": 3.8275862068965516,
"grad_norm": 2.554763078689575,
"learning_rate": 1.0512306450660966e-06,
"loss": 0.0877,
"mean_token_accuracy": 0.9676067233085632,
"step": 111
},
{
"epoch": 3.862068965517241,
"grad_norm": 2.778059482574463,
"learning_rate": 1.0328100665587573e-06,
"loss": 0.0753,
"mean_token_accuracy": 0.9727236032485962,
"step": 112
},
{
"epoch": 3.896551724137931,
"grad_norm": 2.484534502029419,
"learning_rate": 1.0184654907230909e-06,
"loss": 0.0851,
"mean_token_accuracy": 0.9729294180870056,
"step": 113
},
{
"epoch": 3.9310344827586206,
"grad_norm": 2.6316423416137695,
"learning_rate": 1.0082100059950713e-06,
"loss": 0.0697,
"mean_token_accuracy": 0.972983717918396,
"step": 114
},
{
"epoch": 3.9655172413793105,
"grad_norm": 2.666123867034912,
"learning_rate": 1.0020529697970998e-06,
"loss": 0.0832,
"mean_token_accuracy": 0.9710332751274109,
"step": 115
},
{
"epoch": 4.0,
"grad_norm": 2.6746883392333984,
"learning_rate": 1.0000000000000002e-06,
"loss": 0.089,
"mean_token_accuracy": 0.9708768725395203,
"step": 116
},
{
"epoch": 4.0,
"step": 116,
"total_flos": 1.2005510253379584e+17,
"train_loss": 0.40030854754149914,
"train_runtime": 469.3197,
"train_samples_per_second": 7.901,
"train_steps_per_second": 0.247
}
],
"logging_steps": 1,
"max_steps": 116,
"num_input_tokens_seen": 0,
"num_train_epochs": 4,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": false,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 1.2005510253379584e+17,
"train_batch_size": 4,
"trial_name": null,
"trial_params": null
}

237
training.log Normal file
View File

@@ -0,0 +1,237 @@
2026-03-06 06:37:54 - INFO - __main__ - Model parameters ModelConfig(model_name_or_path='Qwen/Qwen1.5-MoE-A2.7B', model_revision='main', torch_dtype='bfloat16', trust_remote_code=True, attn_implementation='flash_attention_2', use_peft=False, lora_r=16, lora_alpha=32, lora_dropout=0.05, lora_target_modules=None, lora_modules_to_save=None, lora_task_type='CAUSAL_LM', use_rslora=False, load_in_8bit=False, load_in_4bit=False, bnb_4bit_quant_type='nf4', use_bnb_nested_quant=False)
2026-03-06 06:37:54 - INFO - __main__ - Script parameters ScriptArguments(dataset_name='RoxanneWsyw/ESFT-law', dataset_config=None, dataset_train_split='train', dataset_test_split='test', gradient_checkpointing_use_reentrant=False, ignore_bias_buffers=False)
2026-03-06 06:37:54 - INFO - __main__ - Training parameters SFTConfig(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
attn_kl_weight=1.0,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
benchmarks=[],
bf16=True,
bf16_full_eval=False,
callbacks=[],
chars_per_token=<CHARS_PER_TOKEN>,
chat_template=None,
cluster_mode=hierarchical-dynamic,
cluster_num_groups=None,
cluster_prune_ratio=None,
cluster_prune_tau=1.0,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
dataset_batch_size=None,
dataset_kwargs=None,
dataset_num_proc=None,
dataset_text_field=text,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800000000,
debug=[],
deepspeed=None,
disable_teacher_dropout=True,
disable_tqdm=False,
dispatch_batches=None,
do_eval=True,
do_predict=False,
do_train=False,
entropy_slope_alpha=1.0,
entropy_slope_beta=1.0,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_packing=None,
eval_steps=None,
eval_strategy=IntervalStrategy.NO,
eval_use_gather_object=False,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=True,
gradient_checkpointing_kwargs={'use_reentrant': False},
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_model_revision=main,
hub_private_repo=None,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
last_entropy_weight=1.0,
layer_entropy_l1_layers=None,
layer_entropy_l1_weight=1.0,
learning_rate=1e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=info,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=/project/flame/haozeh/llm-honing/sft_models/Qwen1.5-MOE-sft-ESFT-law/runs/Mar06_06-37-51_orchard-flame-9,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=1,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_kwargs={'min_lr_rate': 0.1},
lr_scheduler_type=SchedulerType.COSINE_WITH_MIN_LR,
max_grad_norm=1.0,
max_length=4096,
max_seq_length=None,
max_steps=-1,
merging_metrics=None,
metric_for_best_model=None,
model_init_kwargs=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_of_sequences=None,
num_train_epochs=4.0,
optim=OptimizerNames.ADAMW_TORCH,
optim_args=None,
optim_target_modules=None,
output_dir=/project/flame/haozeh/llm-honing/sft_models/Qwen1.5-MOE-sft-ESFT-law,
overwrite_hub_revision=False,
overwrite_output_dir=True,
packing=False,
past_index=-1,
per_device_eval_batch_size=16,
per_device_train_batch_size=4,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_revision=False,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
remove_unused_columns=True,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
router_manual_mask=None,
router_prune_enable=True,
router_prune_expert_per_layer=None,
router_prune_interval=5,
router_prune_min_keep=1,
router_prune_start_step=None,
router_prune_step_size=32,
router_prune_use_plan=True,
run_name=sft-base-ESFT-law-epoch4,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=SaveStrategy.NO,
save_total_limit=None,
seed=1234,
skip_memory_metrics=True,
split_batches=None,
system_prompt=None,
teacher_attn_implementation=None,
teacher_model_name_or_path=None,
teacher_model_revision=None,
teacher_torch_dtype=auto,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger=False,
use_liger_kernel=False,
use_mps_device=False,
wandb_entity=jayzxinkai-uc-san-diego,
wandb_project=moe-honing,
warmup_ratio=0.1,
warmup_steps=0,
weight_decay=0.0,
weight_feature_rank=None,
)
2026-03-06 06:37:55 - INFO - datasets.builder - Generating dataset esft-law (/tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879)
2026-03-06 06:37:55 - INFO - datasets.builder - Downloading and preparing dataset esft-law/default (download: 1.39 MiB, generated: 1.36 MiB, post-processed: Unknown size, total: 2.75 MiB) to /tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879...
2026-03-06 06:37:55 - INFO - datasets.download.download_manager - Downloading took 0.0 min
2026-03-06 06:37:55 - INFO - datasets.download.download_manager - Checksum Computation took 0.0 min
2026-03-06 06:37:55 - INFO - datasets.builder - Generating train split
2026-03-06 06:37:55 - INFO - datasets.builder - Generating test split
2026-03-06 06:37:55 - INFO - datasets.utils.info_utils - All the splits matched successfully.
2026-03-06 06:37:55 - INFO - datasets.builder - Dataset esft-law downloaded and prepared to /tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879. Subsequent calls will reuse this data.
2026-03-06 06:37:55 - INFO - datasets.arrow_dataset - Caching processed dataset at /tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879/cache-8cb72a2fbdf09830.arrow
2026-03-06 06:37:55 - INFO - datasets.arrow_dataset - Caching processed dataset at /tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879/cache-8c77f4e889bc2190.arrow
2026-03-06 06:37:56 - INFO - __main__ - *** Initializing model kwargs ***
2026-03-06 06:38:03 - INFO - datasets.arrow_dataset - Caching processed dataset at /tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879/cache-8f73e46c2590024d.arrow
2026-03-06 06:38:03 - INFO - datasets.arrow_dataset - Caching processed dataset at /tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879/cache-d11f855ddd766bdb.arrow
2026-03-06 06:38:03 - INFO - datasets.arrow_dataset - Caching processed dataset at /tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879/cache-bf96c76aa98c8d10.arrow
2026-03-06 06:38:03 - INFO - datasets.arrow_dataset - Caching processed dataset at /tmp/hf_cache/datasets/RoxanneWsyw___esft-law/default/0.0.0/d99c2b0fe818e485df3cb66c1cdf2db27db6b879/cache-8073ced6ef4756ef.arrow
2026-03-06 06:38:09 - INFO - __main__ - *** Train ***
2026-03-06 06:38:09 - INFO - __main__ - Qwen2MoeForCausalLM(
(model): Qwen2MoeModel(
(embed_tokens): Embedding(151936, 2048)
(layers): ModuleList(
(0-23): 24 x Qwen2MoeDecoderLayer(
(self_attn): Qwen2MoeFlashAttention2(
(q_proj): Linear(in_features=2048, out_features=2048, bias=True)
(k_proj): Linear(in_features=2048, out_features=2048, bias=True)
(v_proj): Linear(in_features=2048, out_features=2048, bias=True)
(o_proj): Linear(in_features=2048, out_features=2048, bias=False)
(rotary_emb): Qwen2MoeRotaryEmbedding()
)
(mlp): Qwen2MoeSparseMoeBlock(
(gate): Linear(in_features=2048, out_features=60, bias=False)
(experts): ModuleList(
(0-59): 60 x Qwen2MoeMLP(
(gate_proj): Linear(in_features=2048, out_features=1408, bias=False)
(up_proj): Linear(in_features=2048, out_features=1408, bias=False)
(down_proj): Linear(in_features=1408, out_features=2048, bias=False)
(act_fn): SiLU()
)
)
(shared_expert): Qwen2MoeMLP(
(gate_proj): Linear(in_features=2048, out_features=5632, bias=False)
(up_proj): Linear(in_features=2048, out_features=5632, bias=False)
(down_proj): Linear(in_features=5632, out_features=2048, bias=False)
(act_fn): SiLU()
)
(shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
)
(input_layernorm): Qwen2MoeRMSNorm((2048,), eps=1e-06)
(post_attention_layernorm): Qwen2MoeRMSNorm((2048,), eps=1e-06)
)
)
(norm): Qwen2MoeRMSNorm((2048,), eps=1e-06)
(rotary_emb): Qwen2MoeRotaryEmbedding()
)
(lm_head): Linear(in_features=2048, out_features=151936, bias=False)
)
2026-03-06 06:47:12 - INFO - __main__ - *** Save model ***
2026-03-06 06:49:31 - INFO - __main__ - Model saved to /project/flame/haozeh/llm-honing/sft_models/Qwen1.5-MOE-sft-ESFT-law

3
training_args.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:027f977808d8eec5acd5b44eb4bea1f8355149ad20555b5746dedf04e8b7c5c7
size 8504

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long