78 lines
118 KiB
Plaintext
78 lines
118 KiB
Plaintext
|
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
|
|||
|
|
`torch_dtype` is deprecated! Use `dtype` instead!
|
|||
|
|
============================================================
|
|||
|
|
DeepSentinel GRPO Training
|
|||
|
|
============================================================
|
|||
|
|
|
|||
|
|
[1] Collecting 120 training episodes...
|
|||
|
|
Collected 120 episodes
|
|||
|
|
Distribution: 40 easy, 40 medium, 40 hard
|
|||
|
|
|
|||
|
|
[2] Evaluating heuristic baseline...
|
|||
|
|
easy: avg_reward=-0.3681
|
|||
|
|
medium: avg_reward=0.0049
|
|||
|
|
hard: avg_reward=0.0833
|
|||
|
|
|
|||
|
|
[3] Loading model: Qwen/Qwen2.5-0.5B-Instruct
|
|||
|
|
Loading weights: 0%| | 0/290 [00:00<?, ?it/s]
Loading weights: 0%| | 1/290 [00:00<00:00, 8738.13it/s, Materializing param=model.embed_tokens.weight]
Loading weights: 0%| | 1/290 [00:00<00:00, 2663.05it/s, Materializing param=model.embed_tokens.weight]
Loading weights: 1%| | 2/290 [00:00<00:05, 49.37it/s, Materializing param=model.layers.0.input_layernorm.weight]
Loading weights: 1%| | 2/290 [00:00<00:05, 49.13it/s, Materializing param=model.layers.0.input_layernorm.weight]
Loading weights: 1%| | 3/290 [00:00<00:03, 73.10it/s, Materializing param=model.layers.0.mlp.down_proj.weight]
Loading weights: 1%| | 3/290 [00:00<00:03, 72.82it/s, Materializing param=model.layers.0.mlp.down_proj.weight]
Loading weights: 1%|▏ | 4/290 [00:00<00:03, 94.80it/s, Materializing param=model.layers.0.mlp.gate_proj.weight]
Loading weights: 1%|▏ | 4/290 [00:00<00:03, 94.30it/s, Materializing param=model.layers.0.mlp.gate_proj.weight]
Loading weights: 2%|▏ | 5/290 [00:00<00:02, 109.99it/s, Materializing param=model.layers.0.mlp.up_proj.weight]
Loading weights: 2%|▏ | 5/290 [00:00<00:02, 109.59it/s, Materializing param=model.layers.0.mlp.up_proj.weight]
Loading weights: 2%|▏ | 6/290 [00:00<00:02, 127.87it/s, Materializing param=model.layers.0.post_attention_layernorm.weight]
Loading weights: 2%|▏ | 6/290 [00:00<00:02, 127.48it/s, Materializing param=model.layers.0.post_attention_layernorm.weight]
Loading weights: 2%|▏ | 7/290 [00:00<00:01, 147.08it/s, Materializing param=model.layers.0.self_attn.k_proj.bias]
Loading weights: 2%|▏ | 7/290 [00:00<00:01, 146.62it/s, Materializing param=model.layers.0.self_attn.k_proj.bias]
Loading weights: 3%|▎ | 8/290 [00:00<00:01, 166.08it/s, Materializing param=model.layers.0.self_attn.k_proj.weight]
Loading weights: 3%|▎ | 8/290 [00:00<00:01, 165.40it/s, Materializing param=model.layers.0.self_attn.k_proj.weight]
Loading weights: 3%|▎ | 9/290 [00:00<00:01, 181.47it/s, Materializing param=model.layers.0.self_attn.o_proj.weight]
Loading weights: 3%|▎ | 9/290 [00:00<00:01, 180.92it/s, Materializing param=model.layers.0.self_attn.o_proj.weight]
Loading weights: 3%|▎ | 10/290 [00:00<00:01, 199.46it/s, Materializing param=model.layers.0.self_attn.q_proj.bias]
Loading weights: 3%|▎ | 10/290 [00:00<00:01, 198.86it/s, Materializing param=model.layers.0.self_attn.q_proj.bias]
Loading weights: 4%|▍ | 11/290 [00:00<00:01, 217.67it/s, Materializing param=model.layers.0.self_attn.q_proj.weight]
Loading weights: 4%|▍ | 11/290 [00:00<00:01, 217.09it/s, Materializing param=model.layers.0.self_attn.q_proj.weight]
Loading weights: 4%|▍ | 12/290 [00:00<00:01, 235.72it/s, Materializing param=model.layers.0.self_attn.v_proj.bias]
Loading weights: 4%|▍ | 12/290 [00:00<00:01, 235.11it/s, Materializing param=model.layers.0.self_attn.v_proj.bias]
Loading weights: 4%|▍ | 13/290 [00:00<00:01, 252.59it/s, Materializing param=model.layers.0.self_attn.v_proj.weight]
Loading weights: 4%|▍ | 13/290 [00:00<00:01, 251.63it/s, Materializing param=model.layers.0.self_attn.v_proj.weight]
Loading weights: 5%|▍ | 14/290 [00:00<00:01, 267.61it/s, Materializing param=model.layers.1.input_layernorm.weight]
Loading weights: 5%|▍ | 14/290 [00:00<00:01, 266.62it/s, Materializing param=model.layers.1.input_layernorm.weight]
Loading weights: 5%|▌ | 15/290 [00:00<00:00, 284.00it/s, Materializing param=model.layers.1.mlp.down_proj.weight]
Loading weights: 5%|▌ | 15/290 [00:00<00:00, 283.19it/s, Materializing param=model.layers.1.mlp.down_proj.weight]
Loading weights: 6%|▌ | 16/290 [00:00<00:00, 278.33it/s, Materializing param=model.layers.1.mlp.gate_proj.weight]
Loading weights: 6%|▌ | 16/290 [00:00<00:00, 277.46it/s, Materializing param=model.layers.1.mlp.ga
|
|||
|
|
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': 151643}.
|
|||
|
|
Model loaded: 494.0M parameters
|
|||
|
|
|
|||
|
|
[4] Building training dataset...
|
|||
|
|
|
|||
|
|
[5] Configuring GRPO trainer...
|
|||
|
|
|
|||
|
|
[6] Training for 250 steps...
|
|||
|
|
Reward will be logged every 10 steps.
|
|||
|
|
Watch for increasing reward_mean over time.
|
|||
|
|
|
|||
|
|
0%| | 0/250 [00:00<?, ?it/s]
0%| | 1/250 [00:19<1:20:59, 19.52s/it]
1%| | 2/250 [00:52<1:54:16, 27.65s/it]
1%| | 3/250 [01:27<2:07:17, 30.92s/it]
2%|▏ | 4/250 [02:02<2:13:52, 32.65s/it]
2%|▏ | 5/250 [02:36<2:15:20, 33.14s/it]
2%|▏ | 6/250 [03:12<2:17:48, 33.89s/it]
3%|▎ | 7/250 [03:47<2:19:03, 34.34s/it]
3%|▎ | 8/250 [04:22<2:19:12, 34.52s/it]
4%|▎ | 9/250 [04:57<2:19:24, 34.71s/it]
4%|▍ | 10/250 [05:32<2:18:48, 34.70s/it]
4%|▍ | 10/250 [05:32<2:18:48, 34.70s/it]
4%|▍ | 11/250 [06:07<2:19:10, 34.94s/it]
5%|▍ | 12/250 [06:43<2:19:03, 35.06s/it]
5%|▌ | 13/250 [07:18<2:18:59, 35.19s/it]
6%|▌ | 14/250 [07:53<2:18:29, 35.21s/it]
6%|▌ | 15/250 [08:28<2:17:46, 35.18s/it]
6%|▋ | 16/250 [09:03<2:16:30, 35.00s/it]
7%|▋ | 17/250 [09:38<2:15:43, 34.95s/it]
7%|▋ | 18/250 [10:13<2:15:07, 34.95s/it]
8%|▊ | 19/250 [10:48<2:15:00, 35.07s/it]
8%|▊ | 20/250 [11:23<2:14:24, 35.06s/it]
8%|▊ | 20/250 [11:23<2:14:24, 35.06s/it]
8%|▊ | 21/250 [11:58<2:13:46, 35.05s/it]
9%|▉ | 22/250 [12:34<2:13:27, 35.12s/it]
9%|▉ | 23/250 [13:08<2:12:30, 35.02s/it]
10%|▉ | 24/250 [13:43<2:11:38, 34.95s/it]
10%|█ | 25/250 [14:19<2:11:54, 35.18s/it]
10%|█ | 26/250 [14:54<2:11:14, 35.15s/it]
11%|█ | 27/250 [15:29<2:10:22, 35.08s/it]
11%|█ | 28/250 [16:04<2:09:38, 35.04s/it]
12%|█▏ | 29/250 [16:39<2:09:23, 35.13s/it]
12%|█▏ | 30/250 [17:14<2:08:41, 35.10s/it]
12%|█▏ | 30/250 [17:14<2:08:41, 35.10s/it]
12%|█▏ | 31/250 [17:49<2:08:03, 35.08s/it]
13%|█▎ | 32/250 [18:24<2:07:05, 34.98s/it]
13%|█▎ | 33/250 [18:59<2:06:15, 34.91s/it]
14%|█▎ | 34/250 [19:34<2:05:52, 34.96s/it]
14%|█▍ | 35/250 [20:09<2:05:14, 34.95s/it]
14%|█▍ | 36/250 [20:44<2:05:11, 35.10s/it]
15%|█▍ | 37/250 [21:20<2:04:57, 35.20s/it]
15%|█▌ | 38/250 [21:55<2:04:30, 35.24s/it]
16%|█▌ | 39/250 [22:30<2:03:19, 35.07s/it]
16%|█▌ | 40/250 [23:04<2:02:28, 34.99s/it]
16%|█▌ | 40/250 [23:04<2:02:28, 34.99s/it]
16%|█▋ | 41/250 [23:39<2:01:43, 34.95s/it]
17%|█▋ | 42/250 [24:15<2:01:36, 35.08s/it]
17%|█▋ | 43/250 [24:50<2:01:32, 35.23s/it]
18%|█▊ | 44/250 [25:25<2:00:41, 35.15s/it]
18%|█▊ | 45/250 [26:00<2:00:14, 35.19s/it]
18%|█▊ | 46/250 [26:36<1:59:31, 35.16s/it]
19%|█▉ | 47/250 [27:10<1:58:37, 35.06s/it]
19%|█▉ | 48/250 [27:46<1:58:30, 35.20s/it]
20%|█▉ | 49/250 [28:21<1:57:43, 35.14s/it]
20%|██ | 50/250 [28:56<1:57:05, 35.13s/it]
20%|██ | 50/250 [28:56<1:57:05, 35.13s/it]
20%|██ | 51/250 [29:32<1:56:55, 35.25s/it]
21%|██ | 52/250 [30:06<1:56:00, 35.15s/it]
21%|██ | 53/250 [30:41<1:54:57, 35.01s/it]
22%|██▏ | 54/250 [31:16<1:54:28, 35.04s/it]
22%|██▏ | 55/250 [31:51<1:54:00, 35.08s/it]
22%|██▏ | 56/250 [32:26<1:53:17, 35.04s/it]
23%|██▎ | 57/250 [33:02<1:53:10, 35.18s/it]
23%|██▎ | 58/250 [33:37<1:52:11, 35.06s/it]
24%|██▎ | 59/250 [34:12<1:51:27, 35.01s/it]
24%|██▍ | 60/250 [34:47<1:51:03, 35.07s/it]
24%|██▍ | 60/250 [34:47<1:51:03, 35.07s/it]
24%|██▍ | 61/250 [35:22<1:50:45, 35.16s/it]
25%|██▍ | 62/250 [35:57<1:50:09, 35.15s/it]
25%|██▌ | 63/250 [36:32<1:49:25, 35.11s/it]
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '1.848e-05', 'num_tokens': '1.268e+05', 'completions/mean_length': '400', 'completions/min_length': '400', 'completions/max_length': '400', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/reward_fn/mean': '0.04825', 'rewards/reward_fn/std': '0.8191', 'reward': '0.04825', 'reward_std': '0.8191', 'frac_reward_zero_std': '1', 'entropy': '0.6594', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.55', 'epoch': '0.6667'}
|
|||
|
|
{'loss': '-0.005067', 'grad_norm': '1.805', 'learning_rate': '1.768e-05', 'num_tokens': '1.915e+05', 'completions/mean_length': '396.2', 'completions/min_length': '369.7', 'completions/max_length': '400', 'completions/clipped_ratio': '0.975', 'completions/mean_terminated_length': '49.7', 'completions/min_terminated_length': '49.7', 'completions/max_terminated_length': '49.7', 'rewards/reward_fn/mean': '0.1105', 'rewards/reward_fn/std': '0.7119', 'reward': '0.1105', 'reward_std': '0.7119', 'frac_reward_zero_std': '0.9', 'entropy': '0.5579', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.49', 'epoch': '1'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '1.688e-05', 'num_tokens': '2.565e+05', 'completions/mean_length': '400', 'completions/min_length': '400', 'completions/max_length': '400', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/reward_fn/mean': '0.07887', 'rewards/reward_fn/std': '0.8938', 'reward': '0.07887', 'reward_std': '0.8938', 'frac_reward_zero_std': '0.95', 'entropy': '0.00434', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.44', 'epoch': '1.333'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0.3848', 'learning_rate': '1.608e-05', 'num_tokens': '3.211e+05', 'completions/mean_length': '395.1', 'completions/min_length': '360.9', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '0.9', 'completions/min_terminated_length': '0.9', 'completions/max_terminated_length': '0.9', 'rewards/reward_fn/mean': '-0.1644', 'rewards/reward_fn/std': '1.064', 'reward': '-0.1644', 'reward_std': '1.064', 'frac_reward_zero_std': '0.975', 'entropy': '0.004431', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.56', 'epoch': '1.667'}
|
|||
|
|
{'loss': '0.009756', 'grad_norm': '0', 'learning_rate': '1.528e-05', 'num_tokens': '3.848e+05', 'completions/mean_length': '385.5', 'completions/min_length': '283.7', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9625', 'completions/mean_terminated_length': '3.7', 'completions/min_terminated_length': '3.7', 'completions/max_terminated_length': '3.7', 'rewards/reward_fn/mean': '-0.1656', 'rewards/reward_fn/std': '0.936', 'reward': '-0.1656', 'reward_std': '0.936', 'frac_reward_zero_std': '0.925', 'entropy': '0.09349', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.49', 'epoch': '2'}
|
|||
|
|
{'loss': '-0.01167', 'grad_norm': '2.406', 'learning_rate': '1.448e-05', 'num_tokens': '4.458e+05', 'completions/mean_length': '351.8', 'completions/min_length': '159.8', 'completions/max_length': '400', 'completions/clipped_ratio': '0.8625', 'completions/mean_terminated_length': '42.27', 'completions/min_terminated_length': '39.8', 'completions/max_terminated_length': '44.3', 'rewards/reward_fn/mean': '0.08913', 'rewards/reward_fn/std': '0.9902', 'reward': '0.08913', 'reward_std': '0.9902', 'frac_reward_zero_std': '0.925', 'entropy': '0.6014', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.48', 'epoch': '2.333'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '1.368e-05', 'num_tokens': '5.079e+05', 'completions/mean_length': '365.5', 'completions/min_length': '192.7', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9', 'completions/mean_terminated_length': '34.75', 'completions/min_terminated_length': '32.7', 'completions/max_terminated_length': '36.8', 'rewards/reward_fn/mean': '-0.1438', 'rewards/reward_fn/std': '0.6797', 'reward': '-0.1438', 'reward_std': '0.6797', 'frac_reward_zero_std': '0.975', 'entropy': '0.7648', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.44', 'epoch': '2.667'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '1.288e-05', 'num_tokens': '5.719e+05', 'completions/mean_length': '386.4', 'completions/min_length': '312.1', 'completions/max_length': '400', 'completions/clipped_ratio': '0.95', 'completions/mean_terminated_length': '35.4', 'completions/min_terminated_length': '32.1', 'completions/max_terminated_length': '38.7', 'rewards/reward_fn/mean': '-0.091', 'rewards/reward_fn/std': '0.7491', 'reward': '-0.091', 'reward_std': '0.7491', 'frac_reward_zero_std': '1', 'entropy': '0.8177', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.42', 'epoch': '3'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '1.208e-05', 'num_tokens': '6.358e+05', 'completions/mean_length': '386.5', 'completions/min_length': '292.1', 'completions/max_length': '400', 'completions/clipped_ratio': '0.95', 'completions/mean_terminated_length': '52.1', 'completions/min_terminated_length': '52.1', 'completions/max_terminated_length': '52.1', 'rewards/reward_fn/mean': '-0.1185', 'rewards/reward_fn/std': '0.978', 'reward': '-0.1185', 'reward_std': '0.978', 'frac_reward_zero_std': '1', 'entropy': '0.7871', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.45', 'epoch': '3.333'}
|
|||
|
|
|
|||
|
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s][A
|
|||
|
|
Writing model shards: 100%|██████████| 1/1 [00:08<00:00, 8.32s/it][A
Writing model shards: 100%|██████████| 1/1 [00:08<00:00, 8.32s/it]
|
|||
|
|
40%|████ | 101/250 [59:05<1:43:00, 41.48s/it]
41%|████ | 102/250 [59:40<1:37:45, 39.63s/it]
41%|████ | 103/250 [1:00:15<1:33:26, 38.14s/it]
42%|████▏ | 104/250 [1:00:50<1:30:29, 37.19s/it]
42%|████▏ | 105/250 [1:01:24<1:28:07, 36.46s/it]
42%|████▏ | 106/250 [1:01:59<1:26:11, 35.91s/it]
43%|████▎ | 107/250 [1:02:34<1:24:45, 35.56s/it]
43%|████▎ | 108/250 [1:03:09<1:23:45, 35.39s/it]
44%|████▎ | 109/250 [1:03:44<1:23:08, 35.38s/it]
44%|████▍ | 110/250 [1:04:19<1:22:30, 35.36s/it]
44%|████▍ | 110/250 [1:04:19<1:22:30, 35.36s/it]
44%|████▍ | 111/250 [1:04:55<1:21:56, 35.37s/it]
45%|████▍ | 112/250 [1:05:29<1:20:50, 35.15s/it]
45%|████▌ | 113/250 [1:06:04<1:20:09, 35.10s/it]
46%|████▌ | 114/250 [1:06:39<1:19:30, 35.07s/it]
46%|████▌ | 115/250 [1:07:15<1:19:08, 35.18s/it]
46%|████▋ | 116/250 [1:07:50<1:18:22, 35.09s/it]
47%|████▋ | 117/250 [1:08:25<1:17:54, 35.14s/it]
47%|████▋ | 118/250 [1:09:00<1:17:10, 35.08s/it]
48%|████▊ | 119/250 [1:09:35<1:16:49, 35.19s/it]
48%|████▊ | 120/250 [1:10:10<1:15:50, 35.00s/it]
48%|████▊ | 120/250 [1:10:10<1:15:50, 35.00s/it]
48%|████▊ | 121/250 [1:10:45<1:15:15, 35.00s/it]
49%|████▉ | 122/250 [1:11:20<1:14:59, 35.16s/it]
49%|████▉ | 123/250 [1:11:55<1:14:21, 35.13s/it]
50%|████▉ | 124/250 [1:12:31<1:13:45, 35.12s/it]
50%|█████ | 125/250 [1:13:05<1:12:59, 35.04s/it]
50%|█████ | 126/250 [1:13:41<1:12:30, 35.09s/it]
51%|█████ | 127/250 [1:14:16<1:11:54, 35.08s/it]
51%|█████ | 128/250 [1:14:51<1:11:28, 35.15s/it]
52%|█████▏ | 129/250 [1:15:26<1:10:52, 35.14s/it]
52%|█████▏ | 130/250 [1:16:01<1:10:00, 35.00s/it]
52%|█████▏ | 130/250 [1:16:01<1:10:00, 35.00s/it]
52%|█████▏ | 131/250 [1:16:36<1:09:35, 35.09s/it]
53%|█████▎ | 132/250 [1:17:11<1:08:41, 34.93s/it]
53%|█████▎ | 133/250 [1:17:46<1:08:14, 35.00s/it]
54%|█████▎ | 134/250 [1:18:21<1:07:43, 35.03s/it]
54%|█████▍ | 135/250 [1:18:56<1:07:24, 35.17s/it]
54%|█████▍ | 136/250 [1:19:31<1:06:42, 35.11s/it]
55%|█████▍ | 137/250 [1:20:06<1:05:54, 34.99s/it]
55%|█████▌ | 138/250 [1:20:41<1:05:28, 35.08s/it]
56%|█████▌ | 139/250 [1:21:17<1:05:06, 35.20s/it]
56%|█████▌ | 140/250 [1:21:52<1:04:23, 35.12s/it]
56%|█████▌ | 140/250 [1:21:52<1:04:23, 35.12s/it]
56%|█████▋ | 141/250 [1:22:27<1:03:54, 35.18s/it]
57%|█████▋ | 142/250 [1:23:02<1:03:25, 35.24s/it]
57%|█████▋ | 143/250 [1:23:37<1:02:38, 35.13s/it]
58%|█████▊ | 144/250 [1:24:12<1:01:53, 35.03s/it]
58%|█████▊ | 145/250 [1:24:47<1:01:18, 35.03s/it]
58%|█████▊ | 146/250 [1:25:22<1:00:39, 34.99s/it]
59%|█████▉ | 147/250 [1:25:58<1:00:19, 35.14s/it]
59%|█████▉ | 148/250 [1:26:32<59:32, 35.03s/it]
60%|█████▉ | 149/250 [1:27:07<58:56, 35.02s/it]
60%|██████ | 150/250 [1:27:42<58:22, 35.02s/it]
60%|██████ | 150/250 [1:27:42<58:22, 35.02s/it]
60%|██████ | 151/250 [1:28:18<57:51, 35.06s/it]
61%|██████ | 152/250 [1:28:52<57:11, 35.01s/it]
61%|██████ | 153/250 [1:29:27<56:27, 34.92s/it]
62%|██████▏ | 154/250 [1:30:02<55:53, 34.93s/it]
62%|█<>
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '1.048e-05', 'num_tokens': '7.647e+05', 'completions/mean_length': '398.6', 'completions/min_length': '389', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '29', 'completions/min_terminated_length': '29', 'completions/max_terminated_length': '29', 'rewards/reward_fn/mean': '-0.05875', 'rewards/reward_fn/std': '0.7074', 'reward': '-0.05875', 'reward_std': '0.7074', 'frac_reward_zero_std': '1', 'entropy': '0.8096', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.48', 'epoch': '4'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '9.68e-06', 'num_tokens': '8.289e+05', 'completions/mean_length': '391.5', 'completions/min_length': '331.8', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9625', 'completions/mean_terminated_length': '51.8', 'completions/min_terminated_length': '51.8', 'completions/max_terminated_length': '51.8', 'rewards/reward_fn/mean': '-0.017', 'rewards/reward_fn/std': '0.7561', 'reward': '-0.017', 'reward_std': '0.7561', 'frac_reward_zero_std': '0.975', 'entropy': '0.8178', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.51', 'epoch': '4.333'}
|
|||
|
|
{'loss': '-0.01235', 'grad_norm': '0', 'learning_rate': '8.88e-06', 'num_tokens': '8.924e+05', 'completions/mean_length': '380.8', 'completions/min_length': '246.2', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9375', 'completions/mean_terminated_length': '46.2', 'completions/min_terminated_length': '46.2', 'completions/max_terminated_length': '46.2', 'rewards/reward_fn/mean': '-0.1348', 'rewards/reward_fn/std': '0.9628', 'reward': '-0.1348', 'reward_std': '0.9628', 'frac_reward_zero_std': '0.95', 'entropy': '0.8855', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.51', 'epoch': '4.667'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '8.08e-06', 'num_tokens': '9.569e+05', 'completions/mean_length': '394.3', 'completions/min_length': '361.9', 'completions/max_length': '400', 'completions/clipped_ratio': '0.975', 'completions/mean_terminated_length': '17.1', 'completions/min_terminated_length': '1.9', 'completions/max_terminated_length': '32.3', 'rewards/reward_fn/mean': '-0.03', 'rewards/reward_fn/std': '0.7298', 'reward': '-0.03', 'reward_std': '0.7298', 'frac_reward_zero_std': '1', 'entropy': '0.8237', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.47', 'epoch': '5'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '7.28e-06', 'num_tokens': '1.021e+06', 'completions/mean_length': '394.8', 'completions/min_length': '358.5', 'completions/max_length': '400', 'completions/clipped_ratio': '0.975', 'completions/mean_terminated_length': '38.5', 'completions/min_terminated_length': '38.5', 'completions/max_terminated_length': '38.5', 'rewards/reward_fn/mean': '-0.0625', 'rewards/reward_fn/std': '0.9605', 'reward': '-0.0625', 'reward_std': '0.9605', 'frac_reward_zero_std': '1', 'entropy': '0.9815', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.41', 'epoch': '5.333'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '6.48e-06', 'num_tokens': '1.086e+06', 'completions/mean_length': '392.6', 'completions/min_length': '341.2', 'completions/max_length': '400', 'completions/clipped_ratio': '0.975', 'completions/mean_terminated_length': '21.2', 'completions/min_terminated_length': '21.2', 'completions/max_terminated_length': '21.2', 'rewards/reward_fn/mean': '-0.1513', 'rewards/reward_fn/std': '0.8553', 'reward': '-0.1513', 'reward_std': '0.8553', 'frac_reward_zero_std': '0.975', 'entropy': '0.907', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.48', 'epoch': '5.667'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '5.68e-06', 'num_tokens': '1.151e+06', 'completions/mean_length': '400', 'completions/min_length': '400', 'completions/max_length': '400', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/reward_fn/mean': '0.082', 'rewards/reward_fn/std': '0.9346', 'reward': '0.082', 'reward_std': '0.9346', 'frac_reward_zero_std': '1', 'entropy': '0.9227', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.47', 'epoch': '6'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '4.88e-06', 'num_tokens': '1.215e+06', 'completions/mean_length': '394.4', 'completions/min_length': '355.5', 'completions/max_length': '400', 'completions/clipped_ratio': '0.975', 'completions/mean_terminated_length': '35.5', 'completions/min_terminated_length': '35.5', 'completions/max_terminated_length': '35.5', 'rewards/reward_fn/mean': '-0.08775', 'rewards/reward_fn/std': '0.8883', 'reward': '-0.08775', 'reward_std': '0.8883', 'frac_reward_zero_std': '0.975', 'entropy': '0.9981', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.42', 'epoch': '6.333'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '4.08e-06', 'num_tokens': '1.28e+06', 'completions/mean_length': '398.8', 'completions/min_length': '390.4', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '30.4', 'completions/min_terminated_length': '30.4', 'completions/max_terminated_length': '30.4', 'rewards/reward_fn/mean': '-0.03525', 'rewards/reward_fn/std': '0.8935', 'reward': '-0.03525', 'reward_std': '0.8935', 'frac_reward_zero_std': '1', 'entropy': '0.979', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.42', 'epoch': '6.667'}
|
|||
|
|
|
|||
|
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s][A
|
|||
|
|
Writing model shards: 100%|██████████| 1/1 [00:16<00:00, 16.63s/it][A
Writing model shards: 100%|██████████| 1/1 [00:16<00:00, 16.63s/it]
|
|||
|
|
80%|████████ | 201/250 [1:58:00<36:25, 44.60s/it]
81%|████████ | 202/250 [1:58:36<33:26, 41.81s/it]
81%|████████ | 203/250 [1:59:11<31:07, 39.73s/it]
82%|████████▏ | 204/250 [1:59:46<29:29, 38.46s/it]
82%|████████▏ | 205/250 [2:00:22<28:10, 37.56s/it]
82%|████████▏ | 206/250 [2:00:57<27:03, 36.91s/it]
83%|████████▎ | 207/250 [2:01:32<25:57, 36.21s/it]
83%|████████▎ | 208/250 [2:02:06<25:00, 35.72s/it]
84%|████████▎ | 209/250 [2:02:41<24:15, 35.50s/it]
84%|████████▍ | 210/250 [2:03:16<23:37, 35.43s/it]
84%|████████▍ | 210/250 [2:03:16<23:37, 35.43s/it]
84%|████████▍ | 211/250 [2:03:52<22:59, 35.38s/it]
85%|████████▍ | 212/250 [2:04:26<22:17, 35.20s/it]
85%|████████▌ | 213/250 [2:05:01<21:38, 35.09s/it]
86%|████████▌ | 214/250 [2:05:36<20:55, 34.87s/it]
86%|████████▌ | 215/250 [2:06:11<20:24, 34.98s/it]
86%|████████▋ | 216/250 [2:06:46<19:49, 34.99s/it]
87%|████████▋ | 217/250 [2:07:21<19:11, 34.91s/it]
87%|████████▋ | 218/250 [2:07:55<18:36, 34.90s/it]
88%|████████▊ | 219/250 [2:08:30<18:01, 34.90s/it]
88%|████████▊ | 220/250 [2:09:05<17:27, 34.91s/it]
88%|████████▊ | 220/250 [2:09:05<17:27, 34.91s/it]
88%|████████▊ | 221/250 [2:09:40<16:52, 34.90s/it]
89%|████████▉ | 222/250 [2:10:16<16:21, 35.06s/it]
89%|████████▉ | 223/250 [2:10:50<15:44, 34.96s/it]
90%|████████▉ | 224/250 [2:11:25<15:09, 34.97s/it]
90%|█████████ | 225/250 [2:12:00<14:32, 34.92s/it]
90%|█████████ | 226/250 [2:12:35<13:57, 34.91s/it]
91%|█████████ | 227/250 [2:13:11<13:27, 35.12s/it]
91%|█████████ | 228/250 [2:13:45<12:49, 34.97s/it]
92%|█████████▏| 229/250 [2:14:21<12:17, 35.10s/it]
92%|█████████▏| 230/250 [2:14:55<11:39, 34.96s/it]
92%|█████████▏| 230/250 [2:14:55<11:39, 34.96s/it]
92%|█████████▏| 231/250 [2:15:31<11:06, 35.08s/it]
93%|█████████▎| 232/250 [2:16:06<10:32, 35.14s/it]
93%|█████████▎| 233/250 [2:16:42<10:00, 35.31s/it]
94%|█████████▎| 234/250 [2:17:17<09:24, 35.27s/it]
94%|█████████▍| 235/250 [2:17:52<08:47, 35.18s/it]
94%|█████████▍| 236/250 [2:18:26<08:09, 34.99s/it]
95%|█████████▍| 237/250 [2:19:01<07:33, 34.91s/it]
95%|█████████▌| 238/250 [2:19:36<06:58, 34.89s/it]
96%|█████████▌| 239/250 [2:20:12<06:26, 35.16s/it]
96%|█████████▌| 240/250 [2:20:47<05:52, 35.24s/it]
96%|█████████▌| 240/250 [2:20:47<05:52, 35.24s/it]
96%|█████████▋| 241/250 [2:21:22<05:16, 35.13s/it]
97%|█████████▋| 242/250 [2:21:57<04:40, 35.06s/it]
97%|█████████▋| 243/250 [2:22:32<04:05, 35.04s/it]
98%|█████████▊| 244/250 [2:23:07<03:30, 35.11s/it]
98%|█████████▊| 245/250 [2:23:42<02:55, 35.18s/it]
98%|█████████▊| 246/250 [2:24:18<02:20, 35.22s/it]
99%|█████████▉| 247/250 [2:24:53<01:45, 35.27s/it]
99%|█████████▉| 248/250 [2:25:28<01:10, 35.27s/it]
100%|█████████▉| 249/250 [2:26:03<00:35, 35.12s/it]
100%|██████████| 250/250 [2:26:38<00:00, 35.17s/it]
100%|<7C><>
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '2.48e-06', 'num_tokens': '1.409e+06', 'completions/mean_length': '400', 'completions/min_length': '400', 'completions/max_length': '400', 'completions/clipped_ratio': '1', 'completions/mean_terminated_length': '0', 'completions/min_terminated_length': '0', 'completions/max_terminated_length': '0', 'rewards/reward_fn/mean': '0.01825', 'rewards/reward_fn/std': '0.8384', 'reward': '0.01825', 'reward_std': '0.8384', 'frac_reward_zero_std': '1', 'entropy': '0.9604', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.32', 'epoch': '7.333'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '1.68e-06', 'num_tokens': '1.474e+06', 'completions/mean_length': '397.4', 'completions/min_length': '379.6', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9875', 'completions/mean_terminated_length': '19.6', 'completions/min_terminated_length': '19.6', 'completions/max_terminated_length': '19.6', 'rewards/reward_fn/mean': '-0.1123', 'rewards/reward_fn/std': '0.8594', 'reward': '-0.1123', 'reward_std': '0.8594', 'frac_reward_zero_std': '1', 'entropy': '0.9672', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.42', 'epoch': '7.667'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '8.8e-07', 'num_tokens': '1.538e+06', 'completions/mean_length': '389.5', 'completions/min_length': '316.1', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9625', 'completions/mean_terminated_length': '36.1', 'completions/min_terminated_length': '36.1', 'completions/max_terminated_length': '36.1', 'rewards/reward_fn/mean': '-0.015', 'rewards/reward_fn/std': '0.9199', 'reward': '-0.015', 'reward_std': '0.9199', 'frac_reward_zero_std': '1', 'entropy': '1.013', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.59', 'epoch': '8'}
|
|||
|
|
{'loss': '0', 'grad_norm': '0', 'learning_rate': '8e-08', 'num_tokens': '1.603e+06', 'completions/mean_length': '391.8', 'completions/min_length': '334.6', 'completions/max_length': '400', 'completions/clipped_ratio': '0.9625', 'completions/mean_terminated_length': '54.6', 'completions/min_terminated_length': '54.6', 'completions/max_terminated_length': '54.6', 'rewards/reward_fn/mean': '0.0445', 'rewards/reward_fn/std': '0.861', 'reward': '0.0445', 'reward_std': '0.861', 'frac_reward_zero_std': '1', 'entropy': '0.9223', 'clip_ratio/low_mean': '0', 'clip_ratio/low_min': '0', 'clip_ratio/high_mean': '0', 'clip_ratio/high_max': '0', 'clip_ratio/region_mean': '0', 'step_time': '33.55', 'epoch': '8.333'}
|
|||
|
|
|
|||
|
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s][A
|
|||
|
|
Writing model shards: 100%|██████████| 1/1 [00:05<00:00, 5.67s/it][A
Writing model shards: 100%|██████████| 1/1 [00:05<00:00, 5.67s/it]
|
|||
|
|
100%|██████████| 250/250 [2:26:57<00:00, 35.17s/it]
100%|██████████| 250/250 [2:26:57<00:00, 35.27s/it]
|
|||
|
|
{'train_runtime': '8818', 'train_samples_per_second': '0.227', 'train_steps_per_second': '0.028', 'train_loss': '-0.001472', 'epoch': '8.333'}
|
|||
|
|
|
|||
|
|
[7] Saving model to ./deepsentinel_model_small
|
|||
|
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|██████████| 1/1 [00:07<00:00, 7.07s/it]
Writing model shards: 100%|██████████| 1/1 [00:07<00:00, 7.08s/it]
|
|||
|
|
Done!
|
|||
|
|
|
|||
|
|
[8] Post-training evaluation...
|
|||
|
|
Baseline vs Post-training (heuristic used for comparison):
|
|||
|
|
easy: -0.3681 → -0.3681 (Δ=+0.0000)
|
|||
|
|
medium: 0.0049 → 0.0049 (Δ=+0.0000)
|
|||
|
|
hard: 0.0833 → 0.0833 (Δ=+0.0000)
|
|||
|
|
|
|||
|
|
Training complete!
|
|||
|
|
Model saved to: ./deepsentinel_model_small
|