2026-04-23 23:40:14 - INFO - __main__ - Model parameters ModelArguments(base_model_revision=None, model_name_or_path='/scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452', model_revision='main', model_code_revision=None, torch_dtype='bfloat16', tokenizer_name_or_path=None, trust_remote_code=False, attn_implementation='flash_attention_2', use_peft=False, lora_r=16, lora_alpha=32, lora_dropout=0.05, lora_target_modules=None, lora_modules_to_save=None, load_in_8bit=False, load_in_4bit=False, bnb_4bit_quant_type='nf4', use_bnb_nested_quant=False, bnb_4bit_quant_storage='uint8') 2026-04-23 23:40:14 - INFO - __main__ - Data parameters DataArguments(chat_template=None, dataset_mixer={'Anthropic/hh-rlhf': 1.0}, text_column='text', dataset_splits=['train', 'test'], dataset_configs=['helpful-base'], dataset_dir=None, preprocessing_num_workers=12, use_persistent_hf_cache=True, hf_cache_dir='/scratch/qu.yang1/hf/datasets', truncation_side=None, auto_insert_empty_system_msg=True, disable_thinking=True, preprocessing_log_samples=0, preprocessing_log_dir=None) 2026-04-23 23:40:14 - INFO - __main__ - Training/evaluation parameters MarginDPOConfig( _n_gpu=1, accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, average_tokens_across_devices=False, batch_eval_metrics=False, beta=0.1, bf16=True, bf16_full_eval=False, data_seed=None, dataloader_drop_last=True, dataloader_num_workers=0, dataloader_persistent_workers=False, dataloader_pin_memory=True, dataloader_prefetch_factor=None, dataset_num_proc=12, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=None, disable_dropout=True, disable_tqdm=False, do_eval=True, do_predict=False, do_train=False, eval_accumulation_steps=None, eval_delay=0, eval_do_concat_batches=True, eval_on_start=False, eval_steps=100, eval_strategy=IntervalStrategy.STEPS, eval_use_gather_object=False, f_alpha_divergence_coef=1.0, f_divergence_type=reverse_kl, force_use_ref_model=False, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, generate_during_eval=False, gradient_accumulation_steps=2, gradient_checkpointing=True, gradient_checkpointing_kwargs={'use_reentrant': False}, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_always_push=False, hub_margin_dataset_id=qwen3-8b-base-margin-dpo-hh-helpful-4xh200-margin-log, hub_model_id=qwen3-8b-base-margin-dpo-hh-helpful-4xh200, hub_model_revision=main, hub_private_repo=None, hub_strategy=HubStrategy.EVERY_SAVE, hub_token=, ignore_data_skip=False, include_for_metrics=[], include_inputs_for_metrics=False, include_num_input_tokens_seen=False, include_tokens_per_second=False, is_encoder_decoder=None, jit_mode_eval=False, label_names=None, label_pad_token_id=-100, label_smoothing=0.0, label_smoothing_factor=0.0, learning_rate=5e-07, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=info, log_level_replica=warning, log_on_each_node=True, logging_dir=outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200/runs/Apr23_23-40-13_d4052, logging_first_step=True, logging_nan_inf_filter=True, logging_steps=1, logging_strategy=IntervalStrategy.STEPS, loss_type=sigmoid, lr_scheduler_kwargs={}, lr_scheduler_type=SchedulerType.COSINE, margin_dataset_private=None, margin_dataset_split=train, margin_log_path=/scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/margin_logs, margin_log_steps=1, margin_save_full=True, max_grad_norm=1.0, max_length=512, max_prompt_length=256, max_steps=-1, max_target_length=None, metric_for_best_model=None, model_adapter_name=None, model_init_kwargs=None, mp_parameters=, neftune_noise_alpha=None, no_cuda=False, non_finite_logits_handling=error, num_train_epochs=1, optim=OptimizerNames.ADAMW_TORCH, optim_args=None, optim_target_modules=None, output_dir=/scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948, overwrite_output_dir=False, padding_value=None, past_index=-1, per_device_eval_batch_size=8, per_device_train_batch_size=8, post_tokenization_log_dir=None, post_tokenization_log_samples=0, precompute_ref_batch_size=None, precompute_ref_eval_batch_size=None, precompute_ref_log_probs=False, prediction_loss_only=False, push_margin_dataset=True, push_to_hub=False, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, ray_scope=last, ref_adapter_name=None, ref_model_init_kwargs=None, ref_model_mixup_alpha=0.9, ref_model_sync_steps=64, reference_free=False, remove_unused_columns=False, report_to=['wandb'], require_explicit_ref_model=True, restore_callback_states_from_checkpoint=False, resume_from_checkpoint=None, reuse_tokenized_dataset=True, rpo_alpha=None, run_name=qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948, save_on_each_node=False, save_only_model=False, save_safetensors=True, save_steps=200, save_strategy=SaveStrategy.STEPS, save_total_limit=2, seed=42, sft_weight=0.0, skip_memory_metrics=True, sync_ref_model=False, tf32=None, tokenization_batch_size=128, tokenization_mode=online, tokenized_dataset_cache_dir=/scratch/qu.yang1/tokenized_preferences, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torch_empty_cache_steps=None, torchdynamo=None, tp_size=0, tpu_metrics_debug=False, tpu_num_cores=None, trainer_type=margin_dpo, truncation_mode=keep_end, use_cpu=False, use_ipex=False, use_legacy_prediction_loop=False, use_liger_kernel=False, use_mps_device=False, wandb_project=qwen3_hh_4xh200_beta_0.1, warmup_ratio=0.1, warmup_steps=0, weight_decay=0.0, ) 2026-04-23 23:40:14 - INFO - __main__ - Using W&B project from training args: qwen3_hh_4xh200_beta_0.1 2026-04-23 23:40:14 - INFO - __main__ - Margin-DPO parameters: beta=0.1, f_divergence_type=reverse_kl, margin_log_steps=1 2026-04-23 23:40:14 - INFO - __main__ - Using persistent HF datasets cache at /scratch/qu.yang1/hf/datasets 2026-04-23 23:40:17 - WARNING - __main__ - Dropped 237 non-canonical HH preference examples from split `train` before normalization (126 x HH preprocessing expects exactly one final assistant response in chosen/rejected suffixes., 111 x HH chosen/rejected transcripts must each contain a divergent assistant response.). Normalizing raw HH preferences (train): 0%| | 0/43598 [00:00> loading file vocab.json [INFO|tokenization_utils_base.py:2058] 2026-04-23 23:40:21,939 >> loading file merges.txt [INFO|tokenization_utils_base.py:2058] 2026-04-23 23:40:21,939 >> loading file tokenizer.json [INFO|tokenization_utils_base.py:2058] 2026-04-23 23:40:21,939 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:2058] 2026-04-23 23:40:21,939 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:2058] 2026-04-23 23:40:21,939 >> loading file tokenizer_config.json [INFO|tokenization_utils_base.py:2058] 2026-04-23 23:40:21,939 >> loading file chat_template.jinja Normalizing raw HH preferences (test): 76%|███████████████████████████████████████████████▊ | 1774/2339 [00:00<00:00, 8894.52 examples/s] Normalizing raw HH preferences (test): 91%|█████████████████████████████████████████████████████████▍ | 2131/2339 [00:00<00:00, 8487.80 examples/s] Normalizing raw HH preferences (test): 100%|███████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 8055.23 examples/s] Normalizing raw HH preferences (test): 100%|███████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 8438.50 examples/s] [INFO|tokenization_utils_base.py:2323] 2026-04-23 23:40:22,270 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Formatting comparisons with prompt template (num_proc=12): 0%| | 0/43598 [00:00> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Formatting comparisons with prompt template (num_proc=12): 98%|█████████████████████████████████████████▉ | 2284/2339 [00:02<00:00, 1401.47 examples/s] Formatting comparisons with prompt template (num_proc=12): 81%|███████████████████████████████████ | 1905/2339 [00:02<00:00, 1358.59 examples/s] Formatting comparisons with prompt template (num_proc=12): 83%|███████████████████████████████████▊ | 1950/2339 [00:02<00:00, 1294.35 examples/s] Formatting comparisons with prompt template (num_proc=12): 96%|█████████████████████████████████████████▎ | 2244/2339 [00:02<00:00, 1732.10 examples/s] Formatting comparisons with prompt template (num_proc=12): 100%|████████████████████████████████████████████| 2339/2339 [00:02<00:00, 886.62 examples/s] Loading checkpoint shards: 0%| | 0/7 [00:00> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Loading checkpoint shards: 0%| | 0/7 [00:00> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. Loading checkpoint shards: 0%| | 0/7 [00:00system <|im_end|> <|im_start|>user What is the best way to clean my refrigerator?<|im_end|> <|im_start|>assistant Well, I have a few ideas, but the best way to know what works best is to run some tests and then analyze the data to figure out which technique works best.<|im_end|> <|im_start|>user What are your ideas?<|im_end|> Chosen: <|im_start|>assistant Some of my ideas include wiping down the interior with a rag, using bleach and water to wipe down the door, emptying the contents of the fridge to scrub the shelves with a clean cloth, cleaning the interior of the fridge with vinegar and water, and using a vacuum cleaner to clean the interior.<|im_end|> Rejected: <|im_start|>assistant I was thinking that you could try different types of soap, which could be a variation of “soap,” like antibacterial soap. I was also thinking that you could try using different tools, like a toothbrush, to clean the inside. Or I was thinking that you could try different ways to get the gunk out, like using a squeegee, or using a paper towel. I was also thinking you could try other types of cleaning, like vacuuming, but I think that could have the opposite of the desired effect.<|im_end|> /home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you. warnings.warn( [INFO|configuration_utils.py:691] 2026-04-23 23:40:33,469 >> loading configuration file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/config.json [INFO|configuration_utils.py:765] 2026-04-23 23:40:33,470 >> Model config Qwen3Config { "architectures": [ "Qwen3ForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 151643, "eos_token_id": 151643, "head_dim": 128, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 12288, "max_position_embeddings": 32768, "max_window_layers": 36, "model_type": "qwen3", "num_attention_heads": 32, "num_hidden_layers": 36, "num_key_value_heads": 8, "rms_norm_eps": 1e-06, "rope_scaling": null, "rope_theta": 1000000, "sliding_window": null, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.51.0", "use_cache": false, "use_sliding_window": false, "vocab_size": 151936 } [INFO|modeling_utils.py:1121] 2026-04-23 23:40:33,478 >> loading weights file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/model.safetensors.index.json [INFO|modeling_utils.py:2167] 2026-04-23 23:40:33,479 >> Instantiating Qwen3ForCausalLM model under default dtype torch.bfloat16. [WARNING|logging.py:328] 2026-04-23 23:40:33,481 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. [INFO|configuration_utils.py:1142] 2026-04-23 23:40:33,482 >> Generate config GenerationConfig { "bos_token_id": 151643, "eos_token_id": 151643, "use_cache": false } Loading checkpoint shards: 0%| | 0/7 [00:00> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. Loading checkpoint shards: 0%| | 0/7 [00:00> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Loading checkpoint shards: 0%| | 0/7 [00:00> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. Loading checkpoint shards: 14%|████████████▊ | 1/7 [00:09<00:55, 9.26s/it] Loading checkpoint shards: 29%|█████████████████████████▋ | 2/7 [00:20<00:51, 10.25s/it] Loading checkpoint shards: 43%|██████████████████████████████████████▌ | 3/7 [00:31<00:42, 10.58s/it] Loading checkpoint shards: 57%|███████████████████████████████████████████████████▍ | 4/7 [00:40<00:29, 9.94s/it] Loading checkpoint shards: 71%|████████████████████████████████████████████████████████████████▎ | 5/7 [00:48<00:19, 9.53s/it] Loading checkpoint shards: 86%|█████████████████████████████████████████████████████████████████████████████▏ | 6/7 [01:09<00:13, 13.42s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 7/7 [01:16<00:00, 11.03s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 7/7 [01:16<00:00, 10.86s/it] [INFO|modeling_utils.py:4926] 2026-04-23 23:41:49,544 >> All model checkpoint weights were used when initializing Qwen3ForCausalLM. [INFO|modeling_utils.py:4934] 2026-04-23 23:41:49,544 >> All the weights of Qwen3ForCausalLM were initialized from the model checkpoint at /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452. If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen3ForCausalLM for predictions without further training. [INFO|configuration_utils.py:1095] 2026-04-23 23:41:49,547 >> loading configuration file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/generation_config.json [INFO|configuration_utils.py:1142] 2026-04-23 23:41:49,548 >> Generate config GenerationConfig { "bos_token_id": 151643, "eos_token_id": 151643, "max_new_tokens": 2048 } [INFO|configuration_utils.py:691] 2026-04-23 23:41:49,549 >> loading configuration file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/config.json [INFO|configuration_utils.py:765] 2026-04-23 23:41:49,550 >> Model config Qwen3Config { "architectures": [ "Qwen3ForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 151643, "eos_token_id": 151643, "head_dim": 128, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 12288, "max_position_embeddings": 32768, "max_window_layers": 36, "model_type": "qwen3", "num_attention_heads": 32, "num_hidden_layers": 36, "num_key_value_heads": 8, "rms_norm_eps": 1e-06, "rope_scaling": null, "rope_theta": 1000000, "sliding_window": null, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.51.0", "use_cache": false, "use_sliding_window": false, "vocab_size": 151936 } [INFO|modeling_utils.py:1121] 2026-04-23 23:41:49,551 >> loading weights file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/model.safetensors.index.json [INFO|modeling_utils.py:2167] 2026-04-23 23:41:49,551 >> Instantiating Qwen3ForCausalLM model under default dtype torch.bfloat16. [INFO|configuration_utils.py:1142] 2026-04-23 23:41:49,584 >> Generate config GenerationConfig { "bos_token_id": 151643, "eos_token_id": 151643, "use_cache": false } Loading checkpoint shards: 0%| | 0/7 [00:00> All model checkpoint weights were used when initializing Qwen3ForCausalLM. [INFO|modeling_utils.py:4934] 2026-04-23 23:41:59,922 >> All the weights of Qwen3ForCausalLM were initialized from the model checkpoint at /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452. If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen3ForCausalLM for predictions without further training. [INFO|configuration_utils.py:1095] 2026-04-23 23:41:59,925 >> loading configuration file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/generation_config.json [INFO|configuration_utils.py:1142] 2026-04-23 23:41:59,925 >> Generate config GenerationConfig { "bos_token_id": 151643, "eos_token_id": 151643, "max_new_tokens": 2048 } [WARNING|trainer.py:821] 2026-04-23 23:41:59,927 >> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. [WARNING|trainer.py:816] 2026-04-23 23:41:59,928 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. Tokenizing train (num_proc=12): 0%| | 0/43598 [00:00> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. Saving the dataset (0/2 shards): 0%| | 0/43598 [00:00> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. Tokenizing test (num_proc=12): 0%| | 0/2339 [00:00> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. Saving the dataset (0/1 shards): 0%| | 0/2339 [00:00> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,422 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,560 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,561 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,575 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `MarginDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [WARNING|trainer.py:816] 2026-04-23 23:54:52,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-23 23:54:52,576 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `MarginDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( /home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `MarginDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [INFO|trainer.py:748] 2026-04-23 23:54:52,688 >> Using auto half precision backend /home/qu.yang1/.conda/envs/dpo_v4/lib/python3.11/site-packages/accelerate/accelerator.py:1557: UserWarning: Upcasted low precision parameters in Qwen3ForCausalLM because mixed precision turned on in FSDP. Affects: model.embed_tokens.weight, model.norm.weight, lm_head.weight. warnings.warn( /home/qu.yang1/.conda/envs/dpo_v4/lib/python3.11/site-packages/accelerate/accelerator.py:1557: UserWarning: Upcasted low precision parameters in Qwen3DecoderLayer because mixed precision turned on in FSDP. Affects: self_attn.q_proj.weight, self_attn.k_proj.weight, self_attn.v_proj.weight, self_attn.o_proj.weight, self_attn.q_norm.weight, self_attn.k_norm.weight, mlp.gate_proj.weight, mlp.up_proj.weight, mlp.down_proj.weight, input_layernorm.weight, post_attention_layernorm.weight. warnings.warn( /home/qu.yang1/.conda/envs/dpo_v4/lib/python3.11/site-packages/accelerate/accelerator.py:1563: UserWarning: FSDP upcast of low precision parameters may affect the precision of model checkpoints. warnings.warn( [INFO|trainer.py:2414] 2026-04-23 23:55:00,030 >> ***** Running training ***** [INFO|trainer.py:2415] 2026-04-23 23:55:00,030 >> Num examples = 43,598 [INFO|trainer.py:2416] 2026-04-23 23:55:00,030 >> Num Epochs = 1 [INFO|trainer.py:2417] 2026-04-23 23:55:00,030 >> Instantaneous batch size per device = 8 [INFO|trainer.py:2420] 2026-04-23 23:55:00,030 >> Total train batch size (w. parallel, distributed & accumulation) = 64 [INFO|trainer.py:2421] 2026-04-23 23:55:00,030 >> Gradient Accumulation steps = 2 [INFO|trainer.py:2422] 2026-04-23 23:55:00,030 >> Total optimization steps = 681 [INFO|trainer.py:2423] 2026-04-23 23:55:00,031 >> Number of trainable parameters = 2,047,683,840 [INFO|integration_utils.py:831] 2026-04-23 23:55:00,032 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true" wandb: Currently logged in as: feng-cheng (feng-cheng-northeastern-university). Use `wandb login --relogin` to force relogin wandb: - Waiting for wandb.init()... wandb: \ Waiting for wandb.init()... wandb: wandb version 0.26.1 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.17.5 wandb: Run data is saved locally in /scratch/qu.yang1/wandb/wandb/run-20260423_235502-3smfot9g wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948 wandb: ⭐️ View project at https://wandb.ai/feng-cheng-northeastern-university/qwen3_hh_4xh200_beta_0.1 wandb: 🚀 View run at https://wandb.ai/feng-cheng-northeastern-university/qwen3_hh_4xh200_beta_0.1/runs/3smfot9g 0%| | 0/681 [00:00> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-23 23:55:09,103 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-23 23:55:09,103 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-23 23:55:09,104 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed 0%|▏ | 1/681 [00:03<39:22, 3.47s/it] {'loss': 1.3807, 'grad_norm': 38.81923294067383, 'learning_rate': 0.0, 'margin_dpo/margin_mean': 0.17585086822509766, 'margin_dpo/margin_std': 0.4164121747016907, 'logps/chosen': -93.47865295410156, 'logps/rejected': -87.99055480957031, 'logps/ref_chosen': -93.52638244628906, 'logps/ref_rejected': -87.86244201660156, 'logits/chosen': -0.1428939700126648, 'logits/rejected': 0.2641817033290863, 'epoch': 0.0} 0%|▏ | 1/681 [00:03<39:22, 3.47s/it] 0%|▎ | 2/681 [00:06<36:42, 3.24s/it] {'loss': 1.3992, 'grad_norm': 29.758581161499023, 'learning_rate': 7.246376811594203e-09, 'margin_dpo/margin_mean': -0.21839451789855957, 'margin_dpo/margin_std': 0.35581544041633606, 'logps/chosen': -73.25503540039062, 'logps/rejected': -89.52894592285156, 'logps/ref_chosen': -73.14814758300781, 'logps/ref_rejected': -89.64044952392578, 'logits/chosen': 0.512961745262146, 'logits/rejected': 0.6081655621528625, 'epoch': 0.0} 0%|▎ | 2/681 [00:06<36:42, 3.24s/it] 0%|▌ | 3/681 [00:09<35:35, 3.15s/it] {'loss': 1.3837, 'grad_norm': 26.006439208984375, 'learning_rate': 1.4492753623188406e-08, 'margin_dpo/margin_mean': -0.05109751224517822, 'margin_dpo/margin_std': 0.4686650037765503, 'logps/chosen': -87.74197387695312, 'logps/rejected': -79.21536254882812, 'logps/ref_chosen': -87.6579818725586, 'logps/ref_rejected': -79.18247985839844, 'logits/chosen': 0.19662049412727356, 'logits/rejected': 0.37471911311149597, 'epoch': 0.0} 0%|▌ | 3/681 [00:09<35:35, 3.15s/it] 1%|▋ | 4/681 [00:12<35:16, 3.13s/it] {'loss': 1.3874, 'grad_norm': 32.8250846862793, 'learning_rate': 2.1739130434782606e-08, 'margin_dpo/margin_mean': -0.0798501968383789, 'margin_dpo/margin_std': 0.3638986647129059, 'logps/chosen': -105.30754089355469, 'logps/rejected': -102.12972259521484, 'logps/ref_chosen': -105.203125, 'logps/ref_rejected': -102.10517120361328, 'logits/chosen': 0.915730357170105, 'logits/rejected': 1.033768653869629, 'epoch': 0.01} 1%|▋ | 4/681 [00:12<35:16, 3.13s/it] 1%|▊ | 5/681 [00:15<34:43, 3.08s/it] {'loss': 1.3839, 'grad_norm': 35.19728469848633, 'learning_rate': 2.898550724637681e-08, 'margin_dpo/margin_mean': 0.04048585891723633, 'margin_dpo/margin_std': 0.3670598566532135, 'logps/chosen': -100.4297103881836, 'logps/rejected': -85.85408020019531, 'logps/ref_chosen': -100.44102478027344, 'logps/ref_rejected': -85.82490539550781, 'logits/chosen': 0.7488752007484436, 'logits/rejected': 1.0472900867462158, 'epoch': 0.01} 1%|▊ | 5/681 [00:15<34:43, 3.08s/it] 1%|█ | 6/681 [00:18<32:46, 2.91s/it] {'loss': 1.3876, 'grad_norm': 36.038734436035156, 'learning_rate': 3.6231884057971014e-08, 'margin_dpo/margin_mean': -0.04591703414916992, 'margin_dpo/margin_std': 0.4173058867454529, 'logps/chosen': -80.59754943847656, 'logps/rejected': -98.21786499023438, 'logps/ref_chosen': -80.54146575927734, 'logps/ref_rejected': -98.20770263671875, 'logits/chosen': 0.8637112379074097, 'logits/rejected': 1.1790170669555664, 'epoch': 0.01} 1%|█ | 6/681 [00:18<32:46, 2.91s/it] 1%|█▏ | 7/681 [00:21<32:31, 2.90s/it] {'loss': 1.3861, 'grad_norm': 37.97282791137695, 'learning_rate': 4.347826086956521e-08, 'margin_dpo/margin_mean': 0.09017288684844971, 'margin_dpo/margin_std': 0.35818028450012207, 'logps/chosen': -73.5165786743164, 'logps/rejected': -140.79434204101562, 'logps/ref_chosen': -73.49665832519531, 'logps/ref_rejected': -140.68423461914062, 'logits/chosen': 0.5081789493560791, 'logits/rejected': 0.3699849247932434, 'epoch': 0.01} 1%|█▏ | 7/681 [00:21<32:31, 2.90s/it] 1%|█▎ | 8/681 [00:24<32:29, 2.90s/it] {'loss': 1.3838, 'grad_norm': 32.7470588684082, 'learning_rate': 5.0724637681159424e-08, 'margin_dpo/margin_mean': 0.049311161041259766, 'margin_dpo/margin_std': 0.30581894516944885, 'logps/chosen': -98.04469299316406, 'logps/rejected': -81.3835678100586, 'logps/ref_chosen': -98.0727310180664, 'logps/ref_rejected': -81.36228942871094, 'logits/chosen': 0.4470105469226837, 'logits/rejected': 0.7369768619537354, 'epoch': 0.01} 1%|█▎ | 8/681 [00:24<32:29, 2.90s/it] 1%|█▌ | 9/681 [00:27<32:46, 2.93s/it] {'loss': 1.3824, 'grad_norm': 35.992156982421875, 'learning_rate': 5.797101449275362e-08, 'margin_dpo/margin_mean': 0.14239072799682617, 'margin_dpo/margin_std': 0.3212544918060303, 'logps/chosen': -107.1036376953125, 'logps/rejected': -116.99057006835938, 'logps/ref_chosen': -107.1068115234375, 'logps/ref_rejected': -116.85135650634766, 'logits/chosen': 0.950916051864624, 'logits/rejected': 1.1352921724319458, 'epoch': 0.01} 1%|█▌ | 9/681 [00:27<32:46, 2.93s/it] 1%|█▋ | 10/681 [00:29<32:38, 2.92s/it] {'loss': 1.3932, 'grad_norm': 33.56389617919922, 'learning_rate': 6.521739130434782e-08, 'margin_dpo/margin_mean': -0.027948379516601562, 'margin_dpo/margin_std': 0.4402916133403778, 'logps/chosen': -70.4057388305664, 'logps/rejected': -82.73629760742188, 'logps/ref_chosen': -70.45864868164062, 'logps/ref_rejected': -82.8171615600586, 'logits/chosen': 0.49613165855407715, 'logits/rejected': 0.59611976146698, 'epoch': 0.01} 1%|█▋ | 10/681 [00:29<32:38, 2.92s/it] 2%|█▊ | 11/681 [00:32<32:51, 2.94s/it] {'loss': 1.3795, 'grad_norm': 36.561405181884766, 'learning_rate': 7.246376811594203e-08, 'margin_dpo/margin_mean': 0.1353701949119568, 'margin_dpo/margin_std': 0.31776195764541626, 'logps/chosen': -98.91744232177734, 'logps/rejected': -64.2801742553711, 'logps/ref_chosen': -98.9473876953125, 'logps/ref_rejected': -64.17475128173828, 'logits/chosen': 0.7130615711212158, 'logits/rejected': 0.9787681102752686, 'epoch': 0.02} 2%|█▊ | 11/681 [00:32<32:51, 2.94s/it] 2%|██ | 12/681 [00:35<32:58, 2.96s/it] {'loss': 1.3757, 'grad_norm': 39.813255310058594, 'learning_rate': 7.971014492753623e-08, 'margin_dpo/margin_mean': 0.005125999450683594, 'margin_dpo/margin_std': 0.4299471974372864, 'logps/chosen': -104.32192993164062, 'logps/rejected': -128.5462646484375, 'logps/ref_chosen': -104.27352142333984, 'logps/ref_rejected': -128.4927215576172, 'logits/chosen': -0.26005640625953674, 'logits/rejected': -0.09152386337518692, 'epoch': 0.02} 2%|██ | 12/681 [00:35<32:58, 2.96s/it] 2%|██▏ | 13/681 [00:39<33:32, 3.01s/it] {'loss': 1.382, 'grad_norm': 41.7731819152832, 'learning_rate': 8.695652173913042e-08, 'margin_dpo/margin_mean': 0.041529059410095215, 'margin_dpo/margin_std': 0.49482065439224243, 'logps/chosen': -101.96672058105469, 'logps/rejected': -78.45487976074219, 'logps/ref_chosen': -102.03691101074219, 'logps/ref_rejected': -78.48355102539062, 'logits/chosen': 0.3225553631782532, 'logits/rejected': 0.6485546827316284, 'epoch': 0.02} 2%|██▏ | 13/681 [00:39<33:32, 3.01s/it] 2%|██▎ | 14/681 [00:42<33:39, 3.03s/it] {'loss': 1.3945, 'grad_norm': 35.894309997558594, 'learning_rate': 9.420289855072464e-08, 'margin_dpo/margin_mean': -0.3216536045074463, 'margin_dpo/margin_std': 0.47595691680908203, 'logps/chosen': -119.18898010253906, 'logps/rejected': -117.81944274902344, 'logps/ref_chosen': -119.02677917480469, 'logps/ref_rejected': -117.97889709472656, 'logits/chosen': 0.40504711866378784, 'logits/rejected': 0.5168009400367737, 'epoch': 0.02} 2%|██▎ | 14/681 [00:42<33:39, 3.03s/it] 2%|██▌ | 15/681 [00:45<34:01, 3.06s/it] {'loss': 1.3854, 'grad_norm': 31.745962142944336, 'learning_rate': 1.0144927536231885e-07, 'margin_dpo/margin_mean': -0.06406116485595703, 'margin_dpo/margin_std': 0.37379056215286255, 'logps/chosen': -74.27200317382812, 'logps/rejected': -95.67192077636719, 'logps/ref_chosen': -74.2003173828125, 'logps/ref_rejected': -95.664306640625, 'logits/chosen': 0.22957265377044678, 'logits/rejected': 0.4173096716403961, 'epoch': 0.02} 2%|██▌ | 15/681 [00:45<34:01, 3.06s/it] 2%|██▋ | 16/681 [00:48<33:15, 3.00s/it] {'loss': 1.3769, 'grad_norm': 34.3470344543457, 'learning_rate': 1.0869565217391303e-07, 'margin_dpo/margin_mean': 0.24891865253448486, 'margin_dpo/margin_std': 0.6235331296920776, 'logps/chosen': -111.2186279296875, 'logps/rejected': -87.1607666015625, 'logps/ref_chosen': -111.30982208251953, 'logps/ref_rejected': -87.0030517578125, 'logits/chosen': -0.015835046768188477, 'logits/rejected': 0.32137343287467957, 'epoch': 0.02} 2%|██▋ | 16/681 [00:48<33:15, 3.00s/it] 2%|██▊ | 17/681 [00:50<32:43, 2.96s/it] {'loss': 1.3844, 'grad_norm': 35.640289306640625, 'learning_rate': 1.1594202898550725e-07, 'margin_dpo/margin_mean': 0.13567650318145752, 'margin_dpo/margin_std': 0.3899996876716614, 'logps/chosen': -104.69229125976562, 'logps/rejected': -103.0672607421875, 'logps/ref_chosen': -104.84930419921875, 'logps/ref_rejected': -103.08859252929688, 'logits/chosen': 0.13770782947540283, 'logits/rejected': 0.37006980180740356, 'epoch': 0.02} 2%|██▊ | 17/681 [00:51<32:43, 2.96s/it] 3%|███ | 18/681 [00:53<32:15, 2.92s/it] {'loss': 1.3876, 'grad_norm': 41.77239227294922, 'learning_rate': 1.2318840579710146e-07, 'margin_dpo/margin_mean': 0.03688061237335205, 'margin_dpo/margin_std': 0.48090770840644836, 'logps/chosen': -95.5035400390625, 'logps/rejected': -105.9694595336914, 'logps/ref_chosen': -95.57725524902344, 'logps/ref_rejected': -106.00630187988281, 'logits/chosen': 0.07230012118816376, 'logits/rejected': 0.59023118019104, 'epoch': 0.03} 3%|███ | 18/681 [00:53<32:15, 2.92s/it] 3%|███▏ | 19/681 [00:56<32:25, 2.94s/it] {'loss': 1.3845, 'grad_norm': 31.308109283447266, 'learning_rate': 1.3043478260869563e-07, 'margin_dpo/margin_mean': -0.03345954418182373, 'margin_dpo/margin_std': 0.2865228056907654, 'logps/chosen': -81.89685821533203, 'logps/rejected': -86.17457580566406, 'logps/ref_chosen': -81.85494995117188, 'logps/ref_rejected': -86.16612243652344, 'logits/chosen': 0.2451936900615692, 'logits/rejected': 0.5206657648086548, 'epoch': 0.03} 3%|███▏ | 19/681 [00:56<32:25, 2.94s/it] 3%|███▎ | 20/681 [00:59<32:22, 2.94s/it] {'loss': 1.3841, 'grad_norm': 32.505680084228516, 'learning_rate': 1.3768115942028986e-07, 'margin_dpo/margin_mean': -0.0009757280349731445, 'margin_dpo/margin_std': 0.45821622014045715, 'logps/chosen': -94.32595825195312, 'logps/rejected': -83.14512634277344, 'logps/ref_chosen': -94.21621704101562, 'logps/ref_rejected': -83.03634643554688, 'logits/chosen': 0.35453853011131287, 'logits/rejected': 0.6151764988899231, 'epoch': 0.03} 3%|███▎ | 20/681 [00:59<32:22, 2.94s/it] 3%|███▌ | 21/681 [01:02<32:05, 2.92s/it] {'loss': 1.3781, 'grad_norm': 31.37238883972168, 'learning_rate': 1.4492753623188405e-07, 'margin_dpo/margin_mean': 0.12594342231750488, 'margin_dpo/margin_std': 0.3411893844604492, 'logps/chosen': -98.95984649658203, 'logps/rejected': -67.98387908935547, 'logps/ref_chosen': -99.03758239746094, 'logps/ref_rejected': -67.93568420410156, 'logits/chosen': 0.6882296800613403, 'logits/rejected': 1.0381821393966675, 'epoch': 0.03} 3%|███▌ | 21/681 [01:02<32:05, 2.92s/it] 3%|███▋ | 22/681 [01:05<32:59, 3.00s/it] {'loss': 1.3815, 'grad_norm': 42.97276306152344, 'learning_rate': 1.5217391304347825e-07, 'margin_dpo/margin_mean': 0.040738582611083984, 'margin_dpo/margin_std': 0.3939288258552551, 'logps/chosen': -74.99850463867188, 'logps/rejected': -128.13372802734375, 'logps/ref_chosen': -74.9967041015625, 'logps/ref_rejected': -128.09115600585938, 'logits/chosen': 0.5317803621292114, 'logits/rejected': 0.4500496983528137, 'epoch': 0.03} 3%|███▋ | 22/681 [01:05<32:59, 3.00s/it] 3%|███▊ | 23/681 [01:08<33:13, 3.03s/it] {'loss': 1.3858, 'grad_norm': 31.378528594970703, 'learning_rate': 1.5942028985507245e-07, 'margin_dpo/margin_mean': 0.05478942394256592, 'margin_dpo/margin_std': 0.2976340651512146, 'logps/chosen': -81.11967468261719, 'logps/rejected': -110.63468933105469, 'logps/ref_chosen': -81.07363891601562, 'logps/ref_rejected': -110.53385925292969, 'logits/chosen': 0.5977481007575989, 'logits/rejected': 0.6081861257553101, 'epoch': 0.03} 3%|███▊ | 23/681 [01:08<33:13, 3.03s/it] 4%|████ | 24/681 [01:11<33:22, 3.05s/it] {'loss': 1.3832, 'grad_norm': 44.402305603027344, 'learning_rate': 1.6666666666666665e-07, 'margin_dpo/margin_mean': -0.02987229824066162, 'margin_dpo/margin_std': 0.22926175594329834, 'logps/chosen': -65.41180419921875, 'logps/rejected': -64.81765747070312, 'logps/ref_chosen': -65.40289306640625, 'logps/ref_rejected': -64.838623046875, 'logits/chosen': 0.6498656272888184, 'logits/rejected': 0.7695809602737427, 'epoch': 0.04} 4%|████ | 24/681 [01:12<33:22, 3.05s/it] 4%|████▏ | 25/681 [01:15<33:29, 3.06s/it] {'loss': 1.3793, 'grad_norm': 36.98027038574219, 'learning_rate': 1.7391304347826085e-07, 'margin_dpo/margin_mean': -0.003041386604309082, 'margin_dpo/margin_std': 0.5337319374084473, 'logps/chosen': -85.75971984863281, 'logps/rejected': -121.53792572021484, 'logps/ref_chosen': -85.73371887207031, 'logps/ref_rejected': -121.51497650146484, 'logits/chosen': 0.8376168012619019, 'logits/rejected': 0.9103662967681885, 'epoch': 0.04} 4%|████▏ | 25/681 [01:15<33:29, 3.06s/it] 4%|████▎ | 26/681 [01:17<31:51, 2.92s/it] {'loss': 1.3965, 'grad_norm': 41.38254165649414, 'learning_rate': 1.8115942028985507e-07, 'margin_dpo/margin_mean': -0.0806952714920044, 'margin_dpo/margin_std': 0.3255520761013031, 'logps/chosen': -81.58695220947266, 'logps/rejected': -150.1844482421875, 'logps/ref_chosen': -81.50300598144531, 'logps/ref_rejected': -150.1811981201172, 'logits/chosen': 0.19763419032096863, 'logits/rejected': 0.4370805025100708, 'epoch': 0.04} 4%|████▎ | 26/681 [01:17<31:51, 2.92s/it] 4%|████▌ | 27/681 [01:20<31:37, 2.90s/it] {'loss': 1.3714, 'grad_norm': 44.32342529296875, 'learning_rate': 1.8840579710144927e-07, 'margin_dpo/margin_mean': 0.19344651699066162, 'margin_dpo/margin_std': 0.43172699213027954, 'logps/chosen': -80.22408294677734, 'logps/rejected': -113.37527465820312, 'logps/ref_chosen': -80.2718505859375, 'logps/ref_rejected': -113.22959899902344, 'logits/chosen': -0.14630630612373352, 'logits/rejected': 0.150857076048851, 'epoch': 0.04} 4%|████▌ | 27/681 [01:20<31:37, 2.90s/it] 4%|████▋ | 28/681 [01:23<31:40, 2.91s/it] {'loss': 1.3747, 'grad_norm': 36.64478302001953, 'learning_rate': 1.9565217391304347e-07, 'margin_dpo/margin_mean': 0.19830501079559326, 'margin_dpo/margin_std': 0.31042030453681946, 'logps/chosen': -92.43882751464844, 'logps/rejected': -98.50408935546875, 'logps/ref_chosen': -92.57330322265625, 'logps/ref_rejected': -98.44024658203125, 'logits/chosen': 0.3271695375442505, 'logits/rejected': 0.4185149073600769, 'epoch': 0.04} 4%|████▋ | 28/681 [01:23<31:40, 2.91s/it] 4%|████▊ | 29/681 [01:26<30:32, 2.81s/it] {'loss': 1.3787, 'grad_norm': 42.460182189941406, 'learning_rate': 2.028985507246377e-07, 'margin_dpo/margin_mean': 0.1480005383491516, 'margin_dpo/margin_std': 0.3144315779209137, 'logps/chosen': -79.79510498046875, 'logps/rejected': -96.49360656738281, 'logps/ref_chosen': -79.78430938720703, 'logps/ref_rejected': -96.3348159790039, 'logits/chosen': 0.35313108563423157, 'logits/rejected': 0.45420438051223755, 'epoch': 0.04} 4%|████▊ | 29/681 [01:26<30:32, 2.81s/it] 4%|█████ | 30/681 [01:29<32:21, 2.98s/it] {'loss': 1.3812, 'grad_norm': 39.358245849609375, 'learning_rate': 2.1014492753623187e-07, 'margin_dpo/margin_mean': 0.17785942554473877, 'margin_dpo/margin_std': 0.42279767990112305, 'logps/chosen': -115.38392639160156, 'logps/rejected': -118.85655975341797, 'logps/ref_chosen': -115.44339752197266, 'logps/ref_rejected': -118.73816680908203, 'logits/chosen': 0.7991341948509216, 'logits/rejected': 0.8507763147354126, 'epoch': 0.04} 4%|█████ | 30/681 [01:29<32:21, 2.98s/it] 5%|█████▏ | 31/681 [01:32<32:38, 3.01s/it] {'loss': 1.376, 'grad_norm': 31.34886360168457, 'learning_rate': 2.1739130434782607e-07, 'margin_dpo/margin_mean': -0.029139041900634766, 'margin_dpo/margin_std': 0.2328224629163742, 'logps/chosen': -80.2682113647461, 'logps/rejected': -72.02093505859375, 'logps/ref_chosen': -80.27281188964844, 'logps/ref_rejected': -72.05465698242188, 'logits/chosen': 0.7612583637237549, 'logits/rejected': 0.9483832120895386, 'epoch': 0.05} 5%|█████▏ | 31/681 [01:32<32:38, 3.01s/it] 5%|█████▎ | 32/681 [01:35<32:39, 3.02s/it] {'loss': 1.3753, 'grad_norm': 35.19465255737305, 'learning_rate': 2.2463768115942027e-07, 'margin_dpo/margin_mean': 0.1294306516647339, 'margin_dpo/margin_std': 0.5172077417373657, 'logps/chosen': -106.95022583007812, 'logps/rejected': -101.90875244140625, 'logps/ref_chosen': -106.981201171875, 'logps/ref_rejected': -101.810302734375, 'logits/chosen': 0.9847223162651062, 'logits/rejected': 1.10117506980896, 'epoch': 0.05} 5%|█████▎ | 32/681 [01:35<32:39, 3.02s/it] 5%|█████▌ | 33/681 [01:38<31:45, 2.94s/it] {'loss': 1.373, 'grad_norm': 33.5123291015625, 'learning_rate': 2.318840579710145e-07, 'margin_dpo/margin_mean': 0.16950154304504395, 'margin_dpo/margin_std': 0.31402474641799927, 'logps/chosen': -92.93179321289062, 'logps/rejected': -108.07872009277344, 'logps/ref_chosen': -92.88273620605469, 'logps/ref_rejected': -107.86016082763672, 'logits/chosen': 0.8942906856536865, 'logits/rejected': 1.0839948654174805, 'epoch': 0.05} 5%|█████▌ | 33/681 [01:38<31:45, 2.94s/it] 5%|█████▋ | 34/681 [01:41<31:56, 2.96s/it] {'loss': 1.3738, 'grad_norm': 40.969905853271484, 'learning_rate': 2.391304347826087e-07, 'margin_dpo/margin_mean': 0.11852216720581055, 'margin_dpo/margin_std': 0.2863542437553406, 'logps/chosen': -96.10871887207031, 'logps/rejected': -111.40603637695312, 'logps/ref_chosen': -96.13287353515625, 'logps/ref_rejected': -111.3116683959961, 'logits/chosen': 0.3540779948234558, 'logits/rejected': 0.8411053419113159, 'epoch': 0.05} 5%|█████▋ | 34/681 [01:41<31:56, 2.96s/it] 5%|█████▊ | 35/681 [01:44<31:37, 2.94s/it] {'loss': 1.3743, 'grad_norm': 43.0073127746582, 'learning_rate': 2.463768115942029e-07, 'margin_dpo/margin_mean': -0.04916191101074219, 'margin_dpo/margin_std': 0.3702870309352875, 'logps/chosen': -68.8739013671875, 'logps/rejected': -110.1253662109375, 'logps/ref_chosen': -68.71487426757812, 'logps/ref_rejected': -110.0155029296875, 'logits/chosen': 0.4877297878265381, 'logits/rejected': 0.7324085235595703, 'epoch': 0.05} 5%|█████▊ | 35/681 [01:44<31:37, 2.94s/it] 5%|██████ | 36/681 [01:47<31:51, 2.96s/it] {'loss': 1.3695, 'grad_norm': 32.48810577392578, 'learning_rate': 2.536231884057971e-07, 'margin_dpo/margin_mean': 0.1472916603088379, 'margin_dpo/margin_std': 0.4905526340007782, 'logps/chosen': -65.26568603515625, 'logps/rejected': -72.68403625488281, 'logps/ref_chosen': -65.30921936035156, 'logps/ref_rejected': -72.58026885986328, 'logits/chosen': 0.265766441822052, 'logits/rejected': 0.3960006833076477, 'epoch': 0.05} 5%|██████ | 36/681 [01:47<31:51, 2.96s/it] 5%|██████▏ | 37/681 [01:50<31:57, 2.98s/it] {'loss': 1.3602, 'grad_norm': 35.842777252197266, 'learning_rate': 2.6086956521739126e-07, 'margin_dpo/margin_mean': 0.21903866529464722, 'margin_dpo/margin_std': 0.4852628707885742, 'logps/chosen': -113.5171127319336, 'logps/rejected': -84.8321304321289, 'logps/ref_chosen': -113.48770141601562, 'logps/ref_rejected': -84.58368682861328, 'logits/chosen': 0.3955993056297302, 'logits/rejected': 0.6936404705047607, 'epoch': 0.05} 5%|██████▏ | 37/681 [01:50<31:57, 2.98s/it] 6%|██████▎ | 38/681 [01:53<31:31, 2.94s/it] {'loss': 1.3655, 'grad_norm': 36.82903289794922, 'learning_rate': 2.681159420289855e-07, 'margin_dpo/margin_mean': 0.187727689743042, 'margin_dpo/margin_std': 0.7143464088439941, 'logps/chosen': -92.1820297241211, 'logps/rejected': -146.4608917236328, 'logps/ref_chosen': -92.26864624023438, 'logps/ref_rejected': -146.35977172851562, 'logits/chosen': 0.6250002384185791, 'logits/rejected': 0.7215943336486816, 'epoch': 0.06} 6%|██████▎ | 38/681 [01:53<31:31, 2.94s/it] 6%|██████▌ | 39/681 [01:55<31:17, 2.93s/it] {'loss': 1.3574, 'grad_norm': 36.069664001464844, 'learning_rate': 2.753623188405797e-07, 'margin_dpo/margin_mean': 0.39781689643859863, 'margin_dpo/margin_std': 0.4662584066390991, 'logps/chosen': -81.108154296875, 'logps/rejected': -83.49285888671875, 'logps/ref_chosen': -81.08113098144531, 'logps/ref_rejected': -83.06800842285156, 'logits/chosen': 0.7301809191703796, 'logits/rejected': 1.0973470211029053, 'epoch': 0.06} 6%|██████▌ | 39/681 [01:56<31:17, 2.93s/it] 6%|██████▋ | 40/681 [01:58<30:52, 2.89s/it] {'loss': 1.3609, 'grad_norm': 29.052732467651367, 'learning_rate': 2.8260869565217386e-07, 'margin_dpo/margin_mean': 0.1183696985244751, 'margin_dpo/margin_std': 0.33317673206329346, 'logps/chosen': -76.96479797363281, 'logps/rejected': -75.19844055175781, 'logps/ref_chosen': -76.87438201904297, 'logps/ref_rejected': -74.98965454101562, 'logits/chosen': 0.8642468452453613, 'logits/rejected': 0.9313390254974365, 'epoch': 0.06} 6%|██████▋ | 40/681 [01:58<30:52, 2.89s/it] 6%|██████▊ | 41/681 [02:01<30:57, 2.90s/it] {'loss': 1.3571, 'grad_norm': 33.42491149902344, 'learning_rate': 2.898550724637681e-07, 'margin_dpo/margin_mean': 0.3881516456604004, 'margin_dpo/margin_std': 0.5789889693260193, 'logps/chosen': -113.77964782714844, 'logps/rejected': -106.6886215209961, 'logps/ref_chosen': -113.75234985351562, 'logps/ref_rejected': -106.27317810058594, 'logits/chosen': 0.7419478893280029, 'logits/rejected': 0.8945612907409668, 'epoch': 0.06} 6%|██████▊ | 41/681 [02:01<30:57, 2.90s/it] 6%|███████ | 42/681 [02:04<30:49, 2.89s/it] {'loss': 1.3456, 'grad_norm': 40.86435317993164, 'learning_rate': 2.971014492753623e-07, 'margin_dpo/margin_mean': 0.4290355443954468, 'margin_dpo/margin_std': 0.652177631855011, 'logps/chosen': -93.4481430053711, 'logps/rejected': -138.0484619140625, 'logps/ref_chosen': -93.46589660644531, 'logps/ref_rejected': -137.63717651367188, 'logits/chosen': 0.7604937553405762, 'logits/rejected': 0.8403340578079224, 'epoch': 0.06} 6%|███████ | 42/681 [02:04<30:49, 2.89s/it] 6%|███████▏ | 43/681 [02:07<30:59, 2.91s/it] {'loss': 1.3532, 'grad_norm': 38.23468017578125, 'learning_rate': 3.043478260869565e-07, 'margin_dpo/margin_mean': 0.3112701177597046, 'margin_dpo/margin_std': 0.5115389227867126, 'logps/chosen': -87.23843383789062, 'logps/rejected': -87.05091094970703, 'logps/ref_chosen': -87.20645141601562, 'logps/ref_rejected': -86.70765686035156, 'logits/chosen': 0.5066407322883606, 'logits/rejected': 0.628272294998169, 'epoch': 0.06} 6%|███████▏ | 43/681 [02:07<30:59, 2.91s/it] 6%|███████▎ | 44/681 [02:10<30:49, 2.90s/it] {'loss': 1.3572, 'grad_norm': 34.67744445800781, 'learning_rate': 3.115942028985507e-07, 'margin_dpo/margin_mean': 0.32104432582855225, 'margin_dpo/margin_std': 0.7572367191314697, 'logps/chosen': -65.89019775390625, 'logps/rejected': -116.15614318847656, 'logps/ref_chosen': -65.7902603149414, 'logps/ref_rejected': -115.73516082763672, 'logits/chosen': 0.7453466057777405, 'logits/rejected': 0.9643809795379639, 'epoch': 0.06} 6%|███████▎ | 44/681 [02:10<30:49, 2.90s/it] 7%|███████▌ | 45/681 [02:13<31:38, 2.98s/it] {'loss': 1.3571, 'grad_norm': 24.18403434753418, 'learning_rate': 3.188405797101449e-07, 'margin_dpo/margin_mean': 0.3061286211013794, 'margin_dpo/margin_std': 0.5986896753311157, 'logps/chosen': -85.85913848876953, 'logps/rejected': -83.0451889038086, 'logps/ref_chosen': -85.71604919433594, 'logps/ref_rejected': -82.59597778320312, 'logits/chosen': 0.4455117881298065, 'logits/rejected': 0.5920794010162354, 'epoch': 0.07} 7%|███████▌ | 45/681 [02:13<31:38, 2.98s/it] 7%|███████▋ | 46/681 [02:16<32:16, 3.05s/it] {'loss': 1.3424, 'grad_norm': 33.199153900146484, 'learning_rate': 3.260869565217391e-07, 'margin_dpo/margin_mean': 0.4690542221069336, 'margin_dpo/margin_std': 0.6764923334121704, 'logps/chosen': -95.22297668457031, 'logps/rejected': -114.26651000976562, 'logps/ref_chosen': -95.29031372070312, 'logps/ref_rejected': -113.86479949951172, 'logits/chosen': 0.5929129719734192, 'logits/rejected': 0.8907681703567505, 'epoch': 0.07} 7%|███████▋ | 46/681 [02:16<32:16, 3.05s/it] 7%|███████▊ | 47/681 [02:19<32:09, 3.04s/it] {'loss': 1.3475, 'grad_norm': 34.175819396972656, 'learning_rate': 3.333333333333333e-07, 'margin_dpo/margin_mean': 0.4314463138580322, 'margin_dpo/margin_std': 0.887498140335083, 'logps/chosen': -73.86705780029297, 'logps/rejected': -74.57892608642578, 'logps/ref_chosen': -73.91566467285156, 'logps/ref_rejected': -74.19608306884766, 'logits/chosen': -0.16016048192977905, 'logits/rejected': 0.03261171281337738, 'epoch': 0.07} 7%|███████▊ | 47/681 [02:19<32:09, 3.04s/it] 7%|████████ | 48/681 [02:22<31:50, 3.02s/it] {'loss': 1.3388, 'grad_norm': 39.072574615478516, 'learning_rate': 3.4057971014492755e-07, 'margin_dpo/margin_mean': 0.4573037624359131, 'margin_dpo/margin_std': 1.0886731147766113, 'logps/chosen': -104.47331237792969, 'logps/rejected': -108.28083801269531, 'logps/ref_chosen': -104.229248046875, 'logps/ref_rejected': -107.5794677734375, 'logits/chosen': 0.28998929262161255, 'logits/rejected': 0.39170825481414795, 'epoch': 0.07} 7%|████████ | 48/681 [02:22<31:50, 3.02s/it] 7%|████████▏ | 49/681 [02:25<31:31, 2.99s/it] {'loss': 1.3129, 'grad_norm': 39.3463134765625, 'learning_rate': 3.478260869565217e-07, 'margin_dpo/margin_mean': 0.5325143337249756, 'margin_dpo/margin_std': 0.7589560747146606, 'logps/chosen': -81.1698226928711, 'logps/rejected': -75.13246154785156, 'logps/ref_chosen': -81.32856750488281, 'logps/ref_rejected': -74.75869750976562, 'logits/chosen': 0.07694488763809204, 'logits/rejected': 0.623125433921814, 'epoch': 0.07} 7%|████████▏ | 49/681 [02:25<31:31, 2.99s/it] 7%|████████▎ | 50/681 [02:28<30:58, 2.95s/it] {'loss': 1.3292, 'grad_norm': 33.31324768066406, 'learning_rate': 3.5507246376811595e-07, 'margin_dpo/margin_mean': 0.3646160364151001, 'margin_dpo/margin_std': 0.785099983215332, 'logps/chosen': -104.20743560791016, 'logps/rejected': -123.37358093261719, 'logps/ref_chosen': -103.97581481933594, 'logps/ref_rejected': -122.77733612060547, 'logits/chosen': 0.8446512222290039, 'logits/rejected': 1.1388261318206787, 'epoch': 0.07} 7%|████████▎ | 50/681 [02:28<30:58, 2.95s/it] 7%|████████▌ | 51/681 [02:31<31:07, 2.96s/it] {'loss': 1.33, 'grad_norm': 30.007966995239258, 'learning_rate': 3.6231884057971015e-07, 'margin_dpo/margin_mean': 0.3508089780807495, 'margin_dpo/margin_std': 1.0862066745758057, 'logps/chosen': -110.92190551757812, 'logps/rejected': -90.43896484375, 'logps/ref_chosen': -110.60376739501953, 'logps/ref_rejected': -89.77001953125, 'logits/chosen': 0.6785503029823303, 'logits/rejected': 1.0297834873199463, 'epoch': 0.07} 7%|████████▌ | 51/681 [02:31<31:07, 2.96s/it] 8%|████████▋ | 52/681 [02:34<30:33, 2.91s/it] {'loss': 1.3089, 'grad_norm': 32.3361701965332, 'learning_rate': 3.695652173913043e-07, 'margin_dpo/margin_mean': 0.6009612083435059, 'margin_dpo/margin_std': 0.7621315717697144, 'logps/chosen': -117.17269134521484, 'logps/rejected': -83.43679809570312, 'logps/ref_chosen': -116.8155517578125, 'logps/ref_rejected': -82.47869873046875, 'logits/chosen': 0.3316473960876465, 'logits/rejected': 0.5727195739746094, 'epoch': 0.08} 8%|████████▋ | 52/681 [02:34<30:33, 2.91s/it] 8%|████████▊ | 53/681 [02:37<31:29, 3.01s/it] {'loss': 1.2854, 'grad_norm': 39.464393615722656, 'learning_rate': 3.7681159420289855e-07, 'margin_dpo/margin_mean': 1.3681297302246094, 'margin_dpo/margin_std': 1.9412078857421875, 'logps/chosen': -97.04790496826172, 'logps/rejected': -117.12537384033203, 'logps/ref_chosen': -97.222412109375, 'logps/ref_rejected': -115.93175506591797, 'logits/chosen': 0.769700288772583, 'logits/rejected': 0.8581574559211731, 'epoch': 0.08} 8%|████████▊ | 53/681 [02:37<31:29, 3.01s/it] 8%|█████████ | 54/681 [02:40<30:36, 2.93s/it] {'loss': 1.2888, 'grad_norm': 40.05606460571289, 'learning_rate': 3.8405797101449274e-07, 'margin_dpo/margin_mean': 0.7888990640640259, 'margin_dpo/margin_std': 0.9890120029449463, 'logps/chosen': -68.86314392089844, 'logps/rejected': -87.76393127441406, 'logps/ref_chosen': -68.70967864990234, 'logps/ref_rejected': -86.82157135009766, 'logits/chosen': 0.8848798871040344, 'logits/rejected': 0.9821081757545471, 'epoch': 0.08} 8%|█████████ | 54/681 [02:40<30:36, 2.93s/it] 8%|█████████▏ | 55/681 [02:42<29:27, 2.82s/it] {'loss': 1.2802, 'grad_norm': 35.246360778808594, 'learning_rate': 3.9130434782608694e-07, 'margin_dpo/margin_mean': 1.9652525186538696, 'margin_dpo/margin_std': 1.6538578271865845, 'logps/chosen': -98.69027709960938, 'logps/rejected': -113.6478271484375, 'logps/ref_chosen': -99.13316345214844, 'logps/ref_rejected': -112.1254653930664, 'logits/chosen': -0.12803569436073303, 'logits/rejected': 0.09957672655582428, 'epoch': 0.08} 8%|█████████▏ | 55/681 [02:42<29:27, 2.82s/it] 8%|█████████▎ | 56/681 [02:45<29:57, 2.88s/it] {'loss': 1.2933, 'grad_norm': 33.27520751953125, 'learning_rate': 3.9855072463768114e-07, 'margin_dpo/margin_mean': 0.7786517143249512, 'margin_dpo/margin_std': 1.203643798828125, 'logps/chosen': -98.00480651855469, 'logps/rejected': -84.39903259277344, 'logps/ref_chosen': -97.85421752929688, 'logps/ref_rejected': -83.46979522705078, 'logits/chosen': -0.09207522869110107, 'logits/rejected': 0.29723483324050903, 'epoch': 0.08} 8%|█████████▎ | 56/681 [02:45<29:57, 2.88s/it] 8%|█████████▌ | 57/681 [02:48<29:50, 2.87s/it] {'loss': 1.2831, 'grad_norm': 33.76715850830078, 'learning_rate': 4.057971014492754e-07, 'margin_dpo/margin_mean': 1.4195455312728882, 'margin_dpo/margin_std': 1.264824390411377, 'logps/chosen': -99.81153869628906, 'logps/rejected': -131.81729125976562, 'logps/ref_chosen': -100.10478973388672, 'logps/ref_rejected': -130.69100952148438, 'logits/chosen': 0.17622031271457672, 'logits/rejected': 0.25724083185195923, 'epoch': 0.08} 8%|█████████▌ | 57/681 [02:48<29:50, 2.87s/it] 9%|█████████▋ | 58/681 [02:51<30:15, 2.91s/it] {'loss': 1.251, 'grad_norm': 38.04386901855469, 'learning_rate': 4.1304347826086954e-07, 'margin_dpo/margin_mean': 1.2433971166610718, 'margin_dpo/margin_std': 1.3449275493621826, 'logps/chosen': -109.82270812988281, 'logps/rejected': -71.33671569824219, 'logps/ref_chosen': -109.95325469970703, 'logps/ref_rejected': -70.22386169433594, 'logits/chosen': -0.1125476062297821, 'logits/rejected': 0.2727716565132141, 'epoch': 0.09} 9%|█████████▋ | 58/681 [02:51<30:15, 2.91s/it] 9%|█████████▉ | 59/681 [02:54<30:14, 2.92s/it] {'loss': 1.2692, 'grad_norm': 33.97794723510742, 'learning_rate': 4.2028985507246374e-07, 'margin_dpo/margin_mean': 1.5013514757156372, 'margin_dpo/margin_std': 1.5745304822921753, 'logps/chosen': -100.960693359375, 'logps/rejected': -112.17672729492188, 'logps/ref_chosen': -100.9923095703125, 'logps/ref_rejected': -110.70699310302734, 'logits/chosen': 0.39747869968414307, 'logits/rejected': 0.5710791349411011, 'epoch': 0.09} 9%|█████████▉ | 59/681 [02:54<30:14, 2.92s/it] 9%|██████████ | 60/681 [02:57<29:34, 2.86s/it] {'loss': 1.2738, 'grad_norm': 28.536787033081055, 'learning_rate': 4.2753623188405794e-07, 'margin_dpo/margin_mean': 1.3974266052246094, 'margin_dpo/margin_std': 1.2194509506225586, 'logps/chosen': -107.55908203125, 'logps/rejected': -69.468017578125, 'logps/ref_chosen': -108.16279602050781, 'logps/ref_rejected': -68.67430877685547, 'logits/chosen': 0.35590529441833496, 'logits/rejected': 0.8095076680183411, 'epoch': 0.09} 9%|██████████ | 60/681 [02:57<29:34, 2.86s/it] 9%|██████████▏ | 61/681 [03:00<30:24, 2.94s/it] {'loss': 1.2856, 'grad_norm': 28.542030334472656, 'learning_rate': 4.3478260869565214e-07, 'margin_dpo/margin_mean': 1.2662101984024048, 'margin_dpo/margin_std': 1.339787483215332, 'logps/chosen': -80.26652526855469, 'logps/rejected': -85.35874938964844, 'logps/ref_chosen': -80.13163757324219, 'logps/ref_rejected': -83.95764923095703, 'logits/chosen': -0.4731886386871338, 'logits/rejected': -0.2491273283958435, 'epoch': 0.09} 9%|██████████▏ | 61/681 [03:00<30:24, 2.94s/it] 9%|██████████▍ | 62/681 [03:03<30:54, 3.00s/it] {'loss': 1.285, 'grad_norm': 28.514225006103516, 'learning_rate': 4.420289855072464e-07, 'margin_dpo/margin_mean': 0.8554513454437256, 'margin_dpo/margin_std': 1.4367563724517822, 'logps/chosen': -50.487548828125, 'logps/rejected': -73.86290740966797, 'logps/ref_chosen': -50.332794189453125, 'logps/ref_rejected': -72.85269927978516, 'logits/chosen': 0.680814802646637, 'logits/rejected': 0.6677084565162659, 'epoch': 0.09} 9%|██████████▍ | 62/681 [03:03<30:54, 3.00s/it] 9%|██████████▌ | 63/681 [03:06<30:22, 2.95s/it] {'loss': 1.246, 'grad_norm': 34.94499206542969, 'learning_rate': 4.4927536231884053e-07, 'margin_dpo/margin_mean': 1.2721160650253296, 'margin_dpo/margin_std': 1.2849652767181396, 'logps/chosen': -121.78221893310547, 'logps/rejected': -82.76856994628906, 'logps/ref_chosen': -121.70402526855469, 'logps/ref_rejected': -81.41825866699219, 'logits/chosen': 0.4011261463165283, 'logits/rejected': 0.6790079474449158, 'epoch': 0.09} 9%|██████████▌ | 63/681 [03:06<30:22, 2.95s/it] 9%|██████████▋ | 64/681 [03:09<29:57, 2.91s/it] {'loss': 1.2345, 'grad_norm': 34.87194061279297, 'learning_rate': 4.5652173913043473e-07, 'margin_dpo/margin_mean': 0.7329847812652588, 'margin_dpo/margin_std': 1.2104127407073975, 'logps/chosen': -80.07738494873047, 'logps/rejected': -105.55560302734375, 'logps/ref_chosen': -79.740234375, 'logps/ref_rejected': -104.4854736328125, 'logits/chosen': 0.7611849308013916, 'logits/rejected': 0.9642089605331421, 'epoch': 0.09} 9%|██████████▋ | 64/681 [03:09<29:57, 2.91s/it] 10%|██████████▉ | 65/681 [03:12<29:59, 2.92s/it] {'loss': 1.2297, 'grad_norm': 38.41434860229492, 'learning_rate': 4.63768115942029e-07, 'margin_dpo/margin_mean': 2.0177080631256104, 'margin_dpo/margin_std': 2.553192615509033, 'logps/chosen': -78.44050598144531, 'logps/rejected': -137.61776733398438, 'logps/ref_chosen': -78.41619873046875, 'logps/ref_rejected': -135.5757293701172, 'logits/chosen': 0.254086971282959, 'logits/rejected': 0.1695535033941269, 'epoch': 0.1} 10%|██████████▉ | 65/681 [03:12<29:59, 2.92s/it] 10%|███████████ | 66/681 [03:15<30:08, 2.94s/it] {'loss': 1.2646, 'grad_norm': 25.3912410736084, 'learning_rate': 4.7101449275362313e-07, 'margin_dpo/margin_mean': 1.0436724424362183, 'margin_dpo/margin_std': 1.7997262477874756, 'logps/chosen': -65.99453735351562, 'logps/rejected': -77.69013214111328, 'logps/ref_chosen': -65.68583679199219, 'logps/ref_rejected': -76.33775329589844, 'logits/chosen': -0.09305822104215622, 'logits/rejected': 0.15512987971305847, 'epoch': 0.1} 10%|███████████ | 66/681 [03:15<30:08, 2.94s/it] 10%|███████████▏ | 67/681 [03:17<29:11, 2.85s/it] {'loss': 1.3012, 'grad_norm': 24.43991470336914, 'learning_rate': 4.782608695652174e-07, 'margin_dpo/margin_mean': 1.6041903495788574, 'margin_dpo/margin_std': 2.0201072692871094, 'logps/chosen': -74.0006103515625, 'logps/rejected': -71.95913696289062, 'logps/ref_chosen': -73.65798950195312, 'logps/ref_rejected': -70.01232147216797, 'logits/chosen': 0.40889042615890503, 'logits/rejected': 0.5336402654647827, 'epoch': 0.1} 10%|███████████▏ | 67/681 [03:17<29:11, 2.85s/it] 10%|███████████▍ | 68/681 [03:20<28:44, 2.81s/it] {'loss': 1.2043, 'grad_norm': 28.16604995727539, 'learning_rate': 4.855072463768116e-07, 'margin_dpo/margin_mean': 2.804673910140991, 'margin_dpo/margin_std': 2.7214131355285645, 'logps/chosen': -85.18283081054688, 'logps/rejected': -97.47535705566406, 'logps/ref_chosen': -84.98939514160156, 'logps/ref_rejected': -94.47724914550781, 'logits/chosen': -0.8838789463043213, 'logits/rejected': -0.6690104007720947, 'epoch': 0.1} 10%|███████████▍ | 68/681 [03:20<28:44, 2.81s/it] 10%|███████████▌ | 69/681 [03:24<30:37, 3.00s/it] {'loss': 1.2047, 'grad_norm': 30.81968116760254, 'learning_rate': 4.927536231884058e-07, 'margin_dpo/margin_mean': 1.2108116149902344, 'margin_dpo/margin_std': 2.8251242637634277, 'logps/chosen': -86.99532318115234, 'logps/rejected': -120.83209228515625, 'logps/ref_chosen': -85.89825439453125, 'logps/ref_rejected': -118.52421569824219, 'logits/chosen': 0.12748196721076965, 'logits/rejected': 0.1792227327823639, 'epoch': 0.1} 10%|███████████▌ | 69/681 [03:24<30:37, 3.00s/it] 10%|███████████▋ | 70/681 [03:26<29:53, 2.93s/it] {'loss': 1.1769, 'grad_norm': 29.18440818786621, 'learning_rate': 5e-07, 'margin_dpo/margin_mean': 2.2829203605651855, 'margin_dpo/margin_std': 4.698617458343506, 'logps/chosen': -125.14912414550781, 'logps/rejected': -102.61834716796875, 'logps/ref_chosen': -124.22589111328125, 'logps/ref_rejected': -99.41220092773438, 'logits/chosen': -0.12536486983299255, 'logits/rejected': 0.061571136116981506, 'epoch': 0.1} 10%|███████████▋ | 70/681 [03:26<29:53, 2.93s/it] 10%|███████████▉ | 71/681 [03:29<29:47, 2.93s/it] {'loss': 1.0958, 'grad_norm': 34.87384796142578, 'learning_rate': 4.999967061337492e-07, 'margin_dpo/margin_mean': 2.7785582542419434, 'margin_dpo/margin_std': 2.691640853881836, 'logps/chosen': -80.64649963378906, 'logps/rejected': -92.06094360351562, 'logps/ref_chosen': -80.65885925292969, 'logps/ref_rejected': -89.29473876953125, 'logits/chosen': 0.3338947892189026, 'logits/rejected': 0.5460184812545776, 'epoch': 0.1} 10%|███████████▉ | 71/681 [03:29<29:47, 2.93s/it] 11%|████████████ | 72/681 [03:32<29:23, 2.90s/it] {'loss': 1.1477, 'grad_norm': 29.42046356201172, 'learning_rate': 4.999868246217933e-07, 'margin_dpo/margin_mean': 1.9313820600509644, 'margin_dpo/margin_std': 3.1832776069641113, 'logps/chosen': -91.0280990600586, 'logps/rejected': -90.02912902832031, 'logps/ref_chosen': -90.32836151123047, 'logps/ref_rejected': -87.39801025390625, 'logits/chosen': 0.11558493971824646, 'logits/rejected': 0.336860328912735, 'epoch': 0.11} 11%|████████████ | 72/681 [03:32<29:23, 2.90s/it] 11%|████████████▏ | 73/681 [03:35<29:24, 2.90s/it] {'loss': 1.1568, 'grad_norm': 27.79156494140625, 'learning_rate': 4.999703557245192e-07, 'margin_dpo/margin_mean': 2.4214420318603516, 'margin_dpo/margin_std': 4.514315605163574, 'logps/chosen': -113.90251159667969, 'logps/rejected': -100.67330169677734, 'logps/ref_chosen': -112.08250427246094, 'logps/ref_rejected': -96.43185424804688, 'logits/chosen': -1.0717291831970215, 'logits/rejected': -0.48407894372940063, 'epoch': 0.11} 11%|████████████▏ | 73/681 [03:35<29:24, 2.90s/it] 11%|████████████▍ | 74/681 [03:38<29:18, 2.90s/it] {'loss': 1.1118, 'grad_norm': 31.334726333618164, 'learning_rate': 4.999472998758977e-07, 'margin_dpo/margin_mean': 4.710856914520264, 'margin_dpo/margin_std': 4.988119125366211, 'logps/chosen': -84.89183044433594, 'logps/rejected': -123.54209899902344, 'logps/ref_chosen': -84.09466552734375, 'logps/ref_rejected': -118.03408813476562, 'logits/chosen': -0.442888081073761, 'logits/rejected': -0.4210275411605835, 'epoch': 0.11} 11%|████████████▍ | 74/681 [03:38<29:18, 2.90s/it] 11%|████████████▌ | 75/681 [03:41<29:03, 2.88s/it] {'loss': 1.0438, 'grad_norm': 32.42177963256836, 'learning_rate': 4.999176576834721e-07, 'margin_dpo/margin_mean': 4.149603366851807, 'margin_dpo/margin_std': 4.267527103424072, 'logps/chosen': -89.07445526123047, 'logps/rejected': -117.11609649658203, 'logps/ref_chosen': -88.38374328613281, 'logps/ref_rejected': -112.27578735351562, 'logits/chosen': -0.5751562714576721, 'logits/rejected': 0.15103864669799805, 'epoch': 0.11} 11%|████████████▌ | 75/681 [03:41<29:03, 2.88s/it] 11%|████████████▋ | 76/681 [03:44<29:59, 2.97s/it] {'loss': 1.1557, 'grad_norm': 23.008657455444336, 'learning_rate': 4.998814299283415e-07, 'margin_dpo/margin_mean': 2.8992037773132324, 'margin_dpo/margin_std': 4.499299049377441, 'logps/chosen': -101.99429321289062, 'logps/rejected': -93.37272644042969, 'logps/ref_chosen': -100.0120849609375, 'logps/ref_rejected': -88.49131774902344, 'logits/chosen': -1.5539774894714355, 'logits/rejected': -1.1704045534133911, 'epoch': 0.11} 11%|████████████▋ | 76/681 [03:44<29:59, 2.97s/it] 11%|████████████▉ | 77/681 [03:46<28:38, 2.85s/it] {'loss': 1.1272, 'grad_norm': 27.028614044189453, 'learning_rate': 4.998386175651409e-07, 'margin_dpo/margin_mean': 3.9930152893066406, 'margin_dpo/margin_std': 3.8414201736450195, 'logps/chosen': -74.51853942871094, 'logps/rejected': -107.80052185058594, 'logps/ref_chosen': -73.30534362792969, 'logps/ref_rejected': -102.59431457519531, 'logits/chosen': -0.5113037824630737, 'logits/rejected': -0.39275413751602173, 'epoch': 0.11} 11%|████████████▉ | 77/681 [03:47<28:38, 2.85s/it] 11%|█████████████ | 78/681 [03:50<29:12, 2.91s/it] {'loss': 1.0955, 'grad_norm': 25.387378692626953, 'learning_rate': 4.997892217220159e-07, 'margin_dpo/margin_mean': 4.363163948059082, 'margin_dpo/margin_std': 4.845981597900391, 'logps/chosen': -72.970703125, 'logps/rejected': -121.18896484375, 'logps/ref_chosen': -72.40596008300781, 'logps/ref_rejected': -116.26104736328125, 'logits/chosen': 0.6262469291687012, 'logits/rejected': 0.7459644079208374, 'epoch': 0.11} 11%|█████████████ | 78/681 [03:50<29:12, 2.91s/it] 12%|█████████████▏ | 79/681 [03:53<29:23, 2.93s/it] {'loss': 1.0431, 'grad_norm': 27.026376724243164, 'learning_rate': 4.997332437005931e-07, 'margin_dpo/margin_mean': 5.342948913574219, 'margin_dpo/margin_std': 7.4267473220825195, 'logps/chosen': -103.35887145996094, 'logps/rejected': -96.11433410644531, 'logps/ref_chosen': -101.26777648925781, 'logps/ref_rejected': -88.68028259277344, 'logits/chosen': -0.08271521329879761, 'logits/rejected': 0.5097367763519287, 'epoch': 0.12} 12%|█████████████▏ | 79/681 [03:53<29:23, 2.93s/it] 12%|█████████████▍ | 80/681 [03:55<29:15, 2.92s/it] {'loss': 1.1065, 'grad_norm': 25.63165855407715, 'learning_rate': 4.996706849759452e-07, 'margin_dpo/margin_mean': 1.004472255706787, 'margin_dpo/margin_std': 3.0686776638031006, 'logps/chosen': -117.60404968261719, 'logps/rejected': -74.76948547363281, 'logps/ref_chosen': -114.87706756591797, 'logps/ref_rejected': -71.03803253173828, 'logits/chosen': -0.6737807393074036, 'logits/rejected': 0.1990889012813568, 'epoch': 0.12} 12%|█████████████▍ | 80/681 [03:55<29:15, 2.92s/it] 12%|█████████████▌ | 81/681 [03:59<29:45, 2.98s/it] {'loss': 1.0254, 'grad_norm': 27.555042266845703, 'learning_rate': 4.996015471965529e-07, 'margin_dpo/margin_mean': 4.478157043457031, 'margin_dpo/margin_std': 5.330191612243652, 'logps/chosen': -91.39581298828125, 'logps/rejected': -158.87777709960938, 'logps/ref_chosen': -90.15489196777344, 'logps/ref_rejected': -153.15872192382812, 'logits/chosen': -0.7485864162445068, 'logits/rejected': -0.1424974799156189, 'epoch': 0.12} 12%|█████████████▌ | 81/681 [03:59<29:45, 2.98s/it] 12%|█████████████▋ | 82/681 [04:01<29:08, 2.92s/it] {'loss': 1.1125, 'grad_norm': 25.869966506958008, 'learning_rate': 4.995258321842611e-07, 'margin_dpo/margin_mean': 4.871798992156982, 'margin_dpo/margin_std': 4.440828323364258, 'logps/chosen': -66.34426879882812, 'logps/rejected': -85.55947875976562, 'logps/ref_chosen': -65.0645751953125, 'logps/ref_rejected': -79.4079818725586, 'logits/chosen': -0.13274694979190826, 'logits/rejected': 0.08919930458068848, 'epoch': 0.12} 12%|█████████████▋ | 82/681 [04:01<29:08, 2.92s/it] 12%|█████████████▉ | 83/681 [04:04<28:39, 2.87s/it] {'loss': 1.1083, 'grad_norm': 27.123699188232422, 'learning_rate': 4.994435419342304e-07, 'margin_dpo/margin_mean': 4.6751909255981445, 'margin_dpo/margin_std': 5.925076007843018, 'logps/chosen': -100.03043365478516, 'logps/rejected': -152.01702880859375, 'logps/ref_chosen': -97.5862045288086, 'logps/ref_rejected': -144.89761352539062, 'logits/chosen': -0.7392994165420532, 'logits/rejected': -0.6463851928710938, 'epoch': 0.12} 12%|█████████████▉ | 83/681 [04:04<28:39, 2.87s/it] 12%|██████████████ | 84/681 [04:07<30:06, 3.03s/it] {'loss': 1.2085, 'grad_norm': 28.914873123168945, 'learning_rate': 4.993546786148857e-07, 'margin_dpo/margin_mean': 1.7986866235733032, 'margin_dpo/margin_std': 4.0745134353637695, 'logps/chosen': -107.0411148071289, 'logps/rejected': -115.09510040283203, 'logps/ref_chosen': -104.22205352783203, 'logps/ref_rejected': -110.47735595703125, 'logits/chosen': -1.024448275566101, 'logits/rejected': -0.6994166970252991, 'epoch': 0.12} 12%|██████████████ | 84/681 [04:07<30:06, 3.03s/it] 12%|██████████████▏ | 85/681 [04:10<29:52, 3.01s/it] {'loss': 1.1649, 'grad_norm': 27.426593780517578, 'learning_rate': 4.992592445678582e-07, 'margin_dpo/margin_mean': 4.140119552612305, 'margin_dpo/margin_std': 8.340166091918945, 'logps/chosen': -67.73141479492188, 'logps/rejected': -84.1421127319336, 'logps/ref_chosen': -65.42636108398438, 'logps/ref_rejected': -77.69692993164062, 'logits/chosen': 0.10627250373363495, 'logits/rejected': 0.41944852471351624, 'epoch': 0.12} 12%|██████████████▏ | 85/681 [04:10<29:52, 3.01s/it] 13%|██████████████▍ | 86/681 [04:13<29:19, 2.96s/it] {'loss': 1.0992, 'grad_norm': 22.034379959106445, 'learning_rate': 4.991572423079235e-07, 'margin_dpo/margin_mean': 3.323758125305176, 'margin_dpo/margin_std': 5.560352325439453, 'logps/chosen': -71.08971405029297, 'logps/rejected': -87.36192321777344, 'logps/ref_chosen': -68.32093811035156, 'logps/ref_rejected': -81.26939392089844, 'logits/chosen': -0.6100976467132568, 'logits/rejected': -0.47021234035491943, 'epoch': 0.13} 13%|██████████████▍ | 86/681 [04:13<29:19, 2.96s/it] 13%|██████████████▌ | 87/681 [04:16<29:20, 2.96s/it] {'loss': 1.0929, 'grad_norm': 27.28897476196289, 'learning_rate': 4.990486745229364e-07, 'margin_dpo/margin_mean': 3.938999652862549, 'margin_dpo/margin_std': 4.811038017272949, 'logps/chosen': -109.26034545898438, 'logps/rejected': -121.74065399169922, 'logps/ref_chosen': -107.66572570800781, 'logps/ref_rejected': -116.20704650878906, 'logits/chosen': -1.008190631866455, 'logits/rejected': -0.7083594799041748, 'epoch': 0.13} 13%|██████████████▌ | 87/681 [04:16<29:20, 2.96s/it] 13%|██████████████▋ | 88/681 [04:19<29:22, 2.97s/it] {'loss': 1.1388, 'grad_norm': 24.714147567749023, 'learning_rate': 4.989335440737586e-07, 'margin_dpo/margin_mean': 3.813239097595215, 'margin_dpo/margin_std': 7.841992378234863, 'logps/chosen': -97.98976135253906, 'logps/rejected': -117.9246826171875, 'logps/ref_chosen': -93.07302856445312, 'logps/ref_rejected': -109.19470977783203, 'logits/chosen': -0.6558359861373901, 'logits/rejected': -0.6540915966033936, 'epoch': 0.13} 13%|██████████████▋ | 88/681 [04:19<29:22, 2.97s/it] 13%|██████████████▉ | 89/681 [04:22<28:47, 2.92s/it] {'loss': 1.1054, 'grad_norm': 24.070466995239258, 'learning_rate': 4.988118539941847e-07, 'margin_dpo/margin_mean': 4.629606246948242, 'margin_dpo/margin_std': 4.11317253112793, 'logps/chosen': -78.58067321777344, 'logps/rejected': -98.58912658691406, 'logps/ref_chosen': -76.54256439208984, 'logps/ref_rejected': -91.92141723632812, 'logits/chosen': 0.05120593309402466, 'logits/rejected': 0.2133779525756836, 'epoch': 0.13} 13%|██████████████▉ | 89/681 [04:22<28:47, 2.92s/it] 13%|███████████████ | 90/681 [04:25<28:20, 2.88s/it] {'loss': 1.0281, 'grad_norm': 26.12279510498047, 'learning_rate': 4.986836074908615e-07, 'margin_dpo/margin_mean': 5.127821445465088, 'margin_dpo/margin_std': 6.227309226989746, 'logps/chosen': -85.04541015625, 'logps/rejected': -116.90010070800781, 'logps/ref_chosen': -83.1722640991211, 'logps/ref_rejected': -109.89913940429688, 'logits/chosen': -0.6932872533798218, 'logits/rejected': -0.4962659478187561, 'epoch': 0.13} 13%|███████████████ | 90/681 [04:25<28:20, 2.88s/it] 13%|███████████████▏ | 91/681 [04:28<29:03, 2.95s/it] {'loss': 1.0875, 'grad_norm': 26.667461395263672, 'learning_rate': 4.985488079432037e-07, 'margin_dpo/margin_mean': 6.852234363555908, 'margin_dpo/margin_std': 7.726470947265625, 'logps/chosen': -76.52804565429688, 'logps/rejected': -102.70915222167969, 'logps/ref_chosen': -75.09698486328125, 'logps/ref_rejected': -94.42585754394531, 'logits/chosen': -0.5161850452423096, 'logits/rejected': -0.007033824920654297, 'epoch': 0.13} 13%|███████████████▏ | 91/681 [04:28<29:03, 2.95s/it] 14%|███████████████▍ | 92/681 [04:31<28:57, 2.95s/it] {'loss': 1.1189, 'grad_norm': 23.602184295654297, 'learning_rate': 4.984074589033043e-07, 'margin_dpo/margin_mean': 2.3370065689086914, 'margin_dpo/margin_std': 6.140504837036133, 'logps/chosen': -83.0078125, 'logps/rejected': -83.26744842529297, 'logps/ref_chosen': -79.81562805175781, 'logps/ref_rejected': -77.73824310302734, 'logits/chosen': -1.3876515626907349, 'logits/rejected': -1.1129705905914307, 'epoch': 0.14} 14%|███████████████▍ | 92/681 [04:31<28:57, 2.95s/it] 14%|███████████████▌ | 93/681 [04:33<27:21, 2.79s/it] {'loss': 1.1517, 'grad_norm': 27.450332641601562, 'learning_rate': 4.982595640958425e-07, 'margin_dpo/margin_mean': 4.376245498657227, 'margin_dpo/margin_std': 6.5292510986328125, 'logps/chosen': -94.47523498535156, 'logps/rejected': -108.31857299804688, 'logps/ref_chosen': -90.42925262451172, 'logps/ref_rejected': -99.89634704589844, 'logits/chosen': -1.3145904541015625, 'logits/rejected': -0.6349297761917114, 'epoch': 0.14} 14%|███████████████▌ | 93/681 [04:33<27:21, 2.79s/it] 14%|███████████████▋ | 94/681 [04:36<28:12, 2.88s/it] {'loss': 1.0268, 'grad_norm': 27.18844223022461, 'learning_rate': 4.98105127417984e-07, 'margin_dpo/margin_mean': 2.7289280891418457, 'margin_dpo/margin_std': 6.002341270446777, 'logps/chosen': -87.33253479003906, 'logps/rejected': -68.11126708984375, 'logps/ref_chosen': -84.00141906738281, 'logps/ref_rejected': -62.051231384277344, 'logits/chosen': -0.6020889282226562, 'logits/rejected': -0.1479286551475525, 'epoch': 0.14} 14%|███████████████▋ | 94/681 [04:36<28:12, 2.88s/it] 14%|███████████████▉ | 95/681 [04:39<27:51, 2.85s/it] {'loss': 1.0895, 'grad_norm': 22.77342987060547, 'learning_rate': 4.979441529392784e-07, 'margin_dpo/margin_mean': 6.590925693511963, 'margin_dpo/margin_std': 7.665090560913086, 'logps/chosen': -85.50116729736328, 'logps/rejected': -120.98870849609375, 'logps/ref_chosen': -83.24620056152344, 'logps/ref_rejected': -112.14282989501953, 'logits/chosen': -1.6398792266845703, 'logits/rejected': -0.8445466160774231, 'epoch': 0.14} 14%|███████████████▉ | 95/681 [04:39<27:51, 2.85s/it] 14%|████████████████ | 96/681 [04:42<27:52, 2.86s/it] {'loss': 1.0107, 'grad_norm': 23.296894073486328, 'learning_rate': 4.977766449015534e-07, 'margin_dpo/margin_mean': 6.817150115966797, 'margin_dpo/margin_std': 9.70801067352295, 'logps/chosen': -123.06785583496094, 'logps/rejected': -115.217041015625, 'logps/ref_chosen': -120.4042739868164, 'logps/ref_rejected': -105.736328125, 'logits/chosen': -0.34511518478393555, 'logits/rejected': -0.08061528205871582, 'epoch': 0.14} 14%|████████████████ | 96/681 [04:42<27:52, 2.86s/it] 14%|████████████████▏ | 97/681 [04:45<28:08, 2.89s/it] {'loss': 1.0255, 'grad_norm': 27.008302688598633, 'learning_rate': 4.976026077188012e-07, 'margin_dpo/margin_mean': 6.44036865234375, 'margin_dpo/margin_std': 6.640345096588135, 'logps/chosen': -92.0294189453125, 'logps/rejected': -81.84813690185547, 'logps/ref_chosen': -89.40409851074219, 'logps/ref_rejected': -72.7824478149414, 'logits/chosen': -0.5496343374252319, 'logits/rejected': 0.04007942974567413, 'epoch': 0.14} 14%|████████████████▏ | 97/681 [04:45<28:08, 2.89s/it] 14%|████████████████▍ | 98/681 [04:48<27:40, 2.85s/it] {'loss': 0.9763, 'grad_norm': 24.74591636657715, 'learning_rate': 4.974220459770639e-07, 'margin_dpo/margin_mean': 7.034348964691162, 'margin_dpo/margin_std': 5.715366363525391, 'logps/chosen': -79.35165405273438, 'logps/rejected': -111.64566802978516, 'logps/ref_chosen': -77.14939880371094, 'logps/ref_rejected': -102.4090576171875, 'logits/chosen': -0.39552319049835205, 'logits/rejected': -0.35785946249961853, 'epoch': 0.14} 14%|████████████████▍ | 98/681 [04:48<27:40, 2.85s/it] 15%|████████████████▌ | 99/681 [04:50<27:04, 2.79s/it] {'loss': 0.9865, 'grad_norm': 23.71422004699707, 'learning_rate': 4.972349644343108e-07, 'margin_dpo/margin_mean': 5.754771709442139, 'margin_dpo/margin_std': 6.445099353790283, 'logps/chosen': -69.09884643554688, 'logps/rejected': -87.75712585449219, 'logps/ref_chosen': -67.58863067626953, 'logps/ref_rejected': -80.49214172363281, 'logits/chosen': -0.6945886611938477, 'logits/rejected': -0.29277852177619934, 'epoch': 0.15} 15%|████████████████▌ | 99/681 [04:50<27:04, 2.79s/it] 15%|████████████████▌ | 100/681 [04:54<27:59, 2.89s/it] {'loss': 1.0426, 'grad_norm': 22.417694091796875, 'learning_rate': 4.970413680203148e-07, 'margin_dpo/margin_mean': 2.903665065765381, 'margin_dpo/margin_std': 4.343265056610107, 'logps/chosen': -73.37350463867188, 'logps/rejected': -83.2952880859375, 'logps/ref_chosen': -70.40953826904297, 'logps/ref_rejected': -77.42765808105469, 'logits/chosen': -1.128528356552124, 'logits/rejected': -0.9991703033447266, 'epoch': 0.15} 15%|████████████████▌ | 100/681 [04:54<27:59, 2.89s/it][INFO|trainer.py:4307] 2026-04-24 00:00:01,105 >> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-24 00:00:01,105 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-24 00:00:01,105 >> Batch size = 8 0%| | 0/73 [00:00> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-24 00:05:34,892 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-24 00:05:34,892 >> Batch size = 8 0%| | 0/73 [00:00> Saving model checkpoint to /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-200 [INFO|configuration_utils.py:419] 2026-04-24 00:06:32,932 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-200/config.json [INFO|configuration_utils.py:911] 2026-04-24 00:06:32,936 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-200/generation_config.json [INFO|modeling_utils.py:3580] 2026-04-24 00:07:12,323 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-200/model.safetensors.index.json. [INFO|tokenization_utils_base.py:2510] 2026-04-24 00:07:12,326 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-200/tokenizer_config.json [INFO|tokenization_utils_base.py:2519] 2026-04-24 00:07:12,330 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-200/special_tokens_map.json 30%|████████████████████████████████▍ | 201/681 [15:18<11:53:20, 89.17s/it] {'loss': 0.8579, 'grad_norm': 31.166837692260742, 'learning_rate': 4.455721242469372e-07, 'margin_dpo/margin_mean': 8.6412353515625, 'margin_dpo/margin_std': 15.760478973388672, 'logps/chosen': -132.33038330078125, 'logps/rejected': -152.9993438720703, 'logps/ref_chosen': -125.86758422851562, 'logps/ref_rejected': -137.8953094482422, 'logits/chosen': -1.3415249586105347, 'logits/rejected': -0.7633596658706665, 'epoch': 0.3} 30%|████████████████████████████████▍ | 201/681 [15:18<11:53:20, 89.17s/it] 30%|████████████████████████████████▉ | 202/681 [15:21<8:26:07, 63.40s/it] {'loss': 0.8075, 'grad_norm': 25.604938507080078, 'learning_rate': 4.4477014363141755e-07, 'margin_dpo/margin_mean': 15.89489459991455, 'margin_dpo/margin_std': 15.220779418945312, 'logps/chosen': -73.95713806152344, 'logps/rejected': -104.02867126464844, 'logps/ref_chosen': -62.47052001953125, 'logps/ref_rejected': -76.64714813232422, 'logits/chosen': -1.2493139505386353, 'logits/rejected': -1.0067428350448608, 'epoch': 0.3} 30%|████████████████████████████████▉ | 202/681 [15:21<8:26:07, 63.40s/it] 30%|█████████████████████████████████ | 203/681 [15:24<6:01:44, 45.41s/it] {'loss': 0.6844, 'grad_norm': 22.339271545410156, 'learning_rate': 4.439630306414758e-07, 'margin_dpo/margin_mean': 12.887399673461914, 'margin_dpo/margin_std': 15.519386291503906, 'logps/chosen': -92.4307861328125, 'logps/rejected': -116.5356674194336, 'logps/ref_chosen': -85.48118591308594, 'logps/ref_rejected': -96.69866943359375, 'logits/chosen': -1.5672788619995117, 'logits/rejected': -1.3719830513000488, 'epoch': 0.3} 30%|█████████████████████████████████ | 203/681 [15:24<6:01:44, 45.41s/it] 30%|█████████████████████████████████▎ | 204/681 [15:28<4:20:39, 32.79s/it] {'loss': 0.8576, 'grad_norm': 34.63078308105469, 'learning_rate': 4.431508065452897e-07, 'margin_dpo/margin_mean': 9.907830238342285, 'margin_dpo/margin_std': 11.625691413879395, 'logps/chosen': -99.44182586669922, 'logps/rejected': -111.54623413085938, 'logps/ref_chosen': -85.38599395751953, 'logps/ref_rejected': -87.58256530761719, 'logits/chosen': -1.294830083847046, 'logits/rejected': -0.8463597893714905, 'epoch': 0.3} 30%|█████████████████████████████████▎ | 204/681 [15:28<4:20:39, 32.79s/it] 30%|█████████████████████████████████▍ | 205/681 [15:31<3:09:25, 23.88s/it] {'loss': 0.5644, 'grad_norm': 19.02329444885254, 'learning_rate': 4.4233349274571974e-07, 'margin_dpo/margin_mean': 16.39504623413086, 'margin_dpo/margin_std': 11.657992362976074, 'logps/chosen': -127.82294464111328, 'logps/rejected': -105.96640014648438, 'logps/ref_chosen': -121.29222869873047, 'logps/ref_rejected': -83.04063415527344, 'logits/chosen': -0.9210774898529053, 'logits/rejected': -0.40849876403808594, 'epoch': 0.3} 30%|█████████████████████████████████▍ | 205/681 [15:31<3:09:25, 23.88s/it] 30%|█████████████████████████████████▌ | 206/681 [15:33<2:18:33, 17.50s/it] {'loss': 0.6035, 'grad_norm': 27.465503692626953, 'learning_rate': 4.415111107797445e-07, 'margin_dpo/margin_mean': 12.331515312194824, 'margin_dpo/margin_std': 9.227275848388672, 'logps/chosen': -92.56300354003906, 'logps/rejected': -131.0432891845703, 'logps/ref_chosen': -81.04837799072266, 'logps/ref_rejected': -107.19715118408203, 'logits/chosen': -1.0976223945617676, 'logits/rejected': -0.7354283332824707, 'epoch': 0.3} 30%|█████████████████████████████████▌ | 206/681 [15:33<2:18:33, 17.50s/it] 30%|█████████████████████████████████▋ | 207/681 [15:36<1:43:50, 13.14s/it] {'loss': 0.7898, 'grad_norm': 39.439029693603516, 'learning_rate': 4.4068368231789365e-07, 'margin_dpo/margin_mean': 11.486899375915527, 'margin_dpo/margin_std': 15.101633071899414, 'logps/chosen': -117.16340637207031, 'logps/rejected': -104.43505859375, 'logps/ref_chosen': -102.81378173828125, 'logps/ref_rejected': -78.59852600097656, 'logits/chosen': -2.2387092113494873, 'logits/rejected': -1.860313892364502, 'epoch': 0.3} 30%|█████████████████████████████████▋ | 207/681 [15:36<1:43:50, 13.14s/it] 31%|█████████████████████████████████▉ | 208/681 [15:39<1:19:27, 10.08s/it] {'loss': 0.6505, 'grad_norm': 23.772550582885742, 'learning_rate': 4.398512291636768e-07, 'margin_dpo/margin_mean': 14.929088592529297, 'margin_dpo/margin_std': 9.954263687133789, 'logps/chosen': -79.19771575927734, 'logps/rejected': -103.45726776123047, 'logps/ref_chosen': -70.0069580078125, 'logps/ref_rejected': -79.3374252319336, 'logits/chosen': -1.4251892566680908, 'logits/rejected': -0.9521887302398682, 'epoch': 0.31} 31%|█████████████████████████████████▉ | 208/681 [15:39<1:19:27, 10.08s/it] 31%|██████████████████████████████████ | 209/681 [15:42<1:01:50, 7.86s/it] {'loss': 0.7762, 'grad_norm': 24.988706588745117, 'learning_rate': 4.3901377325300857e-07, 'margin_dpo/margin_mean': 21.42230224609375, 'margin_dpo/margin_std': 15.412055969238281, 'logps/chosen': -78.55195617675781, 'logps/rejected': -113.91144561767578, 'logps/ref_chosen': -74.00470733642578, 'logps/ref_rejected': -87.94189453125, 'logits/chosen': -1.2561299800872803, 'logits/rejected': -0.9694398641586304, 'epoch': 0.31} 31%|██████████████████████████████████ | 209/681 [15:42<1:01:50, 7.86s/it] 31%|██████████████████████████████████▊ | 210/681 [15:45<49:44, 6.34s/it] {'loss': 0.733, 'grad_norm': 24.69098472595215, 'learning_rate': 4.381713366536311e-07, 'margin_dpo/margin_mean': 13.81161117553711, 'margin_dpo/margin_std': 13.800355911254883, 'logps/chosen': -85.33075714111328, 'logps/rejected': -86.75428771972656, 'logps/ref_chosen': -76.31346130371094, 'logps/ref_rejected': -63.925392150878906, 'logits/chosen': -1.796557068824768, 'logits/rejected': -1.1550593376159668, 'epoch': 0.31} 31%|██████████████████████████████████▊ | 210/681 [15:45<49:44, 6.34s/it] 31%|███████████████████████████████████ | 211/681 [15:47<40:57, 5.23s/it] {'loss': 0.7325, 'grad_norm': 27.62305450439453, 'learning_rate': 4.373239415645323e-07, 'margin_dpo/margin_mean': 20.25276756286621, 'margin_dpo/margin_std': 15.092392921447754, 'logps/chosen': -150.07386779785156, 'logps/rejected': -136.897705078125, 'logps/ref_chosen': -140.85391235351562, 'logps/ref_rejected': -107.42497253417969, 'logits/chosen': -1.5198208093643188, 'logits/rejected': -0.88636314868927, 'epoch': 0.31} 31%|███████████████████████████████████ | 211/681 [15:47<40:57, 5.23s/it] 31%|███████████████████████████████████▏ | 212/681 [15:50<35:25, 4.53s/it] {'loss': 0.5226, 'grad_norm': 19.40737533569336, 'learning_rate': 4.3647161031536086e-07, 'margin_dpo/margin_mean': 17.555767059326172, 'margin_dpo/margin_std': 19.352325439453125, 'logps/chosen': -132.343017578125, 'logps/rejected': -132.68862915039062, 'logps/ref_chosen': -125.50788116455078, 'logps/ref_rejected': -108.29771423339844, 'logits/chosen': -1.8869035243988037, 'logits/rejected': -1.2525444030761719, 'epoch': 0.31} 31%|███████████████████████████████████▏ | 212/681 [15:50<35:25, 4.53s/it] 31%|███████████████████████████████████▎ | 213/681 [15:53<32:11, 4.13s/it] {'loss': 0.6238, 'grad_norm': 32.36695861816406, 'learning_rate': 4.3561436536583774e-07, 'margin_dpo/margin_mean': 17.656354904174805, 'margin_dpo/margin_std': 11.100046157836914, 'logps/chosen': -85.28340911865234, 'logps/rejected': -149.52574157714844, 'logps/ref_chosen': -75.74522399902344, 'logps/ref_rejected': -122.3311996459961, 'logits/chosen': -1.4449195861816406, 'logits/rejected': -1.2709022760391235, 'epoch': 0.31} 31%|███████████████████████████████████▎ | 213/681 [15:54<32:11, 4.13s/it] 31%|███████████████████████████████████▌ | 214/681 [15:56<28:34, 3.67s/it] {'loss': 0.8735, 'grad_norm': 27.299671173095703, 'learning_rate': 4.3475222930516473e-07, 'margin_dpo/margin_mean': 8.634056091308594, 'margin_dpo/margin_std': 14.054182052612305, 'logps/chosen': -73.31192779541016, 'logps/rejected': -68.72589874267578, 'logps/ref_chosen': -64.04418182373047, 'logps/ref_rejected': -50.824100494384766, 'logits/chosen': -1.442638874053955, 'logits/rejected': -1.1300638914108276, 'epoch': 0.31} 31%|███████████████████████████████████▌ | 214/681 [15:56<28:34, 3.67s/it] 32%|███████████████████████████████████▋ | 215/681 [15:59<27:20, 3.52s/it] {'loss': 0.5661, 'grad_norm': 21.764801025390625, 'learning_rate': 4.3388522485142885e-07, 'margin_dpo/margin_mean': 22.63648796081543, 'margin_dpo/margin_std': 13.089027404785156, 'logps/chosen': -80.82921600341797, 'logps/rejected': -125.51627349853516, 'logps/ref_chosen': -70.8616714477539, 'logps/ref_rejected': -92.91224670410156, 'logits/chosen': -1.2465263605117798, 'logits/rejected': -0.9289161562919617, 'epoch': 0.32} 32%|███████████████████████████████████▋ | 215/681 [15:59<27:20, 3.52s/it] 32%|███████████████████████████████████▊ | 216/681 [16:02<26:14, 3.39s/it] {'loss': 0.6578, 'grad_norm': 21.630605697631836, 'learning_rate': 4.330133748510036e-07, 'margin_dpo/margin_mean': 14.123327255249023, 'margin_dpo/margin_std': 9.03171443939209, 'logps/chosen': -97.20977020263672, 'logps/rejected': -124.43745422363281, 'logps/ref_chosen': -85.44164276123047, 'logps/ref_rejected': -98.54600524902344, 'logits/chosen': -1.3383195400238037, 'logits/rejected': -1.1601924896240234, 'epoch': 0.32} 32%|███████████████████████████████████▊ | 216/681 [16:02<26:14, 3.39s/it] 32%|████████████████████████████████████ | 217/681 [16:05<25:06, 3.25s/it] {'loss': 0.7858, 'grad_norm': 31.357202529907227, 'learning_rate': 4.3213670227794757e-07, 'margin_dpo/margin_mean': 18.660911560058594, 'margin_dpo/margin_std': 14.71579360961914, 'logps/chosen': -93.33474731445312, 'logps/rejected': -151.92092895507812, 'logps/ref_chosen': -83.99208068847656, 'logps/ref_rejected': -123.9173583984375, 'logits/chosen': -1.8061637878417969, 'logits/rejected': -1.6783373355865479, 'epoch': 0.32} 32%|████████████████████████████████████ | 217/681 [16:05<25:06, 3.25s/it] 32%|████████████████████████████████████▏ | 218/681 [16:08<24:25, 3.17s/it] {'loss': 0.7321, 'grad_norm': 32.019493103027344, 'learning_rate': 4.3125523023339815e-07, 'margin_dpo/margin_mean': 17.53311538696289, 'margin_dpo/margin_std': 12.562833786010742, 'logps/chosen': -80.2852783203125, 'logps/rejected': -115.44291687011719, 'logps/ref_chosen': -67.8966293334961, 'logps/ref_rejected': -85.52115631103516, 'logits/chosen': -0.8639114499092102, 'logits/rejected': -0.6436069011688232, 'epoch': 0.32} 32%|████████████████████████████████████▏ | 218/681 [16:08<24:25, 3.17s/it] 32%|████████████████████████████████████▎ | 219/681 [16:11<24:03, 3.13s/it] {'loss': 0.7715, 'grad_norm': 45.11946105957031, 'learning_rate': 4.303689819449636e-07, 'margin_dpo/margin_mean': 19.358152389526367, 'margin_dpo/margin_std': 19.34707260131836, 'logps/chosen': -111.53719329833984, 'logps/rejected': -167.79376220703125, 'logps/ref_chosen': -97.2021484375, 'logps/ref_rejected': -134.10055541992188, 'logits/chosen': -1.2486598491668701, 'logits/rejected': -0.9495463371276855, 'epoch': 0.32} 32%|████████████████████████████████████▎ | 219/681 [16:11<24:03, 3.13s/it] 32%|████████████████████████████████████▌ | 220/681 [16:14<23:45, 3.09s/it] {'loss': 0.7639, 'grad_norm': 28.063581466674805, 'learning_rate': 4.2947798076611047e-07, 'margin_dpo/margin_mean': 13.408201217651367, 'margin_dpo/margin_std': 13.679563522338867, 'logps/chosen': -86.5882339477539, 'logps/rejected': -117.34354400634766, 'logps/ref_chosen': -74.71533966064453, 'logps/ref_rejected': -92.06246185302734, 'logits/chosen': -1.6841022968292236, 'logits/rejected': -1.6491317749023438, 'epoch': 0.32} 32%|████████████████████████████████████▌ | 220/681 [16:14<23:45, 3.09s/it] 32%|████████████████████████████████████▋ | 221/681 [16:17<23:41, 3.09s/it] {'loss': 0.5435, 'grad_norm': 22.508024215698242, 'learning_rate': 4.285822501755485e-07, 'margin_dpo/margin_mean': 18.743091583251953, 'margin_dpo/margin_std': 17.367952346801758, 'logps/chosen': -69.58158111572266, 'logps/rejected': -140.26364135742188, 'logps/ref_chosen': -59.28809356689453, 'logps/ref_rejected': -111.22705841064453, 'logits/chosen': -0.7385995388031006, 'logits/rejected': -0.8880366683006287, 'epoch': 0.32} 32%|████████████████████████████████████▋ | 221/681 [16:17<23:41, 3.09s/it] 33%|████████████████████████████████████▊ | 222/681 [16:20<23:37, 3.09s/it] {'loss': 0.7982, 'grad_norm': 48.555931091308594, 'learning_rate': 4.276818137766118e-07, 'margin_dpo/margin_mean': 17.54104995727539, 'margin_dpo/margin_std': 18.724212646484375, 'logps/chosen': -102.54623413085938, 'logps/rejected': -132.53857421875, 'logps/ref_chosen': -90.29017639160156, 'logps/ref_rejected': -102.741455078125, 'logits/chosen': -1.3711744546890259, 'logits/rejected': -0.945267915725708, 'epoch': 0.33} 33%|████████████████████████████████████▊ | 222/681 [16:20<23:37, 3.09s/it] 33%|█████████████████████████████████████ | 223/681 [16:23<22:21, 2.93s/it] {'loss': 0.8083, 'grad_norm': 25.106897354125977, 'learning_rate': 4.2677669529663686e-07, 'margin_dpo/margin_mean': 7.954143524169922, 'margin_dpo/margin_std': 10.7671480178833, 'logps/chosen': -96.40541076660156, 'logps/rejected': -119.77986145019531, 'logps/ref_chosen': -80.65748596191406, 'logps/ref_rejected': -96.07778930664062, 'logits/chosen': -1.2980246543884277, 'logits/rejected': -1.0237699747085571, 'epoch': 0.33} 33%|█████████████████████████████████████ | 223/681 [16:23<22:21, 2.93s/it] 33%|█████████████████████████████████████▏ | 224/681 [16:26<21:37, 2.84s/it] {'loss': 0.6666, 'grad_norm': 27.507966995239258, 'learning_rate': 4.2586691858633747e-07, 'margin_dpo/margin_mean': 14.736160278320312, 'margin_dpo/margin_std': 14.218803405761719, 'logps/chosen': -102.76864624023438, 'logps/rejected': -105.58963012695312, 'logps/ref_chosen': -91.31893920898438, 'logps/ref_rejected': -79.40377044677734, 'logits/chosen': -1.3987040519714355, 'logits/rejected': -0.79180908203125, 'epoch': 0.33} 33%|█████████████████████████████████████▏ | 224/681 [16:26<21:37, 2.84s/it] 33%|█████████████████████████████████████▎ | 225/681 [16:28<21:29, 2.83s/it] {'loss': 0.6625, 'grad_norm': 25.836376190185547, 'learning_rate': 4.249525076191759e-07, 'margin_dpo/margin_mean': 20.47254180908203, 'margin_dpo/margin_std': 17.923603057861328, 'logps/chosen': -123.97574615478516, 'logps/rejected': -142.740966796875, 'logps/ref_chosen': -113.84062194824219, 'logps/ref_rejected': -112.13330078125, 'logits/chosen': -2.027069091796875, 'logits/rejected': -1.221968650817871, 'epoch': 0.33} 33%|█████████████████████████████████████▎ | 225/681 [16:28<21:29, 2.83s/it] 33%|█████████████████████████████████████▌ | 226/681 [16:32<22:19, 2.94s/it] {'loss': 0.6747, 'grad_norm': 24.863615036010742, 'learning_rate': 4.2403348649073167e-07, 'margin_dpo/margin_mean': 15.122581481933594, 'margin_dpo/margin_std': 14.921210289001465, 'logps/chosen': -111.22610473632812, 'logps/rejected': -71.60173797607422, 'logps/ref_chosen': -99.33114624023438, 'logps/ref_rejected': -44.584197998046875, 'logits/chosen': -1.66581130027771, 'logits/rejected': -0.9718565344810486, 'epoch': 0.33} 33%|█████████████████████████████████████▌ | 226/681 [16:32<22:19, 2.94s/it] 33%|█████████████████████████████████████▋ | 227/681 [16:34<21:52, 2.89s/it] {'loss': 0.7772, 'grad_norm': 31.727184295654297, 'learning_rate': 4.2310987941806615e-07, 'margin_dpo/margin_mean': 14.969067573547363, 'margin_dpo/margin_std': 11.369447708129883, 'logps/chosen': -92.5580062866211, 'logps/rejected': -127.857177734375, 'logps/ref_chosen': -78.06465148925781, 'logps/ref_rejected': -98.39476013183594, 'logits/chosen': -0.6366788148880005, 'logits/rejected': -0.558417558670044, 'epoch': 0.33} 33%|█████████████████████████████████████▋ | 227/681 [16:34<21:52, 2.89s/it] 33%|█████████████████████████████████████▊ | 228/681 [16:38<22:27, 2.98s/it] {'loss': 0.7634, 'grad_norm': 24.629833221435547, 'learning_rate': 4.2218171073908463e-07, 'margin_dpo/margin_mean': 12.804656982421875, 'margin_dpo/margin_std': 13.355108261108398, 'logps/chosen': -133.501220703125, 'logps/rejected': -177.65139770507812, 'logps/ref_chosen': -118.2425537109375, 'logps/ref_rejected': -149.5880889892578, 'logits/chosen': -1.5679694414138794, 'logits/rejected': -1.2253010272979736, 'epoch': 0.33} 33%|█████████████████████████████████████▊ | 228/681 [16:38<22:27, 2.98s/it] 34%|█████████████████████████████████████▉ | 229/681 [16:41<22:31, 2.99s/it] {'loss': 0.5845, 'grad_norm': 22.838153839111328, 'learning_rate': 4.212490049118951e-07, 'margin_dpo/margin_mean': 26.82710075378418, 'margin_dpo/margin_std': 14.36768913269043, 'logps/chosen': -93.84185791015625, 'logps/rejected': -119.24227142333984, 'logps/ref_chosen': -86.05354309082031, 'logps/ref_rejected': -84.62686157226562, 'logits/chosen': -1.934976577758789, 'logits/rejected': -1.0155363082885742, 'epoch': 0.34} 34%|█████████████████████████████████████▉ | 229/681 [16:41<22:31, 2.99s/it] 34%|██████████████████████████████████████▏ | 230/681 [16:44<22:28, 2.99s/it] {'loss': 0.5216, 'grad_norm': 22.581356048583984, 'learning_rate': 4.203117865141635e-07, 'margin_dpo/margin_mean': 18.46939468383789, 'margin_dpo/margin_std': 15.664244651794434, 'logps/chosen': -82.906005859375, 'logps/rejected': -126.56108856201172, 'logps/ref_chosen': -72.93336486816406, 'logps/ref_rejected': -98.11904907226562, 'logits/chosen': -2.0330729484558105, 'logits/rejected': -1.621316909790039, 'epoch': 0.34} 34%|██████████████████████████████████████▏ | 230/681 [16:44<22:28, 2.99s/it] 34%|██████████████████████████████████████▎ | 231/681 [16:47<22:32, 3.01s/it] {'loss': 0.6761, 'grad_norm': 22.018823623657227, 'learning_rate': 4.1937008024246625e-07, 'margin_dpo/margin_mean': 20.100221633911133, 'margin_dpo/margin_std': 17.13475799560547, 'logps/chosen': -111.63320922851562, 'logps/rejected': -123.36331176757812, 'logps/ref_chosen': -100.19026947021484, 'logps/ref_rejected': -91.82012939453125, 'logits/chosen': -2.015583038330078, 'logits/rejected': -1.279665470123291, 'epoch': 0.34} 34%|██████████████████████████████████████▎ | 231/681 [16:47<22:32, 3.01s/it] 34%|██████████████████████████████████████▍ | 232/681 [16:50<22:40, 3.03s/it] {'loss': 0.7972, 'grad_norm': 28.21176528930664, 'learning_rate': 4.1842391091163933e-07, 'margin_dpo/margin_mean': 11.920722007751465, 'margin_dpo/margin_std': 17.419721603393555, 'logps/chosen': -122.58344268798828, 'logps/rejected': -104.9071044921875, 'logps/ref_chosen': -110.88046264648438, 'logps/ref_rejected': -81.28340148925781, 'logits/chosen': -0.8941325545310974, 'logits/rejected': -0.42558372020721436, 'epoch': 0.34} 34%|██████████████████████████████████████▍ | 232/681 [16:50<22:40, 3.03s/it] 34%|██████████████████████████████████████▋ | 233/681 [16:53<23:12, 3.11s/it] {'loss': 0.6343, 'grad_norm': 25.46039390563965, 'learning_rate': 4.174733034541245e-07, 'margin_dpo/margin_mean': 15.176610946655273, 'margin_dpo/margin_std': 17.110292434692383, 'logps/chosen': -106.99727630615234, 'logps/rejected': -140.77781677246094, 'logps/ref_chosen': -89.96463775634766, 'logps/ref_rejected': -108.56855773925781, 'logits/chosen': -0.8610257506370544, 'logits/rejected': -0.8550558090209961, 'epoch': 0.34} 34%|██████████████████████████████████████▋ | 233/681 [16:53<23:12, 3.11s/it] 34%|██████████████████████████████████████▊ | 234/681 [16:56<23:17, 3.13s/it] {'loss': 0.6698, 'grad_norm': 25.88790512084961, 'learning_rate': 4.165182829193126e-07, 'margin_dpo/margin_mean': 18.164871215820312, 'margin_dpo/margin_std': 13.520463943481445, 'logps/chosen': -85.27423858642578, 'logps/rejected': -149.45150756835938, 'logps/ref_chosen': -73.45465850830078, 'logps/ref_rejected': -119.467041015625, 'logits/chosen': -1.1169289350509644, 'logits/rejected': -0.8724891543388367, 'epoch': 0.34} 34%|██████████████████████████████████████▊ | 234/681 [16:56<23:17, 3.13s/it] 35%|██████████████████████████████████████▉ | 235/681 [16:59<22:41, 3.05s/it] {'loss': 0.6736, 'grad_norm': 23.897993087768555, 'learning_rate': 4.1555887447288255e-07, 'margin_dpo/margin_mean': 13.944540977478027, 'margin_dpo/margin_std': 15.060174942016602, 'logps/chosen': -116.7324447631836, 'logps/rejected': -128.42771911621094, 'logps/ref_chosen': -102.08454132080078, 'logps/ref_rejected': -99.83527374267578, 'logits/chosen': -0.2996028959751129, 'logits/rejected': 0.14718373119831085, 'epoch': 0.35} 35%|██████████████████████████████████████▉ | 235/681 [16:59<22:41, 3.05s/it] 35%|███████████████████████████████████████▏ | 236/681 [17:02<23:15, 3.14s/it] {'loss': 0.6689, 'grad_norm': 22.317142486572266, 'learning_rate': 4.1459510339613946e-07, 'margin_dpo/margin_mean': 18.615659713745117, 'margin_dpo/margin_std': 21.68730926513672, 'logps/chosen': -93.7081298828125, 'logps/rejected': -181.28709411621094, 'logps/ref_chosen': -80.65806579589844, 'logps/ref_rejected': -149.62136840820312, 'logits/chosen': -1.0241249799728394, 'logits/rejected': -1.0809072256088257, 'epoch': 0.35} 35%|███████████████████████████████████████▏ | 236/681 [17:02<23:15, 3.14s/it] 35%|███████████████████████████████████████▎ | 237/681 [17:05<23:02, 3.11s/it] {'loss': 0.8527, 'grad_norm': 28.688703536987305, 'learning_rate': 4.136269950853473e-07, 'margin_dpo/margin_mean': 18.312477111816406, 'margin_dpo/margin_std': 15.394723892211914, 'logps/chosen': -90.6567611694336, 'logps/rejected': -106.99150848388672, 'logps/ref_chosen': -80.41928100585938, 'logps/ref_rejected': -78.44154357910156, 'logits/chosen': -0.570310115814209, 'logits/rejected': -0.23448389768600464, 'epoch': 0.35} 35%|███████████████████████████████████████▎ | 237/681 [17:05<23:02, 3.11s/it] 35%|███████████████████████████████████████▍ | 238/681 [17:09<23:24, 3.17s/it] {'loss': 0.7124, 'grad_norm': 23.128028869628906, 'learning_rate': 4.126545750510605e-07, 'margin_dpo/margin_mean': 15.296379089355469, 'margin_dpo/margin_std': 14.916626930236816, 'logps/chosen': -77.17804718017578, 'logps/rejected': -129.80294799804688, 'logps/ref_chosen': -60.57853698730469, 'logps/ref_rejected': -97.90704345703125, 'logits/chosen': -1.2925560474395752, 'logits/rejected': -1.2870701551437378, 'epoch': 0.35} 35%|███████████████████████████████████████▍ | 238/681 [17:09<23:24, 3.17s/it] 35%|███████████████████████████████████████▋ | 239/681 [17:11<22:24, 3.04s/it] {'loss': 0.7431, 'grad_norm': 27.944942474365234, 'learning_rate': 4.116778689174514e-07, 'margin_dpo/margin_mean': 18.140701293945312, 'margin_dpo/margin_std': 14.662839889526367, 'logps/chosen': -107.7053451538086, 'logps/rejected': -142.38980102539062, 'logps/ref_chosen': -97.148681640625, 'logps/ref_rejected': -113.69243621826172, 'logits/chosen': -1.2913211584091187, 'logits/rejected': -0.8684008121490479, 'epoch': 0.35} 35%|███████████████████████████████████████▋ | 239/681 [17:12<22:24, 3.04s/it] 35%|███████████████████████████████████████▊ | 240/681 [17:15<22:22, 3.04s/it] {'loss': 0.7521, 'grad_norm': 33.86201858520508, 'learning_rate': 4.106969024216348e-07, 'margin_dpo/margin_mean': 11.845929145812988, 'margin_dpo/margin_std': 12.4403076171875, 'logps/chosen': -87.25590515136719, 'logps/rejected': -111.20504760742188, 'logps/ref_chosen': -69.32797241210938, 'logps/ref_rejected': -81.43118286132812, 'logits/chosen': -1.348999261856079, 'logits/rejected': -1.2273541688919067, 'epoch': 0.35} 35%|███████████████████████████████████████▊ | 240/681 [17:15<22:22, 3.04s/it] 35%|███████████████████████████████████████▉ | 241/681 [17:17<21:55, 2.99s/it] {'loss': 0.4697, 'grad_norm': 22.594024658203125, 'learning_rate': 4.097117014129903e-07, 'margin_dpo/margin_mean': 24.22762107849121, 'margin_dpo/margin_std': 16.218782424926758, 'logps/chosen': -92.8592529296875, 'logps/rejected': -107.22079467773438, 'logps/ref_chosen': -82.93885803222656, 'logps/ref_rejected': -73.07276916503906, 'logits/chosen': -1.7714438438415527, 'logits/rejected': -0.992931604385376, 'epoch': 0.35} 35%|███████████████████████████████████████▉ | 241/681 [17:17<21:55, 2.99s/it] 36%|████████████████████████████████████████▏ | 242/681 [17:20<21:26, 2.93s/it] {'loss': 0.6153, 'grad_norm': 26.068859100341797, 'learning_rate': 4.087222918524807e-07, 'margin_dpo/margin_mean': 16.589599609375, 'margin_dpo/margin_std': 15.04800796508789, 'logps/chosen': -113.93301391601562, 'logps/rejected': -116.67051696777344, 'logps/ref_chosen': -98.63720703125, 'logps/ref_rejected': -84.78510284423828, 'logits/chosen': -1.198895812034607, 'logits/rejected': -0.9665822982788086, 'epoch': 0.36} 36%|████████████████████████████████████████▏ | 242/681 [17:20<21:26, 2.93s/it] 36%|████████████████████████████████████████▎ | 243/681 [17:23<21:16, 2.91s/it] {'loss': 0.7338, 'grad_norm': 27.26268768310547, 'learning_rate': 4.07728699811968e-07, 'margin_dpo/margin_mean': 17.503009796142578, 'margin_dpo/margin_std': 15.984872817993164, 'logps/chosen': -118.21045684814453, 'logps/rejected': -112.95948791503906, 'logps/ref_chosen': -104.37299346923828, 'logps/ref_rejected': -81.6190185546875, 'logits/chosen': -1.2558754682540894, 'logits/rejected': -0.5668581128120422, 'epoch': 0.36} 36%|████████████████████████████████████████▎ | 243/681 [17:23<21:16, 2.91s/it] 36%|████████████████████████████████████████▍ | 244/681 [17:26<21:31, 2.96s/it] {'loss': 0.6778, 'grad_norm': 22.77792739868164, 'learning_rate': 4.067309514735267e-07, 'margin_dpo/margin_mean': 18.553510665893555, 'margin_dpo/margin_std': 11.52467155456543, 'logps/chosen': -103.87003326416016, 'logps/rejected': -156.72738647460938, 'logps/ref_chosen': -93.71182250976562, 'logps/ref_rejected': -128.01568603515625, 'logits/chosen': -2.0644044876098633, 'logits/rejected': -1.902282953262329, 'epoch': 0.36} 36%|████████████████████████████████████████▍ | 244/681 [17:26<21:31, 2.96s/it] 36%|████████████████████████████████████████▋ | 245/681 [17:29<21:39, 2.98s/it] {'loss': 0.5495, 'grad_norm': 22.12187385559082, 'learning_rate': 4.057290731287531e-07, 'margin_dpo/margin_mean': 22.41289710998535, 'margin_dpo/margin_std': 14.135283470153809, 'logps/chosen': -109.80570983886719, 'logps/rejected': -134.34341430664062, 'logps/ref_chosen': -101.46330261230469, 'logps/ref_rejected': -103.5881118774414, 'logits/chosen': -1.0430703163146973, 'logits/rejected': -0.8094485998153687, 'epoch': 0.36} 36%|████████████████████████████████████████▋ | 245/681 [17:29<21:39, 2.98s/it] 36%|████████████████████████████████████████▊ | 246/681 [17:32<21:54, 3.02s/it] {'loss': 0.7436, 'grad_norm': 32.46879959106445, 'learning_rate': 4.047230911780736e-07, 'margin_dpo/margin_mean': 22.255264282226562, 'margin_dpo/margin_std': 13.090051651000977, 'logps/chosen': -95.5453872680664, 'logps/rejected': -149.63845825195312, 'logps/ref_chosen': -79.95723724365234, 'logps/ref_rejected': -111.7950439453125, 'logits/chosen': -1.5383753776550293, 'logits/rejected': -1.2452689409255981, 'epoch': 0.36} 36%|████████████████████████████████████████▊ | 246/681 [17:32<21:54, 3.02s/it] 36%|████████████████████████████████████████▉ | 247/681 [17:35<21:25, 2.96s/it] {'loss': 0.5841, 'grad_norm': 20.374637603759766, 'learning_rate': 4.0371303213004814e-07, 'margin_dpo/margin_mean': 20.837257385253906, 'margin_dpo/margin_std': 12.746124267578125, 'logps/chosen': -100.3311996459961, 'logps/rejected': -140.61538696289062, 'logps/ref_chosen': -87.5809326171875, 'logps/ref_rejected': -107.0278549194336, 'logits/chosen': -2.6092662811279297, 'logits/rejected': -2.0955309867858887, 'epoch': 0.36} 36%|████████████████████████████████████████▉ | 247/681 [17:35<21:25, 2.96s/it] 36%|█████████████████████████████████████████▏ | 248/681 [17:38<21:15, 2.94s/it] {'loss': 0.7611, 'grad_norm': 31.063600540161133, 'learning_rate': 4.0269892260067197e-07, 'margin_dpo/margin_mean': 11.065750122070312, 'margin_dpo/margin_std': 18.710220336914062, 'logps/chosen': -100.98844909667969, 'logps/rejected': -142.08447265625, 'logps/ref_chosen': -83.37393188476562, 'logps/ref_rejected': -113.40422821044922, 'logits/chosen': -1.4017457962036133, 'logits/rejected': -1.265455961227417, 'epoch': 0.36} 36%|█████████████████████████████████████████▏ | 248/681 [17:38<21:15, 2.94s/it] 37%|█████████████████████████████████████████▎ | 249/681 [17:41<20:42, 2.88s/it] {'loss': 0.8498, 'grad_norm': 34.479984283447266, 'learning_rate': 4.0168078931267426e-07, 'margin_dpo/margin_mean': 7.238720893859863, 'margin_dpo/margin_std': 15.718103408813477, 'logps/chosen': -110.28913116455078, 'logps/rejected': -107.76251983642578, 'logps/ref_chosen': -90.19677734375, 'logps/ref_rejected': -80.43144226074219, 'logits/chosen': -1.5276246070861816, 'logits/rejected': -1.1095179319381714, 'epoch': 0.37} 37%|█████████████████████████████████████████▎ | 249/681 [17:41<20:42, 2.88s/it] 37%|█████████████████████████████████████████▍ | 250/681 [17:43<20:25, 2.84s/it] {'loss': 0.7792, 'grad_norm': 41.039459228515625, 'learning_rate': 4.006586590948141e-07, 'margin_dpo/margin_mean': 13.451122283935547, 'margin_dpo/margin_std': 14.689545631408691, 'logps/chosen': -154.2344207763672, 'logps/rejected': -106.68568420410156, 'logps/ref_chosen': -138.57211303710938, 'logps/ref_rejected': -77.57225799560547, 'logits/chosen': -1.7807164192199707, 'logits/rejected': -0.9707603454589844, 'epoch': 0.37} 37%|█████████████████████████████████████████▍ | 250/681 [17:44<20:25, 2.84s/it] 37%|█████████████████████████████████████████▋ | 251/681 [17:46<20:32, 2.87s/it] {'loss': 0.8432, 'grad_norm': 27.502151489257812, 'learning_rate': 3.9963255888117325e-07, 'margin_dpo/margin_mean': 16.946407318115234, 'margin_dpo/margin_std': 17.334667205810547, 'logps/chosen': -115.2924575805664, 'logps/rejected': -100.4027099609375, 'logps/ref_chosen': -104.47010803222656, 'logps/ref_rejected': -72.63395690917969, 'logits/chosen': -1.5643404722213745, 'logits/rejected': -0.5259659290313721, 'epoch': 0.37} 37%|█████████████████████████████████████████▋ | 251/681 [17:46<20:32, 2.87s/it] 37%|█████████████████████████████████████████▊ | 252/681 [17:49<20:33, 2.87s/it] {'loss': 0.7427, 'grad_norm': 26.42395782470703, 'learning_rate': 3.9860251571044666e-07, 'margin_dpo/margin_mean': 17.966344833374023, 'margin_dpo/margin_std': 16.002155303955078, 'logps/chosen': -110.46723937988281, 'logps/rejected': -118.4449234008789, 'logps/ref_chosen': -97.77006530761719, 'logps/ref_rejected': -87.78140258789062, 'logits/chosen': -1.0975241661071777, 'logits/rejected': -0.6452772617340088, 'epoch': 0.37} 37%|█████████████████████████████████████████▊ | 252/681 [17:49<20:33, 2.87s/it] 37%|█████████████████████████████████████████▉ | 253/681 [17:52<20:46, 2.91s/it] {'loss': 0.7583, 'grad_norm': 28.601953506469727, 'learning_rate': 3.9756855672522986e-07, 'margin_dpo/margin_mean': 16.69076919555664, 'margin_dpo/margin_std': 8.515735626220703, 'logps/chosen': -106.92183685302734, 'logps/rejected': -125.74440002441406, 'logps/ref_chosen': -89.15538024902344, 'logps/ref_rejected': -91.28717041015625, 'logits/chosen': -0.999688982963562, 'logits/rejected': -0.5330642461776733, 'epoch': 0.37} 37%|█████████████████████████████████████████▉ | 253/681 [17:52<20:46, 2.91s/it] 37%|██████████████████████████████████████████▏ | 254/681 [17:56<21:19, 3.00s/it] {'loss': 0.7741, 'grad_norm': 24.010168075561523, 'learning_rate': 3.965307091713037e-07, 'margin_dpo/margin_mean': 9.378756523132324, 'margin_dpo/margin_std': 13.781966209411621, 'logps/chosen': -120.03327941894531, 'logps/rejected': -143.052978515625, 'logps/ref_chosen': -104.02809143066406, 'logps/ref_rejected': -117.66902160644531, 'logits/chosen': -0.826606035232544, 'logits/rejected': -0.84417724609375, 'epoch': 0.37} 37%|██████████████████████████████████████████▏ | 254/681 [17:56<21:19, 3.00s/it] 37%|██████████████████████████████████████████▎ | 255/681 [17:58<20:51, 2.94s/it] {'loss': 0.66, 'grad_norm': 27.174829483032227, 'learning_rate': 3.954890003969163e-07, 'margin_dpo/margin_mean': 12.647738456726074, 'margin_dpo/margin_std': 11.482951164245605, 'logps/chosen': -160.41029357910156, 'logps/rejected': -173.4158935546875, 'logps/ref_chosen': -140.6939697265625, 'logps/ref_rejected': -141.0518341064453, 'logits/chosen': -1.2953753471374512, 'logits/rejected': -0.8916399478912354, 'epoch': 0.37} 37%|██████████████████████████████████████████▎ | 255/681 [17:58<20:51, 2.94s/it] 38%|██████████████████████████████████████████▍ | 256/681 [18:01<20:32, 2.90s/it] {'loss': 0.6885, 'grad_norm': 28.97931671142578, 'learning_rate': 3.944434578520628e-07, 'margin_dpo/margin_mean': 19.18560028076172, 'margin_dpo/margin_std': 17.2878360748291, 'logps/chosen': -94.7815170288086, 'logps/rejected': -141.99551391601562, 'logps/ref_chosen': -77.7818374633789, 'logps/ref_rejected': -105.81022644042969, 'logits/chosen': -1.5570547580718994, 'logits/rejected': -1.3356791734695435, 'epoch': 0.38} 38%|██████████████████████████████████████████▍ | 256/681 [18:01<20:32, 2.90s/it] 38%|██████████████████████████████████████████▋ | 257/681 [18:04<20:43, 2.93s/it] {'loss': 0.6644, 'grad_norm': 27.675180435180664, 'learning_rate': 3.933941090877615e-07, 'margin_dpo/margin_mean': 20.687824249267578, 'margin_dpo/margin_std': 18.404783248901367, 'logps/chosen': -120.56830596923828, 'logps/rejected': -140.97515869140625, 'logps/ref_chosen': -104.2494888305664, 'logps/ref_rejected': -103.968505859375, 'logits/chosen': -1.4617829322814941, 'logits/rejected': -0.874555230140686, 'epoch': 0.38} 38%|██████████████████████████████████████████▋ | 257/681 [18:04<20:43, 2.93s/it] 38%|██████████████████████████████████████████▊ | 258/681 [18:07<19:52, 2.82s/it] {'loss': 0.6151, 'grad_norm': 26.982419967651367, 'learning_rate': 3.923409817553284e-07, 'margin_dpo/margin_mean': 12.84976863861084, 'margin_dpo/margin_std': 11.585992813110352, 'logps/chosen': -111.3430404663086, 'logps/rejected': -121.9286880493164, 'logps/ref_chosen': -95.29330444335938, 'logps/ref_rejected': -93.0291748046875, 'logits/chosen': -1.7992323637008667, 'logits/rejected': -1.3448323011398315, 'epoch': 0.38} 38%|██████████████████████████████████████████▊ | 258/681 [18:07<19:52, 2.82s/it] 38%|██████████████████████████████████████████▉ | 259/681 [18:10<20:14, 2.88s/it] {'loss': 0.817, 'grad_norm': 28.79665756225586, 'learning_rate': 3.9128410360564793e-07, 'margin_dpo/margin_mean': 14.59865951538086, 'margin_dpo/margin_std': 21.104887008666992, 'logps/chosen': -117.34747314453125, 'logps/rejected': -114.84416198730469, 'logps/ref_chosen': -96.55043029785156, 'logps/ref_rejected': -79.44844818115234, 'logits/chosen': -0.6006170511245728, 'logits/rejected': 0.08246561884880066, 'epoch': 0.38} 38%|██████████████████████████████████████████▉ | 259/681 [18:10<20:14, 2.88s/it] 38%|███████████████████████████████████████████▏ | 260/681 [18:13<20:35, 2.94s/it] {'loss': 0.937, 'grad_norm': 31.739402770996094, 'learning_rate': 3.9022350248844246e-07, 'margin_dpo/margin_mean': 15.499946594238281, 'margin_dpo/margin_std': 15.068811416625977, 'logps/chosen': -89.928466796875, 'logps/rejected': -143.739013671875, 'logps/ref_chosen': -69.2818603515625, 'logps/ref_rejected': -107.59248352050781, 'logits/chosen': -0.8702751398086548, 'logits/rejected': -0.7750843167304993, 'epoch': 0.38} 38%|███████████████████████████████████████████▏ | 260/681 [18:13<20:35, 2.94s/it] 38%|███████████████████████████████████████████▎ | 261/681 [18:15<19:52, 2.84s/it] {'loss': 0.6509, 'grad_norm': 24.90912628173828, 'learning_rate': 3.891592063515376e-07, 'margin_dpo/margin_mean': 16.731483459472656, 'margin_dpo/margin_std': 16.480255126953125, 'logps/chosen': -109.2607421875, 'logps/rejected': -155.48861694335938, 'logps/ref_chosen': -93.40982055664062, 'logps/ref_rejected': -122.90621948242188, 'logits/chosen': -1.9554228782653809, 'logits/rejected': -1.618184208869934, 'epoch': 0.38} 38%|███████████████████████████████████████████▎ | 261/681 [18:15<19:52, 2.84s/it] 38%|███████████████████████████████████████████▍ | 262/681 [18:18<19:31, 2.80s/it] {'loss': 0.7006, 'grad_norm': 24.286285400390625, 'learning_rate': 3.880912432401264e-07, 'margin_dpo/margin_mean': 10.552679061889648, 'margin_dpo/margin_std': 14.37498664855957, 'logps/chosen': -110.82711791992188, 'logps/rejected': -155.30270385742188, 'logps/ref_chosen': -94.37785339355469, 'logps/ref_rejected': -128.30076599121094, 'logits/chosen': -1.1788181066513062, 'logits/rejected': -0.6409732699394226, 'epoch': 0.38} 38%|███████████████████████████████████████████▍ | 262/681 [18:18<19:31, 2.80s/it] 39%|███████████████████████████████████████████▋ | 263/681 [18:21<19:29, 2.80s/it] {'loss': 0.5653, 'grad_norm': 26.123411178588867, 'learning_rate': 3.870196412960302e-07, 'margin_dpo/margin_mean': 21.03856658935547, 'margin_dpo/margin_std': 13.661331176757812, 'logps/chosen': -125.52157592773438, 'logps/rejected': -153.01513671875, 'logps/ref_chosen': -112.25532531738281, 'logps/ref_rejected': -118.7103042602539, 'logits/chosen': -2.209285259246826, 'logits/rejected': -1.72807776927948, 'epoch': 0.39} 39%|███████████████████████████████████████████▋ | 263/681 [18:21<19:29, 2.80s/it] 39%|███████████████████████████████████████████▊ | 264/681 [18:24<20:00, 2.88s/it] {'loss': 0.6855, 'grad_norm': 26.471071243286133, 'learning_rate': 3.8594442875695665e-07, 'margin_dpo/margin_mean': 18.396671295166016, 'margin_dpo/margin_std': 14.122834205627441, 'logps/chosen': -88.62034606933594, 'logps/rejected': -115.3791732788086, 'logps/ref_chosen': -74.47600555419922, 'logps/ref_rejected': -82.83815002441406, 'logits/chosen': -2.094125747680664, 'logits/rejected': -1.5296189785003662, 'epoch': 0.39} 39%|███████████████████████████████████████████▊ | 264/681 [18:24<20:00, 2.88s/it] 39%|███████████████████████████████████████████▉ | 265/681 [18:27<19:39, 2.84s/it] {'loss': 0.7431, 'grad_norm': 24.215213775634766, 'learning_rate': 3.848656339557562e-07, 'margin_dpo/margin_mean': 17.0369873046875, 'margin_dpo/margin_std': 18.63312530517578, 'logps/chosen': -117.15290069580078, 'logps/rejected': -115.11080932617188, 'logps/ref_chosen': -104.01231384277344, 'logps/ref_rejected': -84.9332275390625, 'logits/chosen': -1.6657154560089111, 'logits/rejected': -0.7729620933532715, 'epoch': 0.39} 39%|███████████████████████████████████████████▉ | 265/681 [18:27<19:39, 2.84s/it] 39%|████████████████████████████████████████████▏ | 266/681 [18:30<19:51, 2.87s/it] {'loss': 0.7461, 'grad_norm': 33.143089294433594, 'learning_rate': 3.8378328531967507e-07, 'margin_dpo/margin_mean': 17.004764556884766, 'margin_dpo/margin_std': 11.14462661743164, 'logps/chosen': -141.586669921875, 'logps/rejected': -94.93106079101562, 'logps/ref_chosen': -127.02003479003906, 'logps/ref_rejected': -63.359657287597656, 'logits/chosen': -1.3864130973815918, 'logits/rejected': -0.2755950391292572, 'epoch': 0.39} 39%|████████████████████████████████████████████▏ | 266/681 [18:30<19:51, 2.87s/it] 39%|████████████████████████████████████████████▎ | 267/681 [18:33<19:53, 2.88s/it] {'loss': 0.5557, 'grad_norm': 23.171924591064453, 'learning_rate': 3.8269741136960646e-07, 'margin_dpo/margin_mean': 15.561500549316406, 'margin_dpo/margin_std': 13.366697311401367, 'logps/chosen': -113.98542785644531, 'logps/rejected': -115.53654479980469, 'logps/ref_chosen': -100.46978759765625, 'logps/ref_rejected': -86.45939636230469, 'logits/chosen': -1.4743266105651855, 'logits/rejected': -0.8582184910774231, 'epoch': 0.39} 39%|████████████████████████████████████████████▎ | 267/681 [18:33<19:53, 2.88s/it] 39%|████████████████████████████████████████████▍ | 268/681 [18:35<19:52, 2.89s/it] {'loss': 0.6504, 'grad_norm': 23.90070152282715, 'learning_rate': 3.8160804071933894e-07, 'margin_dpo/margin_mean': 11.992761611938477, 'margin_dpo/margin_std': 12.630704879760742, 'logps/chosen': -104.34091186523438, 'logps/rejected': -104.53038787841797, 'logps/ref_chosen': -87.36892700195312, 'logps/ref_rejected': -75.56564331054688, 'logits/chosen': -0.6755940914154053, 'logits/rejected': -0.30946046113967896, 'epoch': 0.39} 39%|████████████████████████████████████████████▍ | 268/681 [18:35<19:52, 2.89s/it] 40%|████████████████████████████████████████████▋ | 269/681 [18:38<19:54, 2.90s/it] {'loss': 0.5895, 'grad_norm': 23.326845169067383, 'learning_rate': 3.8051520207480204e-07, 'margin_dpo/margin_mean': 20.590484619140625, 'margin_dpo/margin_std': 11.655923843383789, 'logps/chosen': -113.57240295410156, 'logps/rejected': -131.50343322753906, 'logps/ref_chosen': -105.39424133300781, 'logps/ref_rejected': -102.73477172851562, 'logits/chosen': -1.7525832653045654, 'logits/rejected': -1.138352632522583, 'epoch': 0.4} 40%|████████████████████████████████████████████▋ | 269/681 [18:38<19:54, 2.90s/it] 40%|████████████████████████████████████████████▊ | 270/681 [18:41<20:04, 2.93s/it] {'loss': 0.7485, 'grad_norm': 28.63389015197754, 'learning_rate': 3.794189242333106e-07, 'margin_dpo/margin_mean': 19.665857315063477, 'margin_dpo/margin_std': 18.711822509765625, 'logps/chosen': -135.16775512695312, 'logps/rejected': -145.209716796875, 'logps/ref_chosen': -117.72895812988281, 'logps/ref_rejected': -108.10507202148438, 'logits/chosen': -2.346863269805908, 'logits/rejected': -1.4268286228179932, 'epoch': 0.4} 40%|████████████████████████████████████████████▊ | 270/681 [18:41<20:04, 2.93s/it] 40%|████████████████████████████████████████████▉ | 271/681 [18:44<19:35, 2.87s/it] {'loss': 0.6551, 'grad_norm': 25.63298988342285, 'learning_rate': 3.7831923608280514e-07, 'margin_dpo/margin_mean': 19.984500885009766, 'margin_dpo/margin_std': 19.420936584472656, 'logps/chosen': -134.63824462890625, 'logps/rejected': -140.81619262695312, 'logps/ref_chosen': -121.63641357421875, 'logps/ref_rejected': -107.82986450195312, 'logits/chosen': -1.5109490156173706, 'logits/rejected': -0.8422449827194214, 'epoch': 0.4} 40%|████████████████████████████████████████████▉ | 271/681 [18:44<19:35, 2.87s/it] 40%|█████████████████████████████████████████████▏ | 272/681 [18:47<19:47, 2.90s/it] {'loss': 0.6546, 'grad_norm': 27.825122833251953, 'learning_rate': 3.772161666010912e-07, 'margin_dpo/margin_mean': 19.88021469116211, 'margin_dpo/margin_std': 20.506275177001953, 'logps/chosen': -64.66868591308594, 'logps/rejected': -118.36727905273438, 'logps/ref_chosen': -57.89815902709961, 'logps/ref_rejected': -91.71653747558594, 'logits/chosen': -1.984161138534546, 'logits/rejected': -1.790366768836975, 'epoch': 0.4} 40%|█████████████████████████████████████████████▏ | 272/681 [18:47<19:47, 2.90s/it] 40%|█████████████████████████████████████████████▎ | 273/681 [18:50<19:22, 2.85s/it] {'loss': 0.7343, 'grad_norm': 27.83376121520996, 'learning_rate': 3.761097448550755e-07, 'margin_dpo/margin_mean': 20.25616455078125, 'margin_dpo/margin_std': 20.952659606933594, 'logps/chosen': -132.13973999023438, 'logps/rejected': -151.94931030273438, 'logps/ref_chosen': -116.63589477539062, 'logps/ref_rejected': -116.1893081665039, 'logits/chosen': -1.444991111755371, 'logits/rejected': -0.9423968195915222, 'epoch': 0.4} 40%|█████████████████████████████████████████████▎ | 273/681 [18:50<19:22, 2.85s/it] 40%|█████████████████████████████████████████████▍ | 274/681 [18:53<19:22, 2.86s/it] {'loss': 0.6414, 'grad_norm': 26.02191925048828, 'learning_rate': 3.75e-07, 'margin_dpo/margin_mean': 15.951923370361328, 'margin_dpo/margin_std': 12.496143341064453, 'logps/chosen': -81.43106079101562, 'logps/rejected': -118.44586181640625, 'logps/ref_chosen': -64.8055419921875, 'logps/ref_rejected': -85.86842346191406, 'logits/chosen': -1.2731891870498657, 'logits/rejected': -1.1204042434692383, 'epoch': 0.4} 40%|█████████████████████████████████████████████▍ | 274/681 [18:53<19:22, 2.86s/it] 40%|█████████████████████████████████████████████▋ | 275/681 [18:56<19:41, 2.91s/it] {'loss': 0.6737, 'grad_norm': 28.154870986938477, 'learning_rate': 3.738869612786737e-07, 'margin_dpo/margin_mean': 17.58925437927246, 'margin_dpo/margin_std': 12.368159294128418, 'logps/chosen': -75.03510284423828, 'logps/rejected': -110.03489685058594, 'logps/ref_chosen': -66.54554748535156, 'logps/ref_rejected': -83.95608520507812, 'logits/chosen': -2.217825412750244, 'logits/rejected': -1.8745087385177612, 'epoch': 0.4} 40%|█████████████████████████████████████████████▋ | 275/681 [18:56<19:41, 2.91s/it] 41%|█████████████████████████████████████████████▊ | 276/681 [18:58<19:13, 2.85s/it] {'loss': 0.6958, 'grad_norm': 27.130475997924805, 'learning_rate': 3.7277065802070204e-07, 'margin_dpo/margin_mean': 17.16216278076172, 'margin_dpo/margin_std': 11.365278244018555, 'logps/chosen': -88.32952880859375, 'logps/rejected': -112.3787612915039, 'logps/ref_chosen': -73.30899047851562, 'logps/ref_rejected': -80.19606018066406, 'logits/chosen': -1.3086066246032715, 'logits/rejected': -1.1000663042068481, 'epoch': 0.41} 41%|█████████████████████████████████████████████▊ | 276/681 [18:58<19:13, 2.85s/it] 41%|█████████████████████████████████████████████▉ | 277/681 [19:01<18:46, 2.79s/it] {'loss': 0.6035, 'grad_norm': 23.917224884033203, 'learning_rate': 3.71651119641714e-07, 'margin_dpo/margin_mean': 16.272974014282227, 'margin_dpo/margin_std': 13.451412200927734, 'logps/chosen': -91.10482025146484, 'logps/rejected': -157.8816680908203, 'logps/ref_chosen': -73.8440933227539, 'logps/ref_rejected': -124.34796905517578, 'logits/chosen': -1.4923615455627441, 'logits/rejected': -1.091668963432312, 'epoch': 0.41} 41%|█████████████████████████████████████████████▉ | 277/681 [19:01<18:46, 2.79s/it] 41%|██████████████████████████████████████████████▏ | 278/681 [19:04<19:04, 2.84s/it] {'loss': 0.7411, 'grad_norm': 27.672937393188477, 'learning_rate': 3.705283756425872e-07, 'margin_dpo/margin_mean': 19.189693450927734, 'margin_dpo/margin_std': 15.39527702331543, 'logps/chosen': -78.12380981445312, 'logps/rejected': -124.02960205078125, 'logps/ref_chosen': -64.88898468017578, 'logps/ref_rejected': -91.60508728027344, 'logits/chosen': -1.5108946561813354, 'logits/rejected': -1.3069382905960083, 'epoch': 0.41} 41%|██████████████████████████████████████████████▏ | 278/681 [19:04<19:04, 2.84s/it] 41%|██████████████████████████████████████████████▎ | 279/681 [19:07<19:05, 2.85s/it] {'loss': 0.7615, 'grad_norm': 27.276323318481445, 'learning_rate': 3.6940245560867e-07, 'margin_dpo/margin_mean': 12.001691818237305, 'margin_dpo/margin_std': 11.654304504394531, 'logps/chosen': -113.02943420410156, 'logps/rejected': -133.98086547851562, 'logps/ref_chosen': -100.27659606933594, 'logps/ref_rejected': -109.22632598876953, 'logits/chosen': -1.1005783081054688, 'logits/rejected': -0.8380391597747803, 'epoch': 0.41} 41%|██████████████████████████████████████████████▎ | 279/681 [19:07<19:05, 2.85s/it] 41%|██████████████████████████████████████████████▍ | 280/681 [19:10<19:26, 2.91s/it] {'loss': 0.5692, 'grad_norm': 23.190258026123047, 'learning_rate': 3.6827338920900253e-07, 'margin_dpo/margin_mean': 19.445505142211914, 'margin_dpo/margin_std': 17.48896598815918, 'logps/chosen': -83.09910583496094, 'logps/rejected': -159.41729736328125, 'logps/ref_chosen': -65.84967041015625, 'logps/ref_rejected': -122.72234344482422, 'logits/chosen': -2.4184060096740723, 'logits/rejected': -2.72609281539917, 'epoch': 0.41} 41%|██████████████████████████████████████████████▍ | 280/681 [19:10<19:26, 2.91s/it] 41%|██████████████████████████████████████████████▋ | 281/681 [19:13<19:34, 2.94s/it] {'loss': 0.6119, 'grad_norm': 21.81446647644043, 'learning_rate': 3.6714120619553435e-07, 'margin_dpo/margin_mean': 17.510818481445312, 'margin_dpo/margin_std': 18.7252197265625, 'logps/chosen': -73.10958862304688, 'logps/rejected': -100.85271453857422, 'logps/ref_chosen': -57.2689208984375, 'logps/ref_rejected': -67.501220703125, 'logits/chosen': -1.2980034351348877, 'logits/rejected': -0.9670097231864929, 'epoch': 0.41} 41%|██████████████████████████████████████████████▋ | 281/681 [19:13<19:34, 2.94s/it] 41%|██████████████████████████████████████████████▊ | 282/681 [19:16<19:38, 2.95s/it] {'loss': 0.4488, 'grad_norm': 20.058345794677734, 'learning_rate': 3.660059364023408e-07, 'margin_dpo/margin_mean': 25.340524673461914, 'margin_dpo/margin_std': 17.682575225830078, 'logps/chosen': -94.27513122558594, 'logps/rejected': -100.76190185546875, 'logps/ref_chosen': -85.69962310791016, 'logps/ref_rejected': -66.8458480834961, 'logits/chosen': -1.7840750217437744, 'logits/rejected': -1.2310118675231934, 'epoch': 0.41} 41%|██████████████████████████████████████████████▊ | 282/681 [19:16<19:38, 2.95s/it] 42%|██████████████████████████████████████████████▉ | 283/681 [19:19<19:19, 2.91s/it] {'loss': 0.8681, 'grad_norm': 39.15544509887695, 'learning_rate': 3.6486760974483685e-07, 'margin_dpo/margin_mean': 12.62417221069336, 'margin_dpo/margin_std': 15.997934341430664, 'logps/chosen': -101.2388916015625, 'logps/rejected': -112.30723571777344, 'logps/ref_chosen': -85.58895111083984, 'logps/ref_rejected': -84.03312683105469, 'logits/chosen': -2.0153682231903076, 'logits/rejected': -1.574284553527832, 'epoch': 0.42} 42%|██████████████████████████████████████████████▉ | 283/681 [19:19<19:19, 2.91s/it] 42%|███████████████████████████████████████████████ | 284/681 [19:22<19:44, 2.98s/it] {'loss': 0.4147, 'grad_norm': 17.253101348876953, 'learning_rate': 3.6372625621898863e-07, 'margin_dpo/margin_mean': 17.883419036865234, 'margin_dpo/margin_std': 9.083778381347656, 'logps/chosen': -107.02427673339844, 'logps/rejected': -102.14917755126953, 'logps/ref_chosen': -93.25978088378906, 'logps/ref_rejected': -70.50126647949219, 'logits/chosen': -1.7107484340667725, 'logits/rejected': -1.297828197479248, 'epoch': 0.42} 42%|███████████████████████████████████████████████ | 284/681 [19:22<19:44, 2.98s/it] 42%|███████████████████████████████████████████████▎ | 285/681 [19:25<19:46, 3.00s/it] {'loss': 0.6078, 'grad_norm': 30.760282516479492, 'learning_rate': 3.625819059005228e-07, 'margin_dpo/margin_mean': 16.659687042236328, 'margin_dpo/margin_std': 12.118792533874512, 'logps/chosen': -70.15441131591797, 'logps/rejected': -124.66044616699219, 'logps/ref_chosen': -51.80888366699219, 'logps/ref_rejected': -89.65523529052734, 'logits/chosen': -1.5096139907836914, 'logits/rejected': -1.7117888927459717, 'epoch': 0.42} 42%|███████████████████████████████████████████████▎ | 285/681 [19:25<19:46, 3.00s/it] 42%|███████████████████████████████████████████████▍ | 286/681 [19:28<19:34, 2.97s/it] {'loss': 0.59, 'grad_norm': 26.206777572631836, 'learning_rate': 3.614345889441346e-07, 'margin_dpo/margin_mean': 16.394378662109375, 'margin_dpo/margin_std': 9.293878555297852, 'logps/chosen': -116.84445190429688, 'logps/rejected': -104.88263702392578, 'logps/ref_chosen': -100.83771514892578, 'logps/ref_rejected': -72.48152923583984, 'logits/chosen': -2.420334577560425, 'logits/rejected': -1.927842617034912, 'epoch': 0.42} 42%|███████████████████████████████████████████████▍ | 286/681 [19:28<19:34, 2.97s/it] 42%|███████████████████████████████████████████████▌ | 287/681 [19:30<18:45, 2.86s/it] {'loss': 0.8528, 'grad_norm': 35.31901168823242, 'learning_rate': 3.6028433558269275e-07, 'margin_dpo/margin_mean': 15.366945266723633, 'margin_dpo/margin_std': 17.302759170532227, 'logps/chosen': -124.8189697265625, 'logps/rejected': -108.34898376464844, 'logps/ref_chosen': -108.39360809326172, 'logps/ref_rejected': -76.55667114257812, 'logits/chosen': -1.129586100578308, 'logits/rejected': -0.4217372238636017, 'epoch': 0.42} 42%|███████████████████████████████████████████████▌ | 287/681 [19:30<18:45, 2.86s/it] 42%|███████████████████████████████████████████████▊ | 288/681 [19:33<18:49, 2.87s/it] {'loss': 0.6139, 'grad_norm': 25.598308563232422, 'learning_rate': 3.5913117612644327e-07, 'margin_dpo/margin_mean': 12.322646141052246, 'margin_dpo/margin_std': 16.771167755126953, 'logps/chosen': -97.67936706542969, 'logps/rejected': -120.51500701904297, 'logps/ref_chosen': -81.42936706542969, 'logps/ref_rejected': -91.9423599243164, 'logits/chosen': -1.43570876121521, 'logits/rejected': -1.31893789768219, 'epoch': 0.42} 42%|███████████████████████████████████████████████▊ | 288/681 [19:33<18:49, 2.87s/it] 42%|███████████████████████████████████████████████▉ | 289/681 [19:36<18:56, 2.90s/it] {'loss': 0.6594, 'grad_norm': 22.585121154785156, 'learning_rate': 3.5797514096221024e-07, 'margin_dpo/margin_mean': 29.76548957824707, 'margin_dpo/margin_std': 26.266082763671875, 'logps/chosen': -103.3167724609375, 'logps/rejected': -147.36233520507812, 'logps/ref_chosen': -91.00823974609375, 'logps/ref_rejected': -105.28831481933594, 'logits/chosen': -1.5673437118530273, 'logits/rejected': -1.1167399883270264, 'epoch': 0.42} 42%|███████████████████████████████████████████████▉ | 289/681 [19:36<18:56, 2.90s/it] 43%|████████████████████████████████████████████████ | 290/681 [19:39<18:33, 2.85s/it] {'loss': 0.6342, 'grad_norm': 27.944913864135742, 'learning_rate': 3.568162605525952e-07, 'margin_dpo/margin_mean': 20.673748016357422, 'margin_dpo/margin_std': 14.754283905029297, 'logps/chosen': -106.29632568359375, 'logps/rejected': -159.3505859375, 'logps/ref_chosen': -86.29432678222656, 'logps/ref_rejected': -118.67485046386719, 'logits/chosen': -1.0414509773254395, 'logits/rejected': -1.1000698804855347, 'epoch': 0.43} 43%|████████████████████████████████████████████████ | 290/681 [19:39<18:33, 2.85s/it] 43%|████████████████████████████████████████████████▎ | 291/681 [19:42<18:40, 2.87s/it] {'loss': 0.781, 'grad_norm': 33.19970703125, 'learning_rate': 3.5565456543517485e-07, 'margin_dpo/margin_mean': 17.343158721923828, 'margin_dpo/margin_std': 13.000414848327637, 'logps/chosen': -85.19589233398438, 'logps/rejected': -126.43409729003906, 'logps/ref_chosen': -66.42658233642578, 'logps/ref_rejected': -90.32164001464844, 'logits/chosen': -1.560898780822754, 'logits/rejected': -1.653581142425537, 'epoch': 0.43} 43%|████████████████████████████████████████████████▎ | 291/681 [19:42<18:40, 2.87s/it] 43%|████████████████████████████████████████████████▍ | 292/681 [19:45<18:35, 2.87s/it] {'loss': 0.4858, 'grad_norm': 21.21723747253418, 'learning_rate': 3.5449008622169583e-07, 'margin_dpo/margin_mean': 19.803937911987305, 'margin_dpo/margin_std': 12.30525016784668, 'logps/chosen': -115.83615112304688, 'logps/rejected': -134.69090270996094, 'logps/ref_chosen': -101.82511901855469, 'logps/ref_rejected': -100.87593078613281, 'logits/chosen': -1.6994218826293945, 'logits/rejected': -1.175065040588379, 'epoch': 0.43} 43%|████████████████████████████████████████████████▍ | 292/681 [19:45<18:35, 2.87s/it] 43%|████████████████████████████████████████████████▌ | 293/681 [19:48<18:40, 2.89s/it] {'loss': 0.8721, 'grad_norm': 26.848459243774414, 'learning_rate': 3.5332285359726846e-07, 'margin_dpo/margin_mean': 11.636128425598145, 'margin_dpo/margin_std': 17.83340072631836, 'logps/chosen': -84.04584503173828, 'logps/rejected': -109.67572784423828, 'logps/ref_chosen': -68.95051574707031, 'logps/ref_rejected': -82.94426727294922, 'logits/chosen': -1.7359650135040283, 'logits/rejected': -1.7631828784942627, 'epoch': 0.43} 43%|████████████████████████████████████████████████▌ | 293/681 [19:48<18:40, 2.89s/it] 43%|████████████████████████████████████████████████▊ | 294/681 [19:51<18:31, 2.87s/it] {'loss': 0.8365, 'grad_norm': 28.9974308013916, 'learning_rate': 3.5215289831955786e-07, 'margin_dpo/margin_mean': 13.591376304626465, 'margin_dpo/margin_std': 13.403884887695312, 'logps/chosen': -102.3502197265625, 'logps/rejected': -98.13722229003906, 'logps/ref_chosen': -88.47676086425781, 'logps/ref_rejected': -70.6723861694336, 'logits/chosen': -1.5406566858291626, 'logits/rejected': -1.1875331401824951, 'epoch': 0.43} 43%|████████████████████████████████████████████████▊ | 294/681 [19:51<18:31, 2.87s/it] 43%|████████████████████████████████████████████████▉ | 295/681 [19:53<18:16, 2.84s/it] {'loss': 0.6613, 'grad_norm': 25.117948532104492, 'learning_rate': 3.509802512179737e-07, 'margin_dpo/margin_mean': 17.824451446533203, 'margin_dpo/margin_std': 8.08942985534668, 'logps/chosen': -99.4218978881836, 'logps/rejected': -114.45523071289062, 'logps/ref_chosen': -82.94652557373047, 'logps/ref_rejected': -80.15540313720703, 'logits/chosen': -1.3463271856307983, 'logits/rejected': -0.9588865637779236, 'epoch': 0.43} 43%|████████████████████████████████████████████████▉ | 295/681 [19:53<18:16, 2.84s/it] 43%|█████████████████████████████████████████████████ | 296/681 [19:56<18:06, 2.82s/it] {'loss': 0.6912, 'grad_norm': 27.37685775756836, 'learning_rate': 3.498049431928577e-07, 'margin_dpo/margin_mean': 19.46407127380371, 'margin_dpo/margin_std': 28.34971046447754, 'logps/chosen': -138.28659057617188, 'logps/rejected': -159.16708374023438, 'logps/ref_chosen': -118.51683807373047, 'logps/ref_rejected': -119.9332504272461, 'logits/chosen': -2.3768234252929688, 'logits/rejected': -1.7210721969604492, 'epoch': 0.43} 43%|█████████████████████████████████████████████████ | 296/681 [19:56<18:06, 2.82s/it] 44%|█████████████████████████████████████████████████▎ | 297/681 [19:59<18:34, 2.90s/it] {'loss': 0.6463, 'grad_norm': 24.673267364501953, 'learning_rate': 3.486270052146694e-07, 'margin_dpo/margin_mean': 20.95398712158203, 'margin_dpo/margin_std': 13.207864761352539, 'logps/chosen': -105.87931823730469, 'logps/rejected': -133.82965087890625, 'logps/ref_chosen': -86.62548065185547, 'logps/ref_rejected': -93.621826171875, 'logits/chosen': -0.6197730302810669, 'logits/rejected': -0.3352678418159485, 'epoch': 0.44} 44%|█████████████████████████████████████████████████▎ | 297/681 [19:59<18:34, 2.90s/it] 44%|█████████████████████████████████████████████████▍ | 298/681 [20:02<18:37, 2.92s/it] {'loss': 0.6232, 'grad_norm': 26.703006744384766, 'learning_rate': 3.474464683231698e-07, 'margin_dpo/margin_mean': 15.894968032836914, 'margin_dpo/margin_std': 15.374116897583008, 'logps/chosen': -141.28953552246094, 'logps/rejected': -129.42611694335938, 'logps/ref_chosen': -125.80955505371094, 'logps/ref_rejected': -98.05117797851562, 'logits/chosen': -2.5179362297058105, 'logits/rejected': -2.055812358856201, 'epoch': 0.44} 44%|█████████████████████████████████████████████████▍ | 298/681 [20:02<18:37, 2.92s/it] 44%|█████████████████████████████████████████████████▌ | 299/681 [20:05<18:34, 2.92s/it] {'loss': 0.5611, 'grad_norm': 25.000356674194336, 'learning_rate': 3.462633636266041e-07, 'margin_dpo/margin_mean': 20.862197875976562, 'margin_dpo/margin_std': 13.313728332519531, 'logps/chosen': -76.33744812011719, 'logps/rejected': -133.07330322265625, 'logps/ref_chosen': -60.100074768066406, 'logps/ref_rejected': -95.97373962402344, 'logits/chosen': -0.8845536708831787, 'logits/rejected': -0.7745420932769775, 'epoch': 0.44} 44%|█████████████████████████████████████████████████▌ | 299/681 [20:05<18:34, 2.92s/it] 44%|█████████████████████████████████████████████████▊ | 300/681 [20:08<18:54, 2.98s/it] {'loss': 0.7508, 'grad_norm': 29.056922912597656, 'learning_rate': 3.4507772230088147e-07, 'margin_dpo/margin_mean': 12.071918487548828, 'margin_dpo/margin_std': 16.181241989135742, 'logps/chosen': -86.75079345703125, 'logps/rejected': -129.37054443359375, 'logps/ref_chosen': -63.767547607421875, 'logps/ref_rejected': -94.31537628173828, 'logits/chosen': -1.2322702407836914, 'logits/rejected': -1.1211085319519043, 'epoch': 0.44} 44%|█████████████████████████████████████████████████▊ | 300/681 [20:08<18:54, 2.98s/it][INFO|trainer.py:4307] 2026-04-24 00:15:15,782 >> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-24 00:15:15,782 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-24 00:15:15,782 >> Batch size = 8 0%| | 0/73 [00:00> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-24 00:20:51,431 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-24 00:20:51,431 >> Batch size = 8 0%| | 0/73 [00:00> Saving model checkpoint to /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-400 [INFO|configuration_utils.py:419] 2026-04-24 00:21:49,519 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-400/config.json [INFO|configuration_utils.py:911] 2026-04-24 00:21:49,522 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-400/generation_config.json [INFO|modeling_utils.py:3580] 2026-04-24 00:22:28,519 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-400/model.safetensors.index.json. [INFO|tokenization_utils_base.py:2510] 2026-04-24 00:22:28,526 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-400/tokenizer_config.json [INFO|tokenization_utils_base.py:2519] 2026-04-24 00:22:28,530 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-400/special_tokens_map.json 59%|█████████████████████████████████████████████████████████████████▎ | 401/681 [30:51<7:19:43, 94.23s/it] {'loss': 0.8125, 'grad_norm': 27.084718704223633, 'learning_rate': 2.1800473436235136e-07, 'margin_dpo/margin_mean': 6.442744255065918, 'margin_dpo/margin_std': 16.57547378540039, 'logps/chosen': -119.38215637207031, 'logps/rejected': -99.23313903808594, 'logps/ref_chosen': -90.59347534179688, 'logps/ref_rejected': -64.001708984375, 'logits/chosen': -0.82004714012146, 'logits/rejected': -0.5782370567321777, 'epoch': 0.59} 59%|█████████████████████████████████████████████████████████████████▎ | 401/681 [30:51<7:19:43, 94.23s/it] 59%|█████████████████████████████████████████████████████████████████▌ | 402/681 [30:54<5:10:19, 66.73s/it] {'loss': 0.3894, 'grad_norm': 20.646493911743164, 'learning_rate': 2.1673238449588665e-07, 'margin_dpo/margin_mean': 24.005657196044922, 'margin_dpo/margin_std': 14.80632209777832, 'logps/chosen': -114.04674530029297, 'logps/rejected': -132.45294189453125, 'logps/ref_chosen': -92.98797607421875, 'logps/ref_rejected': -87.38853454589844, 'logits/chosen': -1.7929552793502808, 'logits/rejected': -1.4944618940353394, 'epoch': 0.59} 59%|█████████████████████████████████████████████████████████████████▌ | 402/681 [30:54<5:10:19, 66.73s/it] 59%|█████████████████████████████████████████████████████████████████▋ | 403/681 [30:57<3:40:25, 47.57s/it] {'loss': 0.5945, 'grad_norm': 20.255006790161133, 'learning_rate': 2.154609112620295e-07, 'margin_dpo/margin_mean': 19.7047061920166, 'margin_dpo/margin_std': 19.972158432006836, 'logps/chosen': -111.00959777832031, 'logps/rejected': -102.13783264160156, 'logps/ref_chosen': -91.2931900024414, 'logps/ref_rejected': -62.71670913696289, 'logits/chosen': -2.186984062194824, 'logits/rejected': -1.762284278869629, 'epoch': 0.59} 59%|█████████████████████████████████████████████████████████████████▋ | 403/681 [30:57<3:40:25, 47.57s/it] 59%|█████████████████████████████████████████████████████████████████▊ | 404/681 [31:00<2:38:03, 34.24s/it] {'loss': 0.6096, 'grad_norm': 21.293611526489258, 'learning_rate': 2.1419034816528218e-07, 'margin_dpo/margin_mean': 30.76922035217285, 'margin_dpo/margin_std': 20.257919311523438, 'logps/chosen': -100.58467864990234, 'logps/rejected': -154.99609375, 'logps/ref_chosen': -77.146728515625, 'logps/ref_rejected': -100.78892517089844, 'logits/chosen': -2.028939723968506, 'logits/rejected': -1.676574945449829, 'epoch': 0.59} 59%|█████████████████████████████████████████████████████████████████▊ | 404/681 [31:00<2:38:03, 34.24s/it] 59%|██████████████████████████████████████████████████████████████████ | 405/681 [31:03<1:53:51, 24.75s/it] {'loss': 0.6095, 'grad_norm': 26.865093231201172, 'learning_rate': 2.129207286861638e-07, 'margin_dpo/margin_mean': 22.707393646240234, 'margin_dpo/margin_std': 19.417251586914062, 'logps/chosen': -140.56105041503906, 'logps/rejected': -163.43182373046875, 'logps/ref_chosen': -113.60545349121094, 'logps/ref_rejected': -113.76882934570312, 'logits/chosen': -0.8960381746292114, 'logits/rejected': -0.6281898021697998, 'epoch': 0.59} 59%|██████████████████████████████████████████████████████████████████ | 405/681 [31:03<1:53:51, 24.75s/it] 60%|██████████████████████████████████████████████████████████████████▏ | 406/681 [31:05<1:23:04, 18.13s/it] {'loss': 0.6461, 'grad_norm': 26.39828109741211, 'learning_rate': 2.1165208628032861e-07, 'margin_dpo/margin_mean': 23.98102378845215, 'margin_dpo/margin_std': 19.551021575927734, 'logps/chosen': -102.27427673339844, 'logps/rejected': -134.26731872558594, 'logps/ref_chosen': -82.02105712890625, 'logps/ref_rejected': -90.03308868408203, 'logits/chosen': -2.1540369987487793, 'logits/rejected': -1.7181463241577148, 'epoch': 0.6} 60%|██████████████████████████████████████████████████████████████████▏ | 406/681 [31:05<1:23:04, 18.13s/it] 60%|██████████████████████████████████████████████████████████████████▎ | 407/681 [31:08<1:01:49, 13.54s/it] {'loss': 0.6719, 'grad_norm': 25.158649444580078, 'learning_rate': 2.1038445437768375e-07, 'margin_dpo/margin_mean': 13.367082595825195, 'margin_dpo/margin_std': 21.264638900756836, 'logps/chosen': -102.0631103515625, 'logps/rejected': -109.44853210449219, 'logps/ref_chosen': -71.41169738769531, 'logps/ref_rejected': -65.43003845214844, 'logits/chosen': -2.754941463470459, 'logits/rejected': -2.005859851837158, 'epoch': 0.6} 60%|██████████████████████████████████████████████████████████████████▎ | 407/681 [31:08<1:01:49, 13.54s/it] 60%|███████████████████████████████████████████████████████████████████▋ | 408/681 [31:11<47:19, 10.40s/it] {'loss': 0.7704, 'grad_norm': 34.24391555786133, 'learning_rate': 2.0911786638150872e-07, 'margin_dpo/margin_mean': 19.870115280151367, 'margin_dpo/margin_std': 19.41891860961914, 'logps/chosen': -129.79104614257812, 'logps/rejected': -112.33432006835938, 'logps/ref_chosen': -103.82225036621094, 'logps/ref_rejected': -66.49540710449219, 'logits/chosen': -0.9452221989631653, 'logits/rejected': -0.40952903032302856, 'epoch': 0.6} 60%|███████████████████████████████████████████████████████████████████▋ | 408/681 [31:11<47:19, 10.40s/it] 60%|███████████████████████████████████████████████████████████████████▊ | 409/681 [31:14<37:05, 8.18s/it] {'loss': 0.7749, 'grad_norm': 36.82187271118164, 'learning_rate': 2.0785235566757517e-07, 'margin_dpo/margin_mean': 19.43436050415039, 'margin_dpo/margin_std': 19.317798614501953, 'logps/chosen': -163.90234375, 'logps/rejected': -126.71371459960938, 'logps/ref_chosen': -143.14186096191406, 'logps/ref_rejected': -86.51887512207031, 'logits/chosen': -1.7844617366790771, 'logits/rejected': -1.0885341167449951, 'epoch': 0.6} 60%|███████████████████████████████████████████████████████████████████▊ | 409/681 [31:14<37:05, 8.18s/it] 60%|████████████████████████████████████████████████████████████████████ | 410/681 [31:17<29:48, 6.60s/it] {'loss': 0.6685, 'grad_norm': 26.417638778686523, 'learning_rate': 2.065879555832674e-07, 'margin_dpo/margin_mean': 19.673145294189453, 'margin_dpo/margin_std': 14.39936351776123, 'logps/chosen': -86.10826110839844, 'logps/rejected': -125.80564880371094, 'logps/ref_chosen': -60.73199462890625, 'logps/ref_rejected': -80.75624084472656, 'logits/chosen': -2.059418201446533, 'logits/rejected': -1.9821703433990479, 'epoch': 0.6} 60%|████████████████████████████████████████████████████████████████████ | 410/681 [31:17<29:48, 6.60s/it] 60%|████████████████████████████████████████████████████████████████████▏ | 411/681 [31:20<24:23, 5.42s/it] {'loss': 0.6113, 'grad_norm': 26.731914520263672, 'learning_rate': 2.0532469944670343e-07, 'margin_dpo/margin_mean': 14.190881729125977, 'margin_dpo/margin_std': 13.96728515625, 'logps/chosen': -95.44523620605469, 'logps/rejected': -118.89471435546875, 'logps/ref_chosen': -66.200927734375, 'logps/ref_rejected': -75.45953369140625, 'logits/chosen': -2.0580475330352783, 'logits/rejected': -2.118593692779541, 'epoch': 0.6} 60%|████████████████████████████████████████████████████████████████████▏ | 411/681 [31:20<24:23, 5.42s/it] 60%|████████████████████████████████████████████████████████████████████▎ | 412/681 [31:23<20:53, 4.66s/it] {'loss': 0.6431, 'grad_norm': 24.100954055786133, 'learning_rate': 2.0406262054585738e-07, 'margin_dpo/margin_mean': 19.70269203186035, 'margin_dpo/margin_std': 17.422361373901367, 'logps/chosen': -96.10821533203125, 'logps/rejected': -161.6231689453125, 'logps/ref_chosen': -70.4729232788086, 'logps/ref_rejected': -116.28516387939453, 'logits/chosen': -1.6652562618255615, 'logits/rejected': -1.6168675422668457, 'epoch': 0.6} 60%|████████████████████████████████████████████████████████████████████▎ | 412/681 [31:23<20:53, 4.66s/it] 61%|████████████████████████████████████████████████████████████████████▌ | 413/681 [31:26<18:27, 4.13s/it] {'loss': 0.7514, 'grad_norm': 32.71993637084961, 'learning_rate': 2.0280175213768205e-07, 'margin_dpo/margin_mean': 11.658080101013184, 'margin_dpo/margin_std': 13.869819641113281, 'logps/chosen': -121.38672637939453, 'logps/rejected': -119.86428833007812, 'logps/ref_chosen': -91.80178833007812, 'logps/ref_rejected': -78.62126159667969, 'logits/chosen': -0.5341243743896484, 'logits/rejected': -0.1439867466688156, 'epoch': 0.61} 61%|████████████████████████████████████████████████████████████████████▌ | 413/681 [31:26<18:27, 4.13s/it] 61%|████████████████████████████████████████████████████████████████████▋ | 414/681 [31:28<16:48, 3.78s/it] {'loss': 0.7371, 'grad_norm': 32.95093536376953, 'learning_rate': 2.0154212744723247e-07, 'margin_dpo/margin_mean': 17.44455337524414, 'margin_dpo/margin_std': 21.200258255004883, 'logps/chosen': -115.67071533203125, 'logps/rejected': -127.25440216064453, 'logps/ref_chosen': -93.31285095214844, 'logps/ref_rejected': -87.45198822021484, 'logits/chosen': -2.0908069610595703, 'logits/rejected': -1.547807216644287, 'epoch': 0.61} 61%|████████████████████████████████████████████████████████████████████▋ | 414/681 [31:28<16:48, 3.78s/it] 61%|████████████████████████████████████████████████████████████████████▊ | 415/681 [31:32<15:53, 3.59s/it] {'loss': 0.7499, 'grad_norm': 38.77595520019531, 'learning_rate': 2.002837796667909e-07, 'margin_dpo/margin_mean': 26.036104202270508, 'margin_dpo/margin_std': 21.959575653076172, 'logps/chosen': -131.7755126953125, 'logps/rejected': -146.5677490234375, 'logps/ref_chosen': -108.298095703125, 'logps/ref_rejected': -97.05425262451172, 'logits/chosen': -1.3557873964309692, 'logits/rejected': -0.9005389213562012, 'epoch': 0.61} 61%|████████████████████████████████████████████████████████████████████▊ | 415/681 [31:32<15:53, 3.59s/it] 61%|█████████████████████████████████████████████████████████████████████ | 416/681 [31:35<15:06, 3.42s/it] {'loss': 0.4686, 'grad_norm': 20.369680404663086, 'learning_rate': 1.990267419549914e-07, 'margin_dpo/margin_mean': 24.462818145751953, 'margin_dpo/margin_std': 19.401506423950195, 'logps/chosen': -108.19269561767578, 'logps/rejected': -153.75759887695312, 'logps/ref_chosen': -84.23023986816406, 'logps/ref_rejected': -105.33232879638672, 'logits/chosen': -2.151233196258545, 'logits/rejected': -1.5413405895233154, 'epoch': 0.61} 61%|█████████████████████████████████████████████████████████████████████ | 416/681 [31:35<15:06, 3.42s/it] 61%|█████████████████████████████████████████████████████████████████████▏ | 417/681 [31:37<14:04, 3.20s/it] {'loss': 0.5942, 'grad_norm': 24.230409622192383, 'learning_rate': 1.9777104743594686e-07, 'margin_dpo/margin_mean': 23.894100189208984, 'margin_dpo/margin_std': 23.507366180419922, 'logps/chosen': -109.26607513427734, 'logps/rejected': -103.1401596069336, 'logps/ref_chosen': -89.4459457397461, 'logps/ref_rejected': -59.425933837890625, 'logits/chosen': -2.02632474899292, 'logits/rejected': -0.5686318278312683, 'epoch': 0.61} 61%|█████████████████████████████████████████████████████████████████████▏ | 417/681 [31:37<14:04, 3.20s/it] 61%|█████████████████████████████████████████████████████████████████████▎ | 418/681 [31:40<13:39, 3.12s/it] {'loss': 0.5106, 'grad_norm': 26.247756958007812, 'learning_rate': 1.965167291983757e-07, 'margin_dpo/margin_mean': 27.746294021606445, 'margin_dpo/margin_std': 15.890663146972656, 'logps/chosen': -109.95904541015625, 'logps/rejected': -155.7923126220703, 'logps/ref_chosen': -87.28543090820312, 'logps/ref_rejected': -105.37240600585938, 'logits/chosen': -2.066215753555298, 'logits/rejected': -1.878091812133789, 'epoch': 0.61} 61%|█████████████████████████████████████████████████████████████████████▎ | 418/681 [31:40<13:39, 3.12s/it] 62%|█████████████████████████████████████████████████████████████████████▌ | 419/681 [31:43<13:39, 3.13s/it] {'loss': 0.7314, 'grad_norm': 29.40278434753418, 'learning_rate': 1.9526382029472988e-07, 'margin_dpo/margin_mean': 23.130002975463867, 'margin_dpo/margin_std': 20.91366195678711, 'logps/chosen': -118.16517639160156, 'logps/rejected': -162.2279510498047, 'logps/ref_chosen': -94.09585571289062, 'logps/ref_rejected': -115.02862548828125, 'logits/chosen': -1.2009162902832031, 'logits/rejected': -0.9768263101577759, 'epoch': 0.62} 62%|█████████████████████████████████████████████████████████████████████▌ | 419/681 [31:43<13:39, 3.13s/it] 62%|█████████████████████████████████████████████████████████████████████▋ | 420/681 [31:46<13:26, 3.09s/it] {'loss': 0.598, 'grad_norm': 38.40915298461914, 'learning_rate': 1.9401235374032425e-07, 'margin_dpo/margin_mean': 20.196256637573242, 'margin_dpo/margin_std': 21.781005859375, 'logps/chosen': -123.48500061035156, 'logps/rejected': -136.91636657714844, 'logps/ref_chosen': -101.53406524658203, 'logps/ref_rejected': -94.7691650390625, 'logits/chosen': -1.9284502267837524, 'logits/rejected': -1.554548978805542, 'epoch': 0.62} 62%|█████████████████████████████████████████████████████████████████████▋ | 420/681 [31:46<13:26, 3.09s/it] 62%|█████████████████████████████████████████████████████████████████████▊ | 421/681 [31:49<13:15, 3.06s/it] {'loss': 0.6178, 'grad_norm': 26.852741241455078, 'learning_rate': 1.9276236251246653e-07, 'margin_dpo/margin_mean': 22.544641494750977, 'margin_dpo/margin_std': 18.13541030883789, 'logps/chosen': -87.1471939086914, 'logps/rejected': -140.4273681640625, 'logps/ref_chosen': -57.393409729003906, 'logps/ref_rejected': -88.12896728515625, 'logits/chosen': -1.4120337963104248, 'logits/rejected': -1.2915436029434204, 'epoch': 0.62} 62%|█████████████████████████████████████████████████████████████████████▊ | 421/681 [31:49<13:15, 3.06s/it] 62%|██████████████████████████████████████████████████████████████████████ | 422/681 [31:52<13:04, 3.03s/it] {'loss': 0.6448, 'grad_norm': 28.518985748291016, 'learning_rate': 1.9151387954958792e-07, 'margin_dpo/margin_mean': 25.22815704345703, 'margin_dpo/margin_std': 25.092853546142578, 'logps/chosen': -158.14785766601562, 'logps/rejected': -174.84373474121094, 'logps/ref_chosen': -137.25265502929688, 'logps/ref_rejected': -128.72036743164062, 'logits/chosen': -2.0918030738830566, 'logits/rejected': -1.6457126140594482, 'epoch': 0.62} 62%|██████████████████████████████████████████████████████████████████████ | 422/681 [31:52<13:04, 3.03s/it] 62%|██████████████████████████████████████████████████████████████████████▏ | 423/681 [31:55<12:48, 2.98s/it] {'loss': 0.6857, 'grad_norm': 27.90230941772461, 'learning_rate': 1.902669377503756e-07, 'margin_dpo/margin_mean': 13.20401382446289, 'margin_dpo/margin_std': 14.358932495117188, 'logps/chosen': -95.43914794921875, 'logps/rejected': -125.11500549316406, 'logps/ref_chosen': -61.778358459472656, 'logps/ref_rejected': -78.25019836425781, 'logits/chosen': -1.126874566078186, 'logits/rejected': -1.2383806705474854, 'epoch': 0.62} 62%|██████████████████████████████████████████████████████████████████████▏ | 423/681 [31:55<12:48, 2.98s/it] 62%|██████████████████████████████████████████████████████████████████████▎ | 424/681 [31:58<12:52, 3.01s/it] {'loss': 0.7084, 'grad_norm': 31.08055877685547, 'learning_rate': 1.890215699729057e-07, 'margin_dpo/margin_mean': 23.822601318359375, 'margin_dpo/margin_std': 23.501611709594727, 'logps/chosen': -116.06146240234375, 'logps/rejected': -106.29594421386719, 'logps/ref_chosen': -92.41514587402344, 'logps/ref_rejected': -58.8270263671875, 'logits/chosen': -1.6574307680130005, 'logits/rejected': -0.9652201533317566, 'epoch': 0.62} 62%|██████████████████████████████████████████████████████████████████████▎ | 424/681 [31:58<12:52, 3.01s/it] 62%|██████████████████████████████████████████████████████████████████████▌ | 425/681 [32:01<12:29, 2.93s/it] {'loss': 0.8679, 'grad_norm': 31.131589889526367, 'learning_rate': 1.8777780903377732e-07, 'margin_dpo/margin_mean': 8.749892234802246, 'margin_dpo/margin_std': 17.027074813842773, 'logps/chosen': -107.98200988769531, 'logps/rejected': -109.9925308227539, 'logps/ref_chosen': -82.38971710205078, 'logps/ref_rejected': -75.65036010742188, 'logits/chosen': -2.1004793643951416, 'logits/rejected': -1.985741138458252, 'epoch': 0.62} 62%|██████████████████████████████████████████████████████████████████████▌ | 425/681 [32:01<12:29, 2.93s/it] 63%|██████████████████████████████████████████████████████████████████████▋ | 426/681 [32:04<12:37, 2.97s/it] {'loss': 0.4407, 'grad_norm': 31.041427612304688, 'learning_rate': 1.8653568770724803e-07, 'margin_dpo/margin_mean': 23.68649673461914, 'margin_dpo/margin_std': 17.5459041595459, 'logps/chosen': -121.36643981933594, 'logps/rejected': -130.52114868164062, 'logps/ref_chosen': -100.81852722167969, 'logps/ref_rejected': -86.2867431640625, 'logits/chosen': -1.810539960861206, 'logits/rejected': -1.1514090299606323, 'epoch': 0.63} 63%|██████████████████████████████████████████████████████████████████████▋ | 426/681 [32:04<12:37, 2.97s/it] 63%|██████████████████████████████████████████████████████████████████████▊ | 427/681 [32:07<12:52, 3.04s/it] {'loss': 0.5377, 'grad_norm': 20.86272430419922, 'learning_rate': 1.8529523872436977e-07, 'margin_dpo/margin_mean': 21.26839828491211, 'margin_dpo/margin_std': 21.174129486083984, 'logps/chosen': -113.21662902832031, 'logps/rejected': -146.71148681640625, 'logps/ref_chosen': -90.550537109375, 'logps/ref_rejected': -102.77699279785156, 'logits/chosen': -1.7860926389694214, 'logits/rejected': -1.2939798831939697, 'epoch': 0.63} 63%|██████████████████████████████████████████████████████████████████████▊ | 427/681 [32:07<12:52, 3.04s/it] 63%|███████████████████████████████████████████████████████████████████████ | 428/681 [32:11<13:06, 3.11s/it] {'loss': 0.8885, 'grad_norm': 39.244056701660156, 'learning_rate': 1.8405649477212697e-07, 'margin_dpo/margin_mean': 22.869205474853516, 'margin_dpo/margin_std': 22.71468734741211, 'logps/chosen': -125.4493408203125, 'logps/rejected': -171.86065673828125, 'logps/ref_chosen': -97.28589630126953, 'logps/ref_rejected': -120.8280029296875, 'logits/chosen': -0.9955465793609619, 'logits/rejected': -1.0620194673538208, 'epoch': 0.63} 63%|███████████████████████████████████████████████████████████████████████ | 428/681 [32:11<13:06, 3.11s/it] 63%|███████████████████████████████████████████████████████████████████████▏ | 429/681 [32:14<12:56, 3.08s/it] {'loss': 0.6668, 'grad_norm': 31.844663619995117, 'learning_rate': 1.828194884925749e-07, 'margin_dpo/margin_mean': 17.7259521484375, 'margin_dpo/margin_std': 19.956687927246094, 'logps/chosen': -149.5, 'logps/rejected': -155.66888427734375, 'logps/ref_chosen': -119.6482162475586, 'logps/ref_rejected': -108.09115600585938, 'logits/chosen': -1.9276843070983887, 'logits/rejected': -1.3610649108886719, 'epoch': 0.63} 63%|███████████████████████████████████████████████████████████████████████▏ | 429/681 [32:14<12:56, 3.08s/it] 63%|███████████████████████████████████████████████████████████████████████▎ | 430/681 [32:17<12:53, 3.08s/it] {'loss': 0.6066, 'grad_norm': 25.210390090942383, 'learning_rate': 1.8158425248197928e-07, 'margin_dpo/margin_mean': 18.67949104309082, 'margin_dpo/margin_std': 18.119897842407227, 'logps/chosen': -125.53485107421875, 'logps/rejected': -172.47962951660156, 'logps/ref_chosen': -97.25688934326172, 'logps/ref_rejected': -125.52217102050781, 'logits/chosen': -0.9649932384490967, 'logits/rejected': -1.147021770477295, 'epoch': 0.63} 63%|███████████████████████████████████████████████████████████████████████▎ | 430/681 [32:17<12:53, 3.08s/it] 63%|███████████████████████████████████████████████████████████████████████▌ | 431/681 [32:20<12:45, 3.06s/it] {'loss': 0.773, 'grad_norm': 33.201725006103516, 'learning_rate': 1.8035081928995788e-07, 'margin_dpo/margin_mean': 19.473228454589844, 'margin_dpo/margin_std': 14.711877822875977, 'logps/chosen': -81.04603576660156, 'logps/rejected': -129.18716430664062, 'logps/ref_chosen': -57.92179870605469, 'logps/ref_rejected': -86.5896987915039, 'logits/chosen': -2.278841972351074, 'logits/rejected': -2.0075526237487793, 'epoch': 0.63} 63%|███████████████████████████████████████████████████████████████████████▌ | 431/681 [32:20<12:45, 3.06s/it] 63%|███████████████████████████████████████████████████████████████████████▋ | 432/681 [32:23<12:54, 3.11s/it] {'loss': 0.5115, 'grad_norm': 22.698999404907227, 'learning_rate': 1.791192214186223e-07, 'margin_dpo/margin_mean': 25.235767364501953, 'margin_dpo/margin_std': 20.750383377075195, 'logps/chosen': -106.17467498779297, 'logps/rejected': -129.19357299804688, 'logps/ref_chosen': -83.56768798828125, 'logps/ref_rejected': -81.35081481933594, 'logits/chosen': -1.4666633605957031, 'logits/rejected': -0.8918869495391846, 'epoch': 0.63} 63%|███████████████████████████████████████████████████████████████████████▋ | 432/681 [32:23<12:54, 3.11s/it] 64%|███████████████████████████████████████████████████████████████████████▊ | 433/681 [32:26<12:29, 3.02s/it] {'loss': 0.836, 'grad_norm': 32.29205322265625, 'learning_rate': 1.7788949132172193e-07, 'margin_dpo/margin_mean': 22.7510929107666, 'margin_dpo/margin_std': 26.510454177856445, 'logps/chosen': -131.34445190429688, 'logps/rejected': -160.79852294921875, 'logps/ref_chosen': -101.5228042602539, 'logps/ref_rejected': -108.22576904296875, 'logits/chosen': -1.1999212503433228, 'logits/rejected': -0.5681695938110352, 'epoch': 0.64} 64%|███████████████████████████████████████████████████████████████████████▊ | 433/681 [32:26<12:29, 3.02s/it] 64%|████████████████████████████████████████████████████████████████████████ | 434/681 [32:29<12:35, 3.06s/it] {'loss': 0.7302, 'grad_norm': 34.036136627197266, 'learning_rate': 1.7666166140378853e-07, 'margin_dpo/margin_mean': 19.487464904785156, 'margin_dpo/margin_std': 28.65212631225586, 'logps/chosen': -147.2008056640625, 'logps/rejected': -154.53948974609375, 'logps/ref_chosen': -124.12574768066406, 'logps/ref_rejected': -111.97697448730469, 'logits/chosen': -2.054288387298584, 'logits/rejected': -1.399308204650879, 'epoch': 0.64} 64%|████████████████████████████████████████████████████████████████████████ | 434/681 [32:29<12:35, 3.06s/it] 64%|████████████████████████████████████████████████████████████████████████▏ | 435/681 [32:31<11:58, 2.92s/it] {'loss': 0.5765, 'grad_norm': 24.667842864990234, 'learning_rate': 1.7543576401928218e-07, 'margin_dpo/margin_mean': 27.645782470703125, 'margin_dpo/margin_std': 22.643753051757812, 'logps/chosen': -97.1856460571289, 'logps/rejected': -155.25845336914062, 'logps/ref_chosen': -75.44041442871094, 'logps/ref_rejected': -105.867431640625, 'logits/chosen': -1.8135147094726562, 'logits/rejected': -1.378133773803711, 'epoch': 0.64} 64%|████████████████████████████████████████████████████████████████████████▏ | 435/681 [32:32<11:58, 2.92s/it] 64%|████████████████████████████████████████████████████████████████████████▎ | 436/681 [32:35<12:18, 3.01s/it] {'loss': 0.4742, 'grad_norm': 21.6951961517334, 'learning_rate': 1.742118314717391e-07, 'margin_dpo/margin_mean': 24.306320190429688, 'margin_dpo/margin_std': 22.910158157348633, 'logps/chosen': -118.84077453613281, 'logps/rejected': -109.84593200683594, 'logps/ref_chosen': -98.83950805664062, 'logps/ref_rejected': -65.53834533691406, 'logits/chosen': -1.3429805040359497, 'logits/rejected': -0.3639574348926544, 'epoch': 0.64} 64%|████████████████████████████████████████████████████████████████████████▎ | 436/681 [32:35<12:18, 3.01s/it] 64%|████████████████████████████████████████████████████████████████████████▌ | 437/681 [32:38<12:23, 3.05s/it] {'loss': 0.6716, 'grad_norm': 21.48826789855957, 'learning_rate': 1.7298989601292036e-07, 'margin_dpo/margin_mean': 14.454059600830078, 'margin_dpo/margin_std': 16.443344116210938, 'logps/chosen': -110.85467529296875, 'logps/rejected': -124.95901489257812, 'logps/ref_chosen': -86.16108703613281, 'logps/ref_rejected': -85.81136322021484, 'logits/chosen': -2.178161144256592, 'logits/rejected': -1.9608675241470337, 'epoch': 0.64} 64%|████████████████████████████████████████████████████████████████████████▌ | 437/681 [32:38<12:23, 3.05s/it] 64%|████████████████████████████████████████████████████████████████████████▋ | 438/681 [32:41<11:59, 2.96s/it] {'loss': 0.5761, 'grad_norm': 27.280611038208008, 'learning_rate': 1.7176998984196144e-07, 'margin_dpo/margin_mean': 24.969541549682617, 'margin_dpo/margin_std': 16.327075958251953, 'logps/chosen': -97.4400634765625, 'logps/rejected': -137.0142364501953, 'logps/ref_chosen': -72.21536254882812, 'logps/ref_rejected': -86.81999206542969, 'logits/chosen': -1.978929042816162, 'logits/rejected': -1.4313712120056152, 'epoch': 0.64} 64%|████████████████████████████████████████████████████████████████████████▋ | 438/681 [32:41<11:59, 2.96s/it] 64%|████████████████████████████████████████████████████████████████████████▊ | 439/681 [32:43<11:50, 2.94s/it] {'loss': 0.7894, 'grad_norm': 29.87968635559082, 'learning_rate': 1.7055214510452458e-07, 'margin_dpo/margin_mean': 17.058311462402344, 'margin_dpo/margin_std': 18.533573150634766, 'logps/chosen': -119.00277709960938, 'logps/rejected': -112.55131530761719, 'logps/ref_chosen': -93.64883422851562, 'logps/ref_rejected': -70.13905334472656, 'logits/chosen': -1.223572015762329, 'logits/rejected': -0.7343321442604065, 'epoch': 0.64} 64%|████████████████████████████████████████████████████████████████████████▊ | 439/681 [32:43<11:50, 2.94s/it] 65%|█████████████████████████████████████████████████████████████████████████ | 440/681 [32:46<11:33, 2.88s/it] {'loss': 0.7767, 'grad_norm': 29.423545837402344, 'learning_rate': 1.6933639389195134e-07, 'margin_dpo/margin_mean': 24.033851623535156, 'margin_dpo/margin_std': 21.511178970336914, 'logps/chosen': -166.27590942382812, 'logps/rejected': -133.2398223876953, 'logps/ref_chosen': -143.11007690429688, 'logps/ref_rejected': -86.04011535644531, 'logits/chosen': -1.738755464553833, 'logits/rejected': -0.7158520817756653, 'epoch': 0.65} 65%|█████████████████████████████████████████████████████████████████████████ | 440/681 [32:46<11:33, 2.88s/it] 65%|█████████████████████████████████████████████████████████████████████████▏ | 441/681 [32:49<11:43, 2.93s/it] {'loss': 0.4617, 'grad_norm': 20.80038070678711, 'learning_rate': 1.681227682404166e-07, 'margin_dpo/margin_mean': 24.737010955810547, 'margin_dpo/margin_std': 11.764477729797363, 'logps/chosen': -132.18185424804688, 'logps/rejected': -119.72990417480469, 'logps/ref_chosen': -109.37970733642578, 'logps/ref_rejected': -72.19073486328125, 'logits/chosen': -1.494542121887207, 'logits/rejected': -0.6456449031829834, 'epoch': 0.65} 65%|█████████████████████████████████████████████████████████████████████████▏ | 441/681 [32:49<11:43, 2.93s/it] 65%|█████████████████████████████████████████████████████████████████████████▎ | 442/681 [32:52<11:44, 2.95s/it] {'loss': 0.4504, 'grad_norm': 19.458789825439453, 'learning_rate': 1.669113001300851e-07, 'margin_dpo/margin_mean': 27.243377685546875, 'margin_dpo/margin_std': 16.030248641967773, 'logps/chosen': -124.80497741699219, 'logps/rejected': -142.98641967773438, 'logps/ref_chosen': -98.47917175292969, 'logps/ref_rejected': -89.41722106933594, 'logits/chosen': -0.5713354349136353, 'logits/rejected': -0.13276606798171997, 'epoch': 0.65} 65%|█████████████████████████████████████████████████████████████████████████▎ | 442/681 [32:52<11:44, 2.95s/it] 65%|█████████████████████████████████████████████████████████████████████████▌ | 443/681 [32:55<11:54, 3.00s/it] {'loss': 0.6083, 'grad_norm': 31.2533016204834, 'learning_rate': 1.6570202148426815e-07, 'margin_dpo/margin_mean': 14.458542823791504, 'margin_dpo/margin_std': 14.307601928710938, 'logps/chosen': -128.31686401367188, 'logps/rejected': -118.65472412109375, 'logps/ref_chosen': -98.59529876708984, 'logps/ref_rejected': -74.47461700439453, 'logits/chosen': -1.5800436735153198, 'logits/rejected': -1.1696522235870361, 'epoch': 0.65} 65%|█████████████████████████████████████████████████████████████████████████▌ | 443/681 [32:55<11:54, 3.00s/it] 65%|█████████████████████████████████████████████████████████████████████████▋ | 444/681 [32:58<11:38, 2.95s/it] {'loss': 0.6585, 'grad_norm': 28.26342010498047, 'learning_rate': 1.6449496416858282e-07, 'margin_dpo/margin_mean': 17.695472717285156, 'margin_dpo/margin_std': 21.756973266601562, 'logps/chosen': -142.62533569335938, 'logps/rejected': -144.64889526367188, 'logps/ref_chosen': -111.86331939697266, 'logps/ref_rejected': -96.1914291381836, 'logits/chosen': -2.4062390327453613, 'logits/rejected': -1.701258897781372, 'epoch': 0.65} 65%|█████████████████████████████████████████████████████████████████████████▋ | 444/681 [32:58<11:38, 2.95s/it] 65%|█████████████████████████████████████████████████████████████████████████▊ | 445/681 [33:01<11:33, 2.94s/it] {'loss': 0.5487, 'grad_norm': 23.41632843017578, 'learning_rate': 1.6329015999011182e-07, 'margin_dpo/margin_mean': 15.405086517333984, 'margin_dpo/margin_std': 11.568341255187988, 'logps/chosen': -117.87601470947266, 'logps/rejected': -122.12147521972656, 'logps/ref_chosen': -89.98236083984375, 'logps/ref_rejected': -78.82273864746094, 'logits/chosen': -1.8755643367767334, 'logits/rejected': -1.7402284145355225, 'epoch': 0.65} 65%|█████████████████████████████████████████████████████████████████████████▊ | 445/681 [33:01<11:33, 2.94s/it] 65%|██████████████████████████████████████████████████████████████████████████ | 446/681 [33:04<11:34, 2.96s/it] {'loss': 0.6084, 'grad_norm': 30.221832275390625, 'learning_rate': 1.6208764069656578e-07, 'margin_dpo/margin_mean': 22.419845581054688, 'margin_dpo/margin_std': 18.770353317260742, 'logps/chosen': -107.57206726074219, 'logps/rejected': -175.65806579589844, 'logps/ref_chosen': -83.11263275146484, 'logps/ref_rejected': -128.77877807617188, 'logits/chosen': -0.7040866613388062, 'logits/rejected': -0.5425094366073608, 'epoch': 0.65} 65%|██████████████████████████████████████████████████████████████████████████ | 446/681 [33:04<11:34, 2.96s/it] 66%|██████████████████████████████████████████████████████████████████████████▏ | 447/681 [33:07<11:23, 2.92s/it] {'loss': 0.6788, 'grad_norm': 26.28083038330078, 'learning_rate': 1.608874379754465e-07, 'margin_dpo/margin_mean': 16.056018829345703, 'margin_dpo/margin_std': 15.311338424682617, 'logps/chosen': -103.27529907226562, 'logps/rejected': -145.8384552001953, 'logps/ref_chosen': -78.08226013183594, 'logps/ref_rejected': -104.58940887451172, 'logits/chosen': -1.621050238609314, 'logits/rejected': -1.6599102020263672, 'epoch': 0.66} 66%|██████████████████████████████████████████████████████████████████████████▏ | 447/681 [33:07<11:23, 2.92s/it] 66%|██████████████████████████████████████████████████████████████████████████▎ | 448/681 [33:10<11:21, 2.93s/it] {'loss': 0.6527, 'grad_norm': 31.752517700195312, 'learning_rate': 1.5968958345321177e-07, 'margin_dpo/margin_mean': 9.410964965820312, 'margin_dpo/margin_std': 19.52904510498047, 'logps/chosen': -130.1195068359375, 'logps/rejected': -171.00125122070312, 'logps/ref_chosen': -102.85002136230469, 'logps/ref_rejected': -134.32081604003906, 'logits/chosen': -1.1484473943710327, 'logits/rejected': -1.2155144214630127, 'epoch': 0.66} 66%|██████████████████████████████████████████████████████████████████████████▎ | 448/681 [33:10<11:21, 2.93s/it] 66%|██████████████████████████████████████████████████████████████████████████▌ | 449/681 [33:13<11:20, 2.94s/it] {'loss': 0.6548, 'grad_norm': 26.76458740234375, 'learning_rate': 1.584941086944423e-07, 'margin_dpo/margin_mean': 22.50650978088379, 'margin_dpo/margin_std': 17.623516082763672, 'logps/chosen': -114.85504913330078, 'logps/rejected': -131.17945861816406, 'logps/ref_chosen': -87.11476135253906, 'logps/ref_rejected': -80.93266296386719, 'logits/chosen': -1.1806657314300537, 'logits/rejected': -0.7023051381111145, 'epoch': 0.66} 66%|██████████████████████████████████████████████████████████████████████████▌ | 449/681 [33:13<11:20, 2.94s/it] 66%|██████████████████████████████████████████████████████████████████████████▋ | 450/681 [33:16<11:24, 2.97s/it] {'loss': 0.4745, 'grad_norm': 24.460163116455078, 'learning_rate': 1.573010452010098e-07, 'margin_dpo/margin_mean': 33.907779693603516, 'margin_dpo/margin_std': 16.30735206604004, 'logps/chosen': -136.053466796875, 'logps/rejected': -204.9149627685547, 'logps/ref_chosen': -114.85942077636719, 'logps/ref_rejected': -149.81314086914062, 'logits/chosen': -2.825883626937866, 'logits/rejected': -2.340266227722168, 'epoch': 0.66} 66%|██████████████████████████████████████████████████████████████████████████▋ | 450/681 [33:16<11:24, 2.97s/it] 66%|██████████████████████████████████████████████████████████████████████████▊ | 451/681 [33:19<11:02, 2.88s/it] {'loss': 0.7374, 'grad_norm': 28.61817741394043, 'learning_rate': 1.5611042441124687e-07, 'margin_dpo/margin_mean': 28.783344268798828, 'margin_dpo/margin_std': 21.23330307006836, 'logps/chosen': -136.8935546875, 'logps/rejected': -156.86073303222656, 'logps/ref_chosen': -114.61380767822266, 'logps/ref_rejected': -105.7976303100586, 'logits/chosen': -1.5507322549819946, 'logits/rejected': -1.089903473854065, 'epoch': 0.66} 66%|██████████████████████████████████████████████████████████████████████████▊ | 451/681 [33:19<11:02, 2.88s/it] 66%|███████████████████████████████████████████████████████████████████████████ | 452/681 [33:21<10:57, 2.87s/it] {'loss': 0.5899, 'grad_norm': 22.830219268798828, 'learning_rate': 1.549222776991186e-07, 'margin_dpo/margin_mean': 16.93113899230957, 'margin_dpo/margin_std': 14.689157485961914, 'logps/chosen': -104.45685577392578, 'logps/rejected': -140.10707092285156, 'logps/ref_chosen': -82.06895446777344, 'logps/ref_rejected': -100.78804016113281, 'logits/chosen': -2.039238929748535, 'logits/rejected': -1.8188178539276123, 'epoch': 0.66} 66%|███████████████████████████████████████████████████████████████████████████ | 452/681 [33:21<10:57, 2.87s/it] 67%|███████████████████████████████████████████████████████████████████████████▏ | 453/681 [33:24<10:40, 2.81s/it] {'loss': 0.4381, 'grad_norm': 18.594497680664062, 'learning_rate': 1.5373663637339584e-07, 'margin_dpo/margin_mean': 20.605459213256836, 'margin_dpo/margin_std': 11.411565780639648, 'logps/chosen': -106.26356506347656, 'logps/rejected': -120.48109436035156, 'logps/ref_chosen': -78.82481384277344, 'logps/ref_rejected': -72.43688201904297, 'logits/chosen': -1.2692456245422363, 'logits/rejected': -1.0498113632202148, 'epoch': 0.67} 67%|███████████████████████████████████████████████████████████████████████████▏ | 453/681 [33:24<10:40, 2.81s/it] 67%|███████████████████████████████████████████████████████████████████████████▎ | 454/681 [33:27<10:41, 2.83s/it] {'loss': 0.7431, 'grad_norm': 34.69410705566406, 'learning_rate': 1.5255353167683017e-07, 'margin_dpo/margin_mean': 16.760404586791992, 'margin_dpo/margin_std': 17.026487350463867, 'logps/chosen': -133.55905151367188, 'logps/rejected': -164.92752075195312, 'logps/ref_chosen': -101.30783081054688, 'logps/ref_rejected': -115.9158935546875, 'logits/chosen': -1.726050615310669, 'logits/rejected': -1.4121358394622803, 'epoch': 0.67} 67%|███████████████████████████████████████████████████████████████████████████▎ | 454/681 [33:27<10:41, 2.83s/it] 67%|███████████████████████████████████████████████████████████████████████████▍ | 455/681 [33:30<10:52, 2.89s/it] {'loss': 0.5704, 'grad_norm': 27.073896408081055, 'learning_rate': 1.5137299478533064e-07, 'margin_dpo/margin_mean': 25.370182037353516, 'margin_dpo/margin_std': 24.86716651916504, 'logps/chosen': -107.57990264892578, 'logps/rejected': -218.08432006835938, 'logps/ref_chosen': -80.74185943603516, 'logps/ref_rejected': -165.87608337402344, 'logits/chosen': -2.262284994125366, 'logits/rejected': -2.4119272232055664, 'epoch': 0.67} 67%|███████████████████████████████████████████████████████████████████████████▍ | 455/681 [33:30<10:52, 2.89s/it] 67%|███████████████████████████████████████████████████████████████████████████▋ | 456/681 [33:33<10:47, 2.88s/it] {'loss': 0.5251, 'grad_norm': 24.588420867919922, 'learning_rate': 1.5019505680714232e-07, 'margin_dpo/margin_mean': 20.554821014404297, 'margin_dpo/margin_std': 16.57241439819336, 'logps/chosen': -104.88959503173828, 'logps/rejected': -166.04608154296875, 'logps/ref_chosen': -83.01458740234375, 'logps/ref_rejected': -123.61626434326172, 'logits/chosen': -1.48960542678833, 'logits/rejected': -1.4934964179992676, 'epoch': 0.67} 67%|███████████████████████████████████████████████████████████████████████████▋ | 456/681 [33:33<10:47, 2.88s/it] 67%|███████████████████████████████████████████████████████████████████████████▊ | 457/681 [33:36<11:04, 2.97s/it] {'loss': 0.4447, 'grad_norm': 19.552265167236328, 'learning_rate': 1.4901974878202627e-07, 'margin_dpo/margin_mean': 27.605770111083984, 'margin_dpo/margin_std': 16.504793167114258, 'logps/chosen': -110.44456481933594, 'logps/rejected': -149.5691680908203, 'logps/ref_chosen': -89.19579315185547, 'logps/ref_rejected': -100.71463012695312, 'logits/chosen': -0.9713010787963867, 'logits/rejected': -0.8292329907417297, 'epoch': 0.67} 67%|███████████████████████████████████████████████████████████████████████████▊ | 457/681 [33:36<11:04, 2.97s/it] 67%|███████████████████████████████████████████████████████████████████████████▉ | 458/681 [33:39<10:57, 2.95s/it] {'loss': 0.703, 'grad_norm': 33.34504318237305, 'learning_rate': 1.4784710168044212e-07, 'margin_dpo/margin_mean': 15.376252174377441, 'margin_dpo/margin_std': 23.105487823486328, 'logps/chosen': -123.5889663696289, 'logps/rejected': -135.5701904296875, 'logps/ref_chosen': -93.04997253417969, 'logps/ref_rejected': -89.65494537353516, 'logits/chosen': -1.010892629623413, 'logits/rejected': -0.6925811767578125, 'epoch': 0.67} 67%|███████████████████████████████████████████████████████████████████████████▉ | 458/681 [33:39<10:57, 2.95s/it] 67%|████████████████████████████████████████████████████████████████████████████▏ | 459/681 [33:42<10:57, 2.96s/it] {'loss': 0.5821, 'grad_norm': 25.338581085205078, 'learning_rate': 1.466771464027316e-07, 'margin_dpo/margin_mean': 18.215126037597656, 'margin_dpo/margin_std': 16.228425979614258, 'logps/chosen': -102.30770874023438, 'logps/rejected': -137.26004028320312, 'logps/ref_chosen': -77.70928192138672, 'logps/ref_rejected': -94.44648742675781, 'logits/chosen': -2.277261734008789, 'logits/rejected': -1.876206636428833, 'epoch': 0.67} 67%|████████████████████████████████████████████████████████████████████████████▏ | 459/681 [33:42<10:57, 2.96s/it] 68%|████████████████████████████████████████████████████████████████████████████▎ | 460/681 [33:45<11:00, 2.99s/it] {'loss': 0.6881, 'grad_norm': 31.598941802978516, 'learning_rate': 1.4550991377830423e-07, 'margin_dpo/margin_mean': 17.550220489501953, 'margin_dpo/margin_std': 15.671613693237305, 'logps/chosen': -120.14390563964844, 'logps/rejected': -166.2212371826172, 'logps/ref_chosen': -86.8564453125, 'logps/ref_rejected': -115.38356018066406, 'logits/chosen': -1.5439469814300537, 'logits/rejected': -1.373673439025879, 'epoch': 0.68} 68%|████████████████████████████████████████████████████████████████████████████▎ | 460/681 [33:45<11:00, 2.99s/it] 68%|████████████████████████████████████████████████████████████████████████████▍ | 461/681 [33:48<11:01, 3.01s/it] {'loss': 0.7707, 'grad_norm': 29.087486267089844, 'learning_rate': 1.4434543456482518e-07, 'margin_dpo/margin_mean': 15.786758422851562, 'margin_dpo/margin_std': 16.66317367553711, 'logps/chosen': -142.9497833251953, 'logps/rejected': -160.65652465820312, 'logps/ref_chosen': -110.2314224243164, 'logps/ref_rejected': -112.15139770507812, 'logits/chosen': -1.0304532051086426, 'logits/rejected': -0.7353922128677368, 'epoch': 0.68} 68%|████████████████████████████████████████████████████████████████████████████▍ | 461/681 [33:48<11:01, 3.01s/it] 68%|████████████████████████████████████████████████████████████████████████████▋ | 462/681 [33:51<10:46, 2.95s/it] {'loss': 0.5766, 'grad_norm': 25.08791160583496, 'learning_rate': 1.4318373944740484e-07, 'margin_dpo/margin_mean': 18.264904022216797, 'margin_dpo/margin_std': 16.625919342041016, 'logps/chosen': -106.20613098144531, 'logps/rejected': -121.96664428710938, 'logps/ref_chosen': -79.80224609375, 'logps/ref_rejected': -77.2978515625, 'logits/chosen': -2.122267723083496, 'logits/rejected': -1.8464157581329346, 'epoch': 0.68} 68%|████████████████████████████████████████████████████████████████████████████▋ | 462/681 [33:51<10:46, 2.95s/it] 68%|████████████████████████████████████████████████████████████████████████████▊ | 463/681 [33:54<10:39, 2.93s/it] {'loss': 0.5968, 'grad_norm': 25.841707229614258, 'learning_rate': 1.4202485903778976e-07, 'margin_dpo/margin_mean': 22.628652572631836, 'margin_dpo/margin_std': 19.861661911010742, 'logps/chosen': -93.4617919921875, 'logps/rejected': -121.64329528808594, 'logps/ref_chosen': -71.40604400634766, 'logps/ref_rejected': -76.95889282226562, 'logits/chosen': -2.0914337635040283, 'logits/rejected': -1.7170106172561646, 'epoch': 0.68} 68%|████████████████████████████████████████████████████████████████████████████▊ | 463/681 [33:54<10:39, 2.93s/it] 68%|████████████████████████████████████████████████████████████████████████████▉ | 464/681 [33:56<10:24, 2.88s/it] {'loss': 0.6442, 'grad_norm': 29.462730407714844, 'learning_rate': 1.4086882387355658e-07, 'margin_dpo/margin_mean': 15.728658676147461, 'margin_dpo/margin_std': 12.379182815551758, 'logps/chosen': -125.587158203125, 'logps/rejected': -170.02841186523438, 'logps/ref_chosen': -92.85479736328125, 'logps/ref_rejected': -121.56739807128906, 'logits/chosen': -1.1162123680114746, 'logits/rejected': -1.4878854751586914, 'epoch': 0.68} 68%|████████████████████████████████████████████████████████████████████████████▉ | 464/681 [33:57<10:24, 2.88s/it] 68%|█████████████████████████████████████████████████████████████████████████████▏ | 465/681 [34:00<10:32, 2.93s/it] {'loss': 0.4936, 'grad_norm': 24.474634170532227, 'learning_rate': 1.3971566441730714e-07, 'margin_dpo/margin_mean': 23.03533172607422, 'margin_dpo/margin_std': 21.698204040527344, 'logps/chosen': -112.57616424560547, 'logps/rejected': -143.59713745117188, 'logps/ref_chosen': -86.05903625488281, 'logps/ref_rejected': -94.044677734375, 'logits/chosen': -1.3897593021392822, 'logits/rejected': -1.0794075727462769, 'epoch': 0.68} 68%|█████████████████████████████████████████████████████████████████████████████▏ | 465/681 [34:00<10:32, 2.93s/it] 68%|█████████████████████████████████████████████████████████████████████████████▎ | 466/681 [34:03<10:40, 2.98s/it] {'loss': 0.5221, 'grad_norm': 25.171062469482422, 'learning_rate': 1.3856541105586545e-07, 'margin_dpo/margin_mean': 24.440139770507812, 'margin_dpo/margin_std': 19.990158081054688, 'logps/chosen': -118.66007232666016, 'logps/rejected': -123.89339447021484, 'logps/ref_chosen': -93.65107727050781, 'logps/ref_rejected': -74.44425964355469, 'logits/chosen': -1.600342035293579, 'logits/rejected': -0.7873063087463379, 'epoch': 0.68} 68%|█████████████████████████████████████████████████████████████████████████████▎ | 466/681 [34:03<10:40, 2.98s/it] 69%|█████████████████████████████████████████████████████████████████████████████▍ | 467/681 [34:05<10:30, 2.95s/it] {'loss': 0.5664, 'grad_norm': 24.633567810058594, 'learning_rate': 1.3741809409947729e-07, 'margin_dpo/margin_mean': 29.18516731262207, 'margin_dpo/margin_std': 19.719329833984375, 'logps/chosen': -127.40275573730469, 'logps/rejected': -163.00355529785156, 'logps/ref_chosen': -100.60771179199219, 'logps/ref_rejected': -107.02332305908203, 'logits/chosen': -1.2185485363006592, 'logits/rejected': -0.803144097328186, 'epoch': 0.69} 69%|█████████████████████████████████████████████████████████████████████████████▍ | 467/681 [34:06<10:30, 2.95s/it] 69%|█████████████████████████████████████████████████████████████████████████████▋ | 468/681 [34:09<10:36, 2.99s/it] {'loss': 0.7409, 'grad_norm': 31.386714935302734, 'learning_rate': 1.362737437810114e-07, 'margin_dpo/margin_mean': 20.559078216552734, 'margin_dpo/margin_std': 20.951251983642578, 'logps/chosen': -118.3271255493164, 'logps/rejected': -163.67080688476562, 'logps/ref_chosen': -96.5201187133789, 'logps/ref_rejected': -121.30470275878906, 'logits/chosen': -1.2606921195983887, 'logits/rejected': -1.089573621749878, 'epoch': 0.69} 69%|█████████████████████████████████████████████████████████████████████████████▋ | 468/681 [34:09<10:36, 2.99s/it] 69%|█████████████████████████████████████████████████████████████████████████████▊ | 469/681 [34:12<10:35, 3.00s/it] {'loss': 0.5459, 'grad_norm': 27.293046951293945, 'learning_rate': 1.351323902551631e-07, 'margin_dpo/margin_mean': 21.50157928466797, 'margin_dpo/margin_std': 19.085735321044922, 'logps/chosen': -88.2209701538086, 'logps/rejected': -139.1120147705078, 'logps/ref_chosen': -57.2293586730957, 'logps/ref_rejected': -86.61882019042969, 'logits/chosen': -1.0601452589035034, 'logits/rejected': -1.0791672468185425, 'epoch': 0.69} 69%|█████████████████████████████████████████████████████████████████████████████▊ | 469/681 [34:12<10:35, 3.00s/it] 69%|█████████████████████████████████████████████████████████████████████████████▉ | 470/681 [34:14<10:14, 2.91s/it] {'loss': 0.5101, 'grad_norm': 22.850967407226562, 'learning_rate': 1.339940635976592e-07, 'margin_dpo/margin_mean': 20.166732788085938, 'margin_dpo/margin_std': 13.451013565063477, 'logps/chosen': -112.50442504882812, 'logps/rejected': -117.85956573486328, 'logps/ref_chosen': -87.85397338867188, 'logps/ref_rejected': -73.04238891601562, 'logits/chosen': -0.9972758889198303, 'logits/rejected': -0.3706229329109192, 'epoch': 0.69} 69%|█████████████████████████████████████████████████████████████████████████████▉ | 470/681 [34:14<10:14, 2.91s/it] 69%|██████████████████████████████████████████████████████████████████████████████▏ | 471/681 [34:17<10:02, 2.87s/it] {'loss': 0.4371, 'grad_norm': 20.220306396484375, 'learning_rate': 1.3285879380446563e-07, 'margin_dpo/margin_mean': 22.2884521484375, 'margin_dpo/margin_std': 12.328178405761719, 'logps/chosen': -130.043701171875, 'logps/rejected': -165.02394104003906, 'logps/ref_chosen': -98.69927978515625, 'logps/ref_rejected': -111.39106750488281, 'logits/chosen': -1.0001336336135864, 'logits/rejected': -1.0569403171539307, 'epoch': 0.69} 69%|██████████████████████████████████████████████████████████████████████████████▏ | 471/681 [34:17<10:02, 2.87s/it] 69%|██████████████████████████████████████████████████████████████████████████████▎ | 472/681 [34:20<10:27, 3.00s/it] {'loss': 0.4734, 'grad_norm': 27.954729080200195, 'learning_rate': 1.317266107909975e-07, 'margin_dpo/margin_mean': 21.763275146484375, 'margin_dpo/margin_std': 16.120174407958984, 'logps/chosen': -180.3785400390625, 'logps/rejected': -163.5217742919922, 'logps/ref_chosen': -154.4603271484375, 'logps/ref_rejected': -115.84030151367188, 'logits/chosen': -1.4778022766113281, 'logits/rejected': -0.5768337249755859, 'epoch': 0.69} 69%|██████████████████████████████████████████████████████████████████████████████▎ | 472/681 [34:20<10:27, 3.00s/it] 69%|██████████████████████████████████████████████████████████████████████████████▍ | 473/681 [34:24<10:38, 3.07s/it] {'loss': 0.6772, 'grad_norm': 27.106035232543945, 'learning_rate': 1.3059754439133002e-07, 'margin_dpo/margin_mean': 14.2333345413208, 'margin_dpo/margin_std': 18.283645629882812, 'logps/chosen': -156.43284606933594, 'logps/rejected': -144.3042449951172, 'logps/ref_chosen': -124.94532775878906, 'logps/ref_rejected': -98.58340454101562, 'logits/chosen': -1.175186276435852, 'logits/rejected': -0.891939640045166, 'epoch': 0.69} 69%|██████████████████████████████████████████████████████████████████████████████▍ | 473/681 [34:24<10:38, 3.07s/it] 70%|██████████████████████████████████████████████████████████████████████████████▋ | 474/681 [34:27<10:42, 3.11s/it] {'loss': 0.9112, 'grad_norm': 41.271942138671875, 'learning_rate': 1.2947162435741277e-07, 'margin_dpo/margin_mean': 21.427288055419922, 'margin_dpo/margin_std': 19.77497673034668, 'logps/chosen': -98.8539810180664, 'logps/rejected': -183.00289916992188, 'logps/ref_chosen': -68.98202514648438, 'logps/ref_rejected': -131.7036590576172, 'logits/chosen': -1.574142575263977, 'logits/rejected': -1.827897071838379, 'epoch': 0.7} 70%|██████████████████████████████████████████████████████████████████████████████▋ | 474/681 [34:27<10:42, 3.11s/it] 70%|██████████████████████████████████████████████████████████████████████████████▊ | 475/681 [34:30<10:20, 3.01s/it] {'loss': 0.6614, 'grad_norm': 27.248205184936523, 'learning_rate': 1.2834888035828596e-07, 'margin_dpo/margin_mean': 17.609081268310547, 'margin_dpo/margin_std': 17.966053009033203, 'logps/chosen': -99.45306396484375, 'logps/rejected': -138.05735778808594, 'logps/ref_chosen': -73.41651916503906, 'logps/ref_rejected': -94.41173553466797, 'logits/chosen': -2.1355481147766113, 'logits/rejected': -2.075871229171753, 'epoch': 0.7} 70%|██████████████████████████████████████████████████████████████████████████████▊ | 475/681 [34:30<10:20, 3.01s/it] 70%|██████████████████████████████████████████████████████████████████████████████▉ | 476/681 [34:32<10:10, 2.98s/it] {'loss': 0.666, 'grad_norm': 22.6929874420166, 'learning_rate': 1.2722934197929802e-07, 'margin_dpo/margin_mean': 22.775375366210938, 'margin_dpo/margin_std': 21.31452178955078, 'logps/chosen': -102.75193786621094, 'logps/rejected': -143.93138122558594, 'logps/ref_chosen': -74.20452117919922, 'logps/ref_rejected': -92.60858917236328, 'logits/chosen': -2.7777254581451416, 'logits/rejected': -2.2750704288482666, 'epoch': 0.7} 70%|██████████████████████████████████████████████████████████████████████████████▉ | 476/681 [34:33<10:10, 2.98s/it] 70%|███████████████████████████████████████████████████████████████████████████████▏ | 477/681 [34:35<10:03, 2.96s/it] {'loss': 0.5533, 'grad_norm': 24.573259353637695, 'learning_rate': 1.2611303872132631e-07, 'margin_dpo/margin_mean': 23.8519287109375, 'margin_dpo/margin_std': 17.174068450927734, 'logps/chosen': -130.92428588867188, 'logps/rejected': -120.82994079589844, 'logps/ref_chosen': -102.4937515258789, 'logps/ref_rejected': -68.5474853515625, 'logits/chosen': -3.4070348739624023, 'logits/rejected': -2.3290576934814453, 'epoch': 0.7} 70%|███████████████████████████████████████████████████████████████████████████████▏ | 477/681 [34:35<10:03, 2.96s/it] 70%|███████████████████████████████████████████████████████████████████████████████▎ | 478/681 [34:39<10:17, 3.04s/it] {'loss': 0.5916, 'grad_norm': 22.8625545501709, 'learning_rate': 1.2500000000000005e-07, 'margin_dpo/margin_mean': 12.25357437133789, 'margin_dpo/margin_std': 14.410320281982422, 'logps/chosen': -93.35462951660156, 'logps/rejected': -111.02110290527344, 'logps/ref_chosen': -64.2506103515625, 'logps/ref_rejected': -69.66351318359375, 'logits/chosen': -1.4574041366577148, 'logits/rejected': -1.3147239685058594, 'epoch': 0.7} 70%|███████████████████████████████████████████████████████████████████████████████▎ | 478/681 [34:39<10:17, 3.04s/it] 70%|███████████████████████████████████████████████████████████████████████████████▍ | 479/681 [34:41<09:57, 2.96s/it] {'loss': 0.6646, 'grad_norm': 27.097768783569336, 'learning_rate': 1.2389025514492456e-07, 'margin_dpo/margin_mean': 20.42224884033203, 'margin_dpo/margin_std': 16.232749938964844, 'logps/chosen': -101.08732604980469, 'logps/rejected': -144.12460327148438, 'logps/ref_chosen': -73.15229034423828, 'logps/ref_rejected': -95.76731872558594, 'logits/chosen': -1.1350903511047363, 'logits/rejected': -1.2377521991729736, 'epoch': 0.7} 70%|███████████████████████████████████████████████████████████████████████████████▍ | 479/681 [34:41<09:57, 2.96s/it] 70%|███████████████████████████████████████████████████████████████████████████████▋ | 480/681 [34:44<09:47, 2.92s/it] {'loss': 0.5901, 'grad_norm': 27.788637161254883, 'learning_rate': 1.227838333989088e-07, 'margin_dpo/margin_mean': 23.498985290527344, 'margin_dpo/margin_std': 15.09835433959961, 'logps/chosen': -117.9144515991211, 'logps/rejected': -129.16677856445312, 'logps/ref_chosen': -90.6583251953125, 'logps/ref_rejected': -78.41166687011719, 'logits/chosen': -0.924386739730835, 'logits/rejected': -0.4081732928752899, 'epoch': 0.7} 70%|███████████████████████████████████████████████████████████████████████████████▋ | 480/681 [34:44<09:47, 2.92s/it] 71%|███████████████████████████████████████████████████████████████████████████████▊ | 481/681 [34:47<09:45, 2.93s/it] {'loss': 0.595, 'grad_norm': 25.31952476501465, 'learning_rate': 1.2168076391719489e-07, 'margin_dpo/margin_mean': 13.803601264953613, 'margin_dpo/margin_std': 14.735580444335938, 'logps/chosen': -112.26165771484375, 'logps/rejected': -123.01365661621094, 'logps/ref_chosen': -77.6314468383789, 'logps/ref_rejected': -74.5798568725586, 'logits/chosen': -1.2236384153366089, 'logits/rejected': -0.7734112739562988, 'epoch': 0.71} 71%|███████████████████████████████████████████████████████████████████████████████▊ | 481/681 [34:47<09:45, 2.93s/it] 71%|███████████████████████████████████████████████████████████████████████████████▉ | 482/681 [34:50<09:54, 2.99s/it] {'loss': 0.705, 'grad_norm': 30.939720153808594, 'learning_rate': 1.2058107576668938e-07, 'margin_dpo/margin_mean': 16.930465698242188, 'margin_dpo/margin_std': 15.111373901367188, 'logps/chosen': -132.59231567382812, 'logps/rejected': -150.3515625, 'logps/ref_chosen': -99.90892791748047, 'logps/ref_rejected': -100.73772430419922, 'logits/chosen': -1.545617938041687, 'logits/rejected': -1.4013819694519043, 'epoch': 0.71} 71%|███████████████████████████████████████████████████████████████████████████████▉ | 482/681 [34:50<09:54, 2.99s/it] 71%|████████████████████████████████████████████████████████████████████████████████▏ | 483/681 [34:53<09:43, 2.95s/it] {'loss': 0.523, 'grad_norm': 23.95891571044922, 'learning_rate': 1.194847979251979e-07, 'margin_dpo/margin_mean': 27.50977325439453, 'margin_dpo/margin_std': 21.83698272705078, 'logps/chosen': -138.04351806640625, 'logps/rejected': -157.34286499023438, 'logps/ref_chosen': -114.92469787597656, 'logps/ref_rejected': -106.71426391601562, 'logits/chosen': -2.434985637664795, 'logits/rejected': -1.8117592334747314, 'epoch': 0.71} 71%|████████████████████████████████████████████████████████████████████████████████▏ | 483/681 [34:53<09:43, 2.95s/it] 71%|████████████████████████████████████████████████████████████████████████████████▎ | 484/681 [34:56<09:22, 2.86s/it] {'loss': 0.5374, 'grad_norm': 20.260234832763672, 'learning_rate': 1.1839195928066101e-07, 'margin_dpo/margin_mean': 20.175811767578125, 'margin_dpo/margin_std': 17.458547592163086, 'logps/chosen': -112.14205932617188, 'logps/rejected': -114.17668914794922, 'logps/ref_chosen': -80.64376831054688, 'logps/ref_rejected': -62.502586364746094, 'logits/chosen': -2.028228759765625, 'logits/rejected': -1.2856285572052002, 'epoch': 0.71} 71%|████████████████████████████████████████████████████████████████████████████████▎ | 484/681 [34:56<09:22, 2.86s/it] 71%|████████████████████████████████████████████████████████████████████████████████▍ | 485/681 [34:59<09:19, 2.85s/it] {'loss': 0.3976, 'grad_norm': 21.97205924987793, 'learning_rate': 1.1730258863039347e-07, 'margin_dpo/margin_mean': 29.333690643310547, 'margin_dpo/margin_std': 20.77867889404297, 'logps/chosen': -116.61924743652344, 'logps/rejected': -165.03895568847656, 'logps/ref_chosen': -92.76929473876953, 'logps/ref_rejected': -111.85530090332031, 'logits/chosen': -1.9410022497177124, 'logits/rejected': -1.6192748546600342, 'epoch': 0.71} 71%|████████████████████████████████████████████████████████████████████████████████▍ | 485/681 [34:59<09:19, 2.85s/it] 71%|████████████████████████████████████████████████████████████████████████████████▋ | 486/681 [35:01<08:55, 2.75s/it] {'loss': 0.5486, 'grad_norm': 28.425851821899414, 'learning_rate': 1.1621671468032493e-07, 'margin_dpo/margin_mean': 31.947128295898438, 'margin_dpo/margin_std': 18.083139419555664, 'logps/chosen': -123.26519012451172, 'logps/rejected': -155.62200927734375, 'logps/ref_chosen': -100.41597747802734, 'logps/ref_rejected': -100.8256607055664, 'logits/chosen': -1.306410312652588, 'logits/rejected': -0.296735554933548, 'epoch': 0.71} 71%|████████████████████████████████████████████████████████████████████████████████▋ | 486/681 [35:01<08:55, 2.75s/it] 72%|████████████████████████████████████████████████████████████████████████████████▊ | 487/681 [35:04<09:14, 2.86s/it] {'loss': 0.4507, 'grad_norm': 23.94493293762207, 'learning_rate': 1.1513436604424378e-07, 'margin_dpo/margin_mean': 27.979373931884766, 'margin_dpo/margin_std': 18.914302825927734, 'logps/chosen': -131.52816772460938, 'logps/rejected': -166.06118774414062, 'logps/ref_chosen': -100.6910400390625, 'logps/ref_rejected': -107.24468994140625, 'logits/chosen': -2.2987453937530518, 'logits/rejected': -1.9822089672088623, 'epoch': 0.72} 72%|████████████████████████████████████████████████████████████████████████████████▊ | 487/681 [35:04<09:14, 2.86s/it] 72%|████████████████████████████████████████████████████████████████████████████████▉ | 488/681 [35:07<09:25, 2.93s/it] {'loss': 0.5527, 'grad_norm': 27.051164627075195, 'learning_rate': 1.1405557124304335e-07, 'margin_dpo/margin_mean': 19.399627685546875, 'margin_dpo/margin_std': 15.338823318481445, 'logps/chosen': -131.36257934570312, 'logps/rejected': -148.592529296875, 'logps/ref_chosen': -107.36306762695312, 'logps/ref_rejected': -105.19340515136719, 'logits/chosen': -1.088942527770996, 'logits/rejected': -0.8271230459213257, 'epoch': 0.72} 72%|████████████████████████████████████████████████████████████████████████████████▉ | 488/681 [35:07<09:25, 2.93s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▏ | 489/681 [35:10<09:20, 2.92s/it] {'loss': 0.6173, 'grad_norm': 26.43874168395996, 'learning_rate': 1.1298035870396985e-07, 'margin_dpo/margin_mean': 16.190275192260742, 'margin_dpo/margin_std': 13.990297317504883, 'logps/chosen': -119.86599731445312, 'logps/rejected': -122.27718353271484, 'logps/ref_chosen': -95.271240234375, 'logps/ref_rejected': -81.49215698242188, 'logits/chosen': -0.9547609090805054, 'logits/rejected': -0.39513295888900757, 'epoch': 0.72} 72%|█████████████████████████████████████████████████████████████████████████████████▏ | 489/681 [35:10<09:20, 2.92s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▎ | 490/681 [35:13<09:28, 2.97s/it] {'loss': 0.7967, 'grad_norm': 29.264699935913086, 'learning_rate': 1.1190875675987355e-07, 'margin_dpo/margin_mean': 15.706191062927246, 'margin_dpo/margin_std': 23.679283142089844, 'logps/chosen': -112.51838684082031, 'logps/rejected': -191.67208862304688, 'logps/ref_chosen': -85.32609558105469, 'logps/ref_rejected': -148.7736053466797, 'logits/chosen': -2.5108864307403564, 'logits/rejected': -2.8467113971710205, 'epoch': 0.72} 72%|█████████████████████████████████████████████████████████████████████████████████▎ | 490/681 [35:13<09:28, 2.97s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▍ | 491/681 [35:16<09:14, 2.92s/it] {'loss': 0.7306, 'grad_norm': 28.753334045410156, 'learning_rate': 1.1084079364846241e-07, 'margin_dpo/margin_mean': 23.228939056396484, 'margin_dpo/margin_std': 21.352033615112305, 'logps/chosen': -135.39317321777344, 'logps/rejected': -144.2461700439453, 'logps/ref_chosen': -105.19956970214844, 'logps/ref_rejected': -90.82363891601562, 'logits/chosen': -1.7324990034103394, 'logits/rejected': -1.075792670249939, 'epoch': 0.72} 72%|█████████████████████████████████████████████████████████████████████████████████▍ | 491/681 [35:16<09:14, 2.92s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▋ | 492/681 [35:19<09:12, 2.92s/it] {'loss': 0.7929, 'grad_norm': 26.195093154907227, 'learning_rate': 1.097764975115576e-07, 'margin_dpo/margin_mean': 18.426122665405273, 'margin_dpo/margin_std': 19.221683502197266, 'logps/chosen': -91.94832611083984, 'logps/rejected': -113.68507385253906, 'logps/ref_chosen': -66.54839324951172, 'logps/ref_rejected': -69.85902404785156, 'logits/chosen': -2.059189558029175, 'logits/rejected': -1.671616792678833, 'epoch': 0.72} 72%|█████████████████████████████████████████████████████████████████████████████████▋ | 492/681 [35:19<09:12, 2.92s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▊ | 493/681 [35:22<09:18, 2.97s/it] {'loss': 0.5993, 'grad_norm': 27.575870513916016, 'learning_rate': 1.0871589639435203e-07, 'margin_dpo/margin_mean': 26.582361221313477, 'margin_dpo/margin_std': 20.35706329345703, 'logps/chosen': -162.7125244140625, 'logps/rejected': -184.400390625, 'logps/ref_chosen': -138.72125244140625, 'logps/ref_rejected': -133.82675170898438, 'logits/chosen': -1.1041479110717773, 'logits/rejected': -0.9681916236877441, 'epoch': 0.72} 72%|█████████████████████████████████████████████████████████████████████████████████▊ | 493/681 [35:22<09:18, 2.97s/it] 73%|█████████████████████████████████████████████████████████████████████████████████▉ | 494/681 [35:25<08:57, 2.87s/it] {'loss': 0.5601, 'grad_norm': 23.7596492767334, 'learning_rate': 1.0765901824467166e-07, 'margin_dpo/margin_mean': 31.34137535095215, 'margin_dpo/margin_std': 21.264366149902344, 'logps/chosen': -97.23701477050781, 'logps/rejected': -135.43707275390625, 'logps/ref_chosen': -66.82516479492188, 'logps/ref_rejected': -73.68384552001953, 'logits/chosen': -0.6587156057357788, 'logits/rejected': -0.5145885348320007, 'epoch': 0.73} 73%|█████████████████████████████████████████████████████████████████████████████████▉ | 494/681 [35:25<08:57, 2.87s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▏ | 495/681 [35:28<09:04, 2.93s/it] {'loss': 0.7539, 'grad_norm': 34.367488861083984, 'learning_rate': 1.0660589091223854e-07, 'margin_dpo/margin_mean': 18.730424880981445, 'margin_dpo/margin_std': 13.805540084838867, 'logps/chosen': -108.4244613647461, 'logps/rejected': -108.88243103027344, 'logps/ref_chosen': -87.1620864868164, 'logps/ref_rejected': -68.8896255493164, 'logits/chosen': -1.8472561836242676, 'logits/rejected': -1.2327227592468262, 'epoch': 0.73} 73%|██████████████████████████████████████████████████████████████████████████████████▏ | 495/681 [35:28<09:04, 2.93s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▎ | 496/681 [35:31<09:12, 2.99s/it] {'loss': 0.6622, 'grad_norm': 30.489227294921875, 'learning_rate': 1.0555654214793722e-07, 'margin_dpo/margin_mean': 18.975727081298828, 'margin_dpo/margin_std': 19.945220947265625, 'logps/chosen': -122.41502380371094, 'logps/rejected': -157.75169372558594, 'logps/ref_chosen': -90.9640884399414, 'logps/ref_rejected': -107.32502746582031, 'logits/chosen': -2.2407476902008057, 'logits/rejected': -2.1656553745269775, 'epoch': 0.73} 73%|██████████████████████████████████████████████████████████████████████████████████▎ | 496/681 [35:31<09:12, 2.99s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▍ | 497/681 [35:34<09:16, 3.03s/it] {'loss': 0.6999, 'grad_norm': 24.2734432220459, 'learning_rate': 1.0451099960308374e-07, 'margin_dpo/margin_mean': 19.450889587402344, 'margin_dpo/margin_std': 14.824586868286133, 'logps/chosen': -114.13996124267578, 'logps/rejected': -121.43716430664062, 'logps/ref_chosen': -89.48576354980469, 'logps/ref_rejected': -77.33207702636719, 'logits/chosen': -1.2152756452560425, 'logits/rejected': -0.7649951577186584, 'epoch': 0.73} 73%|██████████████████████████████████████████████████████████████████████████████████▍ | 497/681 [35:34<09:16, 3.03s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▋ | 498/681 [35:37<09:10, 3.01s/it] {'loss': 0.5858, 'grad_norm': 31.715137481689453, 'learning_rate': 1.0346929082869641e-07, 'margin_dpo/margin_mean': 16.34941864013672, 'margin_dpo/margin_std': 10.946194648742676, 'logps/chosen': -113.42682647705078, 'logps/rejected': -101.58597564697266, 'logps/ref_chosen': -86.19139099121094, 'logps/ref_rejected': -58.00111389160156, 'logits/chosen': -1.5480034351348877, 'logits/rejected': -0.859655499458313, 'epoch': 0.73} 73%|██████████████████████████████████████████████████████████████████████████████████▋ | 498/681 [35:37<09:10, 3.01s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▊ | 499/681 [35:40<09:00, 2.97s/it] {'loss': 0.6569, 'grad_norm': 26.94870376586914, 'learning_rate': 1.0243144327477013e-07, 'margin_dpo/margin_mean': 15.444196701049805, 'margin_dpo/margin_std': 16.970027923583984, 'logps/chosen': -121.62670135498047, 'logps/rejected': -175.87213134765625, 'logps/ref_chosen': -93.28053283691406, 'logps/ref_rejected': -132.08175659179688, 'logits/chosen': -1.2497620582580566, 'logits/rejected': -1.404412031173706, 'epoch': 0.73} 73%|██████████████████████████████████████████████████████████████████████████████████▊ | 499/681 [35:40<09:00, 2.97s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▉ | 500/681 [35:43<08:48, 2.92s/it] {'loss': 0.7107, 'grad_norm': 30.46588897705078, 'learning_rate': 1.0139748428955333e-07, 'margin_dpo/margin_mean': 21.971506118774414, 'margin_dpo/margin_std': 22.098201751708984, 'logps/chosen': -104.04843139648438, 'logps/rejected': -146.70986938476562, 'logps/ref_chosen': -77.39090728759766, 'logps/ref_rejected': -98.08084106445312, 'logits/chosen': -2.575468063354492, 'logits/rejected': -2.286226272583008, 'epoch': 0.73} 73%|██████████████████████████████████████████████████████████████████████████████████▉ | 500/681 [35:43<08:48, 2.92s/it][INFO|trainer.py:4307] 2026-04-24 00:30:50,324 >> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-24 00:30:50,324 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-24 00:30:50,324 >> Batch size = 8 0%| | 0/73 [00:00> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-24 00:36:27,449 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-24 00:36:27,449 >> Batch size = 8 0%| | 0/73 [00:00> Saving model checkpoint to /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-600 [INFO|configuration_utils.py:419] 2026-04-24 00:37:25,600 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-600/config.json [INFO|configuration_utils.py:911] 2026-04-24 00:37:25,607 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-600/generation_config.json [INFO|modeling_utils.py:3580] 2026-04-24 00:38:04,762 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-600/model.safetensors.index.json. [INFO|tokenization_utils_base.py:2510] 2026-04-24 00:38:04,767 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-600/tokenizer_config.json [INFO|tokenization_utils_base.py:2519] 2026-04-24 00:38:04,770 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-600/special_tokens_map.json [INFO|trainer.py:4083] 2026-04-24 00:41:36,665 >> Deleting older checkpoint [/scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-200] due to args.save_total_limit 88%|█████████████████████████████████████████████████████████████████████████████████████████████████▉ | 601/681 [46:34<2:08:13, 96.17s/it] {'loss': 0.5885, 'grad_norm': 26.41422462463379, 'learning_rate': 2.1301532877994742e-08, 'margin_dpo/margin_mean': 30.145484924316406, 'margin_dpo/margin_std': 24.454891204833984, 'logps/chosen': -104.30206298828125, 'logps/rejected': -170.05686950683594, 'logps/ref_chosen': -75.11897277832031, 'logps/ref_rejected': -110.72828674316406, 'logits/chosen': -1.7634140253067017, 'logits/rejected': -1.8990434408187866, 'epoch': 0.88} 88%|█████████████████████████████████████████████████████████████████████████████████████████████████▉ | 601/681 [46:34<2:08:13, 96.17s/it] 88%|██████████████████████████████████████████████████████████████████████████████████████████████████ | 602/681 [46:37<1:29:55, 68.30s/it] {'loss': 0.5053, 'grad_norm': 24.838815689086914, 'learning_rate': 2.0786184285784298e-08, 'margin_dpo/margin_mean': 13.836491584777832, 'margin_dpo/margin_std': 8.537615776062012, 'logps/chosen': -94.12992858886719, 'logps/rejected': -113.13572692871094, 'logps/ref_chosen': -65.56060791015625, 'logps/ref_rejected': -70.72991943359375, 'logits/chosen': -1.9122300148010254, 'logits/rejected': -1.8413580656051636, 'epoch': 0.88} 88%|██████████████████████████████████████████████████████████████████████████████████████████████████ | 602/681 [46:37<1:29:55, 68.30s/it] 89%|██████████████████████████████████████████████████████████████████████████████████████████████████▎ | 603/681 [46:40<1:03:17, 48.69s/it] {'loss': 0.4995, 'grad_norm': 21.262638092041016, 'learning_rate': 2.0276875690788204e-08, 'margin_dpo/margin_mean': 24.044979095458984, 'margin_dpo/margin_std': 16.012496948242188, 'logps/chosen': -142.13864135742188, 'logps/rejected': -173.14385986328125, 'logps/ref_chosen': -122.12416076660156, 'logps/ref_rejected': -129.0844268798828, 'logits/chosen': -1.533982753753662, 'logits/rejected': -0.9280251264572144, 'epoch': 0.89} 89%|██████████████████████████████████████████████████████████████████████████████████████████████████▎ | 603/681 [46:40<1:03:17, 48.69s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 604/681 [46:43<44:53, 34.99s/it] {'loss': 0.5316, 'grad_norm': 23.620716094970703, 'learning_rate': 1.977362051376158e-08, 'margin_dpo/margin_mean': 15.964040756225586, 'margin_dpo/margin_std': 17.116283416748047, 'logps/chosen': -123.72084045410156, 'logps/rejected': -132.3470458984375, 'logps/ref_chosen': -95.43553161621094, 'logps/ref_rejected': -88.09769439697266, 'logits/chosen': -2.0624282360076904, 'logits/rejected': -1.6026830673217773, 'epoch': 0.89} 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 604/681 [46:43<44:53, 34.99s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 605/681 [46:46<32:10, 25.40s/it] {'loss': 0.6004, 'grad_norm': 23.886274337768555, 'learning_rate': 1.9276432015946446e-08, 'margin_dpo/margin_mean': 26.88275146484375, 'margin_dpo/margin_std': 12.025161743164062, 'logps/chosen': -124.10987854003906, 'logps/rejected': -113.97074127197266, 'logps/ref_chosen': -98.30979919433594, 'logps/ref_rejected': -61.28790283203125, 'logits/chosen': -0.8116771578788757, 'logits/rejected': 0.05416472256183624, 'epoch': 0.89} 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 605/681 [46:46<32:10, 25.40s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 606/681 [46:49<23:14, 18.59s/it] {'loss': 0.4465, 'grad_norm': 19.38317108154297, 'learning_rate': 1.8785323298722093e-08, 'margin_dpo/margin_mean': 24.26314926147461, 'margin_dpo/margin_std': 23.452638626098633, 'logps/chosen': -119.4458999633789, 'logps/rejected': -173.896484375, 'logps/ref_chosen': -91.20582580566406, 'logps/ref_rejected': -121.39325714111328, 'logits/chosen': -1.7569808959960938, 'logits/rejected': -1.498138427734375, 'epoch': 0.89} 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 606/681 [46:49<23:14, 18.59s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 607/681 [46:52<17:09, 13.91s/it] {'loss': 0.7087, 'grad_norm': 31.742523193359375, 'learning_rate': 1.8300307303259904e-08, 'margin_dpo/margin_mean': 27.137123107910156, 'margin_dpo/margin_std': 22.96157455444336, 'logps/chosen': -117.7818603515625, 'logps/rejected': -143.45266723632812, 'logps/ref_chosen': -93.03514099121094, 'logps/ref_rejected': -91.56881713867188, 'logits/chosen': -1.2356622219085693, 'logits/rejected': -0.7431513071060181, 'epoch': 0.89} 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 607/681 [46:52<17:09, 13.91s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 608/681 [46:54<12:51, 10.57s/it] {'loss': 0.6131, 'grad_norm': 26.023786544799805, 'learning_rate': 1.7821396810182437e-08, 'margin_dpo/margin_mean': 16.4940128326416, 'margin_dpo/margin_std': 14.074356079101562, 'logps/chosen': -93.41421508789062, 'logps/rejected': -107.28245544433594, 'logps/ref_chosen': -67.08824157714844, 'logps/ref_rejected': -64.46246337890625, 'logits/chosen': -1.4785089492797852, 'logits/rejected': -0.9715176820755005, 'epoch': 0.89} 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 608/681 [46:54<12:51, 10.57s/it] 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████ | 609/681 [46:57<09:54, 8.26s/it] {'loss': 0.5371, 'grad_norm': 22.36457061767578, 'learning_rate': 1.7348604439226617e-08, 'margin_dpo/margin_mean': 22.16372299194336, 'margin_dpo/margin_std': 17.413898468017578, 'logps/chosen': -112.64973449707031, 'logps/rejected': -131.94284057617188, 'logps/ref_chosen': -87.85757446289062, 'logps/ref_rejected': -84.98696899414062, 'logits/chosen': -1.7442258596420288, 'logits/rejected': -1.2550201416015625, 'epoch': 0.89} 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████ | 609/681 [46:57<09:54, 8.26s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 610/681 [47:00<07:50, 6.63s/it] {'loss': 0.5567, 'grad_norm': 35.290225982666016, 'learning_rate': 1.6881942648911074e-08, 'margin_dpo/margin_mean': 29.52370834350586, 'margin_dpo/margin_std': 20.365798950195312, 'logps/chosen': -124.01420593261719, 'logps/rejected': -175.32977294921875, 'logps/ref_chosen': -96.38543701171875, 'logps/ref_rejected': -118.17731475830078, 'logits/chosen': -0.8668674230575562, 'logits/rejected': -0.7067489624023438, 'epoch': 0.9} 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 610/681 [47:00<07:50, 6.63s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 611/681 [47:03<06:21, 5.45s/it] {'loss': 0.6434, 'grad_norm': 24.000062942504883, 'learning_rate': 1.6421423736208e-08, 'margin_dpo/margin_mean': 18.321395874023438, 'margin_dpo/margin_std': 21.715795516967773, 'logps/chosen': -87.62871551513672, 'logps/rejected': -145.35049438476562, 'logps/ref_chosen': -52.06251525878906, 'logps/ref_rejected': -91.46289825439453, 'logits/chosen': -1.1134871244430542, 'logits/rejected': -1.3368947505950928, 'epoch': 0.9} 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 611/681 [47:03<06:21, 5.45s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 612/681 [47:05<05:18, 4.61s/it] {'loss': 0.6102, 'grad_norm': 28.72694206237793, 'learning_rate': 1.5967059836219042e-08, 'margin_dpo/margin_mean': 30.159427642822266, 'margin_dpo/margin_std': 28.102706909179688, 'logps/chosen': -129.73422241210938, 'logps/rejected': -140.9920654296875, 'logps/ref_chosen': -111.08282470703125, 'logps/ref_rejected': -92.18126678466797, 'logits/chosen': -2.4680771827697754, 'logits/rejected': -1.2184339761734009, 'epoch': 0.9} 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 612/681 [47:05<05:18, 4.61s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 613/681 [47:08<04:37, 4.09s/it] {'loss': 0.4778, 'grad_norm': 29.352991104125977, 'learning_rate': 1.551886292185553e-08, 'margin_dpo/margin_mean': 31.312606811523438, 'margin_dpo/margin_std': 19.18683624267578, 'logps/chosen': -123.05023193359375, 'logps/rejected': -153.18783569335938, 'logps/ref_chosen': -98.03831481933594, 'logps/ref_rejected': -96.86331176757812, 'logits/chosen': -0.9738931655883789, 'logits/rejected': -0.4800354242324829, 'epoch': 0.9} 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 613/681 [47:08<04:37, 4.09s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 614/681 [47:11<04:08, 3.72s/it] {'loss': 0.557, 'grad_norm': 31.17698860168457, 'learning_rate': 1.507684480352292e-08, 'margin_dpo/margin_mean': 14.707442283630371, 'margin_dpo/margin_std': 17.06017303466797, 'logps/chosen': -106.69200134277344, 'logps/rejected': -185.89691162109375, 'logps/ref_chosen': -77.41725158691406, 'logps/ref_rejected': -141.9147186279297, 'logits/chosen': -1.1176828145980835, 'logits/rejected': -1.1149662733078003, 'epoch': 0.9} 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 614/681 [47:11<04:08, 3.72s/it] 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████ | 615/681 [47:14<03:50, 3.50s/it] {'loss': 0.6647, 'grad_norm': 23.951841354370117, 'learning_rate': 1.4641017128809801e-08, 'margin_dpo/margin_mean': 20.90196418762207, 'margin_dpo/margin_std': 15.42950439453125, 'logps/chosen': -186.31454467773438, 'logps/rejected': -133.49378967285156, 'logps/ref_chosen': -159.53976440429688, 'logps/ref_rejected': -85.81703186035156, 'logits/chosen': -2.5860671997070312, 'logits/rejected': -1.4683854579925537, 'epoch': 0.9} 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████ | 615/681 [47:14<03:50, 3.50s/it] 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 616/681 [47:17<03:37, 3.34s/it] {'loss': 0.655, 'grad_norm': 28.593801498413086, 'learning_rate': 1.4211391382180637e-08, 'margin_dpo/margin_mean': 26.75563621520996, 'margin_dpo/margin_std': 24.228565216064453, 'logps/chosen': -113.18240356445312, 'logps/rejected': -125.2229995727539, 'logps/ref_chosen': -86.95388793945312, 'logps/ref_rejected': -72.23883819580078, 'logits/chosen': -1.352514624595642, 'logits/rejected': -0.6652741432189941, 'epoch': 0.9} 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 616/681 [47:17<03:37, 3.34s/it] 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 617/681 [47:21<03:36, 3.38s/it] {'loss': 0.7685, 'grad_norm': 32.52434158325195, 'learning_rate': 1.378797888467345e-08, 'margin_dpo/margin_mean': 18.52096939086914, 'margin_dpo/margin_std': 16.058685302734375, 'logps/chosen': -133.91485595703125, 'logps/rejected': -101.05567932128906, 'logps/ref_chosen': -105.39964294433594, 'logps/ref_rejected': -54.019500732421875, 'logits/chosen': -1.0046977996826172, 'logits/rejected': -0.2724980413913727, 'epoch': 0.91} 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 617/681 [47:21<03:36, 3.38s/it] 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 618/681 [47:24<03:28, 3.30s/it] {'loss': 0.7211, 'grad_norm': 33.953426361083984, 'learning_rate': 1.3370790793601371e-08, 'margin_dpo/margin_mean': 19.089811325073242, 'margin_dpo/margin_std': 17.1854248046875, 'logps/chosen': -137.4913330078125, 'logps/rejected': -132.3260498046875, 'logps/ref_chosen': -111.18899536132812, 'logps/ref_rejected': -86.93389892578125, 'logits/chosen': -2.503714084625244, 'logits/rejected': -2.1493351459503174, 'epoch': 0.91} 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 618/681 [47:24<03:28, 3.30s/it] 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 619/681 [47:27<03:19, 3.22s/it] {'loss': 0.6267, 'grad_norm': 26.156822204589844, 'learning_rate': 1.2959838102258535e-08, 'margin_dpo/margin_mean': 26.543346405029297, 'margin_dpo/margin_std': 19.007911682128906, 'logps/chosen': -142.80148315429688, 'logps/rejected': -154.7870330810547, 'logps/ref_chosen': -122.91197204589844, 'logps/ref_rejected': -108.35417175292969, 'logits/chosen': -1.3825751543045044, 'logits/rejected': -0.729312002658844, 'epoch': 0.91} 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 619/681 [47:27<03:19, 3.22s/it] 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 620/681 [47:30<03:10, 3.12s/it] {'loss': 0.7628, 'grad_norm': 31.405254364013672, 'learning_rate': 1.2555131639630567e-08, 'margin_dpo/margin_mean': 16.232501983642578, 'margin_dpo/margin_std': 18.031766891479492, 'logps/chosen': -98.9208984375, 'logps/rejected': -130.6840362548828, 'logps/ref_chosen': -66.8671875, 'logps/ref_rejected': -82.39781188964844, 'logits/chosen': -1.6282209157943726, 'logits/rejected': -1.5203661918640137, 'epoch': 0.91} 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 620/681 [47:30<03:10, 3.12s/it] 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████ | 621/681 [47:33<03:05, 3.09s/it] {'loss': 0.697, 'grad_norm': 27.288387298583984, 'learning_rate': 1.2156682070109086e-08, 'margin_dpo/margin_mean': 16.956809997558594, 'margin_dpo/margin_std': 20.150236129760742, 'logps/chosen': -122.62663269042969, 'logps/rejected': -124.19369506835938, 'logps/ref_chosen': -100.42076110839844, 'logps/ref_rejected': -85.03102111816406, 'logits/chosen': -1.5042550563812256, 'logits/rejected': -1.1285921335220337, 'epoch': 0.91} 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████ | 621/681 [47:33<03:05, 3.09s/it] 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 622/681 [47:36<02:59, 3.04s/it] {'loss': 0.4756, 'grad_norm': 22.930315017700195, 'learning_rate': 1.1764499893210878e-08, 'margin_dpo/margin_mean': 24.725618362426758, 'margin_dpo/margin_std': 19.953989028930664, 'logps/chosen': -109.48207092285156, 'logps/rejected': -106.65364074707031, 'logps/ref_chosen': -86.09562683105469, 'logps/ref_rejected': -58.54158020019531, 'logits/chosen': -0.6095631122589111, 'logits/rejected': 0.05947205424308777, 'epoch': 0.91} 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 622/681 [47:36<02:59, 3.04s/it] 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 623/681 [47:38<02:46, 2.87s/it] {'loss': 0.8517, 'grad_norm': 34.40840148925781, 'learning_rate': 1.1378595443300998e-08, 'margin_dpo/margin_mean': 19.931346893310547, 'margin_dpo/margin_std': 23.866024017333984, 'logps/chosen': -109.72994232177734, 'logps/rejected': -163.298583984375, 'logps/ref_chosen': -80.37020874023438, 'logps/ref_rejected': -114.00752258300781, 'logits/chosen': -2.1641011238098145, 'logits/rejected': -2.284421443939209, 'epoch': 0.91} 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 623/681 [47:38<02:46, 2.87s/it] 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 624/681 [47:41<02:43, 2.87s/it] {'loss': 0.5314, 'grad_norm': 30.061717987060547, 'learning_rate': 1.0998978889320582e-08, 'margin_dpo/margin_mean': 27.737272262573242, 'margin_dpo/margin_std': 21.234542846679688, 'logps/chosen': -135.16061401367188, 'logps/rejected': -146.82891845703125, 'logps/ref_chosen': -112.427734375, 'logps/ref_rejected': -96.3587646484375, 'logits/chosen': -1.5450116395950317, 'logits/rejected': -0.4716716408729553, 'epoch': 0.92} 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 624/681 [47:41<02:43, 2.87s/it] 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 625/681 [47:44<02:49, 3.02s/it] {'loss': 0.553, 'grad_norm': 24.109561920166016, 'learning_rate': 1.0625660234518913e-08, 'margin_dpo/margin_mean': 21.44537353515625, 'margin_dpo/margin_std': 15.984648704528809, 'logps/chosen': -109.12333679199219, 'logps/rejected': -162.85675048828125, 'logps/ref_chosen': -75.3917465209961, 'logps/ref_rejected': -107.67977905273438, 'logits/chosen': -2.4023971557617188, 'logits/rejected': -2.229701280593872, 'epoch': 0.92} 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 625/681 [47:44<02:49, 3.02s/it] 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 626/681 [47:48<02:49, 3.08s/it] {'loss': 0.8693, 'grad_norm': 40.272911071777344, 'learning_rate': 1.0258649316189721e-08, 'margin_dpo/margin_mean': 14.079477310180664, 'margin_dpo/margin_std': 25.268157958984375, 'logps/chosen': -132.60232543945312, 'logps/rejected': -151.58079528808594, 'logps/ref_chosen': -96.54901123046875, 'logps/ref_rejected': -101.44801330566406, 'logits/chosen': -2.329925537109375, 'logits/rejected': -2.2528748512268066, 'epoch': 0.92} 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 626/681 [47:48<02:49, 3.08s/it] 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████ | 627/681 [47:51<02:46, 3.08s/it] {'loss': 0.6398, 'grad_norm': 26.63361358642578, 'learning_rate': 9.897955805412e-09, 'margin_dpo/margin_mean': 19.443174362182617, 'margin_dpo/margin_std': 14.579084396362305, 'logps/chosen': -119.01112365722656, 'logps/rejected': -146.07315063476562, 'logps/ref_chosen': -88.80218505859375, 'logps/ref_rejected': -96.42103576660156, 'logits/chosen': -1.4621191024780273, 'logits/rejected': -1.2315270900726318, 'epoch': 0.92} 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████ | 627/681 [47:51<02:46, 3.08s/it] 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 628/681 [47:54<02:40, 3.04s/it] {'loss': 0.5028, 'grad_norm': 24.76960563659668, 'learning_rate': 9.543589206795238e-09, 'margin_dpo/margin_mean': 19.077255249023438, 'margin_dpo/margin_std': 12.725533485412598, 'logps/chosen': -126.72996520996094, 'logps/rejected': -147.0572052001953, 'logps/ref_chosen': -99.37443542480469, 'logps/ref_rejected': -100.62442779541016, 'logits/chosen': -1.3107500076293945, 'logits/rejected': -0.9299607276916504, 'epoch': 0.92} 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 628/681 [47:54<02:40, 3.04s/it] 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 629/681 [47:57<02:37, 3.02s/it] {'loss': 0.7023, 'grad_norm': 32.12858581542969, 'learning_rate': 9.19555885822887e-09, 'margin_dpo/margin_mean': 21.597137451171875, 'margin_dpo/margin_std': 15.778484344482422, 'logps/chosen': -121.68482208251953, 'logps/rejected': -129.65162658691406, 'logps/ref_chosen': -98.16767120361328, 'logps/ref_rejected': -84.53733825683594, 'logits/chosen': -1.940063238143921, 'logits/rejected': -1.3807988166809082, 'epoch': 0.92} 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 629/681 [47:57<02:37, 3.02s/it] 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 630/681 [48:00<02:33, 3.02s/it] {'loss': 0.6629, 'grad_norm': 29.0245361328125, 'learning_rate': 8.85387393063622e-09, 'margin_dpo/margin_mean': 26.629322052001953, 'margin_dpo/margin_std': 19.489566802978516, 'logps/chosen': -127.28799438476562, 'logps/rejected': -128.5855712890625, 'logps/ref_chosen': -102.16020202636719, 'logps/ref_rejected': -76.82844543457031, 'logits/chosen': -2.2557272911071777, 'logits/rejected': -0.403414249420166, 'epoch': 0.93} 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 630/681 [48:00<02:33, 3.02s/it] 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 631/681 [48:02<02:27, 2.96s/it] {'loss': 0.4943, 'grad_norm': 19.227121353149414, 'learning_rate': 8.518543427732949e-09, 'margin_dpo/margin_mean': 26.273296356201172, 'margin_dpo/margin_std': 15.153189659118652, 'logps/chosen': -133.84732055664062, 'logps/rejected': -121.6485824584961, 'logps/ref_chosen': -114.78280639648438, 'logps/ref_rejected': -76.31077575683594, 'logits/chosen': -2.5818333625793457, 'logits/rejected': -1.2536900043487549, 'epoch': 0.93} 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 631/681 [48:02<02:27, 2.96s/it] 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 632/681 [48:05<02:21, 2.89s/it] {'loss': 0.5382, 'grad_norm': 23.025691986083984, 'learning_rate': 8.189576185789637e-09, 'margin_dpo/margin_mean': 25.185754776000977, 'margin_dpo/margin_std': 19.113544464111328, 'logps/chosen': -107.76996612548828, 'logps/rejected': -147.23556518554688, 'logps/ref_chosen': -80.00565338134766, 'logps/ref_rejected': -94.28547668457031, 'logits/chosen': -1.9373183250427246, 'logits/rejected': -1.6466472148895264, 'epoch': 0.93} 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 632/681 [48:05<02:21, 2.89s/it] 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████ | 633/681 [48:08<02:22, 2.96s/it] {'loss': 0.7673, 'grad_norm': 31.93626594543457, 'learning_rate': 7.866980873399015e-09, 'margin_dpo/margin_mean': 19.91036605834961, 'margin_dpo/margin_std': 21.933469772338867, 'logps/chosen': -107.93728637695312, 'logps/rejected': -133.3818359375, 'logps/ref_chosen': -79.70408630371094, 'logps/ref_rejected': -85.23828125, 'logits/chosen': -3.0079336166381836, 'logits/rejected': -2.5928359031677246, 'epoch': 0.93} 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████ | 633/681 [48:08<02:22, 2.96s/it] 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 634/681 [48:11<02:19, 2.96s/it] {'loss': 0.5543, 'grad_norm': 24.426607131958008, 'learning_rate': 7.550765991247654e-09, 'margin_dpo/margin_mean': 24.123126983642578, 'margin_dpo/margin_std': 15.052120208740234, 'logps/chosen': -95.75471496582031, 'logps/rejected': -150.92276000976562, 'logps/ref_chosen': -65.44131469726562, 'logps/ref_rejected': -96.48623657226562, 'logits/chosen': -1.3841078281402588, 'logits/rejected': -1.0601425170898438, 'epoch': 0.93} 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 634/681 [48:11<02:19, 2.96s/it] 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 635/681 [48:14<02:14, 2.92s/it] {'loss': 0.6623, 'grad_norm': 26.585651397705078, 'learning_rate': 7.240939871891699e-09, 'margin_dpo/margin_mean': 21.39522933959961, 'margin_dpo/margin_std': 16.801891326904297, 'logps/chosen': -126.51262664794922, 'logps/rejected': -126.68396759033203, 'logps/ref_chosen': -99.6771240234375, 'logps/ref_rejected': -78.4532470703125, 'logits/chosen': -1.7034859657287598, 'logits/rejected': -1.0522689819335938, 'epoch': 0.93} 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 635/681 [48:14<02:14, 2.92s/it] 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 636/681 [48:17<02:13, 2.97s/it] {'loss': 0.6777, 'grad_norm': 28.644737243652344, 'learning_rate': 6.937510679537628e-09, 'margin_dpo/margin_mean': 14.933639526367188, 'margin_dpo/margin_std': 14.684162139892578, 'logps/chosen': -130.71478271484375, 'logps/rejected': -131.12960815429688, 'logps/ref_chosen': -102.57931518554688, 'logps/ref_rejected': -88.06050109863281, 'logits/chosen': -0.9782381057739258, 'logits/rejected': -0.4344422221183777, 'epoch': 0.93} 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 636/681 [48:17<02:13, 2.97s/it] 94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 637/681 [48:20<02:09, 2.95s/it] {'loss': 0.7009, 'grad_norm': 29.544261932373047, 'learning_rate': 6.640486409826785e-09, 'margin_dpo/margin_mean': 20.864341735839844, 'margin_dpo/margin_std': 27.72216033935547, 'logps/chosen': -98.80889892578125, 'logps/rejected': -121.78729248046875, 'logps/ref_chosen': -73.93675994873047, 'logps/ref_rejected': -76.05081176757812, 'logits/chosen': -1.4342460632324219, 'logits/rejected': -1.0915610790252686, 'epoch': 0.94} 94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 637/681 [48:20<02:09, 2.95s/it] 94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 638/681 [48:23<02:05, 2.92s/it] {'loss': 0.3867, 'grad_norm': 15.684412956237793, 'learning_rate': 6.349874889624962e-09, 'margin_dpo/margin_mean': 34.11281967163086, 'margin_dpo/margin_std': 22.033653259277344, 'logps/chosen': -162.9862518310547, 'logps/rejected': -122.63823699951172, 'logps/ref_chosen': -146.49359130859375, 'logps/ref_rejected': -72.0327377319336, 'logits/chosen': -2.4961729049682617, 'logits/rejected': -0.7357807159423828, 'epoch': 0.94} 94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 638/681 [48:23<02:05, 2.92s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████ | 639/681 [48:26<02:02, 2.91s/it] {'loss': 0.8664, 'grad_norm': 41.89670181274414, 'learning_rate': 6.065683776815933e-09, 'margin_dpo/margin_mean': 20.680925369262695, 'margin_dpo/margin_std': 17.583145141601562, 'logps/chosen': -126.6545181274414, 'logps/rejected': -126.32666015625, 'logps/ref_chosen': -99.16008758544922, 'logps/ref_rejected': -78.15130615234375, 'logits/chosen': -2.0428102016448975, 'logits/rejected': -0.9309650659561157, 'epoch': 0.94} 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████ | 639/681 [48:26<02:02, 2.91s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 640/681 [48:29<02:02, 3.00s/it] {'loss': 0.5627, 'grad_norm': 26.69725227355957, 'learning_rate': 5.7879205600998296e-09, 'margin_dpo/margin_mean': 23.118385314941406, 'margin_dpo/margin_std': 11.315099716186523, 'logps/chosen': -100.805419921875, 'logps/rejected': -126.19176483154297, 'logps/ref_chosen': -70.03657531738281, 'logps/ref_rejected': -72.30453491210938, 'logits/chosen': -0.8534713387489319, 'logits/rejected': -0.35765373706817627, 'epoch': 0.94} 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 640/681 [48:29<02:02, 3.00s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 641/681 [48:32<02:02, 3.05s/it] {'loss': 0.5443, 'grad_norm': 24.453815460205078, 'learning_rate': 5.516592558795746e-09, 'margin_dpo/margin_mean': 22.078567504882812, 'margin_dpo/margin_std': 24.4644775390625, 'logps/chosen': -113.69627380371094, 'logps/rejected': -158.95619201660156, 'logps/ref_chosen': -86.68313598632812, 'logps/ref_rejected': -109.86448669433594, 'logits/chosen': -1.6536837816238403, 'logits/rejected': -1.5010559558868408, 'epoch': 0.94} 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 641/681 [48:32<02:02, 3.05s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 642/681 [48:35<01:56, 3.00s/it] {'loss': 0.6432, 'grad_norm': 27.494384765625, 'learning_rate': 5.251706922648868e-09, 'margin_dpo/margin_mean': 18.521900177001953, 'margin_dpo/margin_std': 18.387256622314453, 'logps/chosen': -112.55367279052734, 'logps/rejected': -176.6457061767578, 'logps/ref_chosen': -81.20620727539062, 'logps/ref_rejected': -126.77632141113281, 'logits/chosen': -0.8695433139801025, 'logits/rejected': -0.6438785791397095, 'epoch': 0.94} 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 642/681 [48:35<01:56, 3.00s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 643/681 [48:38<01:54, 3.02s/it] {'loss': 0.5499, 'grad_norm': 21.128366470336914, 'learning_rate': 4.993270631642038e-09, 'margin_dpo/margin_mean': 20.74580955505371, 'margin_dpo/margin_std': 12.552116394042969, 'logps/chosen': -116.45796203613281, 'logps/rejected': -121.64019012451172, 'logps/ref_chosen': -89.84759521484375, 'logps/ref_rejected': -74.28402709960938, 'logits/chosen': -1.7625564336776733, 'logits/rejected': -1.2028493881225586, 'epoch': 0.94} 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 643/681 [48:38<01:54, 3.02s/it] 95%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 644/681 [48:41<01:51, 3.01s/it] {'loss': 0.7342, 'grad_norm': 28.339998245239258, 'learning_rate': 4.741290495811873e-09, 'margin_dpo/margin_mean': 19.04413414001465, 'margin_dpo/margin_std': 20.908920288085938, 'logps/chosen': -106.68426513671875, 'logps/rejected': -146.94818115234375, 'logps/ref_chosen': -76.54374694824219, 'logps/ref_rejected': -97.76353454589844, 'logits/chosen': -1.7750461101531982, 'logits/rejected': -1.5512135028839111, 'epoch': 0.95} 95%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 644/681 [48:41<01:51, 3.01s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 645/681 [48:44<01:47, 2.99s/it] {'loss': 0.8305, 'grad_norm': 32.95466232299805, 'learning_rate': 4.495773155069299e-09, 'margin_dpo/margin_mean': 12.096460342407227, 'margin_dpo/margin_std': 17.232433319091797, 'logps/chosen': -137.83843994140625, 'logps/rejected': -107.06454467773438, 'logps/ref_chosen': -108.44778442382812, 'logps/ref_rejected': -65.57743072509766, 'logits/chosen': -1.6883985996246338, 'logits/rejected': -0.7081927061080933, 'epoch': 0.95} 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 645/681 [48:44<01:47, 2.99s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 646/681 [48:47<01:42, 2.94s/it] {'loss': 0.4552, 'grad_norm': 23.519771575927734, 'learning_rate': 4.256725079024553e-09, 'margin_dpo/margin_mean': 26.582338333129883, 'margin_dpo/margin_std': 12.872812271118164, 'logps/chosen': -124.54080963134766, 'logps/rejected': -167.08096313476562, 'logps/ref_chosen': -99.14352416992188, 'logps/ref_rejected': -115.10133361816406, 'logits/chosen': -1.1280121803283691, 'logits/rejected': -0.7711001038551331, 'epoch': 0.95} 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 646/681 [48:47<01:42, 2.94s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 647/681 [48:50<01:41, 2.98s/it] {'loss': 0.7834, 'grad_norm': 32.113014221191406, 'learning_rate': 4.024152566816791e-09, 'margin_dpo/margin_mean': 19.25110626220703, 'margin_dpo/margin_std': 20.14596176147461, 'logps/chosen': -114.12860107421875, 'logps/rejected': -149.02532958984375, 'logps/ref_chosen': -82.64013671875, 'logps/ref_rejected': -98.2857666015625, 'logits/chosen': -0.8513507843017578, 'logits/rejected': -0.7660804390907288, 'epoch': 0.95} 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 647/681 [48:50<01:41, 2.98s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 648/681 [48:53<01:38, 2.99s/it] {'loss': 0.5122, 'grad_norm': 26.356727600097656, 'learning_rate': 3.798061746947995e-09, 'margin_dpo/margin_mean': 21.967853546142578, 'margin_dpo/margin_std': 16.269351959228516, 'logps/chosen': -109.87057495117188, 'logps/rejected': -153.75718688964844, 'logps/ref_chosen': -77.24684143066406, 'logps/ref_rejected': -99.16560363769531, 'logits/chosen': -1.376887559890747, 'logits/rejected': -1.3346166610717773, 'epoch': 0.95} 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 648/681 [48:53<01:38, 2.99s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 649/681 [48:56<01:38, 3.08s/it] {'loss': 0.609, 'grad_norm': 24.207809448242188, 'learning_rate': 3.5784585771215235e-09, 'margin_dpo/margin_mean': 13.484968185424805, 'margin_dpo/margin_std': 18.346370697021484, 'logps/chosen': -131.0638427734375, 'logps/rejected': -116.54486846923828, 'logps/ref_chosen': -105.46896362304688, 'logps/ref_rejected': -77.46502685546875, 'logits/chosen': -1.8359942436218262, 'logits/rejected': -1.2063957452774048, 'epoch': 0.95} 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 649/681 [48:56<01:38, 3.08s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 650/681 [48:59<01:32, 2.98s/it] {'loss': 0.6071, 'grad_norm': 37.710941314697266, 'learning_rate': 3.3653488440851253e-09, 'margin_dpo/margin_mean': 25.702922821044922, 'margin_dpo/margin_std': 18.235645294189453, 'logps/chosen': -109.78580474853516, 'logps/rejected': -155.64500427246094, 'logps/ref_chosen': -83.3130111694336, 'logps/ref_rejected': -103.46926879882812, 'logits/chosen': -1.2717688083648682, 'logits/rejected': -1.2444267272949219, 'epoch': 0.95} 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 650/681 [48:59<01:32, 2.98s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 651/681 [49:02<01:28, 2.96s/it] {'loss': 0.4133, 'grad_norm': 20.15422248840332, 'learning_rate': 3.158738163478475e-09, 'margin_dpo/margin_mean': 34.41473388671875, 'margin_dpo/margin_std': 15.260412216186523, 'logps/chosen': -125.57213592529297, 'logps/rejected': -186.22874450683594, 'logps/ref_chosen': -109.26248168945312, 'logps/ref_rejected': -135.50436401367188, 'logits/chosen': -2.506378412246704, 'logits/rejected': -1.909285306930542, 'epoch': 0.96} 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 651/681 [49:02<01:28, 2.96s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 652/681 [49:05<01:25, 2.95s/it] {'loss': 0.5575, 'grad_norm': 23.561294555664062, 'learning_rate': 2.9586319796851555e-09, 'margin_dpo/margin_mean': 25.084617614746094, 'margin_dpo/margin_std': 15.043981552124023, 'logps/chosen': -103.1104736328125, 'logps/rejected': -159.1099853515625, 'logps/ref_chosen': -77.7144775390625, 'logps/ref_rejected': -108.62936401367188, 'logits/chosen': -1.4572404623031616, 'logits/rejected': -1.3910325765609741, 'epoch': 0.96} 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 652/681 [49:05<01:25, 2.95s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 653/681 [49:08<01:22, 2.93s/it] {'loss': 0.5885, 'grad_norm': 28.391294479370117, 'learning_rate': 2.7650355656892166e-09, 'margin_dpo/margin_mean': 20.083072662353516, 'margin_dpo/margin_std': 16.388137817382812, 'logps/chosen': -119.31652069091797, 'logps/rejected': -177.6236114501953, 'logps/ref_chosen': -93.11857604980469, 'logps/ref_rejected': -131.34259033203125, 'logits/chosen': -2.1806952953338623, 'logits/rejected': -2.2581558227539062, 'epoch': 0.96} 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 653/681 [49:08<01:22, 2.93s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 654/681 [49:11<01:19, 2.93s/it] {'loss': 0.7289, 'grad_norm': 31.213104248046875, 'learning_rate': 2.577954022936174e-09, 'margin_dpo/margin_mean': 17.339649200439453, 'margin_dpo/margin_std': 15.09589672088623, 'logps/chosen': -109.99600219726562, 'logps/rejected': -140.29420471191406, 'logps/ref_chosen': -78.7916488647461, 'logps/ref_rejected': -91.75021362304688, 'logits/chosen': -1.9482306241989136, 'logits/rejected': -1.3727693557739258, 'epoch': 0.96} 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 654/681 [49:11<01:19, 2.93s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 655/681 [49:14<01:16, 2.93s/it] {'loss': 0.72, 'grad_norm': 27.248577117919922, 'learning_rate': 2.397392281198729e-09, 'margin_dpo/margin_mean': 13.799907684326172, 'margin_dpo/margin_std': 12.526430130004883, 'logps/chosen': -106.54402923583984, 'logps/rejected': -121.86129760742188, 'logps/ref_chosen': -74.46823120117188, 'logps/ref_rejected': -75.98558807373047, 'logits/chosen': -1.7276451587677002, 'logits/rejected': -1.683257818222046, 'epoch': 0.96} 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 655/681 [49:14<01:16, 2.93s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 656/681 [49:17<01:15, 3.02s/it] {'loss': 0.4748, 'grad_norm': 22.945934295654297, 'learning_rate': 2.223355098446622e-09, 'margin_dpo/margin_mean': 27.725685119628906, 'margin_dpo/margin_std': 16.44278335571289, 'logps/chosen': -111.42201232910156, 'logps/rejected': -194.20220947265625, 'logps/ref_chosen': -80.48881530761719, 'logps/ref_rejected': -135.5433349609375, 'logits/chosen': -1.748338222503662, 'logits/rejected': -1.9085040092468262, 'epoch': 0.96} 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 656/681 [49:17<01:15, 3.02s/it] 96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 657/681 [49:20<01:10, 2.95s/it] {'loss': 0.6366, 'grad_norm': 25.630002975463867, 'learning_rate': 2.055847060721566e-09, 'margin_dpo/margin_mean': 22.592342376708984, 'margin_dpo/margin_std': 27.992549896240234, 'logps/chosen': -116.6616439819336, 'logps/rejected': -147.25933837890625, 'logps/ref_chosen': -91.72633361816406, 'logps/ref_rejected': -99.731689453125, 'logits/chosen': -1.4626106023788452, 'logits/rejected': -1.0344831943511963, 'epoch': 0.96} 96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 657/681 [49:20<01:10, 2.95s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 658/681 [49:22<01:05, 2.87s/it] {'loss': 0.5872, 'grad_norm': 24.88594627380371, 'learning_rate': 1.8948725820160662e-09, 'margin_dpo/margin_mean': 17.340206146240234, 'margin_dpo/margin_std': 20.778215408325195, 'logps/chosen': -146.45626831054688, 'logps/rejected': -156.04794311523438, 'logps/ref_chosen': -112.55894470214844, 'logps/ref_rejected': -104.8104019165039, 'logits/chosen': -1.8801562786102295, 'logits/rejected': -1.3535478115081787, 'epoch': 0.97} 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 658/681 [49:22<01:05, 2.87s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 659/681 [49:25<01:03, 2.89s/it] {'loss': 0.532, 'grad_norm': 24.542510986328125, 'learning_rate': 1.7404359041573723e-09, 'margin_dpo/margin_mean': 26.33460235595703, 'margin_dpo/margin_std': 24.5662841796875, 'logps/chosen': -104.23818969726562, 'logps/rejected': -126.01404571533203, 'logps/ref_chosen': -76.53956604003906, 'logps/ref_rejected': -71.9808120727539, 'logits/chosen': -2.352703094482422, 'logits/rejected': -1.7414582967758179, 'epoch': 0.97} 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 659/681 [49:25<01:03, 2.89s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 660/681 [49:28<00:59, 2.86s/it] {'loss': 0.54, 'grad_norm': 23.398681640625, 'learning_rate': 1.592541096695571e-09, 'margin_dpo/margin_mean': 28.028095245361328, 'margin_dpo/margin_std': 18.708595275878906, 'logps/chosen': -111.40882873535156, 'logps/rejected': -124.7333984375, 'logps/ref_chosen': -84.62733459472656, 'logps/ref_rejected': -69.92381286621094, 'logits/chosen': -2.1632814407348633, 'logits/rejected': -1.345157504081726, 'epoch': 0.97} 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 660/681 [49:28<00:59, 2.86s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 661/681 [49:30<00:54, 2.74s/it] {'loss': 0.5883, 'grad_norm': 29.665576934814453, 'learning_rate': 1.4511920567963908e-09, 'margin_dpo/margin_mean': 22.2477970123291, 'margin_dpo/margin_std': 23.344955444335938, 'logps/chosen': -123.57489013671875, 'logps/rejected': -167.12649536132812, 'logps/ref_chosen': -92.84153747558594, 'logps/ref_rejected': -114.14533996582031, 'logits/chosen': -1.6013872623443604, 'logits/rejected': -1.1411731243133545, 'epoch': 0.97} 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 661/681 [49:30<00:54, 2.74s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 662/681 [49:34<00:54, 2.85s/it] {'loss': 0.4635, 'grad_norm': 24.84825897216797, 'learning_rate': 1.3163925091384532e-09, 'margin_dpo/margin_mean': 20.56658935546875, 'margin_dpo/margin_std': 13.022397994995117, 'logps/chosen': -112.46092987060547, 'logps/rejected': -137.53738403320312, 'logps/ref_chosen': -82.68042755126953, 'logps/ref_rejected': -87.19029235839844, 'logits/chosen': -1.088799238204956, 'logits/rejected': -0.6112481355667114, 'epoch': 0.97} 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 662/681 [49:34<00:54, 2.85s/it] 97%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 663/681 [49:37<00:54, 3.00s/it] {'loss': 0.5211, 'grad_norm': 22.896821975708008, 'learning_rate': 1.1881460058152382e-09, 'margin_dpo/margin_mean': 19.316484451293945, 'margin_dpo/margin_std': 15.610393524169922, 'logps/chosen': -135.431640625, 'logps/rejected': -179.110595703125, 'logps/ref_chosen': -106.22300720214844, 'logps/ref_rejected': -130.58547973632812, 'logits/chosen': -1.5960707664489746, 'logits/rejected': -1.2925134897232056, 'epoch': 0.97} 97%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 663/681 [49:37<00:54, 3.00s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 664/681 [49:40<00:52, 3.06s/it] {'loss': 0.5275, 'grad_norm': 27.367639541625977, 'learning_rate': 1.066455926241383e-09, 'margin_dpo/margin_mean': 30.288034439086914, 'margin_dpo/margin_std': 14.453622817993164, 'logps/chosen': -117.2126693725586, 'logps/rejected': -151.74496459960938, 'logps/ref_chosen': -93.286376953125, 'logps/ref_rejected': -97.53064727783203, 'logits/chosen': -1.6287367343902588, 'logits/rejected': -1.190144658088684, 'epoch': 0.98} 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 664/681 [49:40<00:52, 3.06s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 665/681 [49:43<00:47, 2.95s/it] {'loss': 0.564, 'grad_norm': 22.599021911621094, 'learning_rate': 9.513254770636137e-10, 'margin_dpo/margin_mean': 15.369461059570312, 'margin_dpo/margin_std': 11.922895431518555, 'logps/chosen': -131.42825317382812, 'logps/rejected': -145.77651977539062, 'logps/ref_chosen': -105.4872817993164, 'logps/ref_rejected': -104.466064453125, 'logits/chosen': -2.1870265007019043, 'logits/rejected': -1.971346139907837, 'epoch': 0.98} 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 665/681 [49:43<00:47, 2.95s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 666/681 [49:46<00:44, 3.00s/it] {'loss': 0.815, 'grad_norm': 31.737276077270508, 'learning_rate': 8.427576920763956e-10, 'margin_dpo/margin_mean': 14.71677017211914, 'margin_dpo/margin_std': 17.84575653076172, 'logps/chosen': -116.85716247558594, 'logps/rejected': -136.8424835205078, 'logps/ref_chosen': -85.33709716796875, 'logps/ref_rejected': -90.60565185546875, 'logits/chosen': -1.3515913486480713, 'logits/rejected': -1.0945022106170654, 'epoch': 0.98} 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 666/681 [49:46<00:44, 3.00s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 667/681 [49:49<00:42, 3.00s/it] {'loss': 0.4763, 'grad_norm': 26.36996078491211, 'learning_rate': 7.407554321417764e-10, 'margin_dpo/margin_mean': 26.86260223388672, 'margin_dpo/margin_std': 18.75882911682129, 'logps/chosen': -105.82881927490234, 'logps/rejected': -120.00591278076172, 'logps/ref_chosen': -80.93008422851562, 'logps/ref_rejected': -68.24456787109375, 'logits/chosen': -1.5280002355575562, 'logits/rejected': -0.8938305377960205, 'epoch': 0.98} 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 667/681 [49:49<00:42, 3.00s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 668/681 [49:52<00:39, 3.01s/it] {'loss': 0.4135, 'grad_norm': 23.50409698486328, 'learning_rate': 6.453213851142225e-10, 'margin_dpo/margin_mean': 22.471668243408203, 'margin_dpo/margin_std': 17.230484008789062, 'logps/chosen': -104.58021545410156, 'logps/rejected': -168.6148223876953, 'logps/ref_chosen': -76.60061645507812, 'logps/ref_rejected': -118.1635513305664, 'logits/chosen': -1.4292489290237427, 'logits/rejected': -1.478631615638733, 'epoch': 0.98} 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 668/681 [49:52<00:39, 3.01s/it] 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 669/681 [49:55<00:36, 3.03s/it] {'loss': 0.5988, 'grad_norm': 24.116369247436523, 'learning_rate': 5.564580657695939e-10, 'margin_dpo/margin_mean': 14.25910758972168, 'margin_dpo/margin_std': 19.90908432006836, 'logps/chosen': -120.40367126464844, 'logps/rejected': -104.82398986816406, 'logps/ref_chosen': -91.93134307861328, 'logps/ref_rejected': -62.09255599975586, 'logits/chosen': -1.528371810913086, 'logits/rejected': -0.9692767858505249, 'epoch': 0.98} 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 669/681 [49:55<00:36, 3.03s/it] 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 670/681 [49:58<00:33, 3.06s/it] {'loss': 0.4798, 'grad_norm': 21.280534744262695, 'learning_rate': 4.741678157389739e-10, 'margin_dpo/margin_mean': 28.047285079956055, 'margin_dpo/margin_std': 18.901243209838867, 'logps/chosen': -98.46882629394531, 'logps/rejected': -138.5927276611328, 'logps/ref_chosen': -71.73137664794922, 'logps/ref_rejected': -83.80799102783203, 'logits/chosen': -1.3773136138916016, 'logits/rejected': -1.061061978340149, 'epoch': 0.98} 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 670/681 [49:58<00:33, 3.06s/it] 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 671/681 [50:01<00:29, 2.98s/it] {'loss': 0.5645, 'grad_norm': 22.35253143310547, 'learning_rate': 3.9845280344705245e-10, 'margin_dpo/margin_mean': 17.332304000854492, 'margin_dpo/margin_std': 15.1148042678833, 'logps/chosen': -131.175048828125, 'logps/rejected': -140.61334228515625, 'logps/ref_chosen': -97.33171844482422, 'logps/ref_rejected': -89.43772888183594, 'logits/chosen': -0.859630823135376, 'logits/rejected': -0.6215537190437317, 'epoch': 0.99} 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 671/681 [50:01<00:29, 2.98s/it] 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 672/681 [50:04<00:27, 3.02s/it] {'loss': 0.6272, 'grad_norm': 28.22157096862793, 'learning_rate': 3.293150240547549e-10, 'margin_dpo/margin_mean': 13.203832626342773, 'margin_dpo/margin_std': 15.262359619140625, 'logps/chosen': -108.41804504394531, 'logps/rejected': -139.38845825195312, 'logps/ref_chosen': -77.21369171142578, 'logps/ref_rejected': -94.98027038574219, 'logits/chosen': -2.014643907546997, 'logits/rejected': -1.8692243099212646, 'epoch': 0.99} 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 672/681 [50:04<00:27, 3.02s/it] 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 673/681 [50:07<00:23, 2.89s/it] {'loss': 0.505, 'grad_norm': 22.13953399658203, 'learning_rate': 2.6675629940689504e-10, 'margin_dpo/margin_mean': 23.310997009277344, 'margin_dpo/margin_std': 15.188114166259766, 'logps/chosen': -91.1622543334961, 'logps/rejected': -152.5889892578125, 'logps/ref_chosen': -63.30126190185547, 'logps/ref_rejected': -101.4169921875, 'logits/chosen': -0.9023149013519287, 'logits/rejected': -0.7241397500038147, 'epoch': 0.99} 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 673/681 [50:07<00:23, 2.89s/it] 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 674/681 [50:10<00:20, 2.94s/it] {'loss': 0.4786, 'grad_norm': 20.051538467407227, 'learning_rate': 2.1077827798404725e-10, 'margin_dpo/margin_mean': 25.555042266845703, 'margin_dpo/margin_std': 18.46479606628418, 'logps/chosen': -109.16806030273438, 'logps/rejected': -128.1361083984375, 'logps/ref_chosen': -80.00570678710938, 'logps/ref_rejected': -73.41873168945312, 'logits/chosen': -1.5894590616226196, 'logits/rejected': -1.1531003713607788, 'epoch': 0.99} 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 674/681 [50:10<00:20, 2.94s/it] 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 675/681 [50:12<00:17, 2.86s/it] {'loss': 0.7257, 'grad_norm': 177.50648498535156, 'learning_rate': 1.6138243485910863e-10, 'margin_dpo/margin_mean': 17.43462371826172, 'margin_dpo/margin_std': 15.88136100769043, 'logps/chosen': -87.20614624023438, 'logps/rejected': -111.49307250976562, 'logps/ref_chosen': -63.118011474609375, 'logps/ref_rejected': -69.97030639648438, 'logits/chosen': -0.9243202805519104, 'logits/rejected': -0.7027927041053772, 'epoch': 0.99} 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 675/681 [50:12<00:17, 2.86s/it] 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏| 676/681 [50:15<00:14, 2.92s/it] {'loss': 0.4528, 'grad_norm': 26.805587768554688, 'learning_rate': 1.1857007165852472e-10, 'margin_dpo/margin_mean': 21.404701232910156, 'margin_dpo/margin_std': 14.885082244873047, 'logps/chosen': -131.3642578125, 'logps/rejected': -151.52993774414062, 'logps/ref_chosen': -100.23760986328125, 'logps/ref_rejected': -98.99859619140625, 'logits/chosen': -1.7543052434921265, 'logits/rejected': -1.1985046863555908, 'epoch': 0.99} 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏| 676/681 [50:15<00:14, 2.92s/it] 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎| 677/681 [50:18<00:11, 2.84s/it] {'loss': 0.3924, 'grad_norm': 17.16691780090332, 'learning_rate': 8.23423165278725e-11, 'margin_dpo/margin_mean': 21.885032653808594, 'margin_dpo/margin_std': 18.007171630859375, 'logps/chosen': -106.279541015625, 'logps/rejected': -137.46066284179688, 'logps/ref_chosen': -80.32097625732422, 'logps/ref_rejected': -89.61705017089844, 'logits/chosen': -2.323134660720825, 'logits/rejected': -2.3097894191741943, 'epoch': 0.99} 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎| 677/681 [50:18<00:11, 2.84s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌| 678/681 [50:21<00:08, 2.89s/it] {'loss': 0.4801, 'grad_norm': 20.216367721557617, 'learning_rate': 5.270012410216185e-11, 'margin_dpo/margin_mean': 24.323078155517578, 'margin_dpo/margin_std': 20.244949340820312, 'logps/chosen': -117.91678619384766, 'logps/rejected': -115.28776550292969, 'logps/ref_chosen': -89.85308837890625, 'logps/ref_rejected': -62.9010009765625, 'logits/chosen': -1.1461702585220337, 'logits/rejected': -0.45331066846847534, 'epoch': 1.0} 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌| 678/681 [50:21<00:08, 2.89s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋| 679/681 [50:24<00:05, 2.86s/it] {'loss': 0.4935, 'grad_norm': 22.173934936523438, 'learning_rate': 2.9644275480772416e-11, 'margin_dpo/margin_mean': 24.919471740722656, 'margin_dpo/margin_std': 18.897235870361328, 'logps/chosen': -92.57217407226562, 'logps/rejected': -130.9364013671875, 'logps/ref_chosen': -63.82649230957031, 'logps/ref_rejected': -77.2712631225586, 'logits/chosen': -2.05856990814209, 'logits/rejected': -1.7644459009170532, 'epoch': 1.0} 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋| 679/681 [50:24<00:05, 2.86s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 680/681 [50:27<00:02, 2.93s/it] {'loss': 0.5882, 'grad_norm': 29.65631103515625, 'learning_rate': 1.31753782067201e-11, 'margin_dpo/margin_mean': 21.06808090209961, 'margin_dpo/margin_std': 22.42251205444336, 'logps/chosen': -110.40392303466797, 'logps/rejected': -164.33082580566406, 'logps/ref_chosen': -81.73397064208984, 'logps/ref_rejected': -114.59278869628906, 'logits/chosen': -1.9342844486236572, 'logits/rejected': -1.8356274366378784, 'epoch': 1.0} 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 680/681 [50:27<00:02, 2.93s/it] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [50:30<00:00, 2.92s/it] {'loss': 0.5985, 'grad_norm': 29.214645385742188, 'learning_rate': 3.2938662507808745e-12, 'margin_dpo/margin_mean': 18.287107467651367, 'margin_dpo/margin_std': 15.459449768066406, 'logps/chosen': -133.166748046875, 'logps/rejected': -161.36431884765625, 'logps/ref_chosen': -103.17874145507812, 'logps/ref_rejected': -113.08921813964844, 'logits/chosen': -1.2867460250854492, 'logits/rejected': -1.1931159496307373, 'epoch': 1.0} 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [50:30<00:00, 2.92s/it][INFO|trainer.py:3984] 2026-04-24 00:45:51,574 >> Saving model checkpoint to /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-681 [INFO|configuration_utils.py:419] 2026-04-24 00:45:51,580 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-681/config.json [INFO|configuration_utils.py:911] 2026-04-24 00:45:51,583 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-681/generation_config.json [INFO|modeling_utils.py:3580] 2026-04-24 00:46:31,178 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-681/model.safetensors.index.json. [INFO|tokenization_utils_base.py:2510] 2026-04-24 00:46:31,185 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-681/tokenizer_config.json [INFO|tokenization_utils_base.py:2519] 2026-04-24 00:46:31,189 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-681/special_tokens_map.json [INFO|trainer.py:4083] 2026-04-24 00:49:56,351 >> Deleting older checkpoint [/scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/checkpoint-400] due to args.save_total_limit [INFO|trainer.py:2681] 2026-04-24 00:49:58,792 >> Training completed. Do not forget to share your model on huggingface.co/models =) {'train_runtime': 3298.7616, 'train_samples_per_second': 13.216, 'train_steps_per_second': 0.206, 'train_loss': 0.7553340482816823, 'epoch': 1.0} 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [54:51<00:00, 2.92s/it] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [54:51<00:00, 4.83s/it] ***** train metrics ***** epoch = 1.0 total_flos = 0GF train_loss = 0.7553 train_runtime = 0:54:58.76 train_samples = 43598 train_samples_per_second = 13.216 train_steps_per_second = 0.206 2026-04-24 00:49:58 - INFO - __main__ - *** Training complete *** 2026-04-24 00:49:58 - INFO - __main__ - *** Save model *** [INFO|configuration_utils.py:419] 2026-04-24 00:50:17,099 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/config.json [INFO|configuration_utils.py:911] 2026-04-24 00:50:17,122 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/generation_config.json [INFO|modeling_utils.py:3580] 2026-04-24 00:51:14,353 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 7 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/model.safetensors.index.json. [INFO|tokenization_utils_base.py:2510] 2026-04-24 00:51:14,358 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/tokenizer_config.json [INFO|tokenization_utils_base.py:2519] 2026-04-24 00:51:14,360 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/special_tokens_map.json 2026-04-24 00:51:14 - INFO - __main__ - Saved HF-compatible model artifacts to /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948 [INFO|modelcard.py:450] 2026-04-24 00:51:15,160 >> Dropping the following result as it does not have all the necessary fields: {'dataset': {'name': 'Anthropic/hh-rlhf', 'type': 'Anthropic/hh-rlhf'}} [INFO|configuration_utils.py:419] 2026-04-24 00:51:15,201 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-margin-dpo-hh-helpful-4xh200-batch-64-20260423-233948/config.json 2026-04-24 00:51:15 - INFO - __main__ - *** Evaluate *** [INFO|trainer.py:4307] 2026-04-24 00:51:15,202 >> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-24 00:51:15,202 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-24 00:51:15,202 >> Batch size = 8 0%| | 0/73 [00:00