2026-04-21 21:43:42 - INFO - __main__ - Model parameters ModelArguments(base_model_revision=None, model_name_or_path='/root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200', model_revision='main', model_code_revision=None, torch_dtype='bfloat16', tokenizer_name_or_path=None, trust_remote_code=False, attn_implementation='flash_attention_2', use_peft=False, lora_r=16, lora_alpha=32, lora_dropout=0.05, lora_target_modules=None, lora_modules_to_save=None, load_in_8bit=False, load_in_4bit=False, bnb_4bit_quant_type='nf4', use_bnb_nested_quant=False, bnb_4bit_quant_storage='uint8') 2026-04-21 21:43:42 - INFO - __main__ - Data parameters DataArguments(chat_template=None, dataset_mixer={'Anthropic/hh-rlhf': 1.0}, text_column='text', dataset_splits=['train', 'test'], dataset_configs=['helpful-base'], dataset_dir=None, preprocessing_num_workers=12, use_persistent_hf_cache=True, hf_cache_dir='/root/dynamic-dpo-v4/hf/datasets', truncation_side=None, auto_insert_empty_system_msg=True, preprocessing_log_samples=0, preprocessing_log_dir=None) 2026-04-21 21:43:42 - INFO - __main__ - Training/evaluation parameters NewDPOConfig( _n_gpu=1, accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, average_tokens_across_devices=False, batch_eval_metrics=False, beta=0.1, bf16=True, bf16_full_eval=False, data_seed=None, dataloader_drop_last=True, dataloader_num_workers=0, dataloader_persistent_workers=False, dataloader_pin_memory=True, dataloader_prefetch_factor=None, dataset_num_proc=12, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=None, disable_dropout=True, disable_tqdm=False, do_eval=True, do_predict=False, do_train=False, eta=0.1, eval_accumulation_steps=None, eval_delay=0, eval_do_concat_batches=True, eval_on_start=False, eval_steps=200, eval_strategy=IntervalStrategy.STEPS, eval_use_gather_object=False, f_alpha_divergence_coef=1.0, f_divergence_type=reverse_kl, force_use_ref_model=False, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, generate_during_eval=False, gradient_accumulation_steps=2, gradient_checkpointing=True, gradient_checkpointing_kwargs={'use_reentrant': False}, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_always_push=False, hub_margin_dataset_id=None, hub_model_id=jackf857/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun, hub_model_revision=main, hub_private_repo=None, hub_strategy=HubStrategy.EVERY_SAVE, hub_token=, ignore_data_skip=False, include_for_metrics=[], include_inputs_for_metrics=False, include_num_input_tokens_seen=False, include_tokens_per_second=False, is_encoder_decoder=None, jit_mode_eval=False, label_names=None, label_pad_token_id=-100, label_smoothing=0.0, label_smoothing_factor=0.0, learning_rate=5e-07, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=info, log_level_replica=warning, log_on_each_node=True, logging_dir=outputs/llama3-8b-base-new-method-s_star0.4/runs/Apr21_21-43-41_f6a54ae9d6f6, logging_first_step=True, logging_nan_inf_filter=True, logging_steps=5, logging_strategy=IntervalStrategy.STEPS, loss_type=sigmoid, lr_scheduler_kwargs={}, lr_scheduler_type=SchedulerType.COSINE, margin_dataset_private=None, margin_dataset_split=train, margin_log_path=/root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun/margin_logs, margin_log_steps=1, margin_save_full=True, max_grad_norm=1.0, max_length=512, max_prompt_length=256, max_steps=-1, max_target_length=None, metric_for_best_model=None, model_adapter_name=None, model_init_kwargs=None, mp_parameters=, neftune_noise_alpha=None, no_cuda=False, non_finite_logits_handling=error, num_train_epochs=1, optim=OptimizerNames.ADAMW_TORCH, optim_args=None, optim_target_modules=None, output_dir=/root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun, overwrite_output_dir=False, padding_value=None, past_index=-1, per_device_eval_batch_size=8, per_device_train_batch_size=8, post_tokenization_log_dir=None, post_tokenization_log_samples=0, precompute_ref_batch_size=None, precompute_ref_eval_batch_size=None, precompute_ref_log_probs=False, prediction_loss_only=False, push_margin_dataset=True, push_to_hub=True, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, q_target=0.45, ray_scope=last, ref_adapter_name=None, ref_model_init_kwargs=None, ref_model_mixup_alpha=0.9, ref_model_sync_steps=64, reference_free=False, remove_unused_columns=False, report_to=['wandb'], require_explicit_ref_model=True, restore_callback_states_from_checkpoint=False, resume_from_checkpoint=None, reuse_tokenized_dataset=True, rpo_alpha=None, run_name=llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun, s_star=0.4, save_on_each_node=False, save_only_model=False, save_safetensors=True, save_steps=50, save_strategy=SaveStrategy.NO, save_total_limit=2, seed=42, sft_weight=0.0, skip_memory_metrics=True, sync_ref_model=False, tf32=None, tokenization_batch_size=128, tokenization_mode=online, tokenized_dataset_cache_dir=/root/dynamic-dpo-v4/tokenized_preferences, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torch_empty_cache_steps=None, torchdynamo=None, tp_size=0, tpu_metrics_debug=False, tpu_num_cores=None, trainer_type=new_dpo, truncation_mode=keep_end, use_cpu=False, use_ipex=False, use_legacy_prediction_loop=False, use_liger_kernel=False, use_mps_device=False, wandb_project=llama3-8b-base-new-method-hh-beta-0.1, warmup_ratio=0.1, warmup_steps=0, weight_decay=0.0, ) 2026-04-21 21:43:42 - INFO - __main__ - Using W&B project from training args: llama3-8b-base-new-method-hh-beta-0.1 2026-04-21 21:43:42 - INFO - __main__ - New-DPO parameters: beta=0.1, q_target=0.45, s_star=0.4, eta=0.1 2026-04-21 21:43:42 - INFO - __main__ - Using persistent HF datasets cache at /root/dynamic-dpo-v4/hf/datasets Normalizing raw HH preferences (train): 0%| | 0/43598 [00:00> loading file tokenizer.json [INFO|tokenization_utils_base.py:2058] 2026-04-21 21:43:52,551 >> loading file tokenizer.model [INFO|tokenization_utils_base.py:2058] 2026-04-21 21:43:52,551 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:2058] 2026-04-21 21:43:52,551 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:2058] 2026-04-21 21:43:52,551 >> loading file tokenizer_config.json [INFO|tokenization_utils_base.py:2058] 2026-04-21 21:43:52,551 >> loading file chat_template.jinja Normalizing raw HH preferences (test): 98%|█████████▊| 2298/2339 [00:00<00:00, 11513.59 examples/s] Normalizing raw HH preferences (test): 100%|██████████| 2339/2339 [00:00<00:00, 11256.72 examples/s] [INFO|tokenization_utils_base.py:2323] 2026-04-21 21:43:52,994 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you. warnings.warn( [WARNING|logging.py:328] 2026-04-21 21:43:53,107 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you. warnings.warn( Loading checkpoint shards: 0%| | 0/7 [00:00> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Loading checkpoint shards: 100%|██████████| 7/7 [00:00<00:00, 1480.37it/s] Loading checkpoint shards: 0%| | 0/7 [00:00> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. 2026-04-21 21:43:53 - INFO - __main__ - Processed train sample 41905: Prompt: <|begin_of_text|><|start_header_id|>user<|end_header_id|> What is the best way to clean my refrigerator?<|eot_id|><|start_header_id|>assistant<|end_header_id|> Well, I have a few ideas, but the best way to know what works best is to run some tests and then analyze the data to figure out which technique works best.<|eot_id|><|start_header_id|>user<|end_header_id|> What are your ideas?<|eot_id|> Chosen: <|start_header_id|>assistant<|end_header_id|> Some of my ideas include wiping down the interior with a rag, using bleach and water to wipe down the door, emptying the contents of the fridge to scrub the shelves with a clean cloth, cleaning the interior of the fridge with vinegar and water, and using a vacuum cleaner to clean the interior.<|eot_id|> Rejected: <|start_header_id|>assistant<|end_header_id|> I was thinking that you could try different types of soap, which could be a variation of “soap,” like antibacterial soap. I was also thinking that you could try using different tools, like a toothbrush, to clean the inside. Or I was thinking that you could try different ways to get the gunk out, like using a squeegee, or using a paper towel. I was also thinking you could try other types of cleaning, like vacuuming, but I think that could have the opposite of the desired effect.<|eot_id|> Loading checkpoint shards: 0%| | 0/7 [00:00> loading configuration file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/config.json [INFO|configuration_utils.py:765] 2026-04-21 21:43:53,229 >> Model config LlamaConfig { "architectures": [ "LlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 128000, "eos_token_id": 128001, "head_dim": 128, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 14336, "max_position_embeddings": 8192, "mlp_bias": false, "model_type": "llama", "num_attention_heads": 32, "num_hidden_layers": 32, "num_key_value_heads": 8, "pretraining_tp": 1, "rms_norm_eps": 1e-05, "rope_scaling": null, "rope_theta": 500000.0, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.51.0", "use_cache": false, "vocab_size": 128256 } Loading checkpoint shards: 100%|██████████| 7/7 [00:00<00:00, 1467.86it/s] [WARNING|trainer.py:821] 2026-04-21 21:43:53,235 >> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. [INFO|modeling_utils.py:1121] 2026-04-21 21:43:53,238 >> loading weights file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/model.safetensors.index.json [INFO|modeling_utils.py:2167] 2026-04-21 21:43:53,238 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16. [WARNING|logging.py:328] 2026-04-21 21:43:53,239 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. [INFO|configuration_utils.py:1142] 2026-04-21 21:43:53,241 >> Generate config GenerationConfig { "bos_token_id": 128000, "eos_token_id": 128001, "use_cache": false } Loading checkpoint shards: 0%| | 0/7 [00:00> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Loading checkpoint shards: 0%| | 0/7 [00:00> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. Loading checkpoint shards: 14%|█▍ | 1/7 [00:01<00:10, 1.70s/it] Loading checkpoint shards: 29%|██▊ | 2/7 [00:03<00:08, 1.73s/it] Loading checkpoint shards: 43%|████▎ | 3/7 [00:05<00:06, 1.75s/it] Loading checkpoint shards: 57%|█████▋ | 4/7 [00:07<00:05, 1.78s/it] Loading checkpoint shards: 71%|███████▏ | 5/7 [00:08<00:03, 1.76s/it] Loading checkpoint shards: 86%|████████▌ | 6/7 [00:10<00:01, 1.78s/it] Loading checkpoint shards: 100%|██████████| 7/7 [00:11<00:00, 1.50s/it] Loading checkpoint shards: 100%|██████████| 7/7 [00:11<00:00, 1.64s/it] [INFO|modeling_utils.py:4926] 2026-04-21 21:44:04,766 >> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|modeling_utils.py:4934] 2026-04-21 21:44:04,766 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. [INFO|configuration_utils.py:1095] 2026-04-21 21:44:04,768 >> loading configuration file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/generation_config.json [INFO|configuration_utils.py:1142] 2026-04-21 21:44:04,769 >> Generate config GenerationConfig { "bos_token_id": 128000, "do_sample": true, "eos_token_id": 128001, "max_length": 4096, "temperature": 0.6, "top_p": 0.9 } [INFO|configuration_utils.py:691] 2026-04-21 21:44:04,769 >> loading configuration file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/config.json [INFO|configuration_utils.py:765] 2026-04-21 21:44:04,770 >> Model config LlamaConfig { "architectures": [ "LlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 128000, "eos_token_id": 128001, "head_dim": 128, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 14336, "max_position_embeddings": 8192, "mlp_bias": false, "model_type": "llama", "num_attention_heads": 32, "num_hidden_layers": 32, "num_key_value_heads": 8, "pretraining_tp": 1, "rms_norm_eps": 1e-05, "rope_scaling": null, "rope_theta": 500000.0, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.51.0", "use_cache": false, "vocab_size": 128256 } [INFO|modeling_utils.py:1121] 2026-04-21 21:44:04,771 >> loading weights file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/model.safetensors.index.json [INFO|modeling_utils.py:2167] 2026-04-21 21:44:04,771 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16. [INFO|configuration_utils.py:1142] 2026-04-21 21:44:04,773 >> Generate config GenerationConfig { "bos_token_id": 128000, "eos_token_id": 128001, "use_cache": false } Loading checkpoint shards: 0%| | 0/7 [00:00> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|modeling_utils.py:4934] 2026-04-21 21:44:15,821 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. [INFO|configuration_utils.py:1095] 2026-04-21 21:44:15,824 >> loading configuration file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/generation_config.json [INFO|configuration_utils.py:1142] 2026-04-21 21:44:15,824 >> Generate config GenerationConfig { "bos_token_id": 128000, "do_sample": true, "eos_token_id": 128001, "max_length": 4096, "temperature": 0.6, "top_p": 0.9 } [WARNING|trainer.py:821] 2026-04-21 21:44:15,826 >> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. [WARNING|trainer.py:816] 2026-04-21 21:44:15,826 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:44:15,835 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:44:15,836 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. Tokenizing test (num_proc=12): 0%| | 0/2339 [00:00> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. Saving the dataset (0/1 shards): 0%| | 0/2339 [00:00> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `NewDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [WARNING|trainer.py:816] 2026-04-21 21:54:42,677 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:54:42,678 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:54:42,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:54:42,694 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:54:42,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:54:42,698 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:54:42,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:54:42,699 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-21 21:54:42,706 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `NewDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [WARNING|trainer.py:816] 2026-04-21 21:54:42,710 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `NewDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [WARNING|trainer.py:816] 2026-04-21 21:54:42,711 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `NewDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [INFO|trainer.py:748] 2026-04-21 21:54:47,582 >> Using auto half precision backend /root/dynamic-dpo-v4/.venv/lib/python3.11/site-packages/accelerate/accelerator.py:1557: UserWarning: Upcasted low precision parameters in LlamaForCausalLM because mixed precision turned on in FSDP. Affects: model.embed_tokens.weight, model.norm.weight, lm_head.weight. warnings.warn( /root/dynamic-dpo-v4/.venv/lib/python3.11/site-packages/accelerate/accelerator.py:1557: UserWarning: Upcasted low precision parameters in LlamaDecoderLayer because mixed precision turned on in FSDP. Affects: self_attn.q_proj.weight, self_attn.k_proj.weight, self_attn.v_proj.weight, self_attn.o_proj.weight, mlp.gate_proj.weight, mlp.up_proj.weight, mlp.down_proj.weight, input_layernorm.weight, post_attention_layernorm.weight. warnings.warn( /root/dynamic-dpo-v4/.venv/lib/python3.11/site-packages/accelerate/accelerator.py:1563: UserWarning: FSDP upcast of low precision parameters may affect the precision of model checkpoints. warnings.warn( [INFO|trainer.py:2414] 2026-04-21 21:54:56,469 >> ***** Running training ***** [INFO|trainer.py:2415] 2026-04-21 21:54:56,469 >> Num examples = 43,598 [INFO|trainer.py:2416] 2026-04-21 21:54:56,469 >> Num Epochs = 1 [INFO|trainer.py:2417] 2026-04-21 21:54:56,469 >> Instantaneous batch size per device = 8 [INFO|trainer.py:2420] 2026-04-21 21:54:56,469 >> Total train batch size (w. parallel, distributed & accumulation) = 64 [INFO|trainer.py:2421] 2026-04-21 21:54:56,469 >> Gradient Accumulation steps = 2 [INFO|trainer.py:2422] 2026-04-21 21:54:56,469 >> Total optimization steps = 681 [INFO|trainer.py:2423] 2026-04-21 21:54:56,470 >> Number of trainable parameters = 2,007,565,312 [INFO|integration_utils.py:831] 2026-04-21 21:54:56,471 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true" wandb: Currently logged in as: feng-cheng (feng-cheng-northeastern-university). Use `wandb login --relogin` to force relogin wandb: - Waiting for wandb.init()... wandb: \ Waiting for wandb.init()... wandb: | Waiting for wandb.init()... wandb: / Waiting for wandb.init()... wandb: - Waiting for wandb.init()... wandb: \ Waiting for wandb.init()... wandb: wandb version 0.26.0 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.17.5 wandb: Run data is saved locally in /root/dynamic-dpo-v4/wandb/wandb/run-20260421_215457-yuvsexn3 wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun wandb: ⭐️ View project at https://wandb.ai/feng-cheng-northeastern-university/llama3-8b-base-new-method-hh-beta-0.1 wandb: 🚀 View run at https://wandb.ai/feng-cheng-northeastern-university/llama3-8b-base-new-method-hh-beta-0.1/runs/yuvsexn3 0%| | 0/681 [00:00> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-21 21:55:04,936 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-21 21:55:04,938 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-21 21:55:04,946 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed 0%| | 1/681 [00:02<31:11, 2.75s/it] {'loss': 1.389, 'grad_norm': 83.51022338867188, 'learning_rate': 0.0, 'fcm_dpo/beta': 0.10000000149011612, 'fcm_dpo/q_t': 0.5005706548690796, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': -0.02287006378173828, 'margin_dpo/margin_mean': -0.02287048101425171, 'margin_dpo/margin_std': 0.41920793056488037, 'logps/chosen': -50.1435661315918, 'logps/rejected': -74.09991455078125, 'logps/ref_chosen': -50.14883804321289, 'logps/ref_rejected': -74.1280517578125, 'logits/chosen': -0.4974287748336792, 'logits/rejected': -0.43299180269241333, 'epoch': 0.0} 0%| | 1/681 [00:02<31:11, 2.75s/it] 0%| | 2/681 [00:05<29:28, 2.60s/it] 0%| | 3/681 [00:07<29:13, 2.59s/it] 1%| | 4/681 [00:10<29:27, 2.61s/it] 1%| | 5/681 [00:13<29:18, 2.60s/it] {'loss': 1.3874, 'grad_norm': 89.51734924316406, 'learning_rate': 2.898550724637681e-08, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.500194251537323, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': -0.007774516940116882, 'margin_dpo/margin_mean': -0.007774434983730316, 'margin_dpo/margin_std': 0.37205374240875244, 'logps/chosen': -56.05889892578125, 'logps/rejected': -78.68702697753906, 'logps/ref_chosen': -56.05734634399414, 'logps/ref_rejected': -78.69325256347656, 'logits/chosen': -0.4900396764278412, 'logits/rejected': -0.4533483386039734, 'epoch': 0.01} 1%| | 5/681 [00:13<29:18, 2.60s/it] 1%| | 6/681 [00:15<27:43, 2.46s/it] 1%| | 7/681 [00:17<27:03, 2.41s/it] 1%| | 8/681 [00:19<26:45, 2.38s/it] 1%|▏ | 9/681 [00:22<27:19, 2.44s/it] 1%|▏ | 10/681 [00:25<27:53, 2.49s/it] {'loss': 1.3849, 'grad_norm': 70.37712097167969, 'learning_rate': 6.521739130434782e-08, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.49955543875694275, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.017793914303183556, 'margin_dpo/margin_mean': 0.017794013023376465, 'margin_dpo/margin_std': 0.40502458810806274, 'logps/chosen': -59.53853225708008, 'logps/rejected': -91.18217468261719, 'logps/ref_chosen': -59.54457473754883, 'logps/ref_rejected': -91.17041778564453, 'logits/chosen': -0.49457064270973206, 'logits/rejected': -0.4558273255825043, 'epoch': 0.01} 1%|▏ | 10/681 [00:25<27:53, 2.49s/it] 2%|▏ | 11/681 [00:27<28:39, 2.57s/it] 2%|▏ | 12/681 [00:30<28:43, 2.58s/it] 2%|▏ | 13/681 [00:33<29:02, 2.61s/it] 2%|▏ | 14/681 [00:35<28:40, 2.58s/it] 2%|▏ | 15/681 [00:38<28:33, 2.57s/it] {'loss': 1.3854, 'grad_norm': 64.22943878173828, 'learning_rate': 1.0144927536231885e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.4996658265590668, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.013358664698898792, 'margin_dpo/margin_mean': 0.01335877738893032, 'margin_dpo/margin_std': 0.3935544788837433, 'logps/chosen': -58.85475540161133, 'logps/rejected': -92.97565460205078, 'logps/ref_chosen': -58.83195877075195, 'logps/ref_rejected': -92.93949890136719, 'logits/chosen': -0.48777151107788086, 'logits/rejected': -0.4582846164703369, 'epoch': 0.02} 2%|▏ | 15/681 [00:38<28:33, 2.57s/it] 2%|▏ | 16/681 [00:40<28:04, 2.53s/it] 2%|▏ | 17/681 [00:43<27:48, 2.51s/it] 3%|▎ | 18/681 [00:45<27:41, 2.51s/it] 3%|▎ | 19/681 [00:48<27:49, 2.52s/it] 3%|▎ | 20/681 [00:50<27:48, 2.52s/it] {'loss': 1.3775, 'grad_norm': 73.8072509765625, 'learning_rate': 1.3768115942028986e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.49772581458091736, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.09099045395851135, 'margin_dpo/margin_mean': 0.09099055081605911, 'margin_dpo/margin_std': 0.35805386304855347, 'logps/chosen': -59.618431091308594, 'logps/rejected': -82.83003234863281, 'logps/ref_chosen': -59.6396598815918, 'logps/ref_rejected': -82.76026916503906, 'logits/chosen': -0.5034265518188477, 'logits/rejected': -0.45439839363098145, 'epoch': 0.03} 3%|▎ | 20/681 [00:50<27:48, 2.52s/it] 3%|▎ | 21/681 [00:53<27:33, 2.50s/it] 3%|▎ | 22/681 [00:55<27:54, 2.54s/it] 3%|▎ | 23/681 [00:58<28:52, 2.63s/it] 4%|▎ | 24/681 [01:01<28:54, 2.64s/it] 4%|▎ | 25/681 [01:03<28:50, 2.64s/it] {'loss': 1.3652, 'grad_norm': 73.70997619628906, 'learning_rate': 1.7391304347826085e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.49459710717201233, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.21624115109443665, 'margin_dpo/margin_mean': 0.21624140441417694, 'margin_dpo/margin_std': 0.4134984016418457, 'logps/chosen': -53.1678352355957, 'logps/rejected': -89.17488098144531, 'logps/ref_chosen': -53.205284118652344, 'logps/ref_rejected': -88.99608612060547, 'logits/chosen': -0.5030972361564636, 'logits/rejected': -0.47600775957107544, 'epoch': 0.04} 4%|▎ | 25/681 [01:03<28:50, 2.64s/it] 4%|▍ | 26/681 [01:06<27:35, 2.53s/it] 4%|▍ | 27/681 [01:08<27:21, 2.51s/it] 4%|▍ | 28/681 [01:11<27:32, 2.53s/it] 4%|▍ | 29/681 [01:13<26:26, 2.43s/it] 4%|▍ | 30/681 [01:16<27:04, 2.50s/it] {'loss': 1.3399, 'grad_norm': 87.57738494873047, 'learning_rate': 2.1014492753623187e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.4880555272102356, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.4783309996128082, 'margin_dpo/margin_mean': 0.47833117842674255, 'margin_dpo/margin_std': 0.5532158017158508, 'logps/chosen': -53.46284103393555, 'logps/rejected': -98.26229858398438, 'logps/ref_chosen': -53.5526008605957, 'logps/ref_rejected': -97.87371826171875, 'logits/chosen': -0.5126634240150452, 'logits/rejected': -0.4724017083644867, 'epoch': 0.04} 4%|▍ | 30/681 [01:16<27:04, 2.50s/it] 5%|▍ | 31/681 [01:18<27:36, 2.55s/it] 5%|▍ | 32/681 [01:21<28:13, 2.61s/it] 5%|▍ | 33/681 [01:23<27:44, 2.57s/it] 5%|▍ | 34/681 [01:26<27:52, 2.58s/it] 5%|▌ | 35/681 [01:29<28:04, 2.61s/it] {'loss': 1.3121, 'grad_norm': 82.80741119384766, 'learning_rate': 2.463768115942029e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.48065242171287537, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.7770208120346069, 'margin_dpo/margin_mean': 0.7770206332206726, 'margin_dpo/margin_std': 0.8431359529495239, 'logps/chosen': -56.210533142089844, 'logps/rejected': -92.4262924194336, 'logps/ref_chosen': -56.3298454284668, 'logps/ref_rejected': -91.76858520507812, 'logits/chosen': -0.5122802257537842, 'logits/rejected': -0.4792337417602539, 'epoch': 0.05} 5%|▌ | 35/681 [01:29<28:04, 2.61s/it] 5%|▌ | 36/681 [01:31<28:11, 2.62s/it] 5%|▌ | 37/681 [01:34<28:01, 2.61s/it] 6%|▌ | 38/681 [01:36<26:42, 2.49s/it] 6%|▌ | 39/681 [01:39<26:48, 2.51s/it] 6%|▌ | 40/681 [01:41<27:24, 2.57s/it] {'loss': 1.2605, 'grad_norm': 60.205745697021484, 'learning_rate': 2.8260869565217386e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.46629637479782104, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 1.3613355159759521, 'margin_dpo/margin_mean': 1.3613355159759521, 'margin_dpo/margin_std': 1.4631270170211792, 'logps/chosen': -54.278953552246094, 'logps/rejected': -84.19891357421875, 'logps/ref_chosen': -54.38492965698242, 'logps/ref_rejected': -82.94353485107422, 'logits/chosen': -0.5219647884368896, 'logits/rejected': -0.4850676953792572, 'epoch': 0.06} 6%|▌ | 40/681 [01:41<27:24, 2.57s/it] 6%|▌ | 41/681 [01:44<27:19, 2.56s/it] 6%|▌ | 42/681 [01:46<27:11, 2.55s/it] 6%|▋ | 43/681 [01:49<27:13, 2.56s/it] 6%|▋ | 44/681 [01:52<27:25, 2.58s/it] 7%|▋ | 45/681 [01:54<27:30, 2.59s/it] {'loss': 1.1601, 'grad_norm': 68.1404800415039, 'learning_rate': 3.188405797101449e-07, 'fcm_dpo/beta': 0.10669933259487152, 'fcm_dpo/q_t': 0.4380177855491638, 'fcm_dpo/delta': 0.12330988794565201, 'fcm_dpo/margin': 2.4058146476745605, 'margin_dpo/margin_mean': 2.4058141708374023, 'margin_dpo/margin_std': 2.224526882171631, 'logps/chosen': -54.63410186767578, 'logps/rejected': -100.20404052734375, 'logps/ref_chosen': -54.862335205078125, 'logps/ref_rejected': -98.0264663696289, 'logits/chosen': -0.5121681094169617, 'logits/rejected': -0.4841387867927551, 'epoch': 0.07} 7%|▋ | 45/681 [01:54<27:30, 2.59s/it] 7%|▋ | 46/681 [01:57<27:36, 2.61s/it] 7%|▋ | 47/681 [02:00<27:42, 2.62s/it] 7%|▋ | 48/681 [02:02<28:05, 2.66s/it] 7%|▋ | 49/681 [02:05<27:41, 2.63s/it] 7%|▋ | 50/681 [02:08<27:45, 2.64s/it] {'loss': 1.0694, 'grad_norm': 68.53203582763672, 'learning_rate': 3.5507246376811595e-07, 'fcm_dpo/beta': 0.11535553634166718, 'fcm_dpo/q_t': 0.4073707163333893, 'fcm_dpo/delta': 0.0067912861704826355, 'fcm_dpo/margin': 3.4076931476593018, 'margin_dpo/margin_mean': 3.4076931476593018, 'margin_dpo/margin_std': 3.365309953689575, 'logps/chosen': -58.12030029296875, 'logps/rejected': -94.91819763183594, 'logps/ref_chosen': -58.304595947265625, 'logps/ref_rejected': -91.69480895996094, 'logits/chosen': -0.562477707862854, 'logits/rejected': -0.5116110444068909, 'epoch': 0.07} 7%|▋ | 50/681 [02:08<27:45, 2.64s/it] 7%|▋ | 51/681 [02:10<27:40, 2.64s/it] 8%|▊ | 52/681 [02:13<26:58, 2.57s/it] 8%|▊ | 53/681 [02:15<26:52, 2.57s/it] 8%|▊ | 54/681 [02:18<26:09, 2.50s/it] 8%|▊ | 55/681 [02:20<25:19, 2.43s/it] {'loss': 0.968, 'grad_norm': 52.22880935668945, 'learning_rate': 3.9130434782608694e-07, 'fcm_dpo/beta': 0.10359964519739151, 'fcm_dpo/q_t': 0.3696584403514862, 'fcm_dpo/delta': -0.19395628571510315, 'fcm_dpo/margin': 5.653986930847168, 'margin_dpo/margin_mean': 5.653987407684326, 'margin_dpo/margin_std': 5.778587818145752, 'logps/chosen': -56.34511184692383, 'logps/rejected': -91.62686157226562, 'logps/ref_chosen': -56.06591796875, 'logps/ref_rejected': -85.69367980957031, 'logits/chosen': -0.5961982011795044, 'logits/rejected': -0.5513918399810791, 'epoch': 0.08} 8%|▊ | 55/681 [02:20<25:19, 2.43s/it] 8%|▊ | 56/681 [02:23<26:13, 2.52s/it] 8%|▊ | 57/681 [02:25<26:03, 2.51s/it] 9%|▊ | 58/681 [02:28<26:24, 2.54s/it] 9%|▊ | 59/681 [02:30<26:44, 2.58s/it] 9%|▉ | 60/681 [02:33<26:04, 2.52s/it] {'loss': 0.9663, 'grad_norm': 53.557796478271484, 'learning_rate': 4.2753623188405794e-07, 'fcm_dpo/beta': 0.08351375162601471, 'fcm_dpo/q_t': 0.3656514883041382, 'fcm_dpo/delta': -0.2116357386112213, 'fcm_dpo/margin': 7.191043853759766, 'margin_dpo/margin_mean': 7.191043853759766, 'margin_dpo/margin_std': 7.640544891357422, 'logps/chosen': -62.0140266418457, 'logps/rejected': -98.24510192871094, 'logps/ref_chosen': -60.6871337890625, 'logps/ref_rejected': -89.72715759277344, 'logits/chosen': -0.614372193813324, 'logits/rejected': -0.5657563209533691, 'epoch': 0.09} 9%|▉ | 60/681 [02:33<26:04, 2.52s/it] 9%|▉ | 61/681 [02:35<26:24, 2.56s/it] 9%|▉ | 62/681 [02:38<26:46, 2.60s/it] 9%|▉ | 63/681 [02:41<26:32, 2.58s/it] 9%|▉ | 64/681 [02:43<26:09, 2.54s/it] 10%|▉ | 65/681 [02:46<26:20, 2.57s/it] {'loss': 0.9904, 'grad_norm': 41.2183723449707, 'learning_rate': 4.63768115942029e-07, 'fcm_dpo/beta': 0.06861739605665207, 'fcm_dpo/q_t': 0.37094324827194214, 'fcm_dpo/delta': -0.19986796379089355, 'fcm_dpo/margin': 8.604007720947266, 'margin_dpo/margin_mean': 8.604008674621582, 'margin_dpo/margin_std': 10.206991195678711, 'logps/chosen': -63.8182487487793, 'logps/rejected': -103.9700927734375, 'logps/ref_chosen': -61.75325393676758, 'logps/ref_rejected': -93.30108642578125, 'logits/chosen': -0.6187667846679688, 'logits/rejected': -0.5872984528541565, 'epoch': 0.1} 10%|▉ | 65/681 [02:46<26:20, 2.57s/it] 10%|▉ | 66/681 [02:48<26:30, 2.59s/it] 10%|▉ | 67/681 [02:51<25:30, 2.49s/it] 10%|▉ | 68/681 [02:53<25:12, 2.47s/it] 10%|█ | 69/681 [02:56<26:21, 2.58s/it] 10%|█ | 70/681 [02:58<25:52, 2.54s/it] {'loss': 0.98, 'grad_norm': 39.173946380615234, 'learning_rate': 5e-07, 'fcm_dpo/beta': 0.05661847069859505, 'fcm_dpo/q_t': 0.3718249201774597, 'fcm_dpo/delta': -0.1891469657421112, 'fcm_dpo/margin': 10.259188652038574, 'margin_dpo/margin_mean': 10.259187698364258, 'margin_dpo/margin_std': 11.377448081970215, 'logps/chosen': -63.28239059448242, 'logps/rejected': -98.00967407226562, 'logps/ref_chosen': -59.548004150390625, 'logps/ref_rejected': -84.01609802246094, 'logits/chosen': -0.6201697587966919, 'logits/rejected': -0.5813671350479126, 'epoch': 0.1} 10%|█ | 70/681 [02:58<25:52, 2.54s/it] 10%|█ | 71/681 [03:01<26:09, 2.57s/it] 11%|█ | 72/681 [03:04<26:18, 2.59s/it] 11%|█ | 73/681 [03:06<26:15, 2.59s/it] 11%|█ | 74/681 [03:09<26:07, 2.58s/it] 11%|█ | 75/681 [03:11<26:16, 2.60s/it] {'loss': 0.9415, 'grad_norm': 31.45458221435547, 'learning_rate': 4.999176576834721e-07, 'fcm_dpo/beta': 0.04313134402036667, 'fcm_dpo/q_t': 0.34925174713134766, 'fcm_dpo/delta': -0.354682594537735, 'fcm_dpo/margin': 17.1324405670166, 'margin_dpo/margin_mean': 17.132436752319336, 'margin_dpo/margin_std': 21.05853843688965, 'logps/chosen': -67.56647491455078, 'logps/rejected': -122.8857421875, 'logps/ref_chosen': -59.86931228637695, 'logps/ref_rejected': -98.05613708496094, 'logits/chosen': -0.6573521494865417, 'logits/rejected': -0.6307379603385925, 'epoch': 0.11} 11%|█ | 75/681 [03:11<26:16, 2.60s/it] 11%|█ | 76/681 [03:14<26:05, 2.59s/it] 11%|█▏ | 77/681 [03:16<25:01, 2.49s/it] 11%|█▏ | 78/681 [03:19<25:26, 2.53s/it] 12%|█▏ | 79/681 [03:21<25:39, 2.56s/it] 12%|█▏ | 80/681 [03:24<25:31, 2.55s/it] {'loss': 0.9939, 'grad_norm': 28.407453536987305, 'learning_rate': 4.996706849759452e-07, 'fcm_dpo/beta': 0.03168341889977455, 'fcm_dpo/q_t': 0.3687431514263153, 'fcm_dpo/delta': -0.22349825501441956, 'fcm_dpo/margin': 19.243648529052734, 'margin_dpo/margin_mean': 19.243648529052734, 'margin_dpo/margin_std': 23.982927322387695, 'logps/chosen': -66.81375122070312, 'logps/rejected': -116.2920913696289, 'logps/ref_chosen': -56.18925857543945, 'logps/ref_rejected': -86.42393493652344, 'logits/chosen': -0.6761894822120667, 'logits/rejected': -0.6410226821899414, 'epoch': 0.12} 12%|█▏ | 80/681 [03:24<25:31, 2.55s/it] 12%|█▏ | 81/681 [03:27<26:11, 2.62s/it] 12%|█▏ | 82/681 [03:29<25:56, 2.60s/it] 12%|█▏ | 83/681 [03:32<25:18, 2.54s/it] 12%|█▏ | 84/681 [03:34<25:51, 2.60s/it] 12%|█▏ | 85/681 [03:37<25:48, 2.60s/it] {'loss': 1.0175, 'grad_norm': 26.530624389648438, 'learning_rate': 4.992592445678582e-07, 'fcm_dpo/beta': 0.025587420910596848, 'fcm_dpo/q_t': 0.3756103515625, 'fcm_dpo/delta': -0.18879233300685883, 'fcm_dpo/margin': 22.462379455566406, 'margin_dpo/margin_mean': 22.462383270263672, 'margin_dpo/margin_std': 29.141366958618164, 'logps/chosen': -73.97752380371094, 'logps/rejected': -134.43350219726562, 'logps/ref_chosen': -60.018287658691406, 'logps/ref_rejected': -98.01185607910156, 'logits/chosen': -0.6672359704971313, 'logits/rejected': -0.6387097835540771, 'epoch': 0.12} 12%|█▏ | 85/681 [03:37<25:48, 2.60s/it] 13%|█▎ | 86/681 [03:40<26:05, 2.63s/it] 13%|█▎ | 87/681 [03:42<25:52, 2.61s/it] 13%|█▎ | 88/681 [03:45<25:53, 2.62s/it] 13%|█▎ | 89/681 [03:47<25:23, 2.57s/it] 13%|█▎ | 90/681 [03:50<25:02, 2.54s/it] {'loss': 1.0705, 'grad_norm': 26.1925106048584, 'learning_rate': 4.986836074908615e-07, 'fcm_dpo/beta': 0.02217969484627247, 'fcm_dpo/q_t': 0.38759851455688477, 'fcm_dpo/delta': -0.15027864277362823, 'fcm_dpo/margin': 24.47287368774414, 'margin_dpo/margin_mean': 24.47287368774414, 'margin_dpo/margin_std': 37.618186950683594, 'logps/chosen': -78.13359069824219, 'logps/rejected': -139.5206756591797, 'logps/ref_chosen': -59.8709831237793, 'logps/ref_rejected': -96.78519439697266, 'logits/chosen': -0.6896823644638062, 'logits/rejected': -0.6781142354011536, 'epoch': 0.13} 13%|█▎ | 90/681 [03:50<25:02, 2.54s/it] 13%|█▎ | 91/681 [03:52<25:06, 2.55s/it] 14%|█▎ | 92/681 [03:55<24:50, 2.53s/it] 14%|█▎ | 93/681 [03:57<23:58, 2.45s/it] 14%|█▍ | 94/681 [04:00<25:06, 2.57s/it] 14%|█▍ | 95/681 [04:02<24:42, 2.53s/it] {'loss': 1.0485, 'grad_norm': 21.816125869750977, 'learning_rate': 4.979441529392784e-07, 'fcm_dpo/beta': 0.019353795796632767, 'fcm_dpo/q_t': 0.3899272084236145, 'fcm_dpo/delta': -0.10450585186481476, 'fcm_dpo/margin': 25.787700653076172, 'margin_dpo/margin_mean': 25.787700653076172, 'margin_dpo/margin_std': 34.392967224121094, 'logps/chosen': -74.7010269165039, 'logps/rejected': -128.2239227294922, 'logps/ref_chosen': -55.94385528564453, 'logps/ref_rejected': -83.6790542602539, 'logits/chosen': -0.690959095954895, 'logits/rejected': -0.6617642641067505, 'epoch': 0.14} 14%|█▍ | 95/681 [04:02<24:42, 2.53s/it] 14%|█▍ | 96/681 [04:05<24:45, 2.54s/it] 14%|█▍ | 97/681 [04:08<24:43, 2.54s/it] 14%|█▍ | 98/681 [04:10<24:15, 2.50s/it] 15%|█▍ | 99/681 [04:12<23:22, 2.41s/it] 15%|█▍ | 100/681 [04:15<23:52, 2.47s/it] {'loss': 1.057, 'grad_norm': 23.495582580566406, 'learning_rate': 4.970413680203148e-07, 'fcm_dpo/beta': 0.017355434596538544, 'fcm_dpo/q_t': 0.3915005326271057, 'fcm_dpo/delta': -0.09411351382732391, 'fcm_dpo/margin': 28.09613037109375, 'margin_dpo/margin_mean': 28.09613037109375, 'margin_dpo/margin_std': 37.85677719116211, 'logps/chosen': -78.07569885253906, 'logps/rejected': -135.23020935058594, 'logps/ref_chosen': -57.05888748168945, 'logps/ref_rejected': -86.11727142333984, 'logits/chosen': -0.6664688587188721, 'logits/rejected': -0.6419214010238647, 'epoch': 0.15} 15%|█▍ | 100/681 [04:15<23:52, 2.47s/it] 15%|█▍ | 101/681 [04:17<23:36, 2.44s/it] 15%|█▍ | 102/681 [04:19<23:24, 2.43s/it] 15%|█▌ | 103/681 [04:22<23:07, 2.40s/it] 15%|█▌ | 104/681 [04:24<22:36, 2.35s/it] 15%|█▌ | 105/681 [04:27<23:40, 2.47s/it] {'loss': 1.052, 'grad_norm': 22.918745040893555, 'learning_rate': 4.959758474331832e-07, 'fcm_dpo/beta': 0.01605215296149254, 'fcm_dpo/q_t': 0.38781845569610596, 'fcm_dpo/delta': -0.14830544590950012, 'fcm_dpo/margin': 32.540122985839844, 'margin_dpo/margin_mean': 32.540122985839844, 'margin_dpo/margin_std': 44.27156066894531, 'logps/chosen': -84.82267761230469, 'logps/rejected': -144.65260314941406, 'logps/ref_chosen': -59.20774459838867, 'logps/ref_rejected': -86.49754333496094, 'logits/chosen': -0.6817123889923096, 'logits/rejected': -0.6505634188652039, 'epoch': 0.15} 15%|█▌ | 105/681 [04:27<23:40, 2.47s/it] 16%|█▌ | 106/681 [04:29<23:41, 2.47s/it] 16%|█▌ | 107/681 [04:32<24:07, 2.52s/it] 16%|█▌ | 108/681 [04:34<23:40, 2.48s/it] 16%|█▌ | 109/681 [04:37<24:10, 2.54s/it] 16%|█▌ | 110/681 [04:40<24:27, 2.57s/it] {'loss': 1.0403, 'grad_norm': 18.83212661743164, 'learning_rate': 4.947482930773511e-07, 'fcm_dpo/beta': 0.013644491322338581, 'fcm_dpo/q_t': 0.38538408279418945, 'fcm_dpo/delta': -0.1275637000799179, 'fcm_dpo/margin': 38.23952102661133, 'margin_dpo/margin_mean': 38.23952102661133, 'margin_dpo/margin_std': 50.03200149536133, 'logps/chosen': -88.46613311767578, 'logps/rejected': -157.1068878173828, 'logps/ref_chosen': -60.437957763671875, 'logps/ref_rejected': -90.83917999267578, 'logits/chosen': -0.6280543804168701, 'logits/rejected': -0.5978196859359741, 'epoch': 0.16} 16%|█▌ | 110/681 [04:40<24:27, 2.57s/it] 16%|█▋ | 111/681 [04:42<24:18, 2.56s/it] 16%|█▋ | 112/681 [04:44<23:23, 2.47s/it] 17%|█▋ | 113/681 [04:47<23:47, 2.51s/it] 17%|█▋ | 114/681 [04:49<23:36, 2.50s/it] 17%|█▋ | 115/681 [04:52<24:04, 2.55s/it] {'loss': 1.0871, 'grad_norm': 27.971012115478516, 'learning_rate': 4.933595135901732e-07, 'fcm_dpo/beta': 0.012269817292690277, 'fcm_dpo/q_t': 0.39965710043907166, 'fcm_dpo/delta': -0.06500422954559326, 'fcm_dpo/margin': 37.55778121948242, 'margin_dpo/margin_mean': 37.55778121948242, 'margin_dpo/margin_std': 56.132057189941406, 'logps/chosen': -96.05851745605469, 'logps/rejected': -157.19485473632812, 'logps/ref_chosen': -61.7908821105957, 'logps/ref_rejected': -85.36943054199219, 'logits/chosen': -0.6194974184036255, 'logits/rejected': -0.5893136262893677, 'epoch': 0.17} 17%|█▋ | 115/681 [04:52<24:04, 2.55s/it] 17%|█▋ | 116/681 [04:55<23:26, 2.49s/it] 17%|█▋ | 117/681 [04:57<23:15, 2.48s/it] 17%|█▋ | 118/681 [05:00<23:37, 2.52s/it] 17%|█▋ | 119/681 [05:02<24:23, 2.60s/it] 18%|█▊ | 120/681 [05:05<24:15, 2.59s/it] {'loss': 1.0756, 'grad_norm': 21.68465805053711, 'learning_rate': 4.918104238142103e-07, 'fcm_dpo/beta': 0.01178265642374754, 'fcm_dpo/q_t': 0.39924171566963196, 'fcm_dpo/delta': -0.05727796629071236, 'fcm_dpo/margin': 38.61918640136719, 'margin_dpo/margin_mean': 38.61919021606445, 'margin_dpo/margin_std': 53.98603057861328, 'logps/chosen': -104.28592681884766, 'logps/rejected': -164.33416748046875, 'logps/ref_chosen': -65.3261489868164, 'logps/ref_rejected': -86.75518798828125, 'logits/chosen': -0.6018606424331665, 'logits/rejected': -0.580163836479187, 'epoch': 0.18} 18%|█▊ | 120/681 [05:05<24:15, 2.59s/it] 18%|█▊ | 121/681 [05:07<23:52, 2.56s/it] 18%|█▊ | 122/681 [05:10<23:10, 2.49s/it] 18%|█▊ | 123/681 [05:12<23:42, 2.55s/it] 18%|█▊ | 124/681 [05:15<23:42, 2.55s/it] 18%|█▊ | 125/681 [05:17<23:22, 2.52s/it] {'loss': 1.0102, 'grad_norm': 19.076047897338867, 'learning_rate': 4.90102044194588e-07, 'fcm_dpo/beta': 0.009986856020987034, 'fcm_dpo/q_t': 0.3725253939628601, 'fcm_dpo/delta': -0.2121940553188324, 'fcm_dpo/margin': 59.94220733642578, 'margin_dpo/margin_mean': 59.94221115112305, 'margin_dpo/margin_std': 79.08087158203125, 'logps/chosen': -102.51204681396484, 'logps/rejected': -205.34170532226562, 'logps/ref_chosen': -58.323204040527344, 'logps/ref_rejected': -101.2106704711914, 'logits/chosen': -0.5440986752510071, 'logits/rejected': -0.5444971323013306, 'epoch': 0.18} 18%|█▊ | 125/681 [05:17<23:22, 2.52s/it] 19%|█▊ | 126/681 [05:20<23:41, 2.56s/it] 19%|█▊ | 127/681 [05:23<23:50, 2.58s/it] 19%|█▉ | 128/681 [05:25<24:03, 2.61s/it] 19%|█▉ | 129/681 [05:28<23:51, 2.59s/it] 19%|█▉ | 130/681 [05:30<23:39, 2.58s/it] {'loss': 1.0949, 'grad_norm': 21.011878967285156, 'learning_rate': 4.882355001067891e-07, 'fcm_dpo/beta': 0.00899023748934269, 'fcm_dpo/q_t': 0.4008238911628723, 'fcm_dpo/delta': -0.0574115514755249, 'fcm_dpo/margin': 50.56972122192383, 'margin_dpo/margin_mean': 50.569725036621094, 'margin_dpo/margin_std': 77.31275939941406, 'logps/chosen': -103.4196548461914, 'logps/rejected': -183.76187133789062, 'logps/ref_chosen': -56.38518524169922, 'logps/ref_rejected': -86.15767669677734, 'logits/chosen': -0.5073991417884827, 'logits/rejected': -0.49552878737449646, 'epoch': 0.19} 19%|█▉ | 130/681 [05:31<23:39, 2.58s/it] 19%|█▉ | 131/681 [05:33<23:43, 2.59s/it] 19%|█▉ | 132/681 [05:36<23:19, 2.55s/it] 20%|█▉ | 133/681 [05:38<23:29, 2.57s/it] 20%|█▉ | 134/681 [05:41<22:46, 2.50s/it] 20%|█▉ | 135/681 [05:43<22:36, 2.48s/it] {'loss': 1.0502, 'grad_norm': 21.952133178710938, 'learning_rate': 4.862120211153265e-07, 'fcm_dpo/beta': 0.00818575732409954, 'fcm_dpo/q_t': 0.39124470949172974, 'fcm_dpo/delta': -0.10119257867336273, 'fcm_dpo/margin': 60.596702575683594, 'margin_dpo/margin_mean': 60.596702575683594, 'margin_dpo/margin_std': 81.37339782714844, 'logps/chosen': -103.59019470214844, 'logps/rejected': -204.8570556640625, 'logps/ref_chosen': -54.59065628051758, 'logps/ref_rejected': -95.26080322265625, 'logits/chosen': -0.487496942281723, 'logits/rejected': -0.4898650050163269, 'epoch': 0.2} 20%|█▉ | 135/681 [05:43<22:36, 2.48s/it] 20%|█▉ | 136/681 [05:46<23:09, 2.55s/it] 20%|██ | 137/681 [05:48<23:03, 2.54s/it] 20%|██ | 138/681 [05:51<22:23, 2.47s/it] 20%|██ | 139/681 [05:53<22:07, 2.45s/it] 21%|██ | 140/681 [05:55<21:47, 2.42s/it] {'loss': 1.064, 'grad_norm': 20.813894271850586, 'learning_rate': 4.840329401637809e-07, 'fcm_dpo/beta': 0.007442955859005451, 'fcm_dpo/q_t': 0.39486831426620483, 'fcm_dpo/delta': -0.09183903783559799, 'fcm_dpo/margin': 65.49043273925781, 'margin_dpo/margin_mean': 65.49043273925781, 'margin_dpo/margin_std': 92.36592102050781, 'logps/chosen': -114.61155700683594, 'logps/rejected': -217.3373565673828, 'logps/ref_chosen': -56.04347610473633, 'logps/ref_rejected': -93.27880859375, 'logits/chosen': -0.4298696517944336, 'logits/rejected': -0.4265991747379303, 'epoch': 0.21} 21%|██ | 140/681 [05:55<21:47, 2.42s/it] 21%|██ | 141/681 [05:58<22:31, 2.50s/it] 21%|██ | 142/681 [06:01<23:08, 2.58s/it] 21%|██ | 143/681 [06:04<23:53, 2.67s/it] 21%|██ | 144/681 [06:06<23:58, 2.68s/it] 21%|██▏ | 145/681 [06:09<23:07, 2.59s/it] {'loss': 1.115, 'grad_norm': 23.93450927734375, 'learning_rate': 4.816996926967401e-07, 'fcm_dpo/beta': 0.007036598864942789, 'fcm_dpo/q_t': 0.40865078568458557, 'fcm_dpo/delta': -0.012454366311430931, 'fcm_dpo/margin': 58.48834991455078, 'margin_dpo/margin_mean': 58.48834991455078, 'margin_dpo/margin_std': 92.11347961425781, 'logps/chosen': -129.96218872070312, 'logps/rejected': -213.33724975585938, 'logps/ref_chosen': -61.4414176940918, 'logps/ref_rejected': -86.32813262939453, 'logits/chosen': -0.40944308042526245, 'logits/rejected': -0.39997661113739014, 'epoch': 0.21} 21%|██▏ | 145/681 [06:09<23:07, 2.59s/it] 21%|██▏ | 146/681 [06:11<23:01, 2.58s/it] 22%|██▏ | 147/681 [06:14<23:03, 2.59s/it] 22%|██▏ | 148/681 [06:16<22:55, 2.58s/it] 22%|██▏ | 149/681 [06:19<23:04, 2.60s/it] 22%|██▏ | 150/681 [06:22<22:53, 2.59s/it] {'loss': 1.1066, 'grad_norm': 18.16049575805664, 'learning_rate': 4.792138157142157e-07, 'fcm_dpo/beta': 0.006964278873056173, 'fcm_dpo/q_t': 0.4086775779724121, 'fcm_dpo/delta': -0.03296690061688423, 'fcm_dpo/margin': 59.02332305908203, 'margin_dpo/margin_mean': 59.02332305908203, 'margin_dpo/margin_std': 87.62327575683594, 'logps/chosen': -123.40377044677734, 'logps/rejected': -212.4925079345703, 'logps/ref_chosen': -57.70451736450195, 'logps/ref_rejected': -87.76991271972656, 'logits/chosen': -0.44094228744506836, 'logits/rejected': -0.42822694778442383, 'epoch': 0.22} 22%|██▏ | 150/681 [06:22<22:53, 2.59s/it] 22%|██▏ | 151/681 [06:24<22:26, 2.54s/it] 22%|██▏ | 152/681 [06:27<23:01, 2.61s/it] 22%|██▏ | 153/681 [06:29<22:48, 2.59s/it] 23%|██▎ | 154/681 [06:32<23:01, 2.62s/it] 23%|██▎ | 155/681 [06:35<23:17, 2.66s/it] {'loss': 1.076, 'grad_norm': 19.04676628112793, 'learning_rate': 4.7657694675916247e-07, 'fcm_dpo/beta': 0.006572508253157139, 'fcm_dpo/q_t': 0.3986854553222656, 'fcm_dpo/delta': -0.058337144553661346, 'fcm_dpo/margin': 69.28470611572266, 'margin_dpo/margin_mean': 69.28469848632812, 'margin_dpo/margin_std': 97.56148529052734, 'logps/chosen': -126.15814208984375, 'logps/rejected': -228.1529083251953, 'logps/ref_chosen': -62.08925247192383, 'logps/ref_rejected': -94.79930114746094, 'logits/chosen': -0.4826192855834961, 'logits/rejected': -0.4734383523464203, 'epoch': 0.23} 23%|██▎ | 155/681 [06:35<23:17, 2.66s/it] 23%|██▎ | 156/681 [06:37<23:11, 2.65s/it] 23%|██▎ | 157/681 [06:40<22:10, 2.54s/it] 23%|██▎ | 158/681 [06:42<22:20, 2.56s/it] 23%|██▎ | 159/681 [06:45<22:32, 2.59s/it] 23%|██▎ | 160/681 [06:47<22:16, 2.57s/it] {'loss': 1.1205, 'grad_norm': 23.905847549438477, 'learning_rate': 4.737908228387656e-07, 'fcm_dpo/beta': 0.006233358755707741, 'fcm_dpo/q_t': 0.4046873450279236, 'fcm_dpo/delta': -0.06433447450399399, 'fcm_dpo/margin': 70.75523376464844, 'margin_dpo/margin_mean': 70.75523376464844, 'margin_dpo/margin_std': 118.20479583740234, 'logps/chosen': -155.4814453125, 'logps/rejected': -256.0091552734375, 'logps/ref_chosen': -67.15288543701172, 'logps/ref_rejected': -96.92537689208984, 'logits/chosen': -0.37852075695991516, 'logits/rejected': -0.37446096539497375, 'epoch': 0.23} 23%|██▎ | 160/681 [06:48<22:16, 2.57s/it] 24%|██▎ | 161/681 [06:50<21:21, 2.46s/it] 24%|██▍ | 162/681 [06:52<22:09, 2.56s/it] 24%|██▍ | 163/681 [06:55<22:04, 2.56s/it] 24%|██▍ | 164/681 [06:58<22:02, 2.56s/it] 24%|██▍ | 165/681 [07:00<21:50, 2.54s/it] {'loss': 1.1054, 'grad_norm': 21.921010971069336, 'learning_rate': 4.708572792802069e-07, 'fcm_dpo/beta': 0.005916006397455931, 'fcm_dpo/q_t': 0.4076644480228424, 'fcm_dpo/delta': -0.016961436718702316, 'fcm_dpo/margin': 70.23262023925781, 'margin_dpo/margin_mean': 70.23262023925781, 'margin_dpo/margin_std': 105.99822998046875, 'logps/chosen': -138.60523986816406, 'logps/rejected': -231.7488555908203, 'logps/ref_chosen': -57.40401077270508, 'logps/ref_rejected': -80.31498718261719, 'logits/chosen': -0.38327422738075256, 'logits/rejected': -0.369274765253067, 'epoch': 0.24} 24%|██▍ | 165/681 [07:00<21:50, 2.54s/it] 24%|██▍ | 166/681 [07:02<20:59, 2.44s/it] 25%|██▍ | 167/681 [07:05<21:39, 2.53s/it] 25%|██▍ | 168/681 [07:08<21:46, 2.55s/it] 25%|██▍ | 169/681 [07:10<22:15, 2.61s/it] 25%|██▍ | 170/681 [07:13<22:16, 2.62s/it] {'loss': 1.0613, 'grad_norm': 17.49802589416504, 'learning_rate': 4.6777824852166437e-07, 'fcm_dpo/beta': 0.005466562695801258, 'fcm_dpo/q_t': 0.3945610821247101, 'fcm_dpo/delta': -0.10773168504238129, 'fcm_dpo/margin': 87.78046417236328, 'margin_dpo/margin_mean': 87.78046417236328, 'margin_dpo/margin_std': 120.57357025146484, 'logps/chosen': -131.82830810546875, 'logps/rejected': -253.31906127929688, 'logps/ref_chosen': -52.029144287109375, 'logps/ref_rejected': -85.73944091796875, 'logits/chosen': -0.3227120041847229, 'logits/rejected': -0.3182465136051178, 'epoch': 0.25} 25%|██▍ | 170/681 [07:13<22:16, 2.62s/it] 25%|██▌ | 171/681 [07:15<21:24, 2.52s/it] 25%|██▌ | 172/681 [07:18<21:11, 2.50s/it] 25%|██▌ | 173/681 [07:20<21:03, 2.49s/it] 26%|██▌ | 174/681 [07:23<21:05, 2.50s/it] 26%|██▌ | 175/681 [07:25<21:19, 2.53s/it] {'loss': 1.0959, 'grad_norm': 29.300512313842773, 'learning_rate': 4.645557588393406e-07, 'fcm_dpo/beta': 0.0053649842739105225, 'fcm_dpo/q_t': 0.40541133284568787, 'fcm_dpo/delta': -0.02835622988641262, 'fcm_dpo/margin': 79.64360809326172, 'margin_dpo/margin_mean': 79.64360046386719, 'margin_dpo/margin_std': 116.92179107666016, 'logps/chosen': -156.86700439453125, 'logps/rejected': -266.4975891113281, 'logps/ref_chosen': -62.996971130371094, 'logps/ref_rejected': -92.98394012451172, 'logits/chosen': -0.3354695737361908, 'logits/rejected': -0.3250008523464203, 'epoch': 0.26} 26%|██▌ | 175/681 [07:25<21:19, 2.53s/it] 26%|██▌ | 176/681 [07:28<20:50, 2.48s/it] 26%|██▌ | 177/681 [07:30<21:09, 2.52s/it] 26%|██▌ | 178/681 [07:33<21:09, 2.52s/it] 26%|██▋ | 179/681 [07:36<21:34, 2.58s/it] 26%|██▋ | 180/681 [07:38<21:18, 2.55s/it] {'loss': 1.0541, 'grad_norm': 18.73993492126465, 'learning_rate': 4.611919330113591e-07, 'fcm_dpo/beta': 0.004915366414934397, 'fcm_dpo/q_t': 0.39298662543296814, 'fcm_dpo/delta': -0.08844007551670074, 'fcm_dpo/margin': 98.45795440673828, 'margin_dpo/margin_mean': 98.45795440673828, 'margin_dpo/margin_std': 131.67587280273438, 'logps/chosen': -155.4091796875, 'logps/rejected': -293.9116516113281, 'logps/ref_chosen': -57.0670280456543, 'logps/ref_rejected': -97.1115493774414, 'logits/chosen': -0.2953912019729614, 'logits/rejected': -0.29680582880973816, 'epoch': 0.26} 26%|██▋ | 180/681 [07:38<21:18, 2.55s/it] 27%|██▋ | 181/681 [07:41<21:25, 2.57s/it] 27%|██▋ | 182/681 [07:43<21:04, 2.53s/it] 27%|██▋ | 183/681 [07:45<20:34, 2.48s/it] 27%|██▋ | 184/681 [07:48<21:03, 2.54s/it] 27%|██▋ | 185/681 [07:51<20:42, 2.51s/it] {'loss': 1.1531, 'grad_norm': 19.846059799194336, 'learning_rate': 4.5768898691940836e-07, 'fcm_dpo/beta': 0.00469121104106307, 'fcm_dpo/q_t': 0.423015296459198, 'fcm_dpo/delta': -0.007265460677444935, 'fcm_dpo/margin': 72.3204574584961, 'margin_dpo/margin_mean': 72.3204574584961, 'margin_dpo/margin_std': 118.7593994140625, 'logps/chosen': -150.46461486816406, 'logps/rejected': -243.45437622070312, 'logps/ref_chosen': -54.840736389160156, 'logps/ref_rejected': -75.51002502441406, 'logits/chosen': -0.3211399018764496, 'logits/rejected': -0.31003421545028687, 'epoch': 0.27} 27%|██▋ | 185/681 [07:51<20:42, 2.51s/it] 27%|██▋ | 186/681 [07:53<20:35, 2.50s/it] 27%|██▋ | 187/681 [07:55<20:03, 2.44s/it] 28%|██▊ | 188/681 [07:58<20:18, 2.47s/it] 28%|██▊ | 189/681 [08:00<20:26, 2.49s/it] 28%|██▊ | 190/681 [08:03<19:49, 2.42s/it] {'loss': 1.0829, 'grad_norm': 18.63943862915039, 'learning_rate': 4.5404922808905543e-07, 'fcm_dpo/beta': 0.004630076698958874, 'fcm_dpo/q_t': 0.4038674235343933, 'fcm_dpo/delta': -0.02935381792485714, 'fcm_dpo/margin': 92.37378692626953, 'margin_dpo/margin_mean': 92.37378692626953, 'margin_dpo/margin_std': 125.8380126953125, 'logps/chosen': -155.59190368652344, 'logps/rejected': -277.10418701171875, 'logps/ref_chosen': -57.72148895263672, 'logps/ref_rejected': -86.85997009277344, 'logits/chosen': -0.3224960267543793, 'logits/rejected': -0.31152957677841187, 'epoch': 0.28} 28%|██▊ | 190/681 [08:03<19:49, 2.42s/it] 28%|██▊ | 191/681 [08:05<20:32, 2.51s/it] 28%|██▊ | 192/681 [08:08<20:35, 2.53s/it] 28%|██▊ | 193/681 [08:10<20:12, 2.48s/it] 28%|██▊ | 194/681 [08:13<19:36, 2.42s/it] 29%|██▊ | 195/681 [08:15<20:05, 2.48s/it] {'loss': 1.0971, 'grad_norm': 19.838130950927734, 'learning_rate': 4.5027505416968985e-07, 'fcm_dpo/beta': 0.004372724797576666, 'fcm_dpo/q_t': 0.4060741066932678, 'fcm_dpo/delta': -0.041835661977529526, 'fcm_dpo/margin': 94.63658142089844, 'margin_dpo/margin_mean': 94.63658142089844, 'margin_dpo/margin_std': 133.46847534179688, 'logps/chosen': -171.98052978515625, 'logps/rejected': -297.82037353515625, 'logps/ref_chosen': -58.26164627075195, 'logps/ref_rejected': -89.46485900878906, 'logits/chosen': -0.2853700518608093, 'logits/rejected': -0.2791460156440735, 'epoch': 0.29} 29%|██▊ | 195/681 [08:15<20:05, 2.48s/it] 29%|██▉ | 196/681 [08:18<20:10, 2.50s/it] 29%|██▉ | 197/681 [08:20<20:18, 2.52s/it] 29%|██▉ | 198/681 [08:23<20:25, 2.54s/it] 29%|██▉ | 199/681 [08:26<20:43, 2.58s/it] 29%|██▉ | 200/681 [08:28<20:41, 2.58s/it] {'loss': 1.0801, 'grad_norm': 21.282487869262695, 'learning_rate': 4.4636895135509966e-07, 'fcm_dpo/beta': 0.0042917924001812935, 'fcm_dpo/q_t': 0.40244197845458984, 'fcm_dpo/delta': -0.03731077164411545, 'fcm_dpo/margin': 101.45069122314453, 'margin_dpo/margin_mean': 101.45067596435547, 'margin_dpo/margin_std': 138.32688903808594, 'logps/chosen': -161.27999877929688, 'logps/rejected': -290.16351318359375, 'logps/ref_chosen': -55.71953201293945, 'logps/ref_rejected': -83.15235137939453, 'logits/chosen': -0.29983407258987427, 'logits/rejected': -0.2885088324546814, 'epoch': 0.29} 29%|██▉ | 200/681 [08:28<20:41, 2.58s/it][INFO|trainer.py:4307] 2026-04-21 22:03:32,118 >> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-21 22:03:32,118 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-21 22:03:32,118 >> Batch size = 8 0%| | 0/73 [00:00> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-21 22:12:40,575 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-21 22:12:40,575 >> Batch size = 8 0%| | 0/73 [00:00> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-21 22:21:51,798 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-21 22:21:51,798 >> Batch size = 8 0%| | 0/73 [00:00> Training completed. Do not forget to share your model on huggingface.co/models =) {'train_runtime': 1861.3017, 'train_samples_per_second': 23.423, 'train_steps_per_second': 0.366, 'train_loss': 1.1011297496229893, 'epoch': 1.0} 100%|██████████| 681/681 [30:54<00:00, 2.55s/it] 100%|██████████| 681/681 [30:54<00:00, 2.72s/it] ***** train metrics ***** epoch = 1.0 total_flos = 0GF train_loss = 1.1011 train_runtime = 0:31:01.30 train_samples = 43598 train_samples_per_second = 23.423 train_steps_per_second = 0.366 2026-04-21 22:25:57 - INFO - __main__ - *** Training complete *** 2026-04-21 22:25:57 - INFO - __main__ - *** Save model *** [INFO|configuration_utils.py:419] 2026-04-21 22:26:33,935 >> Configuration saved in /root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun/config.json [INFO|configuration_utils.py:911] 2026-04-21 22:26:33,935 >> Configuration saved in /root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun/generation_config.json [INFO|modeling_utils.py:3580] 2026-04-21 22:27:01,592 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 7 checkpoint shards. You can find where each parameters has been saved in the index located at /root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun/model.safetensors.index.json. [INFO|tokenization_utils_base.py:2510] 2026-04-21 22:27:01,595 >> tokenizer config file saved in /root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun/tokenizer_config.json [INFO|tokenization_utils_base.py:2519] 2026-04-21 22:27:01,595 >> Special tokens file saved in /root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun/special_tokens_map.json 2026-04-21 22:27:01 - INFO - __main__ - Saved HF-compatible model artifacts to /root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun [INFO|modelcard.py:450] 2026-04-21 22:27:03,638 >> Dropping the following result as it does not have all the necessary fields: {'dataset': {'name': 'Anthropic/hh-rlhf', 'type': 'Anthropic/hh-rlhf'}} [INFO|configuration_utils.py:419] 2026-04-21 22:27:03,642 >> Configuration saved in /root/dynamic-dpo-v4/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star0.4-4xh200-batch-64-20260421-214335-rerun/config.json 2026-04-21 22:27:03 - INFO - __main__ - *** Evaluate *** [INFO|trainer.py:4307] 2026-04-21 22:27:03,643 >> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-21 22:27:03,644 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-21 22:27:03,644 >> Batch size = 8 0%| | 0/73 [00:00