2026-04-22 00:39:43 - INFO - __main__ - Model parameters ModelArguments(base_model_revision=None, model_name_or_path='/root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200', model_revision='main', model_code_revision=None, torch_dtype='bfloat16', tokenizer_name_or_path=None, trust_remote_code=False, attn_implementation='flash_attention_2', use_peft=False, lora_r=16, lora_alpha=32, lora_dropout=0.05, lora_target_modules=None, lora_modules_to_save=None, load_in_8bit=False, load_in_4bit=False, bnb_4bit_quant_type='nf4', use_bnb_nested_quant=False, bnb_4bit_quant_storage='uint8') 2026-04-22 00:39:43 - INFO - __main__ - Data parameters DataArguments(chat_template=None, dataset_mixer={'Anthropic/hh-rlhf': 1.0}, text_column='text', dataset_splits=['train', 'test'], dataset_configs=['helpful-base'], dataset_dir=None, preprocessing_num_workers=12, use_persistent_hf_cache=True, hf_cache_dir='/root/dynamic-dpo-v4/outputs/hf/datasets', truncation_side=None, auto_insert_empty_system_msg=True, preprocessing_log_samples=0, preprocessing_log_dir=None) 2026-04-22 00:39:43 - INFO - __main__ - Training/evaluation parameters NewDPOConfig( _n_gpu=1, accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False}, adafactor=False, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, auto_find_batch_size=False, average_tokens_across_devices=False, batch_eval_metrics=False, beta=0.1, bf16=True, bf16_full_eval=False, data_seed=None, dataloader_drop_last=True, dataloader_num_workers=0, dataloader_persistent_workers=False, dataloader_pin_memory=True, dataloader_prefetch_factor=None, dataset_num_proc=12, ddp_backend=None, ddp_broadcast_buffers=None, ddp_bucket_cap_mb=None, ddp_find_unused_parameters=None, ddp_timeout=1800, debug=[], deepspeed=None, disable_dropout=True, disable_tqdm=False, do_eval=True, do_predict=False, do_train=False, eta=0.1, eval_accumulation_steps=None, eval_delay=0, eval_do_concat_batches=True, eval_on_start=False, eval_steps=200, eval_strategy=IntervalStrategy.STEPS, eval_use_gather_object=False, f_alpha_divergence_coef=1.0, f_divergence_type=reverse_kl, force_use_ref_model=False, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, fsdp=[], fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, fsdp_min_num_params=0, fsdp_transformer_layer_cls_to_wrap=None, full_determinism=False, generate_during_eval=False, gradient_accumulation_steps=2, gradient_checkpointing=True, gradient_checkpointing_kwargs={'use_reentrant': False}, greater_is_better=None, group_by_length=False, half_precision_backend=auto, hub_always_push=False, hub_margin_dataset_id=None, hub_model_id=jackf857/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802, hub_model_revision=main, hub_private_repo=None, hub_strategy=HubStrategy.EVERY_SAVE, hub_token=, ignore_data_skip=False, include_for_metrics=[], include_inputs_for_metrics=False, include_num_input_tokens_seen=False, include_tokens_per_second=False, is_encoder_decoder=None, jit_mode_eval=False, label_names=None, label_pad_token_id=-100, label_smoothing=0.0, label_smoothing_factor=0.0, learning_rate=5e-07, length_column_name=length, load_best_model_at_end=False, local_rank=0, log_level=info, log_level_replica=warning, log_on_each_node=True, logging_dir=outputs/llama3-8b-base-new-method-s_star1.0/runs/Apr22_00-39-43_4ab23a513308, logging_first_step=True, logging_nan_inf_filter=True, logging_steps=5, logging_strategy=IntervalStrategy.STEPS, loss_type=sigmoid, lr_scheduler_kwargs={}, lr_scheduler_type=SchedulerType.COSINE, margin_dataset_private=None, margin_dataset_split=train, margin_log_path=/root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802/margin_logs, margin_log_steps=1, margin_save_full=True, max_grad_norm=1.0, max_length=512, max_prompt_length=256, max_steps=-1, max_target_length=None, metric_for_best_model=None, model_adapter_name=None, model_init_kwargs=None, mp_parameters=, neftune_noise_alpha=None, no_cuda=False, non_finite_logits_handling=error, num_train_epochs=1, optim=OptimizerNames.ADAMW_TORCH, optim_args=None, optim_target_modules=None, output_dir=/root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802, overwrite_output_dir=False, padding_value=None, past_index=-1, per_device_eval_batch_size=8, per_device_train_batch_size=8, post_tokenization_log_dir=None, post_tokenization_log_samples=0, precompute_ref_batch_size=None, precompute_ref_eval_batch_size=None, precompute_ref_log_probs=False, prediction_loss_only=False, push_margin_dataset=True, push_to_hub=True, push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=, q_target=0.45, ray_scope=last, ref_adapter_name=None, ref_model_init_kwargs=None, ref_model_mixup_alpha=0.9, ref_model_sync_steps=64, reference_free=False, remove_unused_columns=False, report_to=['wandb'], require_explicit_ref_model=True, restore_callback_states_from_checkpoint=False, resume_from_checkpoint=None, reuse_tokenized_dataset=True, rpo_alpha=None, run_name=llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802, s_star=1.0, save_on_each_node=False, save_only_model=False, save_safetensors=True, save_steps=50, save_strategy=SaveStrategy.NO, save_total_limit=2, seed=42, sft_weight=0.0, skip_memory_metrics=True, sync_ref_model=False, tf32=None, tokenization_batch_size=128, tokenization_mode=online, tokenized_dataset_cache_dir=/root/dynamic-dpo-v4/outputs/tokenized_preferences, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, torch_empty_cache_steps=None, torchdynamo=None, tp_size=0, tpu_metrics_debug=False, tpu_num_cores=None, trainer_type=new_dpo, truncation_mode=keep_end, use_cpu=False, use_ipex=False, use_legacy_prediction_loop=False, use_liger_kernel=False, use_mps_device=False, wandb_project=llama3-8b-base-new-method-hh-beta-0.1, warmup_ratio=0.1, warmup_steps=0, weight_decay=0.0, ) 2026-04-22 00:39:43 - INFO - __main__ - Using W&B project from training args: llama3-8b-base-new-method-hh-beta-0.1 2026-04-22 00:39:43 - INFO - __main__ - New-DPO parameters: beta=0.1, q_target=0.45, s_star=1.0, eta=0.1 2026-04-22 00:39:43 - INFO - __main__ - Using persistent HF datasets cache at /root/dynamic-dpo-v4/outputs/hf/datasets 2026-04-22 00:39:47 - WARNING - __main__ - Dropped 237 non-canonical HH preference examples from split `train` before normalization (126 x HH preprocessing expects exactly one final assistant response in chosen/rejected suffixes., 111 x HH chosen/rejected transcripts must each contain a divergent assistant response.). Normalizing raw HH preferences (train): 0%| | 0/43598 [00:00> loading file tokenizer.json [INFO|tokenization_utils_base.py:2058] 2026-04-22 00:39:51,607 >> loading file tokenizer.model [INFO|tokenization_utils_base.py:2058] 2026-04-22 00:39:51,607 >> loading file added_tokens.json [INFO|tokenization_utils_base.py:2058] 2026-04-22 00:39:51,607 >> loading file special_tokens_map.json [INFO|tokenization_utils_base.py:2058] 2026-04-22 00:39:51,607 >> loading file tokenizer_config.json [INFO|tokenization_utils_base.py:2058] 2026-04-22 00:39:51,607 >> loading file chat_template.jinja Normalizing raw HH preferences (test): 74%|█████████████████████████████████████████████████████████████████████████████████▉ | 1741/2339 [00:00<00:00, 17333.27 examples/s] Normalizing raw HH preferences (test): 0%| | 0/2339 [00:00> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you. warnings.warn( [WARNING|logging.py:328] 2026-04-22 00:39:52,114 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Loading checkpoint shards: 0%| | 0/7 [00:00> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. 2026-04-22 00:39:52 - INFO - __main__ - Processed train sample 41905: Prompt: <|begin_of_text|><|start_header_id|>user<|end_header_id|> What is the best way to clean my refrigerator?<|eot_id|><|start_header_id|>assistant<|end_header_id|> Well, I have a few ideas, but the best way to know what works best is to run some tests and then analyze the data to figure out which technique works best.<|eot_id|><|start_header_id|>user<|end_header_id|> What are your ideas?<|eot_id|> Chosen: <|start_header_id|>assistant<|end_header_id|> Some of my ideas include wiping down the interior with a rag, using bleach and water to wipe down the door, emptying the contents of the fridge to scrub the shelves with a clean cloth, cleaning the interior of the fridge with vinegar and water, and using a vacuum cleaner to clean the interior.<|eot_id|> Rejected: <|start_header_id|>assistant<|end_header_id|> I was thinking that you could try different types of soap, which could be a variation of “soap,” like antibacterial soap. I was also thinking that you could try using different tools, like a toothbrush, to clean the inside. Or I was thinking that you could try different ways to get the gunk out, like using a squeegee, or using a paper towel. I was also thinking you could try other types of cleaning, like vacuuming, but I think that could have the opposite of the desired effect.<|eot_id|> /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you. warnings.warn( [INFO|configuration_utils.py:691] 2026-04-22 00:39:52,198 >> loading configuration file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/config.json [INFO|configuration_utils.py:765] 2026-04-22 00:39:52,199 >> Model config LlamaConfig { "architectures": [ "LlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 128000, "eos_token_id": 128001, "head_dim": 128, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 14336, "max_position_embeddings": 8192, "mlp_bias": false, "model_type": "llama", "num_attention_heads": 32, "num_hidden_layers": 32, "num_key_value_heads": 8, "pretraining_tp": 1, "rms_norm_eps": 1e-05, "rope_scaling": null, "rope_theta": 500000.0, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.51.0", "use_cache": false, "vocab_size": 128256 } [INFO|modeling_utils.py:1121] 2026-04-22 00:39:52,207 >> loading weights file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/model.safetensors.index.json [INFO|modeling_utils.py:2167] 2026-04-22 00:39:52,207 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16. [WARNING|logging.py:328] 2026-04-22 00:39:52,209 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. [INFO|configuration_utils.py:1142] 2026-04-22 00:39:52,210 >> Generate config GenerationConfig { "bos_token_id": 128000, "eos_token_id": 128001, "use_cache": false } Loading checkpoint shards: 0%| | 0/7 [00:00> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Loading checkpoint shards: 0%| | 0/7 [00:00> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you. warnings.warn( [WARNING|logging.py:328] 2026-04-22 00:39:52,374 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Loading checkpoint shards: 0%| | 0/7 [00:00> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. Loading checkpoint shards: 14%|███████████████████▋ | 1/7 [00:01<00:09, 1.61s/it] Loading checkpoint shards: 29%|███████████████████████████████████████▍ | 2/7 [00:03<00:08, 1.61s/it] Loading checkpoint shards: 43%|███████████████████████████████████████████████████████████▏ | 3/7 [00:04<00:06, 1.66s/it] Loading checkpoint shards: 57%|██████████████████████████████████████████████████████████████████████████████▊ | 4/7 [00:06<00:05, 1.69s/it] Loading checkpoint shards: 71%|██████████████████████████████████████████████████████████████████████████████████████████████████▌ | 5/7 [00:08<00:03, 1.66s/it] Loading checkpoint shards: 86%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 6/7 [00:10<00:01, 1.68s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:10<00:00, 1.43s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:10<00:00, 1.56s/it] [INFO|modeling_utils.py:4926] 2026-04-22 00:40:03,141 >> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|modeling_utils.py:4934] 2026-04-22 00:40:03,142 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. [INFO|configuration_utils.py:1095] 2026-04-22 00:40:03,144 >> loading configuration file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/generation_config.json [INFO|configuration_utils.py:1142] 2026-04-22 00:40:03,144 >> Generate config GenerationConfig { "bos_token_id": 128000, "do_sample": true, "eos_token_id": 128001, "max_length": 4096, "temperature": 0.6, "top_p": 0.9 } [INFO|configuration_utils.py:691] 2026-04-22 00:40:03,145 >> loading configuration file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/config.json [INFO|configuration_utils.py:765] 2026-04-22 00:40:03,145 >> Model config LlamaConfig { "architectures": [ "LlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 128000, "eos_token_id": 128001, "head_dim": 128, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 14336, "max_position_embeddings": 8192, "mlp_bias": false, "model_type": "llama", "num_attention_heads": 32, "num_hidden_layers": 32, "num_key_value_heads": 8, "pretraining_tp": 1, "rms_norm_eps": 1e-05, "rope_scaling": null, "rope_theta": 500000.0, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.51.0", "use_cache": false, "vocab_size": 128256 } [INFO|modeling_utils.py:1121] 2026-04-22 00:40:03,146 >> loading weights file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/model.safetensors.index.json [INFO|modeling_utils.py:2167] 2026-04-22 00:40:03,147 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16. [INFO|configuration_utils.py:1142] 2026-04-22 00:40:03,149 >> Generate config GenerationConfig { "bos_token_id": 128000, "eos_token_id": 128001, "use_cache": false } Loading checkpoint shards: 0%| | 0/7 [00:00> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|modeling_utils.py:4934] 2026-04-22 00:40:14,091 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. [INFO|configuration_utils.py:1095] 2026-04-22 00:40:14,093 >> loading configuration file /root/dynamic-dpo-v4/sft-checkpoints/llama-3-8b-base-sft-hh-helpful-4xh200/generation_config.json [INFO|configuration_utils.py:1142] 2026-04-22 00:40:14,094 >> Generate config GenerationConfig { "bos_token_id": 128000, "do_sample": true, "eos_token_id": 128001, "max_length": 4096, "temperature": 0.6, "top_p": 0.9 } [WARNING|trainer.py:821] 2026-04-22 00:40:14,095 >> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead. [WARNING|trainer.py:816] 2026-04-22 00:40:14,096 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:14,104 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:14,106 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:14,109 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `NewDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [WARNING|trainer.py:816] 2026-04-22 00:40:16,074 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,075 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,081 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,085 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `NewDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [WARNING|trainer.py:816] 2026-04-22 00:40:16,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,087 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,088 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. [WARNING|trainer.py:816] 2026-04-22 00:40:16,094 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `NewDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [WARNING|trainer.py:816] 2026-04-22 00:40:16,095 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead. /root/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `NewDPOTrainer.__init__`. Use `processing_class` instead. super().__init__( [INFO|trainer.py:748] 2026-04-22 00:40:16,644 >> Using auto half precision backend /root/dynamic-dpo-v4/.venv/lib/python3.11/site-packages/accelerate/accelerator.py:1557: UserWarning: Upcasted low precision parameters in LlamaForCausalLM because mixed precision turned on in FSDP. Affects: model.embed_tokens.weight, model.norm.weight, lm_head.weight. warnings.warn( /root/dynamic-dpo-v4/.venv/lib/python3.11/site-packages/accelerate/accelerator.py:1557: UserWarning: Upcasted low precision parameters in LlamaDecoderLayer because mixed precision turned on in FSDP. Affects: self_attn.q_proj.weight, self_attn.k_proj.weight, self_attn.v_proj.weight, self_attn.o_proj.weight, mlp.gate_proj.weight, mlp.up_proj.weight, mlp.down_proj.weight, input_layernorm.weight, post_attention_layernorm.weight. warnings.warn( /root/dynamic-dpo-v4/.venv/lib/python3.11/site-packages/accelerate/accelerator.py:1563: UserWarning: FSDP upcast of low precision parameters may affect the precision of model checkpoints. warnings.warn( [INFO|trainer.py:2414] 2026-04-22 00:40:24,115 >> ***** Running training ***** [INFO|trainer.py:2415] 2026-04-22 00:40:24,115 >> Num examples = 43,598 [INFO|trainer.py:2416] 2026-04-22 00:40:24,116 >> Num Epochs = 1 [INFO|trainer.py:2417] 2026-04-22 00:40:24,116 >> Instantaneous batch size per device = 8 [INFO|trainer.py:2420] 2026-04-22 00:40:24,116 >> Total train batch size (w. parallel, distributed & accumulation) = 64 [INFO|trainer.py:2421] 2026-04-22 00:40:24,116 >> Gradient Accumulation steps = 2 [INFO|trainer.py:2422] 2026-04-22 00:40:24,116 >> Total optimization steps = 681 [INFO|trainer.py:2423] 2026-04-22 00:40:24,116 >> Number of trainable parameters = 2,007,565,312 [INFO|integration_utils.py:831] 2026-04-22 00:40:24,117 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true" wandb: Currently logged in as: feng-cheng (feng-cheng-northeastern-university). Use `wandb login --relogin` to force relogin wandb: wandb version 0.26.0 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.17.5 wandb: Run data is saved locally in /root/dynamic-dpo-v4/outputs/wandb/wandb/run-20260422_004024-d5n2lrl2 wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802 wandb: ⭐️ View project at https://wandb.ai/feng-cheng-northeastern-university/llama3-8b-base-new-method-hh-beta-0.1 wandb: 🚀 View run at https://wandb.ai/feng-cheng-northeastern-university/llama3-8b-base-new-method-hh-beta-0.1/runs/d5n2lrl2 0%| | 0/681 [00:00> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-22 00:40:27,727 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-22 00:40:27,735 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed [WARNING|modeling_utils.py:1713] 2026-04-22 00:40:27,741 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed 0%|▏ | 1/681 [00:02<30:07, 2.66s/it] {'loss': 1.389, 'grad_norm': 83.51715850830078, 'learning_rate': 0.0, 'fcm_dpo/beta': 0.10000000149011612, 'fcm_dpo/q_t': 0.5005706548690796, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': -0.02287006378173828, 'margin_dpo/margin_mean': -0.02287048101425171, 'margin_dpo/margin_std': 0.41920793056488037, 'logps/chosen': -50.1435661315918, 'logps/rejected': -74.09991455078125, 'logps/ref_chosen': -50.14883804321289, 'logps/ref_rejected': -74.1280517578125, 'logits/chosen': -0.4974287748336792, 'logits/rejected': -0.43299180269241333, 'epoch': 0.0} 0%|▏ | 1/681 [00:02<30:07, 2.66s/it] 0%|▍ | 2/681 [00:05<29:01, 2.56s/it] 0%|▋ | 3/681 [00:07<28:45, 2.54s/it] 1%|▉ | 4/681 [00:10<29:06, 2.58s/it] 1%|█▏ | 5/681 [00:12<28:52, 2.56s/it] {'loss': 1.3883, 'grad_norm': 89.5003433227539, 'learning_rate': 2.898550724637681e-08, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.5004419088363647, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': -0.01767297089099884, 'margin_dpo/margin_mean': -0.017673060297966003, 'margin_dpo/margin_std': 0.3324979245662689, 'logps/chosen': -56.052879333496094, 'logps/rejected': -78.6711196899414, 'logps/ref_chosen': -56.05734634399414, 'logps/ref_rejected': -78.69325256347656, 'logits/chosen': -0.4901275634765625, 'logits/rejected': -0.45340436697006226, 'epoch': 0.01} 1%|█▏ | 5/681 [00:12<28:52, 2.56s/it] 1%|█▍ | 6/681 [00:15<27:24, 2.44s/it] 1%|█▋ | 7/681 [00:17<26:47, 2.39s/it] 1%|█▉ | 8/681 [00:19<26:35, 2.37s/it] 1%|██▏ | 9/681 [00:22<27:06, 2.42s/it] 1%|██▍ | 10/681 [00:24<27:43, 2.48s/it] {'loss': 1.3879, 'grad_norm': 70.89155578613281, 'learning_rate': 6.521739130434782e-08, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.5002920031547546, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': -0.011697113513946533, 'margin_dpo/margin_mean': -0.011697125621140003, 'margin_dpo/margin_std': 0.3836909830570221, 'logps/chosen': -59.539703369140625, 'logps/rejected': -91.15385437011719, 'logps/ref_chosen': -59.54457473754883, 'logps/ref_rejected': -91.17041778564453, 'logits/chosen': -0.5073713660240173, 'logits/rejected': -0.4694274961948395, 'epoch': 0.01} 1%|██▍ | 10/681 [00:24<27:43, 2.48s/it] 2%|██▌ | 11/681 [00:27<28:28, 2.55s/it] 2%|██▊ | 12/681 [00:30<28:29, 2.55s/it] 2%|███ | 13/681 [00:32<28:51, 2.59s/it] 2%|███▎ | 14/681 [00:35<28:26, 2.56s/it] 2%|███▌ | 15/681 [00:37<28:18, 2.55s/it] {'loss': 1.3861, 'grad_norm': 64.05209350585938, 'learning_rate': 1.0144927536231885e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.4998478293418884, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.0060859620571136475, 'margin_dpo/margin_mean': 0.006086033768951893, 'margin_dpo/margin_std': 0.40277695655822754, 'logps/chosen': -58.84619140625, 'logps/rejected': -92.95982360839844, 'logps/ref_chosen': -58.83195877075195, 'logps/ref_rejected': -92.93949890136719, 'logits/chosen': -0.4684832990169525, 'logits/rejected': -0.43799877166748047, 'epoch': 0.02} 2%|███▌ | 15/681 [00:37<28:18, 2.55s/it] 2%|███▊ | 16/681 [00:40<27:53, 2.52s/it] 2%|████ | 17/681 [00:42<27:38, 2.50s/it] 3%|████▎ | 18/681 [00:45<27:32, 2.49s/it] 3%|████▌ | 19/681 [00:47<27:37, 2.50s/it] 3%|████▊ | 20/681 [00:50<27:36, 2.51s/it] {'loss': 1.3816, 'grad_norm': 73.78077697753906, 'learning_rate': 1.3768115942028986e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.4987525939941406, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.04989497736096382, 'margin_dpo/margin_mean': 0.04989498108625412, 'margin_dpo/margin_std': 0.3380239009857178, 'logps/chosen': -59.63946533203125, 'logps/rejected': -82.80996704101562, 'logps/ref_chosen': -59.6396598815918, 'logps/ref_rejected': -82.76026916503906, 'logits/chosen': -0.5167207717895508, 'logits/rejected': -0.46918559074401855, 'epoch': 0.03} 3%|████▊ | 20/681 [00:50<27:36, 2.51s/it] 3%|████▉ | 21/681 [00:52<27:25, 2.49s/it] 3%|█████▏ | 22/681 [00:55<27:54, 2.54s/it] 3%|█████▍ | 23/681 [00:58<28:54, 2.64s/it] 4%|█████▋ | 24/681 [01:00<28:48, 2.63s/it] 4%|█████▉ | 25/681 [01:03<28:47, 2.63s/it] {'loss': 1.3661, 'grad_norm': 73.6851577758789, 'learning_rate': 1.7391304347826085e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.4948153495788574, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.20749202370643616, 'margin_dpo/margin_mean': 0.20749172568321228, 'margin_dpo/margin_std': 0.39785444736480713, 'logps/chosen': -53.1756591796875, 'logps/rejected': -89.1739501953125, 'logps/ref_chosen': -53.205284118652344, 'logps/ref_rejected': -88.99608612060547, 'logits/chosen': -0.49890923500061035, 'logits/rejected': -0.47215452790260315, 'epoch': 0.04} 4%|█████▉ | 25/681 [01:03<28:47, 2.63s/it] 4%|██████▏ | 26/681 [01:05<27:27, 2.52s/it] 4%|██████▍ | 27/681 [01:08<27:13, 2.50s/it] 4%|██████▋ | 28/681 [01:10<27:18, 2.51s/it] 4%|██████▉ | 29/681 [01:12<26:13, 2.41s/it] 4%|███████▏ | 30/681 [01:15<26:50, 2.47s/it] {'loss': 1.3383, 'grad_norm': 87.87821960449219, 'learning_rate': 2.1014492753623187e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.4876500964164734, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.4946414828300476, 'margin_dpo/margin_mean': 0.4946417212486267, 'margin_dpo/margin_std': 0.575846791267395, 'logps/chosen': -53.45599365234375, 'logps/rejected': -98.2717514038086, 'logps/ref_chosen': -53.5526008605957, 'logps/ref_rejected': -97.87371826171875, 'logits/chosen': -0.5192720890045166, 'logits/rejected': -0.4791105389595032, 'epoch': 0.04} 4%|███████▏ | 30/681 [01:15<26:50, 2.47s/it] 5%|███████▎ | 31/681 [01:18<27:24, 2.53s/it] 5%|███████▌ | 32/681 [01:20<28:00, 2.59s/it] 5%|███████▊ | 33/681 [01:23<27:45, 2.57s/it] 5%|████████ | 34/681 [01:25<27:43, 2.57s/it] 5%|████████▎ | 35/681 [01:28<27:58, 2.60s/it] {'loss': 1.3125, 'grad_norm': 83.32990264892578, 'learning_rate': 2.463768115942029e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.4807465672492981, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 0.7729043960571289, 'margin_dpo/margin_mean': 0.7729043960571289, 'margin_dpo/margin_std': 0.8287278413772583, 'logps/chosen': -56.183860778808594, 'logps/rejected': -92.39549255371094, 'logps/ref_chosen': -56.3298454284668, 'logps/ref_rejected': -91.76858520507812, 'logits/chosen': -0.48504266142845154, 'logits/rejected': -0.45020541548728943, 'epoch': 0.05} 5%|████████▎ | 35/681 [01:28<27:58, 2.60s/it] 5%|████████▌ | 36/681 [01:31<28:03, 2.61s/it] 5%|████████▊ | 37/681 [01:33<27:54, 2.60s/it] 6%|█████████ | 38/681 [01:36<26:34, 2.48s/it] 6%|█████████▎ | 39/681 [01:38<26:37, 2.49s/it] 6%|█████████▌ | 40/681 [01:41<27:12, 2.55s/it] {'loss': 1.2611, 'grad_norm': 60.212345123291016, 'learning_rate': 2.8260869565217386e-07, 'fcm_dpo/beta': 0.10000000894069672, 'fcm_dpo/q_t': 0.46647366881370544, 'fcm_dpo/delta': 0.0, 'fcm_dpo/margin': 1.3533049821853638, 'margin_dpo/margin_mean': 1.3533051013946533, 'margin_dpo/margin_std': 1.4385766983032227, 'logps/chosen': -54.291542053222656, 'logps/rejected': -84.20346069335938, 'logps/ref_chosen': -54.38492965698242, 'logps/ref_rejected': -82.94353485107422, 'logits/chosen': -0.5429744720458984, 'logits/rejected': -0.5070551037788391, 'epoch': 0.06} 6%|█████████▌ | 40/681 [01:41<27:12, 2.55s/it] 6%|█████████▊ | 41/681 [01:43<27:05, 2.54s/it] 6%|█████████▉ | 42/681 [01:46<27:00, 2.54s/it] 6%|██████████▏ | 43/681 [01:48<27:00, 2.54s/it] 6%|██████████▍ | 44/681 [01:51<27:20, 2.57s/it] 7%|██████████▋ | 45/681 [01:54<27:20, 2.58s/it] {'loss': 1.1013, 'grad_norm': 92.142822265625, 'learning_rate': 3.188405797101449e-07, 'fcm_dpo/beta': 0.13899105787277222, 'fcm_dpo/q_t': 0.4230595529079437, 'fcm_dpo/delta': 0.597256600856781, 'fcm_dpo/margin': 2.4335668087005615, 'margin_dpo/margin_mean': 2.4335670471191406, 'margin_dpo/margin_std': 2.2672231197357178, 'logps/chosen': -54.635047912597656, 'logps/rejected': -100.23272705078125, 'logps/ref_chosen': -54.862335205078125, 'logps/ref_rejected': -98.0264663696289, 'logits/chosen': -0.5021598935127258, 'logits/rejected': -0.4736817479133606, 'epoch': 0.07} 7%|██████████▋ | 45/681 [01:54<27:20, 2.58s/it] 7%|██████████▉ | 46/681 [01:56<27:28, 2.60s/it] 7%|███████████▏ | 47/681 [01:59<27:28, 2.60s/it] 7%|███████████▍ | 48/681 [02:02<27:58, 2.65s/it] 7%|███████████▋ | 49/681 [02:04<27:31, 2.61s/it] 7%|███████████▉ | 50/681 [02:07<27:34, 2.62s/it] {'loss': 0.8932, 'grad_norm': 102.62498474121094, 'learning_rate': 3.5507246376811595e-07, 'fcm_dpo/beta': 0.22112059593200684, 'fcm_dpo/q_t': 0.34648868441581726, 'fcm_dpo/delta': 0.2686576247215271, 'fcm_dpo/margin': 3.3537793159484863, 'margin_dpo/margin_mean': 3.3537793159484863, 'margin_dpo/margin_std': 3.3193202018737793, 'logps/chosen': -58.14952850341797, 'logps/rejected': -94.89351654052734, 'logps/ref_chosen': -58.304595947265625, 'logps/ref_rejected': -91.69480895996094, 'logits/chosen': -0.5621516108512878, 'logits/rejected': -0.5111474394798279, 'epoch': 0.07} 7%|███████████▉ | 50/681 [02:07<27:34, 2.62s/it] 7%|████████████▏ | 51/681 [02:09<27:25, 2.61s/it] 8%|████████████▎ | 52/681 [02:12<26:43, 2.55s/it] 8%|████████████▌ | 53/681 [02:14<26:36, 2.54s/it] 8%|████████████▊ | 54/681 [02:17<25:59, 2.49s/it] 8%|█████████████ | 55/681 [02:19<25:01, 2.40s/it] {'loss': 0.7614, 'grad_norm': 89.94622039794922, 'learning_rate': 3.9130434782608694e-07, 'fcm_dpo/beta': 0.21576957404613495, 'fcm_dpo/q_t': 0.2881360948085785, 'fcm_dpo/delta': -0.21076150238513947, 'fcm_dpo/margin': 5.520001411437988, 'margin_dpo/margin_mean': 5.520001411437988, 'margin_dpo/margin_std': 5.638136863708496, 'logps/chosen': -56.29509735107422, 'logps/rejected': -91.44287109375, 'logps/ref_chosen': -56.06591796875, 'logps/ref_rejected': -85.69367980957031, 'logits/chosen': -0.5873750448226929, 'logits/rejected': -0.5421477556228638, 'epoch': 0.08} 8%|█████████████ | 55/681 [02:19<25:01, 2.40s/it] 8%|█████████████▎ | 56/681 [02:22<25:59, 2.49s/it] 8%|█████████████▌ | 57/681 [02:24<25:52, 2.49s/it] 9%|█████████████▊ | 58/681 [02:27<26:12, 2.52s/it] 9%|██████████████ | 59/681 [02:29<26:43, 2.58s/it] 9%|██████████████▎ | 60/681 [02:32<25:58, 2.51s/it] {'loss': 0.7907, 'grad_norm': 94.9372329711914, 'learning_rate': 4.2753623188405794e-07, 'fcm_dpo/beta': 0.17349031567573547, 'fcm_dpo/q_t': 0.28984424471855164, 'fcm_dpo/delta': -0.17873340845108032, 'fcm_dpo/margin': 6.684614658355713, 'margin_dpo/margin_mean': 6.684614658355713, 'margin_dpo/margin_std': 7.005400657653809, 'logps/chosen': -61.69890213012695, 'logps/rejected': -97.42354583740234, 'logps/ref_chosen': -60.6871337890625, 'logps/ref_rejected': -89.72715759277344, 'logits/chosen': -0.6055638194084167, 'logits/rejected': -0.5562660098075867, 'epoch': 0.09} 9%|██████████████▎ | 60/681 [02:32<25:58, 2.51s/it] 9%|██████████████▌ | 61/681 [02:34<26:10, 2.53s/it] 9%|██████████████▋ | 62/681 [02:37<26:38, 2.58s/it] 9%|██████████████▉ | 63/681 [02:39<26:23, 2.56s/it] 9%|███████████████▏ | 64/681 [02:42<25:54, 2.52s/it] 10%|███████████████▍ | 65/681 [02:44<26:01, 2.53s/it] {'loss': 0.8297, 'grad_norm': 67.76667022705078, 'learning_rate': 4.63768115942029e-07, 'fcm_dpo/beta': 0.1508425772190094, 'fcm_dpo/q_t': 0.29980701208114624, 'fcm_dpo/delta': -0.1438160538673401, 'fcm_dpo/margin': 7.480526924133301, 'margin_dpo/margin_mean': 7.480526924133301, 'margin_dpo/margin_std': 8.598801612854004, 'logps/chosen': -63.0954475402832, 'logps/rejected': -102.12379455566406, 'logps/ref_chosen': -61.75325393676758, 'logps/ref_rejected': -93.30108642578125, 'logits/chosen': -0.5985899567604065, 'logits/rejected': -0.5659457445144653, 'epoch': 0.1} 10%|███████████████▍ | 65/681 [02:44<26:01, 2.53s/it] 10%|███████████████▋ | 66/681 [02:47<26:09, 2.55s/it] 10%|███████████████▉ | 67/681 [02:49<25:09, 2.46s/it] 10%|████████████████▏ | 68/681 [02:52<24:51, 2.43s/it] 10%|████████████████▍ | 69/681 [02:54<26:04, 2.56s/it] 10%|████████████████▋ | 70/681 [02:57<25:37, 2.52s/it] {'loss': 0.7832, 'grad_norm': 65.84673309326172, 'learning_rate': 5e-07, 'fcm_dpo/beta': 0.13242055475711823, 'fcm_dpo/q_t': 0.2980775237083435, 'fcm_dpo/delta': -0.1166788712143898, 'fcm_dpo/margin': 8.34177017211914, 'margin_dpo/margin_mean': 8.34177017211914, 'margin_dpo/margin_std': 8.739455223083496, 'logps/chosen': -61.53076934814453, 'logps/rejected': -94.34062194824219, 'logps/ref_chosen': -59.548004150390625, 'logps/ref_rejected': -84.01609802246094, 'logits/chosen': -0.6093414425849915, 'logits/rejected': -0.5704778432846069, 'epoch': 0.1} 10%|████████████████▋ | 70/681 [02:57<25:37, 2.52s/it] 10%|████████████████▉ | 71/681 [03:00<25:57, 2.55s/it] 11%|█████████████████▏ | 72/681 [03:02<26:07, 2.57s/it] 11%|█████████████████▎ | 73/681 [03:05<26:00, 2.57s/it] 11%|█████████████████▌ | 74/681 [03:07<25:45, 2.55s/it] 11%|█████████████████▊ | 75/681 [03:10<25:54, 2.57s/it] {'loss': 0.7689, 'grad_norm': 47.23740005493164, 'learning_rate': 4.999176576834721e-07, 'fcm_dpo/beta': 0.10968731343746185, 'fcm_dpo/q_t': 0.283592164516449, 'fcm_dpo/delta': -0.25056737661361694, 'fcm_dpo/margin': 11.169679641723633, 'margin_dpo/margin_mean': 11.169679641723633, 'margin_dpo/margin_std': 11.989259719848633, 'logps/chosen': -62.6329460144043, 'logps/rejected': -111.98944091796875, 'logps/ref_chosen': -59.86931228637695, 'logps/ref_rejected': -98.05613708496094, 'logits/chosen': -0.6356680989265442, 'logits/rejected': -0.6052416563034058, 'epoch': 0.11} 11%|█████████████████▊ | 75/681 [03:10<25:54, 2.57s/it] 11%|██████████████████ | 76/681 [03:12<25:48, 2.56s/it] 11%|██████████████████▎ | 77/681 [03:15<24:43, 2.46s/it] 11%|██████████████████▌ | 78/681 [03:17<25:12, 2.51s/it] 12%|██████████████████▊ | 79/681 [03:20<25:28, 2.54s/it] 12%|███████████████████ | 80/681 [03:22<25:21, 2.53s/it] {'loss': 0.7882, 'grad_norm': 47.55595397949219, 'learning_rate': 4.996706849759452e-07, 'fcm_dpo/beta': 0.08976603299379349, 'fcm_dpo/q_t': 0.29587632417678833, 'fcm_dpo/delta': -0.12075959146022797, 'fcm_dpo/margin': 12.325640678405762, 'margin_dpo/margin_mean': 12.325640678405762, 'margin_dpo/margin_std': 12.580961227416992, 'logps/chosen': -59.39103317260742, 'logps/rejected': -101.95135498046875, 'logps/ref_chosen': -56.18925857543945, 'logps/ref_rejected': -86.42393493652344, 'logits/chosen': -0.6552311182022095, 'logits/rejected': -0.6144415736198425, 'epoch': 0.12} 12%|███████████████████ | 80/681 [03:22<25:21, 2.53s/it] 12%|███████████████████▎ | 81/681 [03:25<26:02, 2.60s/it] 12%|███████████████████▌ | 82/681 [03:28<25:51, 2.59s/it] 12%|███████████████████▋ | 83/681 [03:30<25:09, 2.52s/it] 12%|███████████████████▉ | 84/681 [03:33<25:41, 2.58s/it] 12%|████████████████████▏ | 85/681 [03:35<25:35, 2.58s/it] {'loss': 0.808, 'grad_norm': 51.494895935058594, 'learning_rate': 4.992592445678582e-07, 'fcm_dpo/beta': 0.0777033269405365, 'fcm_dpo/q_t': 0.2961280643939972, 'fcm_dpo/delta': -0.14079028367996216, 'fcm_dpo/margin': 14.418505668640137, 'margin_dpo/margin_mean': 14.418505668640137, 'margin_dpo/margin_std': 15.550962448120117, 'logps/chosen': -64.86982727050781, 'logps/rejected': -117.28189849853516, 'logps/ref_chosen': -60.018287658691406, 'logps/ref_rejected': -98.01185607910156, 'logits/chosen': -0.6709647178649902, 'logits/rejected': -0.6374942064285278, 'epoch': 0.12} 12%|████████████████████▏ | 85/681 [03:35<25:35, 2.58s/it] 13%|████████████████████▍ | 86/681 [03:38<25:51, 2.61s/it] 13%|████████████████████▋ | 87/681 [03:41<25:40, 2.59s/it] 13%|████████████████████▉ | 88/681 [03:43<25:41, 2.60s/it] 13%|█████████████████████▏ | 89/681 [03:46<25:08, 2.55s/it] 13%|█████████████████████▍ | 90/681 [03:48<24:40, 2.51s/it] {'loss': 0.8807, 'grad_norm': 49.612510681152344, 'learning_rate': 4.986836074908615e-07, 'fcm_dpo/beta': 0.07141976058483124, 'fcm_dpo/q_t': 0.31418612599372864, 'fcm_dpo/delta': -0.08756770193576813, 'fcm_dpo/margin': 15.070363998413086, 'margin_dpo/margin_mean': 15.070365905761719, 'margin_dpo/margin_std': 18.86634063720703, 'logps/chosen': -67.00579071044922, 'logps/rejected': -118.99037170410156, 'logps/ref_chosen': -59.8709831237793, 'logps/ref_rejected': -96.78519439697266, 'logits/chosen': -0.6960055232048035, 'logits/rejected': -0.6736189723014832, 'epoch': 0.13} 13%|█████████████████████▍ | 90/681 [03:48<24:40, 2.51s/it] 13%|█████████████████████▋ | 91/681 [03:51<24:48, 2.52s/it] 14%|█████████████████████▉ | 92/681 [03:53<24:31, 2.50s/it] 14%|██████████████████████ | 93/681 [03:55<23:44, 2.42s/it] 14%|██████████████████████▎ | 94/681 [03:58<24:56, 2.55s/it] 14%|██████████████████████▌ | 95/681 [04:01<24:30, 2.51s/it] {'loss': 0.82, 'grad_norm': 39.34260559082031, 'learning_rate': 4.979441529392784e-07, 'fcm_dpo/beta': 0.06466929614543915, 'fcm_dpo/q_t': 0.3058759272098541, 'fcm_dpo/delta': -0.06842272728681564, 'fcm_dpo/margin': 16.387496948242188, 'margin_dpo/margin_mean': 16.387496948242188, 'margin_dpo/margin_std': 17.77635955810547, 'logps/chosen': -63.07061004638672, 'logps/rejected': -107.19331359863281, 'logps/ref_chosen': -55.94385528564453, 'logps/ref_rejected': -83.6790542602539, 'logits/chosen': -0.7063266634941101, 'logits/rejected': -0.6684988737106323, 'epoch': 0.14} 14%|██████████████████████▌ | 95/681 [04:01<24:30, 2.51s/it] 14%|██████████████████████▊ | 96/681 [04:03<24:26, 2.51s/it] 14%|███████████████████████ | 97/681 [04:06<24:27, 2.51s/it] 14%|███████████████████████▎ | 98/681 [04:08<23:59, 2.47s/it] 15%|███████████████████████▌ | 99/681 [04:10<23:06, 2.38s/it] 15%|███████████████████████▋ | 100/681 [04:13<23:41, 2.45s/it] {'loss': 0.8241, 'grad_norm': 48.312198638916016, 'learning_rate': 4.970413680203148e-07, 'fcm_dpo/beta': 0.0604403130710125, 'fcm_dpo/q_t': 0.3058018684387207, 'fcm_dpo/delta': -0.049155093729496, 'fcm_dpo/margin': 17.22031021118164, 'margin_dpo/margin_mean': 17.22031021118164, 'margin_dpo/margin_std': 18.75308609008789, 'logps/chosen': -64.37776184082031, 'logps/rejected': -110.65645599365234, 'logps/ref_chosen': -57.05888748168945, 'logps/ref_rejected': -86.11727142333984, 'logits/chosen': -0.680860698223114, 'logits/rejected': -0.6464991569519043, 'epoch': 0.15} 15%|███████████████████████▋ | 100/681 [04:13<23:41, 2.45s/it] 15%|███████████████████████▉ | 101/681 [04:15<23:27, 2.43s/it] 15%|████████████████████████ | 102/681 [04:17<23:18, 2.42s/it] 15%|████████████████████████▎ | 103/681 [04:20<22:58, 2.38s/it] 15%|████████████████████████▌ | 104/681 [04:22<22:22, 2.33s/it] 15%|████████████████████████▊ | 105/681 [04:25<23:27, 2.44s/it] {'loss': 0.8003, 'grad_norm': 33.41164016723633, 'learning_rate': 4.959758474331832e-07, 'fcm_dpo/beta': 0.05940910056233406, 'fcm_dpo/q_t': 0.29566460847854614, 'fcm_dpo/delta': -0.1656493991613388, 'fcm_dpo/margin': 19.471708297729492, 'margin_dpo/margin_mean': 19.471708297729492, 'margin_dpo/margin_std': 20.485849380493164, 'logps/chosen': -67.66845703125, 'logps/rejected': -114.4299545288086, 'logps/ref_chosen': -59.20774459838867, 'logps/ref_rejected': -86.49754333496094, 'logits/chosen': -0.7165222764015198, 'logits/rejected': -0.6761705875396729, 'epoch': 0.15} 15%|████████████████████████▊ | 105/681 [04:25<23:27, 2.44s/it] 16%|█████████████████████████ | 106/681 [04:27<23:28, 2.45s/it] 16%|█████████████████████████▎ | 107/681 [04:30<23:56, 2.50s/it] 16%|█████████████████████████▌ | 108/681 [04:32<23:30, 2.46s/it] 16%|█████████████████████████▊ | 109/681 [04:35<24:02, 2.52s/it] 16%|██████████████████████████ | 110/681 [04:37<24:19, 2.56s/it] {'loss': 0.8078, 'grad_norm': 33.63520050048828, 'learning_rate': 4.947482930773511e-07, 'fcm_dpo/beta': 0.05002494528889656, 'fcm_dpo/q_t': 0.3023180663585663, 'fcm_dpo/delta': -0.06976357847452164, 'fcm_dpo/margin': 21.24454116821289, 'margin_dpo/margin_mean': 21.24454116821289, 'margin_dpo/margin_std': 22.012073516845703, 'logps/chosen': -69.47651672363281, 'logps/rejected': -121.12227630615234, 'logps/ref_chosen': -60.437957763671875, 'logps/ref_rejected': -90.83917999267578, 'logits/chosen': -0.6657849550247192, 'logits/rejected': -0.6188558340072632, 'epoch': 0.16} 16%|██████████████████████████ | 110/681 [04:37<24:19, 2.56s/it] 16%|██████████████████████████▏ | 111/681 [04:40<24:06, 2.54s/it] 16%|██████████████████████████▍ | 112/681 [04:42<23:10, 2.44s/it] 17%|██████████████████████████▋ | 113/681 [04:45<23:32, 2.49s/it] 17%|██████████████████████████▉ | 114/681 [04:47<23:22, 2.47s/it] 17%|███████████████████████████▏ | 115/681 [04:50<23:45, 2.52s/it] {'loss': 0.8539, 'grad_norm': 48.63422393798828, 'learning_rate': 4.933595135901732e-07, 'fcm_dpo/beta': 0.04783306270837784, 'fcm_dpo/q_t': 0.31595808267593384, 'fcm_dpo/delta': -0.005023984704166651, 'fcm_dpo/margin': 20.951387405395508, 'margin_dpo/margin_mean': 20.951387405395508, 'margin_dpo/margin_std': 24.009220123291016, 'logps/chosen': -73.18070983886719, 'logps/rejected': -117.71064758300781, 'logps/ref_chosen': -61.7908821105957, 'logps/ref_rejected': -85.36943054199219, 'logits/chosen': -0.7007675766944885, 'logits/rejected': -0.6588722467422485, 'epoch': 0.17} 17%|███████████████████████████▏ | 115/681 [04:50<23:45, 2.52s/it] 17%|███████████████████████████▍ | 116/681 [04:52<23:09, 2.46s/it] 17%|███████████████████████████▋ | 117/681 [04:55<22:56, 2.44s/it] 17%|███████████████████████████▉ | 118/681 [04:57<23:21, 2.49s/it] 17%|████████████████████████████▏ | 119/681 [05:00<24:08, 2.58s/it] 18%|████████████████████████████▎ | 120/681 [05:02<24:00, 2.57s/it] {'loss': 0.8219, 'grad_norm': 37.845821380615234, 'learning_rate': 4.918104238142103e-07, 'fcm_dpo/beta': 0.04857005923986435, 'fcm_dpo/q_t': 0.31097596883773804, 'fcm_dpo/delta': -0.008426040410995483, 'fcm_dpo/margin': 20.727174758911133, 'margin_dpo/margin_mean': 20.727174758911133, 'margin_dpo/margin_std': 22.264766693115234, 'logps/chosen': -78.19169616699219, 'logps/rejected': -120.3479232788086, 'logps/ref_chosen': -65.3261489868164, 'logps/ref_rejected': -86.75518798828125, 'logits/chosen': -0.6924178600311279, 'logits/rejected': -0.6578742265701294, 'epoch': 0.18} 18%|████████████████████████████▎ | 120/681 [05:03<24:00, 2.57s/it] 18%|████████████████████████████▌ | 121/681 [05:05<23:39, 2.53s/it] 18%|████████████████████████████▊ | 122/681 [05:07<22:57, 2.46s/it] 18%|█████████████████████████████ | 123/681 [05:10<23:31, 2.53s/it] 18%|█████████████████████████████▎ | 124/681 [05:12<23:32, 2.54s/it] 18%|█████████████████████████████▌ | 125/681 [05:15<23:13, 2.51s/it] {'loss': 0.7748, 'grad_norm': 35.820945739746094, 'learning_rate': 4.90102044194588e-07, 'fcm_dpo/beta': 0.040757957845926285, 'fcm_dpo/q_t': 0.28754281997680664, 'fcm_dpo/delta': -0.2243596315383911, 'fcm_dpo/margin': 29.304885864257812, 'margin_dpo/margin_mean': 29.304885864257812, 'margin_dpo/margin_std': 30.770614624023438, 'logps/chosen': -72.17552947998047, 'logps/rejected': -144.36788940429688, 'logps/ref_chosen': -58.323204040527344, 'logps/ref_rejected': -101.2106704711914, 'logits/chosen': -0.6601894497871399, 'logits/rejected': -0.6469351053237915, 'epoch': 0.18} 18%|█████████████████████████████▌ | 125/681 [05:15<23:13, 2.51s/it] 19%|█████████████████████████████▊ | 126/681 [05:18<23:37, 2.55s/it] 19%|██████████████████████████████ | 127/681 [05:20<23:45, 2.57s/it] 19%|██████████████████████████████▎ | 128/681 [05:23<23:58, 2.60s/it] 19%|██████████████████████████████▍ | 129/681 [05:25<23:42, 2.58s/it] 19%|██████████████████████████████▋ | 130/681 [05:28<23:24, 2.55s/it] {'loss': 0.8458, 'grad_norm': 32.028621673583984, 'learning_rate': 4.882355001067891e-07, 'fcm_dpo/beta': 0.038607753813266754, 'fcm_dpo/q_t': 0.31171610951423645, 'fcm_dpo/delta': -0.023931104689836502, 'fcm_dpo/margin': 26.40726661682129, 'margin_dpo/margin_mean': 26.40726661682129, 'margin_dpo/margin_std': 28.96893310546875, 'logps/chosen': -70.66328430175781, 'logps/rejected': -126.8430404663086, 'logps/ref_chosen': -56.38518524169922, 'logps/ref_rejected': -86.15767669677734, 'logits/chosen': -0.6508277058601379, 'logits/rejected': -0.6233072280883789, 'epoch': 0.19} 19%|██████████████████████████████▋ | 130/681 [05:28<23:24, 2.55s/it] 19%|██████████████████████████████▉ | 131/681 [05:30<23:28, 2.56s/it] 19%|███████████████████████████████▏ | 132/681 [05:33<23:05, 2.52s/it] 20%|███████████████████████████████▍ | 133/681 [05:35<23:14, 2.55s/it] 20%|███████████████████████████████▋ | 134/681 [05:38<22:35, 2.48s/it] 20%|███████████████████████████████▉ | 135/681 [05:40<22:25, 2.46s/it] {'loss': 0.7941, 'grad_norm': 33.19447708129883, 'learning_rate': 4.862120211153265e-07, 'fcm_dpo/beta': 0.03552158921957016, 'fcm_dpo/q_t': 0.30137279629707336, 'fcm_dpo/delta': -0.07289578765630722, 'fcm_dpo/margin': 29.98138427734375, 'margin_dpo/margin_mean': 29.98138427734375, 'margin_dpo/margin_std': 31.074054718017578, 'logps/chosen': -69.44171905517578, 'logps/rejected': -140.09323120117188, 'logps/ref_chosen': -54.59065628051758, 'logps/ref_rejected': -95.26080322265625, 'logits/chosen': -0.6592376828193665, 'logits/rejected': -0.6462143659591675, 'epoch': 0.2} 20%|███████████████████████████████▉ | 135/681 [05:40<22:25, 2.46s/it] 20%|████████████████████████████████▏ | 136/681 [05:43<22:58, 2.53s/it] 20%|████████████████████████████████▍ | 137/681 [05:45<22:54, 2.53s/it] 20%|████████████████████████████████▋ | 138/681 [05:48<22:10, 2.45s/it] 20%|████████████████████████████████▊ | 139/681 [05:50<21:52, 2.42s/it] 21%|█████████████████████████████████ | 140/681 [05:52<21:34, 2.39s/it] {'loss': 0.8325, 'grad_norm': 34.031742095947266, 'learning_rate': 4.840329401637809e-07, 'fcm_dpo/beta': 0.033399857580661774, 'fcm_dpo/q_t': 0.3098406195640564, 'fcm_dpo/delta': -0.06322337687015533, 'fcm_dpo/margin': 31.61517906188965, 'margin_dpo/margin_mean': 31.61517906188965, 'margin_dpo/margin_std': 35.689491271972656, 'logps/chosen': -75.58506774902344, 'logps/rejected': -144.43556213378906, 'logps/ref_chosen': -56.04347610473633, 'logps/ref_rejected': -93.27880859375, 'logits/chosen': -0.6483560800552368, 'logits/rejected': -0.6316757202148438, 'epoch': 0.21} 21%|█████████████████████████████████ | 140/681 [05:52<21:34, 2.39s/it] 21%|█████████████████████████████████▎ | 141/681 [05:55<22:21, 2.48s/it] 21%|█████████████████████████████████▌ | 142/681 [05:58<22:58, 2.56s/it] 21%|█████████████████████████████████▊ | 143/681 [06:01<23:50, 2.66s/it] 21%|██████████████████████████████████ | 144/681 [06:03<23:46, 2.66s/it] 21%|██████████████████████████████████▎ | 145/681 [06:06<22:55, 2.57s/it] {'loss': 0.8557, 'grad_norm': 36.38671112060547, 'learning_rate': 4.816996926967401e-07, 'fcm_dpo/beta': 0.031947266310453415, 'fcm_dpo/q_t': 0.3131418824195862, 'fcm_dpo/delta': -0.00863505620509386, 'fcm_dpo/margin': 31.503803253173828, 'margin_dpo/margin_mean': 31.503803253173828, 'margin_dpo/margin_std': 35.5254020690918, 'logps/chosen': -82.8645248413086, 'logps/rejected': -139.25503540039062, 'logps/ref_chosen': -61.4414176940918, 'logps/ref_rejected': -86.32813262939453, 'logits/chosen': -0.6257671117782593, 'logits/rejected': -0.5978158116340637, 'epoch': 0.21} 21%|██████████████████████████████████▎ | 145/681 [06:06<22:55, 2.57s/it] 21%|██████████████████████████████████▌ | 146/681 [06:08<22:45, 2.55s/it] 22%|██████████████████████████████████▊ | 147/681 [06:11<22:46, 2.56s/it] 22%|██████████████████████████████████▉ | 148/681 [06:13<22:39, 2.55s/it] 22%|███████████████████████████████████▏ | 149/681 [06:16<22:52, 2.58s/it] 22%|███████████████████████████████████▍ | 150/681 [06:19<22:48, 2.58s/it] {'loss': 0.8533, 'grad_norm': 36.64207077026367, 'learning_rate': 4.792138157142157e-07, 'fcm_dpo/beta': 0.03161965310573578, 'fcm_dpo/q_t': 0.3133260905742645, 'fcm_dpo/delta': -0.0391874685883522, 'fcm_dpo/margin': 32.65111541748047, 'margin_dpo/margin_mean': 32.65111541748047, 'margin_dpo/margin_std': 37.58909225463867, 'logps/chosen': -81.85924530029297, 'logps/rejected': -144.57577514648438, 'logps/ref_chosen': -57.70451736450195, 'logps/ref_rejected': -87.76991271972656, 'logits/chosen': -0.6312776803970337, 'logits/rejected': -0.6036733984947205, 'epoch': 0.22} 22%|███████████████████████████████████▍ | 150/681 [06:19<22:48, 2.58s/it] 22%|███████████████████████████████████▋ | 151/681 [06:21<22:10, 2.51s/it] 22%|███████████████████████████████████▉ | 152/681 [06:24<22:47, 2.59s/it] 22%|████████████████████████████████████▏ | 153/681 [06:26<22:38, 2.57s/it] 23%|████████████████████████████████████▍ | 154/681 [06:29<22:52, 2.61s/it] 23%|████████████████████████████████████▋ | 155/681 [06:32<23:06, 2.64s/it] {'loss': 0.824, 'grad_norm': 31.77535629272461, 'learning_rate': 4.7657694675916247e-07, 'fcm_dpo/beta': 0.029207954183220863, 'fcm_dpo/q_t': 0.3046707212924957, 'fcm_dpo/delta': -0.06974439322948456, 'fcm_dpo/margin': 36.313804626464844, 'margin_dpo/margin_mean': 36.313804626464844, 'margin_dpo/margin_std': 39.620338439941406, 'logps/chosen': -83.7581787109375, 'logps/rejected': -152.7820281982422, 'logps/ref_chosen': -62.08925247192383, 'logps/ref_rejected': -94.79930114746094, 'logits/chosen': -0.6591686010360718, 'logits/rejected': -0.6376873850822449, 'epoch': 0.23} 23%|████████████████████████████████████▋ | 155/681 [06:32<23:06, 2.64s/it] 23%|████████████████████████████████████▉ | 156/681 [06:34<22:58, 2.63s/it] 23%|█████████████████████████████████████ | 157/681 [06:36<22:00, 2.52s/it] 23%|█████████████████████████████████████▎ | 158/681 [06:39<22:05, 2.53s/it] 23%|█████████████████████████████████████▌ | 159/681 [06:42<22:16, 2.56s/it] 23%|█████████████████████████████████████▊ | 160/681 [06:44<21:58, 2.53s/it] {'loss': 0.8984, 'grad_norm': 35.20014572143555, 'learning_rate': 4.737908228387656e-07, 'fcm_dpo/beta': 0.028771813958883286, 'fcm_dpo/q_t': 0.32106462121009827, 'fcm_dpo/delta': -0.017074979841709137, 'fcm_dpo/margin': 35.24787139892578, 'margin_dpo/margin_mean': 35.24787139892578, 'margin_dpo/margin_std': 44.607139587402344, 'logps/chosen': -93.72312927246094, 'logps/rejected': -158.7434844970703, 'logps/ref_chosen': -67.15288543701172, 'logps/ref_rejected': -96.92537689208984, 'logits/chosen': -0.6497837901115417, 'logits/rejected': -0.6263854503631592, 'epoch': 0.23} 23%|█████████████████████████████████████▊ | 160/681 [06:44<21:58, 2.53s/it] 24%|██████████████████████████████████████ | 161/681 [06:46<21:05, 2.43s/it] 24%|██████████████████████████████████████▎ | 162/681 [06:49<21:52, 2.53s/it] 24%|██████████████████████████████████████▌ | 163/681 [06:52<21:44, 2.52s/it] 24%|██████████████████████████████████████▊ | 164/681 [06:54<21:50, 2.54s/it] 24%|███████████████████████████████████████ | 165/681 [06:57<21:40, 2.52s/it] {'loss': 0.8543, 'grad_norm': 43.92022705078125, 'learning_rate': 4.708572792802069e-07, 'fcm_dpo/beta': 0.0279733594506979, 'fcm_dpo/q_t': 0.3189981281757355, 'fcm_dpo/delta': 0.028953036293387413, 'fcm_dpo/margin': 34.75352478027344, 'margin_dpo/margin_mean': 34.7535285949707, 'margin_dpo/margin_std': 39.33875274658203, 'logps/chosen': -80.7635269165039, 'logps/rejected': -138.42803955078125, 'logps/ref_chosen': -57.40401077270508, 'logps/ref_rejected': -80.31498718261719, 'logits/chosen': -0.6410833597183228, 'logits/rejected': -0.6081478595733643, 'epoch': 0.24} 24%|███████████████████████████████████████ | 165/681 [06:57<21:40, 2.52s/it] 24%|███████████████████████████████████████▏ | 166/681 [06:59<20:46, 2.42s/it] 25%|███████████████████████████████████████▍ | 167/681 [07:02<21:27, 2.50s/it] 25%|███████████████████████████████████████▋ | 168/681 [07:04<21:30, 2.51s/it] 25%|███████████████████████████████████████▉ | 169/681 [07:07<22:01, 2.58s/it] 25%|████████████████████████████████████████▏ | 170/681 [07:09<22:05, 2.59s/it] {'loss': 0.802, 'grad_norm': 28.440479278564453, 'learning_rate': 4.6777824852166437e-07, 'fcm_dpo/beta': 0.025926318019628525, 'fcm_dpo/q_t': 0.29947513341903687, 'fcm_dpo/delta': -0.12728442251682281, 'fcm_dpo/margin': 42.69086456298828, 'margin_dpo/margin_mean': 42.69086456298828, 'margin_dpo/margin_std': 46.09825897216797, 'logps/chosen': -76.37390899658203, 'logps/rejected': -152.7750701904297, 'logps/ref_chosen': -52.029144287109375, 'logps/ref_rejected': -85.73944091796875, 'logits/chosen': -0.5913532376289368, 'logits/rejected': -0.576996922492981, 'epoch': 0.25} 25%|████████████████████████████████████████▏ | 170/681 [07:09<22:05, 2.59s/it] 25%|████████████████████████████████████████▍ | 171/681 [07:12<21:15, 2.50s/it] 25%|████████████████████████████████████████▋ | 172/681 [07:14<21:03, 2.48s/it] 25%|████████████████████████████████████████▉ | 173/681 [07:17<20:53, 2.47s/it] 26%|█████████████████████████████████████████▏ | 174/681 [07:19<20:58, 2.48s/it] 26%|█████████████████████████████████████████▎ | 175/681 [07:22<21:12, 2.52s/it] {'loss': 0.8288, 'grad_norm': 40.217952728271484, 'learning_rate': 4.645557588393406e-07, 'fcm_dpo/beta': 0.025716686621308327, 'fcm_dpo/q_t': 0.30824658274650574, 'fcm_dpo/delta': -0.06026681512594223, 'fcm_dpo/margin': 41.025394439697266, 'margin_dpo/margin_mean': 41.025394439697266, 'margin_dpo/margin_std': 45.7181396484375, 'logps/chosen': -94.55276489257812, 'logps/rejected': -165.5651397705078, 'logps/ref_chosen': -62.996971130371094, 'logps/ref_rejected': -92.98394012451172, 'logits/chosen': -0.6109501123428345, 'logits/rejected': -0.5846124887466431, 'epoch': 0.26} 26%|█████████████████████████████████████████▎ | 175/681 [07:22<21:12, 2.52s/it] 26%|█████████████████████████████████████████▌ | 176/681 [07:24<20:48, 2.47s/it] 26%|█████████████████████████████████████████▊ | 177/681 [07:27<21:05, 2.51s/it] 26%|██████████████████████████████████████████ | 178/681 [07:29<21:09, 2.52s/it] 26%|██████████████████████████████████████████▎ | 179/681 [07:32<21:34, 2.58s/it] 26%|██████████████████████████████████████████▌ | 180/681 [07:34<21:17, 2.55s/it] {'loss': 0.7974, 'grad_norm': 28.27334976196289, 'learning_rate': 4.611919330113591e-07, 'fcm_dpo/beta': 0.021963346749544144, 'fcm_dpo/q_t': 0.2960316240787506, 'fcm_dpo/delta': -0.16025081276893616, 'fcm_dpo/margin': 52.09009552001953, 'margin_dpo/margin_mean': 52.09009552001953, 'margin_dpo/margin_std': 56.285667419433594, 'logps/chosen': -94.24219512939453, 'logps/rejected': -186.37680053710938, 'logps/ref_chosen': -57.0670280456543, 'logps/ref_rejected': -97.1115493774414, 'logits/chosen': -0.5492862462997437, 'logits/rejected': -0.5417381525039673, 'epoch': 0.26} 26%|██████████████████████████████████████████▌ | 180/681 [07:34<21:17, 2.55s/it] 27%|██████████████████████████████████████████▊ | 181/681 [07:37<21:18, 2.56s/it] 27%|███████████████████████████████████████████ | 182/681 [07:39<20:59, 2.52s/it] 27%|███████████████████████████████████████████▎ | 183/681 [07:42<20:23, 2.46s/it] 27%|███████████████████████████████████████████▌ | 184/681 [07:44<20:56, 2.53s/it] 27%|███████████████████████████████████████████▋ | 185/681 [07:47<20:36, 2.49s/it] {'loss': 0.9251, 'grad_norm': 41.11544418334961, 'learning_rate': 4.5768898691940836e-07, 'fcm_dpo/beta': 0.021468380466103554, 'fcm_dpo/q_t': 0.33963409066200256, 'fcm_dpo/delta': 0.1422530859708786, 'fcm_dpo/margin': 40.60783386230469, 'margin_dpo/margin_mean': 40.60783767700195, 'margin_dpo/margin_std': 49.8930778503418, 'logps/chosen': -89.1233901977539, 'logps/rejected': -150.4005126953125, 'logps/ref_chosen': -54.840736389160156, 'logps/ref_rejected': -75.51002502441406, 'logits/chosen': -0.5713580250740051, 'logits/rejected': -0.5433865785598755, 'epoch': 0.27} 27%|███████████████████████████████████████████▋ | 185/681 [07:47<20:36, 2.49s/it] 27%|███████████████████████████████████████████▉ | 186/681 [07:49<20:32, 2.49s/it] 27%|████████████████████████████████████████████▏ | 187/681 [07:52<19:58, 2.43s/it] 28%|████████████████████████████████████████████▍ | 188/681 [07:54<20:09, 2.45s/it] 28%|████████████████████████████████████████████▋ | 189/681 [07:57<20:27, 2.49s/it] 28%|████████████████████████████████████████████▉ | 190/681 [07:59<19:43, 2.41s/it] {'loss': 0.8272, 'grad_norm': 36.971946716308594, 'learning_rate': 4.5404922808905543e-07, 'fcm_dpo/beta': 0.02240898832678795, 'fcm_dpo/q_t': 0.3056829571723938, 'fcm_dpo/delta': -0.08936971426010132, 'fcm_dpo/margin': 48.08588790893555, 'margin_dpo/margin_mean': 48.08588790893555, 'margin_dpo/margin_std': 53.611724853515625, 'logps/chosen': -92.63221740722656, 'logps/rejected': -169.85658264160156, 'logps/ref_chosen': -57.72148895263672, 'logps/ref_rejected': -86.85997009277344, 'logits/chosen': -0.5829431414604187, 'logits/rejected': -0.5550256371498108, 'epoch': 0.28} 28%|████████████████████████████████████████████▉ | 190/681 [07:59<19:43, 2.41s/it] 28%|█████████████████████████████████████████████▏ | 191/681 [08:02<20:28, 2.51s/it] 28%|█████████████████████████████████████████████▍ | 192/681 [08:04<20:33, 2.52s/it] 28%|█████████████████████████████████████████████▋ | 193/681 [08:07<20:08, 2.48s/it] 28%|█████████████████████████████████████████████▊ | 194/681 [08:09<19:28, 2.40s/it] 29%|██████████████████████████████████████████████ | 195/681 [08:11<19:55, 2.46s/it] {'loss': 0.829, 'grad_norm': 41.08122253417969, 'learning_rate': 4.5027505416968985e-07, 'fcm_dpo/beta': 0.020605169236660004, 'fcm_dpo/q_t': 0.30824965238571167, 'fcm_dpo/delta': -0.03353874385356903, 'fcm_dpo/margin': 49.802146911621094, 'margin_dpo/margin_mean': 49.802146911621094, 'margin_dpo/margin_std': 54.20235061645508, 'logps/chosen': -97.18587493896484, 'logps/rejected': -178.19125366210938, 'logps/ref_chosen': -58.26164627075195, 'logps/ref_rejected': -89.46485900878906, 'logits/chosen': -0.5500985383987427, 'logits/rejected': -0.5345898270606995, 'epoch': 0.29} 29%|██████████████████████████████████████████████ | 195/681 [08:11<19:55, 2.46s/it] 29%|██████████████████████████████████████████████▎ | 196/681 [08:14<20:02, 2.48s/it] 29%|██████████████████████████████████████████████▌ | 197/681 [08:16<20:10, 2.50s/it] 29%|██████████████████████████████████████████████▊ | 198/681 [08:19<20:17, 2.52s/it] 29%|███████████████████████████████████████████████ | 199/681 [08:22<20:35, 2.56s/it] 29%|███████████████████████████████████████████████▎ | 200/681 [08:24<20:31, 2.56s/it] {'loss': 0.8238, 'grad_norm': 37.28245162963867, 'learning_rate': 4.4636895135509966e-07, 'fcm_dpo/beta': 0.019669184461236, 'fcm_dpo/q_t': 0.30649372935295105, 'fcm_dpo/delta': -0.09520086646080017, 'fcm_dpo/margin': 55.09596633911133, 'margin_dpo/margin_mean': 55.09596633911133, 'margin_dpo/margin_std': 62.59626007080078, 'logps/chosen': -96.5008316040039, 'logps/rejected': -179.0296173095703, 'logps/ref_chosen': -55.71953201293945, 'logps/ref_rejected': -83.15235137939453, 'logits/chosen': -0.5128642320632935, 'logits/rejected': -0.4953628480434418, 'epoch': 0.29} 29%|███████████████████████████████████████████████▎ | 200/681 [08:24<20:31, 2.56s/it][INFO|trainer.py:4307] 2026-04-22 00:48:51,020 >> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-22 00:48:51,020 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-22 00:48:51,020 >> Batch size = 8 0%| | 0/73 [00:00> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-22 00:57:56,127 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-22 00:57:56,127 >> Batch size = 8 0%| | 0/73 [00:00> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-22 01:07:04,749 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-22 01:07:04,749 >> Batch size = 8 0%| | 0/73 [00:00> Training completed. Do not forget to share your model on huggingface.co/models =) {'train_runtime': 1845.654, 'train_samples_per_second': 23.622, 'train_steps_per_second': 0.369, 'train_loss': 0.846195821282447, 'epoch': 1.0} 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [30:43<00:00, 2.54s/it] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [30:43<00:00, 2.71s/it] ***** train metrics ***** epoch = 1.0 total_flos = 0GF train_loss = 0.8462 train_runtime = 0:30:45.65 train_samples = 43598 train_samples_per_second = 23.622 train_steps_per_second = 0.369 2026-04-22 01:11:09 - INFO - __main__ - *** Training complete *** 2026-04-22 01:11:09 - INFO - __main__ - *** Save model *** [INFO|configuration_utils.py:419] 2026-04-22 01:11:40,782 >> Configuration saved in /root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802/config.json [INFO|configuration_utils.py:911] 2026-04-22 01:11:40,783 >> Configuration saved in /root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802/generation_config.json [INFO|modeling_utils.py:3580] 2026-04-22 01:12:00,634 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 7 checkpoint shards. You can find where each parameters has been saved in the index located at /root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802/model.safetensors.index.json. [INFO|tokenization_utils_base.py:2510] 2026-04-22 01:12:00,637 >> tokenizer config file saved in /root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802/tokenizer_config.json [INFO|tokenization_utils_base.py:2519] 2026-04-22 01:12:00,637 >> Special tokens file saved in /root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802/special_tokens_map.json 2026-04-22 01:12:00 - INFO - __main__ - Saved HF-compatible model artifacts to /root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802 [INFO|modelcard.py:450] 2026-04-22 01:12:02,751 >> Dropping the following result as it does not have all the necessary fields: {'dataset': {'name': 'Anthropic/hh-rlhf', 'type': 'Anthropic/hh-rlhf'}} [INFO|configuration_utils.py:419] 2026-04-22 01:12:02,754 >> Configuration saved in /root/dynamic-dpo-v4/outputs/outputs/llama-3-8b-base-new-dpo-hh-helpful-s_star1.0-4xh200-batch-64-20260421-233802/config.json 2026-04-22 01:12:02 - INFO - __main__ - *** Evaluate *** [INFO|trainer.py:4307] 2026-04-22 01:12:02,755 >> ***** Running Evaluation ***** [INFO|trainer.py:4309] 2026-04-22 01:12:02,755 >> Num examples = 2339 [INFO|trainer.py:4312] 2026-04-22 01:12:02,755 >> Batch size = 8 0%| | 0/73 [00:00