jackf857/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
3b8f28089ffade7833976da4f1d33c46220ee373
qwen3-8b-base-epsilon-dpo-h…/train.log
ModelHub XC 3b8f28089f 初始化项目，由ModelHub XC社区提供模型 
Model: jackf857/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306
Source: Original Platform
2026-05-14 01:00:02 +08:00

1734 lines
1.1 MiB
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

2026-04-24 04:03:22 - INFO - __main__ - Model parameters ModelArguments(base_model_revision=None, model_name_or_path='/scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452', model_revision='main', model_code_revision=None, torch_dtype='bfloat16', tokenizer_name_or_path=None, trust_remote_code=False, attn_implementation='flash_attention_2', use_peft=False, lora_r=16, lora_alpha=32, lora_dropout=0.05, lora_target_modules=None, lora_modules_to_save=None, load_in_8bit=False, load_in_4bit=False, bnb_4bit_quant_type='nf4', use_bnb_nested_quant=False, bnb_4bit_quant_storage='uint8')
2026-04-24 04:03:22 - INFO - __main__ - Data parameters DataArguments(chat_template=None, dataset_mixer={'Anthropic/hh-rlhf': 1.0}, text_column='text', dataset_splits=['train', 'test'], dataset_configs=['helpful-base'], dataset_dir=None, preprocessing_num_workers=12, use_persistent_hf_cache=True, hf_cache_dir='/scratch/qu.yang1/hf/datasets', truncation_side=None, auto_insert_empty_system_msg=True, disable_thinking=True, preprocessing_log_samples=0, preprocessing_log_dir=None)
2026-04-24 04:03:22 - INFO - __main__ - Training/evaluation parameters EpsilonDPOConfig(
_n_gpu=1,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
average_tokens_across_devices=False,
batch_eval_metrics=False,
beta=0.1,
bf16=True,
bf16_full_eval=False,
data_seed=None,
dataloader_drop_last=True,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
dataset_num_proc=12,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_dropout=True,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=False,
epsilon=0.01,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=100,
eval_strategy=IntervalStrategy.STEPS,
eval_use_gather_object=False,
f_alpha_divergence_coef=1.0,
f_divergence_type=FDivergenceType.REVERSE_KL,
force_use_ref_model=False,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
generate_during_eval=False,
gradient_accumulation_steps=2,
gradient_checkpointing=True,
gradient_checkpointing_kwargs={'use_reentrant': False},
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_margin_dataset_id=qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-margin-log,
hub_model_id=qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200,
hub_model_revision=main,
hub_private_repo=None,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_for_metrics=[],
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
is_encoder_decoder=None,
jit_mode_eval=False,
label_names=None,
label_pad_token_id=-100,
label_smoothing=0.0,
label_smoothing_factor=0.0,
learning_rate=5e-07,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=info,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200/runs/Apr24_04-03-22_d4054,
logging_first_step=True,
logging_nan_inf_filter=True,
logging_steps=1,
logging_strategy=IntervalStrategy.STEPS,
loss_type=sigmoid,
lr_scheduler_kwargs={},
lr_scheduler_type=SchedulerType.COSINE,
margin_dataset_private=None,
margin_dataset_split=train,
max_grad_norm=1.0,
max_length=512,
max_prompt_length=256,
max_steps=-1,
max_target_length=None,
metric_for_best_model=None,
model_adapter_name=None,
model_init_kwargs=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
non_finite_logits_handling=error,
num_train_epochs=1,
optim=OptimizerNames.ADAMW_TORCH,
optim_args=None,
optim_target_modules=None,
output_dir=/scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306,
overwrite_output_dir=False,
padding_value=None,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
post_tokenization_log_dir=None,
post_tokenization_log_samples=0,
precompute_ref_batch_size=None,
precompute_ref_eval_batch_size=None,
precompute_ref_log_probs=False,
prediction_loss_only=False,
push_margin_dataset=True,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
ref_adapter_name=None,
ref_model_init_kwargs=None,
ref_model_mixup_alpha=0.9,
ref_model_sync_steps=64,
reference_free=False,
remove_unused_columns=False,
report_to=['wandb'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
reuse_tokenized_dataset=True,
rpo_alpha=None,
run_name=qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=200,
save_strategy=SaveStrategy.STEPS,
save_total_limit=2,
seed=42,
sft_weight=0.0,
skip_memory_metrics=True,
sync_ref_model=False,
tf32=None,
tokenization_batch_size=128,
tokenization_mode=online,
tokenized_dataset_cache_dir=/scratch/qu.yang1/tokenized_preferences,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tp_size=0,
tpu_metrics_debug=False,
tpu_num_cores=None,
trainer_type=epsilon_dpo,
truncation_mode=keep_end,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
wandb_project=qwen3_hh_4xh200_beta_0.1,
warmup_ratio=0.1,
warmup_steps=0,
weight_decay=0.0,
)
2026-04-24 04:03:22 - INFO - __main__ - Using W&B project from training args: qwen3_hh_4xh200_beta_0.1
2026-04-24 04:03:22 - INFO - __main__ - Epsilon-DPO parameters: beta=0.1, epsilon=0.01, gradient_accumulation_steps=2
2026-04-24 04:03:22 - INFO - __main__ - Using persistent HF datasets cache at /scratch/qu.yang1/hf/datasets
2026-04-24 04:03:25 - WARNING - __main__ - Dropped 237 non-canonical HH preference examples from split `train` before normalization (126 x HH preprocessing expects exactly one final assistant response in chosen/rejected suffixes., 111 x HH chosen/rejected transcripts must each contain a divergent assistant response.).
Normalizing raw HH preferences (train): 0%| | 0/43598 [00:00<?, ? examples/s] Normalizing raw HH preferences (train): 0%| | 0/43598 [00:00<?, ? examples/s] Normalizing raw HH preferences (train): 0%| | 0/43598 [00:00<?, ? examples/s] Normalizing raw HH preferences (train): 3%|█▌ | 1129/43598 [00:00<00:03, 11234.17 examples/s] Normalizing raw HH preferences (train): 3%|█▌ | 1120/43598 [00:00<00:03, 11147.92 examples/s] Normalizing raw HH preferences (train): 0%| | 0/43598 [00:00<?, ? examples/s] Normalizing raw HH preferences (train): 3%|█▋ | 1202/43598 [00:00<00:03, 11960.78 examples/s] Normalizing raw HH preferences (train): 5%|███▎ | 2382/43598 [00:00<00:03, 11986.18 examples/s] Normalizing raw HH preferences (train): 5%|███▎ | 2371/43598 [00:00<00:03, 11946.46 examples/s] Normalizing raw HH preferences (train): 3%|█▌ | 1112/43598 [00:00<00:03, 11074.46 examples/s] Normalizing raw HH preferences (train): 6%|███▍ | 2518/43598 [00:00<00:03, 12658.35 examples/s] Normalizing raw HH preferences (train): 8%|█████ | 3678/43598 [00:00<00:03, 12210.49 examples/s] Normalizing raw HH preferences (train): 8%|█████ | 3681/43598 [00:00<00:03, 12189.99 examples/s] Normalizing raw HH preferences (train): 5%|███▎ | 2382/43598 [00:00<00:03, 12021.56 examples/s] Normalizing raw HH preferences (train): 9%|█████▏ | 3814/43598 [00:00<00:03, 12793.33 examples/s] Normalizing raw HH preferences (train): 11%|██████▊ | 4907/43598 [00:00<00:03, 12237.53 examples/s] Normalizing raw HH preferences (train): 11%|██████▊ | 4909/43598 [00:00<00:03, 12220.65 examples/s] Normalizing raw HH preferences (train): 8%|█████ | 3687/43598 [00:00<00:03, 12295.66 examples/s] Normalizing raw HH preferences (train): 13%|███████▊ | 5717/43598 [00:00<00:02, 12718.81 examples/s] Normalizing raw HH preferences (train): 15%|█████████▏ | 6700/43598 [00:00<00:03, 12103.75 examples/s] Normalizing raw HH preferences (train): 11%|██████▊ | 4928/43598 [00:00<00:03, 12337.19 examples/s] Normalizing raw HH preferences (train): 15%|█████████▏ | 6703/43598 [00:00<00:03, 12095.35 examples/s] Normalizing raw HH preferences (train): 15%|█████████▎ | 6727/43598 [00:00<00:03, 12174.50 examples/s] Normalizing raw HH preferences (train): 18%|███████████▏ | 7980/43598 [00:00<00:04, 8217.10 examples/s] Normalizing raw HH preferences (train): 17%|██████████▎ | 7343/43598 [00:00<00:04, 8631.55 examples/s] Normalizing raw HH preferences (train): 18%|███████████▏ | 7975/43598 [00:00<00:04, 8074.77 examples/s] Normalizing raw HH preferences (train): 21%|████████████▋ | 9097/43598 [00:00<00:03, 8875.54 examples/s] Normalizing raw HH preferences (train): 20%|████████████▏ | 8698/43598 [00:00<00:03, 9607.30 examples/s] Normalizing raw HH preferences (train): 18%|███████████▏ | 7992/43598 [00:00<00:03, 8983.99 examples/s] Normalizing raw HH preferences (train): 21%|████████████▋ | 9027/43598 [00:00<00:04, 8612.99 examples/s] Normalizing raw HH preferences (train): 24%|██████████████▏ | 10342/43598 [00:01<00:03, 9738.24 examples/s] Normalizing raw HH preferences (train): 23%|█████████████▊ | 9997/43598 [00:00<00:03, 10410.02 examples/s] Normalizing raw HH preferences (train): 21%|████████████▊ | 9133/43598 [00:00<00:03, 9563.54 examples/s] Normalizing raw HH preferences (train): 24%|██████████████ | 10258/43598 [00:01<00:03, 9498.98 examples/s] Normalizing raw HH preferences (train): 27%|███████████████▊ | 11670/43598 [00:01<00:03, 10462.74 examples/s] Normalizing raw HH preferences (train): 26%|███████████████▏ | 11195/43598 [00:01<00:02, 10804.52 examples/s] Normalizing raw HH preferences (train): 24%|█████████████▉ | 10322/43598 [00:00<00:03, 10153.63 examples/s] Normalizing raw HH preferences (train): 26%|███████████████▌ | 11499/43598 [00:01<00:03, 10239.40 examples/s] Normalizing raw HH preferences (train): 30%|█████████████████▍ | 12902/43598 [00:01<00:02, 10953.93 examples/s] Normalizing raw HH preferences (train): 29%|████████████████▉ | 12483/43598 [00:01<00:02, 11354.64 examples/s] Normalizing raw HH preferences (train): 29%|█████████████████▏ | 12746/43598 [00:01<00:02, 10833.88 examples/s] Normalizing raw HH preferences (train): 27%|███████████████▊ | 11675/43598 [00:01<00:02, 10821.25 examples/s] Normalizing raw HH preferences (train): 32%|██████████████████▋ | 13769/43598 [00:01<00:02, 11766.55 examples/s] Normalizing raw HH preferences (train): 32%|██████████████████▉ | 13973/43598 [00:01<00:02, 11228.02 examples/s] Normalizing raw HH preferences (train): 30%|█████████████████▍ | 12925/43598 [00:01<00:02, 11272.62 examples/s] Normalizing raw HH preferences (train): 34%|███████████████████▉ | 14693/43598 [00:01<00:02, 11308.86 examples/s] Normalizing raw HH preferences (train): 37%|█████████████████████▌ | 15950/43598 [00:01<00:02, 11631.20 examples/s] Normalizing raw HH preferences (train): 36%|█████████████████████▎ | 15707/43598 [00:01<00:02, 12094.36 examples/s] Normalizing raw HH preferences (train): 36%|█████████████████████▍ | 15798/43598 [00:01<00:02, 11572.48 examples/s] Normalizing raw HH preferences (train): 34%|███████████████████▉ | 14742/43598 [00:01<00:02, 11576.00 examples/s] Normalizing raw HH preferences (train): 39%|███████████████████████ | 17000/43598 [00:01<00:02, 12105.60 examples/s] Normalizing raw HH preferences (train): 39%|███████████████████████ | 17000/43598 [00:01<00:02, 11553.49 examples/s] Normalizing raw HH preferences (train): 37%|█████████████████████▋ | 16000/43598 [00:01<00:02, 11633.73 examples/s] Normalizing raw HH preferences (train): 41%|████████████████████████ | 17738/43598 [00:01<00:02, 11728.75 examples/s] Normalizing raw HH preferences (train): 42%|████████████████████████▋ | 18272/43598 [00:01<00:02, 12267.48 examples/s] Normalizing raw HH preferences (train): 42%|████████████████████████▋ | 18201/43598 [00:01<00:02, 11675.07 examples/s] Normalizing raw HH preferences (train): 40%|███████████████████████▎ | 17248/43598 [00:01<00:02, 11858.18 examples/s] Normalizing raw HH preferences (train): 43%|█████████████████████████▋ | 18956/43598 [00:01<00:02, 11839.45 examples/s] Normalizing raw HH preferences (train): 45%|██████████████████████████▎ | 19428/43598 [00:01<00:02, 11838.20 examples/s] Normalizing raw HH preferences (train): 42%|█████████████████████████ | 18474/43598 [00:01<00:02, 11964.56 examples/s] Normalizing raw HH preferences (train): 46%|███████████████████████████▏ | 20089/43598 [00:01<00:01, 12210.60 examples/s] Normalizing raw HH preferences (train): 48%|████████████████████████████ | 20769/43598 [00:01<00:01, 11921.12 examples/s] Normalizing raw HH preferences (train): 47%|███████████████████████████▉ | 20683/43598 [00:01<00:01, 12036.02 examples/s] Normalizing raw HH preferences (train): 45%|██████████████████████████▋ | 19721/43598 [00:01<00:01, 12106.11 examples/s] Normalizing raw HH preferences (train): 49%|████████████████████████████▉ | 21379/43598 [00:01<00:01, 12387.61 examples/s] Normalizing raw HH preferences (train): 50%|█████████████████████████████▊ | 21995/43598 [00:01<00:01, 12004.84 examples/s] Normalizing raw HH preferences (train): 50%|█████████████████████████████▋ | 21916/43598 [00:01<00:01, 12118.44 examples/s] Normalizing raw HH preferences (train): 48%|████████████████████████████▎ | 20966/43598 [00:01<00:01, 12201.21 examples/s] Normalizing raw HH preferences (train): 52%|██████████████████████████████▋ | 22696/43598 [00:01<00:01, 12528.90 examples/s] Normalizing raw HH preferences (train): 55%|████████████████████████████████▏ | 23823/43598 [00:02<00:01, 12065.17 examples/s] Normalizing raw HH preferences (train): 55%|████████████████████████████████▍ | 23998/43598 [00:02<00:01, 12656.06 examples/s] Normalizing raw HH preferences (train): 54%|████████████████████████████████ | 23738/43598 [00:02<00:01, 12126.08 examples/s] Normalizing raw HH preferences (train): 52%|██████████████████████████████▊ | 22791/43598 [00:01<00:01, 12181.72 examples/s] Normalizing raw HH preferences (train): 57%|█████████████████████████████████▊ | 24963/43598 [00:02<00:01, 12157.70 examples/s] Normalizing raw HH preferences (train): 59%|██████████████████████████████████▋ | 25675/43598 [00:02<00:01, 12036.06 examples/s] Normalizing raw HH preferences (train): 59%|██████████████████████████████████▉ | 25854/43598 [00:02<00:01, 12548.06 examples/s] Normalizing raw HH preferences (train): 57%|█████████████████████████████████▍ | 24670/43598 [00:02<00:01, 12163.53 examples/s] Normalizing raw HH preferences (train): 62%|████████████████████████████████████▍ | 26887/43598 [00:02<00:01, 12054.38 examples/s] Normalizing raw HH preferences (train): 61%|████████████████████████████████████▏ | 26765/43598 [00:02<00:01, 12101.95 examples/s] Normalizing raw HH preferences (train): 59%|███████████████████████████████████ | 25901/43598 [00:02<00:01, 12197.95 examples/s] Normalizing raw HH preferences (train): 64%|█████████████████████████████████████▍ | 27710/43598 [00:02<00:01, 12484.53 examples/s] Normalizing raw HH preferences (train): 64%|█████████████████████████████████████▉ | 27992/43598 [00:02<00:01, 12142.00 examples/s] Normalizing raw HH preferences (train): 66%|██████████████████████████████████████▊ | 28680/43598 [00:02<00:01, 12016.62 examples/s] Normalizing raw HH preferences (train): 67%|███████████████████████████████████████▏ | 28997/43598 [00:02<00:01, 12577.84 examples/s] Normalizing raw HH preferences (train): 64%|█████████████████████████████████████▍ | 27698/43598 [00:02<00:01, 12117.72 examples/s] Normalizing raw HH preferences (train): 69%|████████████████████████████████████████▌ | 29932/43598 [00:02<00:01, 12138.50 examples/s] Normalizing raw HH preferences (train): 68%|████████████████████████████████████████▎ | 29827/43598 [00:02<00:01, 12172.75 examples/s] Normalizing raw HH preferences (train): 66%|███████████████████████████████████████▏ | 28946/43598 [00:02<00:01, 12206.60 examples/s] Normalizing raw HH preferences (train): 71%|█████████████████████████████████████████▊ | 30875/43598 [00:02<00:01, 12553.64 examples/s] Normalizing raw HH preferences (train): 73%|██████████████████████████████████████████▉ | 31749/43598 [00:02<00:00, 12125.78 examples/s] Normalizing raw HH preferences (train): 73%|██████████████████████████████████████████▊ | 31680/43598 [00:02<00:00, 12189.08 examples/s] Normalizing raw HH preferences (train): 71%|█████████████████████████████████████████▋ | 30790/43598 [00:02<00:01, 12232.65 examples/s] Normalizing raw HH preferences (train): 75%|████████████████████████████████████████████▎ | 32738/43598 [00:02<00:00, 12506.84 examples/s] Normalizing raw HH preferences (train): 76%|████████████████████████████████████████████▋ | 32989/43598 [00:02<00:00, 12191.93 examples/s] Normalizing raw HH preferences (train): 76%|████████████████████████████████████████████▌ | 32927/43598 [00:02<00:00, 12255.26 examples/s] Normalizing raw HH preferences (train): 78%|██████████████████████████████████████████████ | 34000/43598 [00:02<00:00, 12315.33 examples/s] Normalizing raw HH preferences (train): 75%|████████████████████████████████████████████▏ | 32689/43598 [00:02<00:00, 12247.20 examples/s] Normalizing raw HH preferences (train): 80%|███████████████████████████████████████████████ | 34738/43598 [00:03<00:00, 12003.96 examples/s] Normalizing raw HH preferences (train): 81%|███████████████████████████████████████████████▊ | 35295/43598 [00:02<00:00, 12472.02 examples/s] Normalizing raw HH preferences (train): 80%|███████████████████████████████████████████████ | 34742/43598 [00:03<00:00, 12200.83 examples/s] Normalizing raw HH preferences (train): 78%|█████████████████████████████████████████████▉ | 33917/43598 [00:02<00:00, 12251.96 examples/s] Normalizing raw HH preferences (train): 84%|█████████████████████████████████████████████████▍ | 36562/43598 [00:03<00:00, 12523.50 examples/s] Normalizing raw HH preferences (train): 84%|█████████████████████████████████████████████████▍ | 36500/43598 [00:03<00:00, 11918.58 examples/s] Normalizing raw HH preferences (train): 84%|█████████████████████████████████████████████████▍ | 36513/43598 [00:03<00:00, 12069.65 examples/s] Normalizing raw HH preferences (train): 82%|████████████████████████████████████████████████▎ | 35734/43598 [00:03<00:00, 12203.54 examples/s] Normalizing raw HH preferences (train): 87%|███████████████████████████████████████████████████▏ | 37836/43598 [00:03<00:00, 12581.26 examples/s] Normalizing raw HH preferences (train): 87%|███████████████████████████████████████████████████ | 37745/43598 [00:03<00:00, 12044.65 examples/s] Normalizing raw HH preferences (train): 87%|███████████████████████████████████████████████████ | 37762/43598 [00:03<00:00, 12167.07 examples/s] Normalizing raw HH preferences (train): 85%|██████████████████████████████████████████████████ | 36958/43598 [00:03<00:00, 12209.46 examples/s] Normalizing raw HH preferences (train): 89%|████████████████████████████████████████████████████▋ | 38967/43598 [00:03<00:00, 12086.90 examples/s] Normalizing raw HH preferences (train): 91%|█████████████████████████████████████████████████████▋ | 39701/43598 [00:03<00:00, 12524.14 examples/s] Normalizing raw HH preferences (train): 91%|█████████████████████████████████████████████████████▌ | 39540/43598 [00:03<00:00, 12057.60 examples/s] Normalizing raw HH preferences (train): 89%|████████████████████████████████████████████████████▍ | 38763/43598 [00:03<00:00, 12145.92 examples/s] Normalizing raw HH preferences (train): 94%|███████████████████████████████████████████████████████▍ | 40963/43598 [00:03<00:00, 12547.56 examples/s] Normalizing raw HH preferences (train): 93%|███████████████████████████████████████████████████████▏ | 40762/43598 [00:03<00:00, 12041.18 examples/s] Normalizing raw HH preferences (train): 94%|███████████████████████████████████████████████████████▏ | 40774/43598 [00:03<00:00, 12124.59 examples/s] Normalizing raw HH preferences (train): 92%|██████████████████████████████████████████████████████▏ | 39996/43598 [00:03<00:00, 12189.66 examples/s] Normalizing raw HH preferences (train): 96%|████████████████████████████████████████████████████████▊ | 41994/43598 [00:03<00:00, 12107.28 examples/s] Normalizing raw HH preferences (train): 98%|█████████████████████████████████████████████████████████▉ | 42833/43598 [00:03<00:00, 12514.03 examples/s] Normalizing raw HH preferences (train): 96%|████████████████████████████████████████████████████████▊ | 42000/43598 [00:03<00:00, 11960.89 examples/s] Normalizing raw HH preferences (train): 96%|████████████████████████████████████████████████████████▌ | 41818/43598 [00:03<00:00, 12170.96 examples/s] Normalizing raw HH preferences (train): 99%|███████████████████████████████████████████████████████████▎| 43083/43598 [00:03<00:00, 8530.40 examples/s] Normalizing raw HH preferences (train): 100%|████████████████████████████████████████████████████████████| 43598/43598 [00:04<00:00, 7576.49 examples/s] Normalizing raw HH preferences (train): 100%|████████████████████████████████████████████████████████████| 43598/43598 [00:04<00:00, 7648.78 examples/s] Normalizing raw HH preferences (train): 100%|███████████████████████████████████████████████████████████| 43598/43598 [00:04<00:00, 10824.39 examples/s]
Normalizing raw HH preferences (train): 100%|███████████████████████████████████████████████████████████| 43598/43598 [00:04<00:00, 10519.86 examples/s]
Normalizing raw HH preferences (train): 100%|███████████████████████████████████████████████████████████| 43598/43598 [00:04<00:00, 10468.33 examples/s]
Normalizing raw HH preferences (train): 100%|███████████████████████████████████████████████████████████| 43598/43598 [00:04<00:00, 10827.46 examples/s]
Normalizing raw HH preferences (test): 0%| | 0/2339 [00:00<?, ? examples/s] Normalizing raw HH preferences (test): 0%| | 0/2339 [00:00<?, ? examples/s] Normalizing raw HH preferences (test): 0%| | 0/2339 [00:00<?, ? examples/s] Normalizing raw HH preferences (test): 37%|███████████████████████▋ | 867/2339 [00:00<00:00, 8615.53 examples/s] Normalizing raw HH preferences (test): 49%|██████████████████████████████▌ | 1155/2339 [00:00<00:00, 11502.24 examples/s] Normalizing raw HH preferences (test): 48%|██████████████████████████████ | 1133/2339 [00:00<00:00, 11287.92 examples/s]2026-04-24 04:03:30 - WARNING - __main__ - Dropped 15 non-canonical HH preference examples from split `test` before normalization (9 x HH preprocessing expects exactly one final assistant response in chosen/rejected suffixes., 6 x HH chosen/rejected transcripts must each contain a divergent assistant response.).
Normalizing raw HH preferences (test): 0%| | 0/2339 [00:00<?, ? examples/s] Normalizing raw HH preferences (test): 100%|██████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 11269.84 examples/s] Normalizing raw HH preferences (test): 100%|██████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 11250.21 examples/s] Normalizing raw HH preferences (test): 35%|██████████████████████▍ | 822/2339 [00:00<00:00, 8174.95 examples/s] Normalizing raw HH preferences (test): 100%|██████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 10574.07 examples/s]
Normalizing raw HH preferences (test): 100%|██████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 10543.45 examples/s]
Normalizing raw HH preferences (test): 86%|█████████████████████████████████████████████████████▊ | 2000/2339 [00:00<00:00, 7508.62 examples/s] Normalizing raw HH preferences (test): 100%|███████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 7356.82 examples/s]
Normalizing raw HH preferences (test): 72%|█████████████████████████████████████████████ | 1673/2339 [00:00<00:00, 8367.44 examples/s] Normalizing raw HH preferences (test): 100%|███████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 8337.13 examples/s]
2026-04-24 04:03:30 - INFO - __main__ - Training on the following splits: ['train : 43598', 'test : 2339']
[INFO|tokenization_utils_base.py:2058] 2026-04-24 04:03:30,350 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2058] 2026-04-24 04:03:30,350 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2058] 2026-04-24 04:03:30,350 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2058] 2026-04-24 04:03:30,350 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2058] 2026-04-24 04:03:30,350 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2058] 2026-04-24 04:03:30,350 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2058] 2026-04-24 04:03:30,350 >> loading file chat_template.jinja
[INFO|tokenization_utils_base.py:2323] 2026-04-24 04:03:30,646 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you.
warnings.warn(
/home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you.
warnings.warn(
/home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you.
warnings.warn(
[WARNING|logging.py:328] 2026-04-24 04:03:30,762 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[WARNING|logging.py:328] 2026-04-24 04:03:30,762 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[WARNING|logging.py:328] 2026-04-24 04:03:30,773 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s] Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s] Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 526.67it/s]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 527.89it/s]
2026-04-24 04:03:30 - INFO - __main__ - Processed train sample 41905:
Prompt:
<|im_start|>system
<|im_end|>
<|im_start|>user
What is the best way to clean my refrigerator?<|im_end|>
<|im_start|>assistant
Well, I have a few ideas, but the best way to know what works best is to run some tests and then analyze the data to figure out which technique works best.<|im_end|>
<|im_start|>user
What are your ideas?<|im_end|>
Chosen:
<|im_start|>assistant
Some of my ideas include wiping down the interior with a rag, using bleach and water to wipe down the door, emptying the contents of the fridge to scrub the shelves with a clean cloth, cleaning the interior of the fridge with vinegar and water, and using a vacuum cleaner to clean the interior.<|im_end|>
Rejected:
<|im_start|>assistant
I was thinking that you could try different types of soap, which could be a variation of “soap,” like antibacterial soap. I was also thinking that you could try using different tools, like a toothbrush, to clean the inside. Or I was thinking that you could try different ways to get the gunk out, like using a squeegee, or using a paper towel. I was also thinking you could try other types of cleaning, like vacuuming, but I think that could have the opposite of the desired effect.<|im_end|>
/home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:391: UserWarning: You passed a model_id to the trainer. This will automatically create an `AutoModelForCausalLM` or a `PeftModel` (if you passed a `peft_config`) for you.
warnings.warn(
[INFO|configuration_utils.py:691] 2026-04-24 04:03:30,830 >> loading configuration file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/config.json
[INFO|configuration_utils.py:765] 2026-04-24 04:03:30,831 >> Model config Qwen3Config {
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 12288,
"max_position_embeddings": 32768,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.51.0",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 151936
}
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 755.85it/s]
[INFO|modeling_utils.py:1121] 2026-04-24 04:03:30,838 >> loading weights file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/model.safetensors.index.json
[INFO|modeling_utils.py:2167] 2026-04-24 04:03:30,838 >> Instantiating Qwen3ForCausalLM model under default dtype torch.bfloat16.
[WARNING|logging.py:328] 2026-04-24 04:03:30,839 >> You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
[INFO|configuration_utils.py:1142] 2026-04-24 04:03:30,840 >> Generate config GenerationConfig {
"bos_token_id": 151643,
"eos_token_id": 151643,
"use_cache": false
}
Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s] Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s] Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s] Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 465.00it/s] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 484.06it/s]
[WARNING|trainer.py:821] 2026-04-24 04:03:30,892 >> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead.
[WARNING|trainer.py:821] 2026-04-24 04:03:30,892 >> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<00:00, 439.10it/s]
[WARNING|trainer.py:821] 2026-04-24 04:03:30,901 >> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead.
Loading checkpoint shards: 14%|████████████▊ | 1/7 [00:08<00:51, 8.54s/it] Loading checkpoint shards: 29%|█████████████████████████▋ | 2/7 [00:09<00:21, 4.30s/it] Loading checkpoint shards: 43%|██████████████████████████████████████▌ | 3/7 [00:11<00:11, 2.90s/it] Loading checkpoint shards: 57%|███████████████████████████████████████████████████▍ | 4/7 [00:12<00:06, 2.24s/it] Loading checkpoint shards: 71%|████████████████████████████████████████████████████████████████▎ | 5/7 [00:13<00:03, 1.83s/it] Loading checkpoint shards: 86%|█████████████████████████████████████████████████████████████████████████████▏ | 6/7 [00:14<00:01, 1.59s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:15<00:00, 1.32s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:15<00:00, 2.19s/it]
[INFO|modeling_utils.py:4926] 2026-04-24 04:03:46,199 >> All model checkpoint weights were used when initializing Qwen3ForCausalLM.
[INFO|modeling_utils.py:4934] 2026-04-24 04:03:46,199 >> All the weights of Qwen3ForCausalLM were initialized from the model checkpoint at /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen3ForCausalLM for predictions without further training.
[INFO|configuration_utils.py:1095] 2026-04-24 04:03:46,201 >> loading configuration file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/generation_config.json
[INFO|configuration_utils.py:1142] 2026-04-24 04:03:46,202 >> Generate config GenerationConfig {
"bos_token_id": 151643,
"eos_token_id": 151643,
"max_new_tokens": 2048
}
[INFO|configuration_utils.py:691] 2026-04-24 04:03:46,203 >> loading configuration file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/config.json
[INFO|configuration_utils.py:765] 2026-04-24 04:03:46,203 >> Model config Qwen3Config {
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 12288,
"max_position_embeddings": 32768,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.51.0",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 151936
}
[INFO|modeling_utils.py:1121] 2026-04-24 04:03:46,204 >> loading weights file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/model.safetensors.index.json
[INFO|modeling_utils.py:2167] 2026-04-24 04:03:46,205 >> Instantiating Qwen3ForCausalLM model under default dtype torch.bfloat16.
[INFO|configuration_utils.py:1142] 2026-04-24 04:03:46,207 >> Generate config GenerationConfig {
"bos_token_id": 151643,
"eos_token_id": 151643,
"use_cache": false
}
Loading checkpoint shards: 0%| | 0/7 [00:00<?, ?it/s] Loading checkpoint shards: 14%|████████████▊ | 1/7 [00:01<00:08, 1.36s/it] Loading checkpoint shards: 29%|█████████████████████████▋ | 2/7 [00:02<00:06, 1.34s/it] Loading checkpoint shards: 43%|██████████████████████████████████████▌ | 3/7 [00:03<00:05, 1.29s/it] Loading checkpoint shards: 57%|███████████████████████████████████████████████████▍ | 4/7 [00:05<00:03, 1.25s/it] Loading checkpoint shards: 71%|████████████████████████████████████████████████████████████████▎ | 5/7 [00:06<00:02, 1.20s/it] Loading checkpoint shards: 86%|█████████████████████████████████████████████████████████████████████████████▏ | 6/7 [00:07<00:01, 1.16s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:08<00:00, 1.03s/it] Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:08<00:00, 1.15s/it]
[INFO|modeling_utils.py:4926] 2026-04-24 04:03:54,298 >> All model checkpoint weights were used when initializing Qwen3ForCausalLM.
[INFO|modeling_utils.py:4934] 2026-04-24 04:03:54,298 >> All the weights of Qwen3ForCausalLM were initialized from the model checkpoint at /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen3ForCausalLM for predictions without further training.
[INFO|configuration_utils.py:1095] 2026-04-24 04:03:54,300 >> loading configuration file /scratch/qu.yang1/dynamic-dpo-v4/outputs/qwen3-8b-base-sft-hh-helpful-4xh200-batch-64-20260417-214452/generation_config.json
[INFO|configuration_utils.py:1142] 2026-04-24 04:03:54,300 >> Generate config GenerationConfig {
"bos_token_id": 151643,
"eos_token_id": 151643,
"max_new_tokens": 2048
}
[WARNING|trainer.py:821] 2026-04-24 04:03:54,302 >> Trainer.tokenizer is now deprecated. You should use `Trainer.processing_class = processing_class` instead.
[WARNING|trainer.py:816] 2026-04-24 04:03:54,302 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
Tokenizing train (num_proc=12): 0%| | 0/43598 [00:00<?, ? examples/s] Tokenizing train (num_proc=12): 0%|▏ | 128/43598 [00:33<3:10:37, 3.80 examples/s] Tokenizing train (num_proc=12): 1%|▍ | 256/43598 [00:33<1:18:40, 9.18 examples/s] Tokenizing train (num_proc=12): 1%|▋ | 384/43598 [00:33<42:56, 16.77 examples/s] Tokenizing train (num_proc=12): 1%|▊ | 512/43598 [00:34<26:11, 27.41 examples/s] Tokenizing train (num_proc=12): 1%|█ | 640/43598 [00:34<16:59, 42.14 examples/s] Tokenizing train (num_proc=12): 2%|█▎ | 768/43598 [00:34<11:25, 62.48 examples/s] Tokenizing train (num_proc=12): 2%|█▍ | 896/43598 [00:34<07:54, 90.05 examples/s] Tokenizing train (num_proc=12): 2%|█▋ | 1024/43598 [00:34<05:36, 126.70 examples/s] Tokenizing train (num_proc=12): 3%|█▊ | 1152/43598 [00:34<04:03, 174.02 examples/s] Tokenizing train (num_proc=12): 3%|██ | 1280/43598 [00:34<03:01, 232.65 examples/s] Tokenizing train (num_proc=12): 3%|██▎ | 1408/43598 [00:34<02:18, 304.41 examples/s] Tokenizing train (num_proc=12): 4%|██▍ | 1536/43598 [00:35<01:48, 386.86 examples/s] Tokenizing train (num_proc=12): 4%|██▋ | 1664/43598 [00:35<01:28, 474.31 examples/s] Tokenizing train (num_proc=12): 4%|██▉ | 1792/43598 [00:35<01:14, 560.64 examples/s] Tokenizing train (num_proc=12): 4%|███ | 1920/43598 [00:35<01:05, 635.06 examples/s] Tokenizing train (num_proc=12): 5%|███▎ | 2048/43598 [00:35<00:58, 707.31 examples/s] Tokenizing train (num_proc=12): 5%|███▍ | 2176/43598 [00:35<00:54, 763.75 examples/s] Tokenizing train (num_proc=12): 5%|███▋ | 2304/43598 [00:35<00:51, 807.06 examples/s] Tokenizing train (num_proc=12): 6%|███▉ | 2432/43598 [00:36<00:48, 847.88 examples/s] Tokenizing train (num_proc=12): 6%|████ | 2560/43598 [00:36<00:47, 858.65 examples/s] Tokenizing train (num_proc=12): 6%|████▎ | 2688/43598 [00:36<00:45, 896.33 examples/s] Tokenizing train (num_proc=12): 6%|████▌ | 2816/43598 [00:36<00:45, 897.46 examples/s] Tokenizing train (num_proc=12): 7%|████▋ | 2944/43598 [00:36<00:43, 933.44 examples/s] Tokenizing train (num_proc=12): 7%|████▉ | 3072/43598 [00:36<00:44, 918.31 examples/s] Tokenizing train (num_proc=12): 7%|█████▏ | 3200/43598 [00:36<00:43, 918.15 examples/s] Tokenizing train (num_proc=12): 8%|█████▎ | 3328/43598 [00:37<00:44, 905.12 examples/s] Tokenizing train (num_proc=12): 8%|█████▌ | 3456/43598 [00:37<00:43, 925.97 examples/s] Tokenizing train (num_proc=12): 8%|█████▊ | 3584/43598 [00:37<00:42, 939.37 examples/s] Tokenizing train (num_proc=12): 8%|█████▊ | 3634/43598 [00:49<00:42, 939.37 examples/s] Tokenizing train (num_proc=12): 9%|██████▏ | 3762/43598 [00:59<31:15, 21.24 examples/s] Tokenizing train (num_proc=12): 9%|██████▎ | 3890/43598 [00:59<22:42, 29.13 examples/s] Tokenizing train (num_proc=12): 9%|██████▌ | 4018/43598 [00:59<16:23, 40.26 examples/s] Tokenizing train (num_proc=12): 10%|██████▊ | 4146/43598 [00:59<11:47, 55.74 examples/s] Tokenizing train (num_proc=12): 10%|██████▉ | 4274/43598 [00:59<08:30, 77.08 examples/s] Tokenizing train (num_proc=12): 10%|███████ | 4402/43598 [01:00<06:10, 105.85 examples/s] Tokenizing train (num_proc=12): 10%|███████▎ | 4530/43598 [01:00<04:31, 144.11 examples/s] Tokenizing train (num_proc=12): 11%|███████▍ | 4658/43598 [01:00<03:20, 193.74 examples/s] Tokenizing train (num_proc=12): 11%|███████▋ | 4786/43598 [01:00<02:32, 254.60 examples/s] Tokenizing train (num_proc=12): 11%|███████▉ | 4914/43598 [01:00<01:59, 324.07 examples/s] Tokenizing train (num_proc=12): 12%|████████ | 5042/43598 [01:00<01:35, 404.25 examples/s] Tokenizing train (num_proc=12): 12%|████████▎ | 5170/43598 [01:00<01:18, 490.54 examples/s] Tokenizing train (num_proc=12): 12%|████████▌ | 5298/43598 [01:00<01:07, 567.17 examples/s] Tokenizing train (num_proc=12): 12%|████████▋ | 5426/43598 [01:01<00:58, 652.14 examples/s] Tokenizing train (num_proc=12): 13%|████████▉ | 5554/43598 [01:01<00:53, 712.14 examples/s] Tokenizing train (num_proc=12): 13%|█████████ | 5682/43598 [01:01<00:50, 751.94 examples/s] Tokenizing train (num_proc=12): 13%|█████████▎ | 5810/43598 [01:01<00:46, 817.82 examples/s] Tokenizing train (num_proc=12): 14%|█████████▌ | 5938/43598 [01:01<00:43, 856.92 examples/s] Tokenizing train (num_proc=12): 14%|█████████▋ | 6066/43598 [01:01<00:42, 885.03 examples/s] Tokenizing train (num_proc=12): 14%|█████████▉ | 6194/43598 [01:01<00:41, 898.50 examples/s] Tokenizing train (num_proc=12): 15%|██████████▏ | 6322/43598 [01:02<00:41, 888.22 examples/s] Tokenizing train (num_proc=12): 15%|██████████▎ | 6450/43598 [01:02<00:41, 904.94 examples/s] Tokenizing train (num_proc=12): 15%|██████████▌ | 6578/43598 [01:02<00:39, 927.17 examples/s] Tokenizing train (num_proc=12): 15%|██████████▊ | 6706/43598 [01:02<00:39, 925.30 examples/s] Tokenizing train (num_proc=12): 16%|██████████▉ | 6834/43598 [01:02<00:39, 936.64 examples/s] Tokenizing train (num_proc=12): 16%|███████████▏ | 6962/43598 [01:02<00:39, 935.45 examples/s] Tokenizing train (num_proc=12): 16%|███████████▍ | 7090/43598 [01:02<00:38, 942.35 examples/s] Tokenizing train (num_proc=12): 17%|███████████▌ | 7218/43598 [01:02<00:38, 949.02 examples/s] Tokenizing train (num_proc=12): 17%|███████████▋ | 7268/43598 [01:16<00:38, 949.02 examples/s] Tokenizing train (num_proc=12): 17%|████████████ | 7396/43598 [01:25<28:29, 21.18 examples/s] Tokenizing train (num_proc=12): 17%|████████████▎ | 7524/43598 [01:25<20:40, 29.07 examples/s] Tokenizing train (num_proc=12): 18%|████████████▍ | 7652/43598 [01:25<14:58, 39.99 examples/s] Tokenizing train (num_proc=12): 18%|████████████▋ | 7780/43598 [01:25<10:50, 55.08 examples/s] Tokenizing train (num_proc=12): 18%|████████████▉ | 7908/43598 [01:25<07:50, 75.87 examples/s] Tokenizing train (num_proc=12): 18%|████████████▉ | 8036/43598 [01:25<05:45, 102.83 examples/s] Tokenizing train (num_proc=12): 19%|█████████████ | 8164/43598 [01:26<04:16, 137.92 examples/s] Tokenizing train (num_proc=12): 19%|█████████████▎ | 8292/43598 [01:26<03:10, 185.26 examples/s] Tokenizing train (num_proc=12): 19%|█████████████▌ | 8420/43598 [01:26<02:24, 243.77 examples/s] Tokenizing train (num_proc=12): 20%|█████████████▋ | 8548/43598 [01:26<01:52, 312.52 examples/s] Tokenizing train (num_proc=12): 20%|█████████████▉ | 8676/43598 [01:26<01:28, 395.92 examples/s] Tokenizing train (num_proc=12): 20%|██████████████▏ | 8804/43598 [01:26<01:12, 479.82 examples/s] Tokenizing train (num_proc=12): 20%|██████████████▎ | 8932/43598 [01:26<01:00, 574.28 examples/s] Tokenizing train (num_proc=12): 21%|██████████████▌ | 9060/43598 [01:27<00:52, 652.93 examples/s] Tokenizing train (num_proc=12): 21%|██████████████▊ | 9188/43598 [01:27<00:47, 719.87 examples/s] Tokenizing train (num_proc=12): 21%|██████████████▉ | 9316/43598 [01:27<00:43, 781.28 examples/s] Tokenizing train (num_proc=12): 22%|███████████████▏ | 9444/43598 [01:27<00:41, 830.72 examples/s] Tokenizing train (num_proc=12): 22%|███████████████▎ | 9572/43598 [01:27<00:39, 868.99 examples/s] Tokenizing train (num_proc=12): 22%|███████████████▌ | 9700/43598 [01:27<00:37, 901.79 examples/s] Tokenizing train (num_proc=12): 23%|███████████████▊ | 9828/43598 [01:27<00:37, 909.22 examples/s] Tokenizing train (num_proc=12): 23%|███████████████▉ | 9956/43598 [01:27<00:36, 923.42 examples/s] Tokenizing train (num_proc=12): 23%|███████████████▉ | 10084/43598 [01:28<00:36, 923.23 examples/s] Tokenizing train (num_proc=12): 23%|████████████████▏ | 10212/43598 [01:28<00:35, 928.25 examples/s] Tokenizing train (num_proc=12): 24%|████████████████▎ | 10340/43598 [01:28<00:35, 949.67 examples/s] Tokenizing train (num_proc=12): 24%|████████████████▌ | 10468/43598 [01:28<00:33, 976.65 examples/s] Tokenizing train (num_proc=12): 24%|████████████████▊ | 10596/43598 [01:28<00:33, 979.44 examples/s] Tokenizing train (num_proc=12): 25%|████████████████▉ | 10724/43598 [01:28<00:33, 992.16 examples/s] Tokenizing train (num_proc=12): 25%|█████████████████▏ | 10852/43598 [01:28<00:33, 977.32 examples/s] Tokenizing train (num_proc=12): 25%|█████████████████▎ | 10901/43598 [01:39<00:33, 977.32 examples/s] Tokenizing train (num_proc=12): 25%|█████████████████▋ | 11029/43598 [01:49<23:43, 22.88 examples/s] Tokenizing train (num_proc=12): 26%|█████████████████▉ | 11157/43598 [01:49<17:12, 31.41 examples/s] Tokenizing train (num_proc=12): 26%|██████████████████ | 11285/43598 [01:49<12:25, 43.32 examples/s] Tokenizing train (num_proc=12): 26%|██████████████████▎ | 11413/43598 [01:49<08:58, 59.78 examples/s] Tokenizing train (num_proc=12): 26%|██████████████████▌ | 11541/43598 [01:49<06:29, 82.31 examples/s] Tokenizing train (num_proc=12): 27%|██████████████████▍ | 11669/43598 [01:50<04:43, 112.64 examples/s] Tokenizing train (num_proc=12): 27%|██████████████████▋ | 11797/43598 [01:50<03:28, 152.57 examples/s] Tokenizing train (num_proc=12): 27%|██████████████████▊ | 11925/43598 [01:50<02:35, 203.65 examples/s] Tokenizing train (num_proc=12): 28%|███████████████████ | 12053/43598 [01:50<01:59, 264.35 examples/s] Tokenizing train (num_proc=12): 28%|███████████████████▎ | 12181/43598 [01:50<01:31, 343.16 examples/s] Tokenizing train (num_proc=12): 28%|███████████████████▍ | 12309/43598 [01:50<01:12, 428.72 examples/s] Tokenizing train (num_proc=12): 29%|███████████████████▋ | 12437/43598 [01:50<01:00, 513.72 examples/s] Tokenizing train (num_proc=12): 29%|███████████████████▉ | 12565/43598 [01:50<00:51, 606.66 examples/s] Tokenizing train (num_proc=12): 29%|████████████████████ | 12693/43598 [01:51<00:44, 701.16 examples/s] Tokenizing train (num_proc=12): 29%|████████████████████▎ | 12821/43598 [01:51<00:39, 773.15 examples/s] Tokenizing train (num_proc=12): 30%|████████████████████▍ | 12949/43598 [01:51<00:37, 817.58 examples/s] Tokenizing train (num_proc=12): 30%|████████████████████▋ | 13077/43598 [01:51<00:35, 866.70 examples/s] Tokenizing train (num_proc=12): 30%|████████████████████▉ | 13205/43598 [01:51<00:37, 819.67 examples/s] Tokenizing train (num_proc=12): 31%|█████████████████████ | 13333/43598 [01:51<00:37, 798.17 examples/s] Tokenizing train (num_proc=12): 31%|█████████████████████▎ | 13461/43598 [01:51<00:38, 778.49 examples/s] Tokenizing train (num_proc=12): 31%|█████████████████████▌ | 13589/43598 [01:52<00:38, 776.06 examples/s] Tokenizing train (num_proc=12): 31%|█████████████████████▋ | 13717/43598 [01:52<00:38, 769.36 examples/s] Tokenizing train (num_proc=12): 32%|█████████████████████▉ | 13845/43598 [01:52<00:38, 763.04 examples/s] Tokenizing train (num_proc=12): 32%|██████████████████████ | 13973/43598 [01:52<00:39, 759.37 examples/s] Tokenizing train (num_proc=12): 32%|██████████████████████▎ | 14101/43598 [01:52<00:39, 750.26 examples/s] Tokenizing train (num_proc=12): 33%|██████████████████████▌ | 14229/43598 [01:52<00:38, 759.30 examples/s] Tokenizing train (num_proc=12): 33%|██████████████████████▋ | 14357/43598 [01:53<00:37, 770.34 examples/s] Tokenizing train (num_proc=12): 33%|██████████████████████▉ | 14485/43598 [01:53<00:36, 797.42 examples/s] Tokenizing train (num_proc=12): 33%|███████████████████████ | 14534/43598 [02:07<00:36, 797.42 examples/s] Tokenizing train (num_proc=12): 34%|███████████████████████▌ | 14662/43598 [02:13<21:23, 22.55 examples/s] Tokenizing train (num_proc=12): 34%|███████████████████████▋ | 14790/43598 [02:14<15:30, 30.95 examples/s] Tokenizing train (num_proc=12): 34%|███████████████████████▉ | 14918/43598 [02:14<11:11, 42.73 examples/s] Tokenizing train (num_proc=12): 35%|████████████████████████▏ | 15046/43598 [02:14<08:02, 59.15 examples/s] Tokenizing train (num_proc=12): 35%|████████████████████████▎ | 15174/43598 [02:14<05:47, 81.70 examples/s] Tokenizing train (num_proc=12): 35%|████████████████████████▏ | 15302/43598 [02:14<04:13, 111.74 examples/s] Tokenizing train (num_proc=12): 35%|████████████████████████▍ | 15430/43598 [02:14<03:06, 151.31 examples/s] Tokenizing train (num_proc=12): 36%|████████████████████████▌ | 15558/43598 [02:14<02:19, 201.32 examples/s] Tokenizing train (num_proc=12): 36%|████████████████████████▊ | 15686/43598 [02:15<01:45, 263.80 examples/s] Tokenizing train (num_proc=12): 36%|█████████████████████████ | 15814/43598 [02:15<01:22, 336.90 examples/s] Tokenizing train (num_proc=12): 37%|█████████████████████████▏ | 15942/43598 [02:15<01:06, 417.94 examples/s] Tokenizing train (num_proc=12): 37%|█████████████████████████▍ | 16070/43598 [02:15<00:54, 504.79 examples/s] Tokenizing train (num_proc=12): 37%|█████████████████████████▋ | 16198/43598 [02:15<00:47, 582.09 examples/s] Tokenizing train (num_proc=12): 37%|█████████████████████████▊ | 16326/43598 [02:15<00:40, 668.70 examples/s] Tokenizing train (num_proc=12): 38%|██████████████████████████ | 16454/43598 [02:15<00:36, 749.00 examples/s] Tokenizing train (num_proc=12): 38%|██████████████████████████▏ | 16582/43598 [02:15<00:33, 807.76 examples/s] Tokenizing train (num_proc=12): 38%|██████████████████████████▍ | 16710/43598 [02:16<00:31, 840.79 examples/s] Tokenizing train (num_proc=12): 39%|██████████████████████████▋ | 16838/43598 [02:16<00:30, 877.64 examples/s] Tokenizing train (num_proc=12): 39%|██████████████████████████▊ | 16966/43598 [02:16<00:28, 918.53 examples/s] Tokenizing train (num_proc=12): 39%|███████████████████████████ | 17094/43598 [02:16<00:29, 911.55 examples/s] Tokenizing train (num_proc=12): 40%|███████████████████████████▎ | 17222/43598 [02:16<00:28, 932.47 examples/s] Tokenizing train (num_proc=12): 40%|███████████████████████████▍ | 17350/43598 [02:16<00:27, 941.54 examples/s] Tokenizing train (num_proc=12): 40%|███████████████████████████▋ | 17478/43598 [02:16<00:27, 946.70 examples/s] Tokenizing train (num_proc=12): 40%|███████████████████████████▊ | 17606/43598 [02:16<00:27, 960.64 examples/s] Tokenizing train (num_proc=12): 41%|████████████████████████████ | 17734/43598 [02:17<00:27, 957.35 examples/s] Tokenizing train (num_proc=12): 41%|████████████████████████████▎ | 17862/43598 [02:17<00:26, 987.28 examples/s] Tokenizing train (num_proc=12): 41%|████████████████████████████▍ | 17990/43598 [02:17<00:25, 991.64 examples/s] Tokenizing train (num_proc=12): 42%|████████████████████████████▋ | 18118/43598 [02:17<00:25, 998.28 examples/s] Tokenizing train (num_proc=12): 42%|████████████████████████████▊ | 18167/43598 [02:29<00:25, 998.28 examples/s] Tokenizing train (num_proc=12): 42%|█████████████████████████████▎ | 18295/43598 [02:38<18:33, 22.72 examples/s] Tokenizing train (num_proc=12): 42%|█████████████████████████████▌ | 18423/43598 [02:38<13:26, 31.20 examples/s] Tokenizing train (num_proc=12): 43%|█████████████████████████████▊ | 18551/43598 [02:38<09:41, 43.08 examples/s] Tokenizing train (num_proc=12): 43%|█████████████████████████████▉ | 18679/43598 [02:38<06:57, 59.62 examples/s] Tokenizing train (num_proc=12): 43%|██████████████████████████████▏ | 18807/43598 [02:38<05:01, 82.10 examples/s] Tokenizing train (num_proc=12): 43%|█████████████████████████████▉ | 18935/43598 [02:38<03:39, 112.57 examples/s] Tokenizing train (num_proc=12): 44%|██████████████████████████████▏ | 19063/43598 [02:38<02:41, 152.00 examples/s] Tokenizing train (num_proc=12): 44%|██████████████████████████████▎ | 19191/43598 [02:38<01:58, 205.15 examples/s] Tokenizing train (num_proc=12): 44%|██████████████████████████████▌ | 19319/43598 [02:39<01:30, 267.61 examples/s] Tokenizing train (num_proc=12): 45%|██████████████████████████████▊ | 19447/43598 [02:39<01:10, 341.18 examples/s] Tokenizing train (num_proc=12): 45%|██████████████████████████████▉ | 19575/43598 [02:39<00:56, 422.67 examples/s] Tokenizing train (num_proc=12): 45%|███████████████████████████████▏ | 19703/43598 [02:39<00:46, 512.25 examples/s] Tokenizing train (num_proc=12): 45%|███████████████████████████████▍ | 19831/43598 [02:39<00:40, 591.22 examples/s] Tokenizing train (num_proc=12): 46%|███████████████████████████████▌ | 19959/43598 [02:39<00:35, 664.69 examples/s] Tokenizing train (num_proc=12): 46%|███████████████████████████████▊ | 20087/43598 [02:39<00:32, 725.95 examples/s] Tokenizing train (num_proc=12): 46%|███████████████████████████████▉ | 20215/43598 [02:40<00:30, 775.90 examples/s] Tokenizing train (num_proc=12): 47%|████████████████████████████████▏ | 20343/43598 [02:40<00:28, 828.24 examples/s] Tokenizing train (num_proc=12): 47%|████████████████████████████████▍ | 20471/43598 [02:40<00:25, 897.25 examples/s] Tokenizing train (num_proc=12): 47%|████████████████████████████████▌ | 20599/43598 [02:40<00:25, 899.18 examples/s] Tokenizing train (num_proc=12): 48%|████████████████████████████████▊ | 20727/43598 [02:40<00:25, 902.06 examples/s] Tokenizing train (num_proc=12): 48%|█████████████████████████████████ | 20855/43598 [02:40<00:25, 889.93 examples/s] Tokenizing train (num_proc=12): 48%|█████████████████████████████████▏ | 20983/43598 [02:40<00:24, 915.57 examples/s] Tokenizing train (num_proc=12): 48%|█████████████████████████████████▍ | 21111/43598 [02:40<00:23, 945.41 examples/s] Tokenizing train (num_proc=12): 49%|█████████████████████████████████▌ | 21239/43598 [02:41<00:23, 966.38 examples/s] Tokenizing train (num_proc=12): 49%|█████████████████████████████████▊ | 21367/43598 [02:41<00:22, 993.68 examples/s] Tokenizing train (num_proc=12): 49%|██████████████████████████████████ | 21495/43598 [02:41<00:22, 995.92 examples/s] Tokenizing train (num_proc=12): 50%|██████████████████████████████████▏ | 21623/43598 [02:41<00:22, 984.93 examples/s] Tokenizing train (num_proc=12): 50%|██████████████████████████████████▍ | 21751/43598 [02:41<00:22, 972.69 examples/s] Tokenizing train (num_proc=12): 50%|██████████████████████████████████▌ | 21800/43598 [02:57<00:22, 972.69 examples/s] Tokenizing train (num_proc=12): 50%|███████████████████████████████████▏ | 21928/43598 [03:02<16:26, 21.97 examples/s] Tokenizing train (num_proc=12): 51%|███████████████████████████████████▍ | 22056/43598 [03:03<11:54, 30.13 examples/s] Tokenizing train (num_proc=12): 51%|███████████████████████████████████▌ | 22184/43598 [03:03<08:34, 41.60 examples/s] Tokenizing train (num_proc=12): 51%|███████████████████████████████████▊ | 22312/43598 [03:03<06:09, 57.54 examples/s] Tokenizing train (num_proc=12): 51%|████████████████████████████████████ | 22440/43598 [03:03<04:26, 79.37 examples/s] Tokenizing train (num_proc=12): 52%|███████████████████████████████████▋ | 22568/43598 [03:03<03:13, 108.55 examples/s] Tokenizing train (num_proc=12): 52%|███████████████████████████████████▉ | 22696/43598 [03:03<02:21, 147.37 examples/s] Tokenizing train (num_proc=12): 52%|████████████████████████████████████ | 22824/43598 [03:03<01:45, 196.40 examples/s] Tokenizing train (num_proc=12): 53%|████████████████████████████████████▎ | 22952/43598 [03:04<01:20, 256.26 examples/s] Tokenizing train (num_proc=12): 53%|████████████████████████████████████▌ | 23080/43598 [03:04<01:02, 326.21 examples/s] Tokenizing train (num_proc=12): 53%|████████████████████████████████████▋ | 23208/43598 [03:04<00:50, 405.95 examples/s] Tokenizing train (num_proc=12): 54%|████████████████████████████████████▉ | 23336/43598 [03:04<00:41, 489.91 examples/s] Tokenizing train (num_proc=12): 54%|█████████████████████████████████████▏ | 23464/43598 [03:04<00:35, 569.91 examples/s] Tokenizing train (num_proc=12): 54%|█████████████████████████████████████▎ | 23592/43598 [03:04<00:31, 642.54 examples/s] Tokenizing train (num_proc=12): 54%|█████████████████████████████████████▌ | 23720/43598 [03:04<00:28, 704.73 examples/s] Tokenizing train (num_proc=12): 55%|█████████████████████████████████████▋ | 23848/43598 [03:05<00:25, 763.88 examples/s] Tokenizing train (num_proc=12): 55%|█████████████████████████████████████▉ | 23976/43598 [03:05<00:24, 806.91 examples/s] Tokenizing train (num_proc=12): 55%|██████████████████████████████████████▏ | 24104/43598 [03:05<00:23, 846.84 examples/s] Tokenizing train (num_proc=12): 56%|██████████████████████████████████████▎ | 24232/43598 [03:05<00:22, 870.67 examples/s] Tokenizing train (num_proc=12): 56%|██████████████████████████████████████▌ | 24360/43598 [03:05<00:21, 892.72 examples/s] Tokenizing train (num_proc=12): 56%|██████████████████████████████████████▊ | 24488/43598 [03:05<00:20, 920.04 examples/s] Tokenizing train (num_proc=12): 56%|██████████████████████████████████████▉ | 24616/43598 [03:05<00:20, 935.59 examples/s] Tokenizing train (num_proc=12): 57%|███████████████████████████████████████▏ | 24744/43598 [03:05<00:19, 950.95 examples/s] Tokenizing train (num_proc=12): 57%|███████████████████████████████████████▎ | 24872/43598 [03:06<00:19, 954.55 examples/s] Tokenizing train (num_proc=12): 57%|███████████████████████████████████████▌ | 25000/43598 [03:06<00:19, 969.14 examples/s] Tokenizing train (num_proc=12): 58%|███████████████████████████████████████▊ | 25128/43598 [03:06<00:19, 945.68 examples/s] Tokenizing train (num_proc=12): 58%|███████████████████████████████████████▉ | 25256/43598 [03:06<00:19, 939.13 examples/s] Tokenizing train (num_proc=12): 58%|████████████████████████████████████████▏ | 25384/43598 [03:06<00:19, 921.55 examples/s] Tokenizing train (num_proc=12): 58%|████████████████████████████████████████▎ | 25433/43598 [03:17<00:19, 921.55 examples/s] Tokenizing train (num_proc=12): 59%|█████████████████████████████████████████ | 25561/43598 [03:26<12:57, 23.19 examples/s] Tokenizing train (num_proc=12): 59%|█████████████████████████████████████████▏ | 25689/43598 [03:26<09:23, 31.79 examples/s] Tokenizing train (num_proc=12): 59%|█████████████████████████████████████████▍ | 25817/43598 [03:27<06:46, 43.75 examples/s] Tokenizing train (num_proc=12): 60%|█████████████████████████████████████████▋ | 25945/43598 [03:27<04:51, 60.52 examples/s] Tokenizing train (num_proc=12): 60%|█████████████████████████████████████████▊ | 26073/43598 [03:27<03:29, 83.47 examples/s] Tokenizing train (num_proc=12): 60%|█████████████████████████████████████████▍ | 26201/43598 [03:27<02:32, 114.19 examples/s] Tokenizing train (num_proc=12): 60%|█████████████████████████████████████████▋ | 26329/43598 [03:27<01:51, 154.73 examples/s] Tokenizing train (num_proc=12): 61%|█████████████████████████████████████████▊ | 26457/43598 [03:27<01:22, 207.22 examples/s] Tokenizing train (num_proc=12): 61%|██████████████████████████████████████████ | 26585/43598 [03:27<01:02, 271.70 examples/s] Tokenizing train (num_proc=12): 61%|██████████████████████████████████████████▎ | 26713/43598 [03:28<00:48, 345.78 examples/s] Tokenizing train (num_proc=12): 62%|██████████████████████████████████████████▍ | 26841/43598 [03:28<00:39, 424.95 examples/s] Tokenizing train (num_proc=12): 62%|██████████████████████████████████████████▋ | 26969/43598 [03:28<00:32, 506.48 examples/s] Tokenizing train (num_proc=12): 62%|██████████████████████████████████████████▉ | 27097/43598 [03:28<00:28, 578.24 examples/s] Tokenizing train (num_proc=12): 62%|███████████████████████████████████████████ | 27225/43598 [03:28<00:25, 649.89 examples/s] Tokenizing train (num_proc=12): 63%|███████████████████████████████████████████▎ | 27353/43598 [03:28<00:22, 712.57 examples/s] Tokenizing train (num_proc=12): 63%|███████████████████████████████████████████▍ | 27481/43598 [03:28<00:20, 773.54 examples/s] Tokenizing train (num_proc=12): 63%|███████████████████████████████████████████▋ | 27609/43598 [03:28<00:19, 805.17 examples/s] Tokenizing train (num_proc=12): 64%|███████████████████████████████████████████▉ | 27737/43598 [03:29<00:19, 833.01 examples/s] Tokenizing train (num_proc=12): 64%|████████████████████████████████████████████ | 27865/43598 [03:29<00:18, 835.46 examples/s] Tokenizing train (num_proc=12): 64%|████████████████████████████████████████████▎ | 27993/43598 [03:29<00:18, 846.45 examples/s] Tokenizing train (num_proc=12): 65%|████████████████████████████████████████████▌ | 28121/43598 [03:29<00:17, 860.10 examples/s] Tokenizing train (num_proc=12): 65%|████████████████████████████████████████████▋ | 28249/43598 [03:29<00:17, 869.16 examples/s] Tokenizing train (num_proc=12): 65%|████████████████████████████████████████████▉ | 28377/43598 [03:29<00:17, 889.34 examples/s] Tokenizing train (num_proc=12): 65%|█████████████████████████████████████████████ | 28505/43598 [03:29<00:16, 910.00 examples/s] Tokenizing train (num_proc=12): 66%|█████████████████████████████████████████████▎ | 28633/43598 [03:30<00:15, 937.79 examples/s] Tokenizing train (num_proc=12): 66%|█████████████████████████████████████████████▌ | 28761/43598 [03:30<00:15, 937.09 examples/s] Tokenizing train (num_proc=12): 66%|█████████████████████████████████████████████▋ | 28889/43598 [03:30<00:15, 922.74 examples/s] Tokenizing train (num_proc=12): 67%|█████████████████████████████████████████████▉ | 29017/43598 [03:30<00:16, 897.64 examples/s] Tokenizing train (num_proc=12): 67%|██████████████████████████████████████████████ | 29066/43598 [03:47<00:16, 897.64 examples/s] Tokenizing train (num_proc=12): 67%|██████████████████████████████████████████████▊ | 29194/43598 [03:52<11:24, 21.06 examples/s] Tokenizing train (num_proc=12): 67%|███████████████████████████████████████████████ | 29322/43598 [03:52<08:13, 28.93 examples/s] Tokenizing train (num_proc=12): 68%|███████████████████████████████████████████████▎ | 29450/43598 [03:53<05:54, 39.94 examples/s] Tokenizing train (num_proc=12): 68%|███████████████████████████████████████████████▍ | 29578/43598 [03:53<04:13, 55.25 examples/s] Tokenizing train (num_proc=12): 68%|███████████████████████████████████████████████▋ | 29706/43598 [03:53<03:01, 76.41 examples/s] Tokenizing train (num_proc=12): 68%|███████████████████████████████████████████████▏ | 29834/43598 [03:53<02:10, 105.11 examples/s] Tokenizing train (num_proc=12): 69%|███████████████████████████████████████████████▍ | 29962/43598 [03:53<01:35, 143.10 examples/s] Tokenizing train (num_proc=12): 69%|███████████████████████████████████████████████▌ | 30090/43598 [03:53<01:10, 192.21 examples/s] Tokenizing train (num_proc=12): 69%|███████████████████████████████████████████████▊ | 30218/43598 [03:53<00:53, 252.23 examples/s] Tokenizing train (num_proc=12): 70%|████████████████████████████████████████████████ | 30346/43598 [03:53<00:40, 327.17 examples/s] Tokenizing train (num_proc=12): 70%|████████████████████████████████████████████████▏ | 30474/43598 [03:54<00:32, 405.00 examples/s] Tokenizing train (num_proc=12): 70%|████████████████████████████████████████████████▍ | 30602/43598 [03:54<00:26, 490.26 examples/s] Tokenizing train (num_proc=12): 70%|████████████████████████████████████████████████▋ | 30730/43598 [03:54<00:22, 580.99 examples/s] Tokenizing train (num_proc=12): 71%|████████████████████████████████████████████████▊ | 30858/43598 [03:54<00:19, 659.44 examples/s] Tokenizing train (num_proc=12): 71%|█████████████████████████████████████████████████ | 30986/43598 [03:54<00:17, 730.16 examples/s] Tokenizing train (num_proc=12): 71%|█████████████████████████████████████████████████▏ | 31114/43598 [03:54<00:15, 786.26 examples/s] Tokenizing train (num_proc=12): 72%|█████████████████████████████████████████████████▍ | 31242/43598 [03:54<00:14, 828.18 examples/s] Tokenizing train (num_proc=12): 72%|█████████████████████████████████████████████████▋ | 31370/43598 [03:54<00:14, 871.58 examples/s] Tokenizing train (num_proc=12): 72%|█████████████████████████████████████████████████▊ | 31498/43598 [03:55<00:13, 903.36 examples/s] Tokenizing train (num_proc=12): 73%|██████████████████████████████████████████████████ | 31626/43598 [03:55<00:13, 906.06 examples/s] Tokenizing train (num_proc=12): 73%|██████████████████████████████████████████████████▎ | 31754/43598 [03:55<00:12, 941.87 examples/s] Tokenizing train (num_proc=12): 73%|██████████████████████████████████████████████████▍ | 31882/43598 [03:55<00:12, 927.11 examples/s] Tokenizing train (num_proc=12): 73%|██████████████████████████████████████████████████▋ | 32010/43598 [03:55<00:12, 931.76 examples/s] Tokenizing train (num_proc=12): 74%|██████████████████████████████████████████████████▊ | 32138/43598 [03:55<00:12, 953.15 examples/s] Tokenizing train (num_proc=12): 74%|███████████████████████████████████████████████████ | 32266/43598 [03:55<00:12, 932.13 examples/s] Tokenizing train (num_proc=12): 74%|███████████████████████████████████████████████████▎ | 32394/43598 [03:56<00:11, 973.86 examples/s] Tokenizing train (num_proc=12): 75%|███████████████████████████████████████████████████▍ | 32522/43598 [03:56<00:11, 952.68 examples/s] Tokenizing train (num_proc=12): 75%|███████████████████████████████████████████████████▋ | 32650/43598 [03:56<00:11, 924.67 examples/s] Tokenizing train (num_proc=12): 75%|███████████████████████████████████████████████████▊ | 32699/43598 [04:07<00:11, 924.67 examples/s] Tokenizing train (num_proc=12): 75%|████████████████████████████████████████████████████▋ | 32827/43598 [04:16<07:50, 22.89 examples/s] Tokenizing train (num_proc=12): 76%|████████████████████████████████████████████████████▉ | 32955/43598 [04:16<05:39, 31.37 examples/s] Tokenizing train (num_proc=12): 76%|█████████████████████████████████████████████████████ | 33083/43598 [04:17<04:03, 43.23 examples/s] Tokenizing train (num_proc=12): 76%|█████████████████████████████████████████████████████▎ | 33211/43598 [04:17<02:53, 59.71 examples/s] Tokenizing train (num_proc=12): 76%|█████████████████████████████████████████████████████▌ | 33339/43598 [04:17<02:04, 82.32 examples/s] Tokenizing train (num_proc=12): 77%|████████████████████████████████████████████████████▉ | 33467/43598 [04:17<01:29, 112.67 examples/s] Tokenizing train (num_proc=12): 77%|█████████████████████████████████████████████████████▏ | 33595/43598 [04:17<01:05, 152.85 examples/s] Tokenizing train (num_proc=12): 77%|█████████████████████████████████████████████████████▎ | 33723/43598 [04:17<00:48, 203.80 examples/s] Tokenizing train (num_proc=12): 78%|█████████████████████████████████████████████████████▌ | 33851/43598 [04:17<00:36, 264.96 examples/s] Tokenizing train (num_proc=12): 78%|█████████████████████████████████████████████████████▊ | 33979/43598 [04:18<00:28, 337.25 examples/s] Tokenizing train (num_proc=12): 78%|█████████████████████████████████████████████████████▉ | 34107/43598 [04:18<00:22, 417.23 examples/s] Tokenizing train (num_proc=12): 79%|██████████████████████████████████████████████████████▏ | 34235/43598 [04:18<00:18, 502.08 examples/s] Tokenizing train (num_proc=12): 79%|██████████████████████████████████████████████████████▍ | 34363/43598 [04:18<00:16, 575.86 examples/s] Tokenizing train (num_proc=12): 79%|██████████████████████████████████████████████████████▌ | 34491/43598 [04:18<00:14, 650.18 examples/s] Tokenizing train (num_proc=12): 79%|██████████████████████████████████████████████████████▊ | 34619/43598 [04:18<00:12, 699.65 examples/s] Tokenizing train (num_proc=12): 80%|██████████████████████████████████████████████████████▉ | 34747/43598 [04:18<00:11, 762.72 examples/s] Tokenizing train (num_proc=12): 80%|███████████████████████████████████████████████████████▏ | 34875/43598 [04:18<00:11, 782.96 examples/s] Tokenizing train (num_proc=12): 80%|███████████████████████████████████████████████████████▍ | 35003/43598 [04:19<00:10, 800.58 examples/s] Tokenizing train (num_proc=12): 81%|███████████████████████████████████████████████████████▌ | 35131/43598 [04:19<00:10, 831.49 examples/s] Tokenizing train (num_proc=12): 81%|███████████████████████████████████████████████████████▊ | 35259/43598 [04:19<00:10, 829.18 examples/s] Tokenizing train (num_proc=12): 81%|████████████████████████████████████████████████████████ | 35387/43598 [04:19<00:09, 850.70 examples/s] Tokenizing train (num_proc=12): 81%|████████████████████████████████████████████████████████▏ | 35515/43598 [04:19<00:09, 861.92 examples/s] Tokenizing train (num_proc=12): 82%|████████████████████████████████████████████████████████▍ | 35643/43598 [04:19<00:09, 862.21 examples/s] Tokenizing train (num_proc=12): 82%|████████████████████████████████████████████████████████▌ | 35771/43598 [04:20<00:08, 881.97 examples/s] Tokenizing train (num_proc=12): 82%|████████████████████████████████████████████████████████▊ | 35899/43598 [04:20<00:08, 876.43 examples/s] Tokenizing train (num_proc=12): 83%|█████████████████████████████████████████████████████████ | 36027/43598 [04:20<00:08, 876.81 examples/s] Tokenizing train (num_proc=12): 83%|█████████████████████████████████████████████████████████▏ | 36155/43598 [04:20<00:08, 908.70 examples/s] Tokenizing train (num_proc=12): 83%|█████████████████████████████████████████████████████████▍ | 36283/43598 [04:20<00:07, 924.03 examples/s] Tokenizing train (num_proc=12): 83%|█████████████████████████████████████████████████████████▌ | 36332/43598 [04:37<00:07, 924.03 examples/s] Tokenizing train (num_proc=12): 84%|██████████████████████████████████████████████████████████▌ | 36460/43598 [04:43<05:41, 20.89 examples/s] Tokenizing train (num_proc=12): 84%|██████████████████████████████████████████████████████████▋ | 36588/43598 [04:43<04:04, 28.69 examples/s] Tokenizing train (num_proc=12): 84%|██████████████████████████████████████████████████████████▉ | 36716/43598 [04:43<02:53, 39.59 examples/s] Tokenizing train (num_proc=12): 85%|███████████████████████████████████████████████████████████▏ | 36844/43598 [04:43<02:03, 54.75 examples/s] Tokenizing train (num_proc=12): 85%|███████████████████████████████████████████████████████████▎ | 36972/43598 [04:43<01:27, 75.69 examples/s] Tokenizing train (num_proc=12): 85%|██████████████████████████████████████████████████████████▋ | 37100/43598 [04:43<01:02, 103.77 examples/s] Tokenizing train (num_proc=12): 85%|██████████████████████████████████████████████████████████▉ | 37228/43598 [04:43<00:45, 141.30 examples/s] Tokenizing train (num_proc=12): 86%|███████████████████████████████████████████████████████████ | 37356/43598 [04:43<00:33, 188.71 examples/s] Tokenizing train (num_proc=12): 86%|███████████████████████████████████████████████████████████▎ | 37484/43598 [04:44<00:24, 249.11 examples/s] Tokenizing train (num_proc=12): 86%|███████████████████████████████████████████████████████████▌ | 37612/43598 [04:44<00:18, 318.42 examples/s] Tokenizing train (num_proc=12): 87%|███████████████████████████████████████████████████████████▋ | 37740/43598 [04:44<00:14, 398.78 examples/s] Tokenizing train (num_proc=12): 87%|███████████████████████████████████████████████████████████▉ | 37868/43598 [04:44<00:11, 479.55 examples/s] Tokenizing train (num_proc=12): 87%|████████████████████████████████████████████████████████████▏ | 37996/43598 [04:44<00:10, 549.38 examples/s] Tokenizing train (num_proc=12): 87%|████████████████████████████████████████████████████████████▎ | 38124/43598 [04:44<00:08, 630.65 examples/s] Tokenizing train (num_proc=12): 88%|████████████████████████████████████████████████████████████▌ | 38252/43598 [04:44<00:07, 678.69 examples/s] Tokenizing train (num_proc=12): 88%|████████████████████████████████████████████████████████████▋ | 38380/43598 [04:45<00:07, 730.18 examples/s] Tokenizing train (num_proc=12): 88%|████████████████████████████████████████████████████████████▉ | 38508/43598 [04:45<00:06, 768.80 examples/s] Tokenizing train (num_proc=12): 89%|█████████████████████████████████████████████████████████████▏ | 38636/43598 [04:45<00:06, 824.15 examples/s] Tokenizing train (num_proc=12): 89%|█████████████████████████████████████████████████████████████▎ | 38764/43598 [04:45<00:05, 838.06 examples/s] Tokenizing train (num_proc=12): 89%|█████████████████████████████████████████████████████████████▌ | 38892/43598 [04:45<00:05, 860.26 examples/s] Tokenizing train (num_proc=12): 89%|█████████████████████████████████████████████████████████████▊ | 39020/43598 [04:45<00:05, 852.83 examples/s] Tokenizing train (num_proc=12): 90%|█████████████████████████████████████████████████████████████▉ | 39148/43598 [04:45<00:05, 865.11 examples/s] Tokenizing train (num_proc=12): 90%|██████████████████████████████████████████████████████████████▏ | 39276/43598 [04:46<00:04, 880.58 examples/s] Tokenizing train (num_proc=12): 90%|██████████████████████████████████████████████████████████████▎ | 39404/43598 [04:46<00:04, 892.62 examples/s] Tokenizing train (num_proc=12): 91%|██████████████████████████████████████████████████████████████▌ | 39532/43598 [04:46<00:04, 896.01 examples/s] Tokenizing train (num_proc=12): 91%|██████████████████████████████████████████████████████████████▊ | 39660/43598 [04:46<00:04, 870.33 examples/s] Tokenizing train (num_proc=12): 91%|██████████████████████████████████████████████████████████████▉ | 39788/43598 [04:46<00:04, 911.29 examples/s] Tokenizing train (num_proc=12): 92%|███████████████████████████████████████████████████████████████▏ | 39916/43598 [04:46<00:04, 891.55 examples/s] Tokenizing train (num_proc=12): 92%|███████████████████████████████████████████████████████████████▎ | 39965/43598 [04:57<00:04, 891.55 examples/s] Tokenizing train (num_proc=12): 92%|████████████████████████████████████████████████████████████████▎ | 40093/43598 [05:07<02:32, 22.94 examples/s] Tokenizing train (num_proc=12): 92%|████████████████████████████████████████████████████████████████▌ | 40221/43598 [05:07<01:47, 31.49 examples/s] Tokenizing train (num_proc=12): 93%|████████████████████████████████████████████████████████████████▊ | 40349/43598 [05:07<01:14, 43.50 examples/s] Tokenizing train (num_proc=12): 93%|████████████████████████████████████████████████████████████████▉ | 40477/43598 [05:07<00:51, 60.16 examples/s] Tokenizing train (num_proc=12): 93%|█████████████████████████████████████████████████████████████████▏ | 40605/43598 [05:07<00:36, 83.08 examples/s] Tokenizing train (num_proc=12): 93%|████████████████████████████████████████████████████████████████▍ | 40733/43598 [05:07<00:25, 114.21 examples/s] Tokenizing train (num_proc=12): 94%|████████████████████████████████████████████████████████████████▋ | 40861/43598 [05:07<00:17, 155.95 examples/s] Tokenizing train (num_proc=12): 94%|████████████████████████████████████████████████████████████████▊ | 40989/43598 [05:07<00:12, 208.92 examples/s] Tokenizing train (num_proc=12): 94%|█████████████████████████████████████████████████████████████████ | 41117/43598 [05:08<00:09, 273.97 examples/s] Tokenizing train (num_proc=12): 95%|█████████████████████████████████████████████████████████████████▎ | 41245/43598 [05:08<00:06, 348.25 examples/s] Tokenizing train (num_proc=12): 95%|█████████████████████████████████████████████████████████████████▍ | 41373/43598 [05:08<00:05, 435.91 examples/s] Tokenizing train (num_proc=12): 95%|█████████████████████████████████████████████████████████████████▋ | 41501/43598 [05:08<00:03, 529.58 examples/s] Tokenizing train (num_proc=12): 95%|█████████████████████████████████████████████████████████████████▉ | 41629/43598 [05:08<00:03, 622.40 examples/s] Tokenizing train (num_proc=12): 96%|██████████████████████████████████████████████████████████████████ | 41757/43598 [05:08<00:02, 709.84 examples/s] Tokenizing train (num_proc=12): 96%|██████████████████████████████████████████████████████████████████▎ | 41885/43598 [05:08<00:02, 793.46 examples/s] Tokenizing train (num_proc=12): 96%|██████████████████████████████████████████████████████████████████▍ | 42013/43598 [05:08<00:01, 862.17 examples/s] Tokenizing train (num_proc=12): 97%|██████████████████████████████████████████████████████████████████▋ | 42141/43598 [05:09<00:01, 913.49 examples/s] Tokenizing train (num_proc=12): 97%|██████████████████████████████████████████████████████████████████▉ | 42269/43598 [05:09<00:01, 958.33 examples/s] Tokenizing train (num_proc=12): 97%|██████████████████████████████████████████████████████████████████▏ | 42397/43598 [05:09<00:01, 1018.22 examples/s] Tokenizing train (num_proc=12): 98%|██████████████████████████████████████████████████████████████████▎ | 42525/43598 [05:09<00:01, 1015.48 examples/s] Tokenizing train (num_proc=12): 98%|██████████████████████████████████████████████████████████████████▌ | 42653/43598 [05:09<00:00, 1030.18 examples/s] Tokenizing train (num_proc=12): 98%|██████████████████████████████████████████████████████████████████▋ | 42781/43598 [05:09<00:00, 1049.53 examples/s] Tokenizing train (num_proc=12): 98%|██████████████████████████████████████████████████████████████████▉ | 42909/43598 [05:09<00:00, 1066.51 examples/s] Tokenizing train (num_proc=12): 99%|███████████████████████████████████████████████████████████████████▏| 43037/43598 [05:09<00:00, 1069.38 examples/s] Tokenizing train (num_proc=12): 99%|███████████████████████████████████████████████████████████████████▎| 43165/43598 [05:10<00:00, 1122.54 examples/s] Tokenizing train (num_proc=12): 99%|███████████████████████████████████████████████████████████████████▌| 43293/43598 [05:10<00:00, 1153.15 examples/s] Tokenizing train (num_proc=12): 100%|███████████████████████████████████████████████████████████████████▋| 43421/43598 [05:10<00:00, 1113.36 examples/s] Tokenizing train (num_proc=12): 100%|███████████████████████████████████████████████████████████████████▉| 43549/43598 [05:10<00:00, 1116.86 examples/s] Tokenizing train (num_proc=12): 100%|█████████████████████████████████████████████████████████████████████| 43598/43598 [05:10<00:00, 140.39 examples/s]
[WARNING|trainer.py:816] 2026-04-24 04:09:55,066 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
Saving the dataset (0/2 shards): 0%| | 0/43598 [00:00<?, ? examples/s] Saving the dataset (0/2 shards): 21%|█████████████▊ | 9000/43598 [00:00<00:00, 72936.95 examples/s] Saving the dataset (0/2 shards): 44%|████████████████████████████▊ | 19000/43598 [00:00<00:00, 80392.12 examples/s] Saving the dataset (1/2 shards): 50%|█████████████████████████████████ | 21799/43598 [00:00<00:00, 80392.12 examples/s] Saving the dataset (1/2 shards): 71%|██████████████████████████████████████████████▌ | 30799/43598 [00:00<00:00, 45409.33 examples/s] Saving the dataset (1/2 shards): 94%|█████████████████████████████████████████████████████████████▊ | 40799/43598 [00:00<00:00, 56167.66 examples/s] Saving the dataset (2/2 shards): 100%|██████████████████████████████████████████████████████████████████| 43598/43598 [00:00<00:00, 56167.66 examples/s] Saving the dataset (2/2 shards): 100%|██████████████████████████████████████████████████████████████████| 43598/43598 [00:00<00:00, 44759.04 examples/s]
[WARNING|trainer.py:816] 2026-04-24 04:09:56,718 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
Tokenizing test (num_proc=12): 0%| | 0/2339 [00:00<?, ? examples/s] Tokenizing test (num_proc=12): 5%|████ | 128/2339 [00:33<09:44, 3.79 examples/s] Tokenizing test (num_proc=12): 14%|██████████▏ | 323/2339 [00:59<05:52, 5.72 examples/s] Tokenizing test (num_proc=12): 22%|████████████████▍ | 518/2339 [01:25<04:40, 6.49 examples/s] Tokenizing test (num_proc=12): 30%|██████████████████████▌ | 713/2339 [01:52<03:57, 6.83 examples/s] Tokenizing test (num_proc=12): 39%|████████████████████████████▋ | 908/2339 [02:18<03:22, 7.06 examples/s] Tokenizing test (num_proc=12): 47%|██████████████████████████████████▍ | 1103/2339 [02:44<02:51, 7.20 examples/s] Tokenizing test (num_proc=12): 55%|████████████████████████████████████████▌ | 1298/2339 [03:10<02:23, 7.27 examples/s] Tokenizing test (num_proc=12): 64%|██████████████████████████████████████████████▌ | 1493/2339 [03:37<01:56, 7.29 examples/s] Tokenizing test (num_proc=12): 72%|████████████████████████████████████████████████████▋ | 1688/2339 [04:03<01:28, 7.34 examples/s] Tokenizing test (num_proc=12): 81%|██████████████████████████████████████████████████████████▊ | 1883/2339 [04:30<01:02, 7.33 examples/s] Tokenizing test (num_proc=12): 89%|████████████████████████████████████████████████████████████████▊ | 2078/2339 [04:56<00:35, 7.32 examples/s] Tokenizing test (num_proc=12): 97%|██████████████████████████████████████████████████████████████████████▉ | 2273/2339 [05:22<00:08, 7.38 examples/s] Tokenizing test (num_proc=12): 100%|█████████████████████████████████████████████████████████████████████████| 2339/2339 [05:22<00:00, 7.24 examples/s]
[WARNING|trainer.py:816] 2026-04-24 04:15:56,509 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
Saving the dataset (0/1 shards): 0%| | 0/2339 [00:00<?, ? examples/s] Saving the dataset (1/1 shards): 100%|████████████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 32415.47 examples/s] Saving the dataset (1/1 shards): 100%|████████████████████████████████████████████████████████████████████| 2339/2339 [00:00<00:00, 32343.23 examples/s]
/home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `EpsilonDPOTrainer.__init__`. Use `processing_class` instead.
super().__init__(
[WARNING|trainer.py:816] 2026-04-24 04:15:57,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,794 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,899 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
[WARNING|trainer.py:816] 2026-04-24 04:15:57,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
/home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `EpsilonDPOTrainer.__init__`. Use `processing_class` instead.
super().__init__(
/home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `EpsilonDPOTrainer.__init__`. Use `processing_class` instead.
super().__init__(
[WARNING|trainer.py:816] 2026-04-24 04:15:57,913 >> Trainer.tokenizer is now deprecated. You should use Trainer.processing_class instead.
/home/qu.yang1/dpo-test/dynamic-dpo-v4/scripts/tokenized_dpo_trainer.py:518: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `EpsilonDPOTrainer.__init__`. Use `processing_class` instead.
super().__init__(
[INFO|trainer.py:748] 2026-04-24 04:15:57,938 >> Using auto half precision backend
/home/qu.yang1/.conda/envs/dpo_v4/lib/python3.11/site-packages/accelerate/accelerator.py:1557: UserWarning: Upcasted low precision parameters in Qwen3ForCausalLM because mixed precision turned on in FSDP. Affects: model.embed_tokens.weight, model.norm.weight, lm_head.weight.
warnings.warn(
/home/qu.yang1/.conda/envs/dpo_v4/lib/python3.11/site-packages/accelerate/accelerator.py:1557: UserWarning: Upcasted low precision parameters in Qwen3DecoderLayer because mixed precision turned on in FSDP. Affects: self_attn.q_proj.weight, self_attn.k_proj.weight, self_attn.v_proj.weight, self_attn.o_proj.weight, self_attn.q_norm.weight, self_attn.k_norm.weight, mlp.gate_proj.weight, mlp.up_proj.weight, mlp.down_proj.weight, input_layernorm.weight, post_attention_layernorm.weight.
warnings.warn(
/home/qu.yang1/.conda/envs/dpo_v4/lib/python3.11/site-packages/accelerate/accelerator.py:1563: UserWarning: FSDP upcast of low precision parameters may affect the precision of model checkpoints.
warnings.warn(
[INFO|trainer.py:2414] 2026-04-24 04:16:02,138 >> ***** Running training *****
[INFO|trainer.py:2415] 2026-04-24 04:16:02,138 >> Num examples = 43,598
[INFO|trainer.py:2416] 2026-04-24 04:16:02,138 >> Num Epochs = 1
[INFO|trainer.py:2417] 2026-04-24 04:16:02,138 >> Instantaneous batch size per device = 8
[INFO|trainer.py:2420] 2026-04-24 04:16:02,138 >> Total train batch size (w. parallel, distributed & accumulation) = 64
[INFO|trainer.py:2421] 2026-04-24 04:16:02,138 >> Gradient Accumulation steps = 2
[INFO|trainer.py:2422] 2026-04-24 04:16:02,138 >> Total optimization steps = 681
[INFO|trainer.py:2423] 2026-04-24 04:16:02,139 >> Number of trainable parameters = 2,047,683,840
[INFO|integration_utils.py:831] 2026-04-24 04:16:02,140 >> Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
wandb: Currently logged in as: feng-cheng (feng-cheng-northeastern-university). Use `wandb login --relogin` to force relogin
wandb: wandb version 0.26.1 is available! To upgrade, please run:
wandb: $ pip install wandb --upgrade
wandb: Tracking run with wandb version 0.17.5
wandb: Run data is saved locally in /scratch/qu.yang1/wandb/wandb/run-20260424_041603-gfncx0q7
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306
wandb: ⭐️ View project at https://wandb.ai/feng-cheng-northeastern-university/qwen3_hh_4xh200_beta_0.1
wandb: 🚀 View run at https://wandb.ai/feng-cheng-northeastern-university/qwen3_hh_4xh200_beta_0.1/runs/gfncx0q7
0%| | 0/681 [00:00<?, ?it/s][WARNING|modeling_utils.py:1713] 2026-04-24 04:16:08,808 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed
[WARNING|modeling_utils.py:1713] 2026-04-24 04:16:08,817 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed
[WARNING|modeling_utils.py:1713] 2026-04-24 04:16:08,829 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed
[WARNING|modeling_utils.py:1713] 2026-04-24 04:16:08,835 >> Could not estimate the number of tokens of the input, floating-point operations will not be computed
0%|▏ | 1/681 [00:03<34:44, 3.06s/it] {'loss': 1.381, 'grad_norm': 38.745460510253906, 'learning_rate': 0.0, 'rewards/chosen': 0.005238114856183529, 'rewards/rejected': -0.0005494409706443548, 'rewards/accuracies': 0.5625, 'rewards/margins': 0.005787555128335953, 'logps/chosen': -85.37664031982422, 'logps/rejected': -79.91163635253906, 'logps/ref_chosen': -85.43083190917969, 'logps/ref_rejected': -79.90458679199219, 'logits/chosen': -1.585817575454712, 'logits/rejected': -0.5333532691001892, 'kl/p_epsilon_steps': 0.546875, 'kl/n_epsilon_steps': 0.453125, 'kl/beta': 0.10000000149011612, 'kl/avg_steps': 0.09375, 'epoch': 0.0}
0%|▏ | 1/681 [00:03<34:44, 3.06s/it] 0%|▎ | 2/681 [00:06<35:07, 3.10s/it] {'loss': 1.3995, 'grad_norm': 29.78634262084961, 'learning_rate': 7.246376811594203e-09, 'rewards/chosen': -0.004650775343179703, 'rewards/rejected': 0.008120683953166008, 'rewards/accuracies': 0.375, 'rewards/margins': -0.012771460227668285, 'logps/chosen': -82.11383056640625, 'logps/rejected': -81.57505798339844, 'logps/ref_chosen': -82.06892395019531, 'logps/ref_rejected': -81.65457153320312, 'logits/chosen': -0.7526164054870605, 'logits/rejected': -0.3610996603965759, 'kl/p_epsilon_steps': 0.34375, 'kl/n_epsilon_steps': 0.65625, 'kl/beta': 0.09990634024143219, 'kl/avg_steps': -0.3125, 'epoch': 0.0}
0%|▎ | 2/681 [00:06<35:07, 3.10s/it] 0%|▌ | 3/681 [00:09<35:13, 3.12s/it] {'loss': 1.3813, 'grad_norm': 26.100635528564453, 'learning_rate': 1.4492753623188406e-08, 'rewards/chosen': 0.005534623749554157, 'rewards/rejected': 7.430883124470711e-05, 'rewards/accuracies': 0.578125, 'rewards/margins': 0.00546031491830945, 'logps/chosen': -93.7535629272461, 'logps/rejected': -74.23006439208984, 'logps/ref_chosen': -93.81098937988281, 'logps/ref_rejected': -74.22950744628906, 'logits/chosen': -1.002709150314331, 'logits/rejected': -0.5633082985877991, 'kl/p_epsilon_steps': 0.546875, 'kl/n_epsilon_steps': 0.453125, 'kl/beta': 0.10021952539682388, 'kl/avg_steps': 0.09375, 'epoch': 0.0}
0%|▌ | 3/681 [00:09<35:13, 3.12s/it] 1%|▋ | 4/681 [00:12<35:29, 3.15s/it] {'loss': 1.3929, 'grad_norm': 32.92469024658203, 'learning_rate': 2.1739130434782606e-08, 'rewards/chosen': -0.003004699246957898, 'rewards/rejected': 0.0031696748919785023, 'rewards/accuracies': 0.5625, 'rewards/margins': -0.006174374371767044, 'logps/chosen': -87.32073211669922, 'logps/rejected': -93.79373168945312, 'logps/ref_chosen': -87.29246520996094, 'logps/ref_rejected': -93.82425689697266, 'logits/chosen': -0.8497915267944336, 'logits/rejected': -0.17156964540481567, 'kl/p_epsilon_steps': 0.546875, 'kl/n_epsilon_steps': 0.453125, 'kl/beta': 0.10012565553188324, 'kl/avg_steps': 0.09375, 'epoch': 0.01}
1%|▋ | 4/681 [00:12<35:29, 3.15s/it] 1%|▊ | 5/681 [00:15<35:15, 3.13s/it] {'loss': 1.3853, 'grad_norm': 35.190330505371094, 'learning_rate': 2.898550724637681e-08, 'rewards/chosen': -0.0021656095050275326, 'rewards/rejected': -0.0035966699942946434, 'rewards/accuracies': 0.5625, 'rewards/margins': 0.0014310609549283981, 'logps/chosen': -89.35664367675781, 'logps/rejected': -88.785400390625, 'logps/ref_chosen': -89.33675384521484, 'logps/ref_rejected': -88.74783325195312, 'logits/chosen': -1.187368392944336, 'logits/rejected': -0.5294585227966309, 'kl/p_epsilon_steps': 0.515625, 'kl/n_epsilon_steps': 0.484375, 'kl/beta': 0.10003187507390976, 'kl/avg_steps': 0.03125, 'epoch': 0.01}
1%|▊ | 5/681 [00:15<35:15, 3.13s/it] 1%|█ | 6/681 [00:18<33:23, 2.97s/it] {'loss': 1.3866, 'grad_norm': 36.109169006347656, 'learning_rate': 3.6231884057971014e-08, 'rewards/chosen': -0.0005067111924290657, 'rewards/rejected': -0.0007012896239757538, 'rewards/accuracies': 0.59375, 'rewards/margins': 0.00019457843154668808, 'logps/chosen': -97.32476043701172, 'logps/rejected': -97.89209747314453, 'logps/ref_chosen': -97.32147216796875, 'logps/ref_rejected': -97.88345336914062, 'logits/chosen': -1.2944458723068237, 'logits/rejected': -0.41752371191978455, 'kl/p_epsilon_steps': 0.546875, 'kl/n_epsilon_steps': 0.4375, 'kl/beta': 0.10000062733888626, 'kl/avg_steps': 0.109375, 'epoch': 0.01}
1%|█ | 6/681 [00:18<33:23, 2.97s/it] 1%|█▏ | 7/681 [00:21<33:51, 3.01s/it] {'loss': 1.3805, 'grad_norm': 37.86967086791992, 'learning_rate': 4.347826086956521e-08, 'rewards/chosen': 0.004504315089434385, 'rewards/rejected': -0.0016694795340299606, 'rewards/accuracies': 0.53125, 'rewards/margins': 0.006173794623464346, 'logps/chosen': -86.60205078125, 'logps/rejected': -109.63433837890625, 'logps/ref_chosen': -86.64852905273438, 'logps/ref_rejected': -109.61618041992188, 'logits/chosen': -0.6984870433807373, 'logits/rejected': -0.42031070590019226, 'kl/p_epsilon_steps': 0.546875, 'kl/n_epsilon_steps': 0.453125, 'kl/beta': 0.09989137202501297, 'kl/avg_steps': 0.09375, 'epoch': 0.01}
1%|█▏ | 7/681 [00:21<33:51, 3.01s/it] 1%|█▎ | 8/681 [00:24<33:42, 3.01s/it] {'loss': 1.3968, 'grad_norm': 32.797054290771484, 'learning_rate': 5.0724637681159424e-08, 'rewards/chosen': -0.002112824469804764, 'rewards/rejected': 0.008059106767177582, 'rewards/accuracies': 0.421875, 'rewards/margins': -0.010171930305659771, 'logps/chosen': -89.94332885742188, 'logps/rejected': -86.1485366821289, 'logps/ref_chosen': -89.9236831665039, 'logps/ref_rejected': -86.22803497314453, 'logits/chosen': -1.414137363433838, 'logits/rejected': -0.4588002562522888, 'kl/p_epsilon_steps': 0.4375, 'kl/n_epsilon_steps': 0.5625, 'kl/beta': 0.0997978076338768, 'kl/avg_steps': -0.125, 'epoch': 0.01}
1%|█▎ | 8/681 [00:24<33:42, 3.01s/it] 1%|█▌ | 9/681 [00:27<33:56, 3.03s/it] {'loss': 1.378, 'grad_norm': 35.72300338745117, 'learning_rate': 5.797101449275362e-08, 'rewards/chosen': 0.006141543388366699, 'rewards/rejected': -0.002803270472213626, 'rewards/accuracies': 0.59375, 'rewards/margins': 0.008944813162088394, 'logps/chosen': -103.79417419433594, 'logps/rejected': -104.34971618652344, 'logps/ref_chosen': -103.85713195800781, 'logps/ref_rejected': -104.31932067871094, 'logits/chosen': -0.9759007692337036, 'logits/rejected': -0.3913915753364563, 'kl/p_epsilon_steps': 0.578125, 'kl/n_epsilon_steps': 0.421875, 'kl/beta': 0.09992270916700363, 'kl/avg_steps': 0.15625, 'epoch': 0.01}
1%|█▌ | 9/681 [00:27<33:56, 3.03s/it] 1%|█▋ | 10/681 [00:30<33:52, 3.03s/it] {'loss': 1.401, 'grad_norm': 33.81359100341797, 'learning_rate': 6.521739130434782e-08, 'rewards/chosen': -0.00028783950256183743, 'rewards/rejected': 0.014049299992620945, 'rewards/accuracies': 0.359375, 'rewards/margins': -0.014337141066789627, 'logps/chosen': -76.20588684082031, 'logps/rejected': -87.01283264160156, 'logps/ref_chosen': -76.20436096191406, 'logps/ref_rejected': -87.15210723876953, 'logits/chosen': -1.2568838596343994, 'logits/rejected': -0.34031057357788086, 'kl/p_epsilon_steps': 0.375, 'kl/n_epsilon_steps': 0.625, 'kl/beta': 0.09976682811975479, 'kl/avg_steps': -0.25, 'epoch': 0.01}
1%|█▋ | 10/681 [00:30<33:52, 3.03s/it] 2%|█▊ | 11/681 [00:33<33:58, 3.04s/it] {'loss': 1.3819, 'grad_norm': 37.09693908691406, 'learning_rate': 7.246376811594203e-08, 'rewards/chosen': 0.006203308701515198, 'rewards/rejected': 0.0014935237122699618, 'rewards/accuracies': 0.515625, 'rewards/margins': 0.004709784872829914, 'logps/chosen': -82.30293273925781, 'logps/rejected': -94.2509536743164, 'logps/ref_chosen': -82.36649322509766, 'logps/ref_rejected': -94.26461791992188, 'logits/chosen': -1.4052127599716187, 'logits/rejected': -0.392024964094162, 'kl/p_epsilon_steps': 0.484375, 'kl/n_epsilon_steps': 0.5, 'kl/beta': 0.10001686960458755, 'kl/avg_steps': -0.015625, 'epoch': 0.02}
2%|█▊ | 11/681 [00:33<33:58, 3.04s/it] 2%|██ | 12/681 [00:36<34:14, 3.07s/it] {'loss': 1.3785, 'grad_norm': 39.88624572753906, 'learning_rate': 7.971014492753623e-08, 'rewards/chosen': 0.003762049600481987, 'rewards/rejected': -0.0046038273721933365, 'rewards/accuracies': 0.578125, 'rewards/margins': 0.008365876972675323, 'logps/chosen': -99.06604766845703, 'logps/rejected': -110.31909942626953, 'logps/ref_chosen': -99.10549926757812, 'logps/ref_rejected': -110.27140808105469, 'logits/chosen': -0.7849316596984863, 'logits/rejected': -0.267448753118515, 'kl/p_epsilon_steps': 0.5625, 'kl/n_epsilon_steps': 0.4375, 'kl/beta': 0.1000325009226799, 'kl/avg_steps': 0.125, 'epoch': 0.02}
2%|██ | 12/681 [00:36<34:14, 3.07s/it] 2%|██▏ | 13/681 [00:39<34:50, 3.13s/it] {'loss': 1.3812, 'grad_norm': 41.430633544921875, 'learning_rate': 8.695652173913042e-08, 'rewards/chosen': 0.002787390723824501, 'rewards/rejected': -0.0029441972728818655, 'rewards/accuracies': 0.578125, 'rewards/margins': 0.00573158822953701, 'logps/chosen': -90.52992248535156, 'logps/rejected': -93.72262573242188, 'logps/ref_chosen': -90.55973052978516, 'logps/ref_rejected': -93.69110107421875, 'logits/chosen': -1.5717546939849854, 'logits/rejected': -0.7266464829444885, 'kl/p_epsilon_steps': 0.578125, 'kl/n_epsilon_steps': 0.421875, 'kl/beta': 0.09990761429071426, 'kl/avg_steps': 0.15625, 'epoch': 0.02}
2%|██▏ | 13/681 [00:39<34:50, 3.13s/it] 2%|██▎ | 14/681 [00:42<34:25, 3.10s/it] {'loss': 1.3898, 'grad_norm': 35.816402435302734, 'learning_rate': 9.420289855072464e-08, 'rewards/chosen': -0.003366068471223116, 'rewards/rejected': -0.00033931387588381767, 'rewards/accuracies': 0.515625, 'rewards/margins': -0.0030267564579844475, 'logps/chosen': -99.85889434814453, 'logps/rejected': -108.9466552734375, 'logps/ref_chosen': -99.82717895507812, 'logps/ref_rejected': -108.94200134277344, 'logits/chosen': -0.5540125370025635, 'logits/rejected': -0.28503644466400146, 'kl/p_epsilon_steps': 0.515625, 'kl/n_epsilon_steps': 0.484375, 'kl/beta': 0.09975175559520721, 'kl/avg_steps': 0.03125, 'epoch': 0.02}
2%|██▎ | 14/681 [00:43<34:25, 3.10s/it] 2%|██▌ | 15/681 [00:46<35:53, 3.23s/it] {'loss': 1.3795, 'grad_norm': 31.496644973754883, 'learning_rate': 1.0144927536231885e-07, 'rewards/chosen': 0.004188378341495991, 'rewards/rejected': -0.0030877136159688234, 'rewards/accuracies': 0.4375, 'rewards/margins': 0.007276091258972883, 'logps/chosen': -78.86597442626953, 'logps/rejected': -90.09466552734375, 'logps/ref_chosen': -78.90997314453125, 'logps/ref_rejected': -90.06234741210938, 'logits/chosen': -0.8750624060630798, 'logits/rejected': -0.2879735827445984, 'kl/p_epsilon_steps': 0.40625, 'kl/n_epsilon_steps': 0.59375, 'kl/beta': 0.09972058981657028, 'kl/avg_steps': -0.1875, 'epoch': 0.02}
2%|██▌ | 15/681 [00:46<35:53, 3.23s/it] 2%|██▋ | 16/681 [00:49<34:46, 3.14s/it] {'loss': 1.3824, 'grad_norm': 34.249393463134766, 'learning_rate': 1.0869565217391303e-07, 'rewards/chosen': 0.0043681650422513485, 'rewards/rejected': -0.0002953286748379469, 'rewards/accuracies': 0.578125, 'rewards/margins': 0.004663495346903801, 'logps/chosen': -97.3776626586914, 'logps/rejected': -90.60440826416016, 'logps/ref_chosen': -97.42327880859375, 'logps/ref_rejected': -90.59945678710938, 'logits/chosen': -0.9704724550247192, 'logits/rejected': -0.0901266559958458, 'kl/p_epsilon_steps': 0.578125, 'kl/n_epsilon_steps': 0.421875, 'kl/beta': 0.09990791976451874, 'kl/avg_steps': 0.15625, 'epoch': 0.02}
2%|██▋ | 16/681 [00:49<34:46, 3.14s/it] 2%|██▊ | 17/681 [00:52<34:06, 3.08s/it] {'loss': 1.3875, 'grad_norm': 35.82853698730469, 'learning_rate': 1.1594202898550725e-07, 'rewards/chosen': -0.0006631199503317475, 'rewards/rejected': -8.023856207728386e-05, 'rewards/accuracies': 0.515625, 'rewards/margins': -0.0005828813882544637, 'logps/chosen': -104.36908721923828, 'logps/rejected': -90.47051239013672, 'logps/ref_chosen': -104.36431121826172, 'logps/ref_rejected': -90.46772766113281, 'logits/chosen': -1.0136702060699463, 'logits/rejected': -0.5759009122848511, 'kl/p_epsilon_steps': 0.515625, 'kl/n_epsilon_steps': 0.484375, 'kl/beta': 0.09975205361843109, 'kl/avg_steps': 0.03125, 'epoch': 0.02}
2%|██▊ | 17/681 [00:52<34:06, 3.08s/it] 3%|███ | 18/681 [00:55<33:32, 3.04s/it] {'loss': 1.383, 'grad_norm': 41.663047790527344, 'learning_rate': 1.2318840579710146e-07, 'rewards/chosen': 0.0020006708800792694, 'rewards/rejected': -0.001984333386644721, 'rewards/accuracies': 0.5625, 'rewards/margins': 0.003985004499554634, 'logps/chosen': -87.06967163085938, 'logps/rejected': -81.87223052978516, 'logps/ref_chosen': -87.09195709228516, 'logps/ref_rejected': -81.85072326660156, 'logits/chosen': -1.795450210571289, 'logits/rejected': -0.8455245494842529, 'kl/p_epsilon_steps': 0.53125, 'kl/n_epsilon_steps': 0.46875, 'kl/beta': 0.09972089529037476, 'kl/avg_steps': 0.0625, 'epoch': 0.03}
3%|███ | 18/681 [00:55<33:32, 3.04s/it] 3%|███▏ | 19/681 [00:58<34:43, 3.15s/it] {'loss': 1.3826, 'grad_norm': 31.180570602416992, 'learning_rate': 1.3043478260869563e-07, 'rewards/chosen': 0.0018369832541793585, 'rewards/rejected': -0.0022847556974738836, 'rewards/accuracies': 0.546875, 'rewards/margins': 0.004121738485991955, 'logps/chosen': -105.85330963134766, 'logps/rejected': -96.95452880859375, 'logps/ref_chosen': -105.87354278564453, 'logps/ref_rejected': -96.93023681640625, 'logits/chosen': -1.0867879390716553, 'logits/rejected': -0.03352098539471626, 'kl/p_epsilon_steps': 0.53125, 'kl/n_epsilon_steps': 0.46875, 'kl/beta': 0.09965860843658447, 'kl/avg_steps': 0.0625, 'epoch': 0.03}
3%|███▏ | 19/681 [00:58<34:43, 3.15s/it] 3%|███▎ | 20/681 [01:01<34:23, 3.12s/it] {'loss': 1.3791, 'grad_norm': 32.30035400390625, 'learning_rate': 1.3768115942028986e-07, 'rewards/chosen': 0.003147183684632182, 'rewards/rejected': -0.004679815378040075, 'rewards/accuracies': 0.546875, 'rewards/margins': 0.007826998829841614, 'logps/chosen': -90.72392272949219, 'logps/rejected': -85.96060180664062, 'logps/ref_chosen': -90.75811767578125, 'logps/ref_rejected': -85.91232299804688, 'logits/chosen': -1.2128328084945679, 'logits/rejected': -0.32826870679855347, 'kl/p_epsilon_steps': 0.546875, 'kl/n_epsilon_steps': 0.453125, 'kl/beta': 0.09959635883569717, 'kl/avg_steps': 0.09375, 'epoch': 0.03}
3%|███▎ | 20/681 [01:01<34:23, 3.12s/it] 3%|███▌ | 21/681 [01:05<35:01, 3.18s/it] {'loss': 1.3808, 'grad_norm': 31.352293014526367, 'learning_rate': 1.4492753623188405e-07, 'rewards/chosen': 0.008116335608065128, 'rewards/rejected': 0.002345857210457325, 'rewards/accuracies': 0.59375, 'rewards/margins': 0.005770478397607803, 'logps/chosen': -80.2506332397461, 'logps/rejected': -83.91175842285156, 'logps/ref_chosen': -80.33346557617188, 'logps/ref_rejected': -83.9337387084961, 'logits/chosen': -0.9726300239562988, 'logits/rejected': -0.411948561668396, 'kl/p_epsilon_steps': 0.5625, 'kl/n_epsilon_steps': 0.4375, 'kl/beta': 0.09950307011604309, 'kl/avg_steps': 0.125, 'epoch': 0.03}
3%|███▌ | 21/681 [01:05<35:01, 3.18s/it] 3%|███▋ | 22/681 [01:08<35:30, 3.23s/it] {'loss': 1.3819, 'grad_norm': 42.61378860473633, 'learning_rate': 1.5217391304347825e-07, 'rewards/chosen': 0.0037977853789925575, 'rewards/rejected': -0.0010757955024018884, 'rewards/accuracies': 0.546875, 'rewards/margins': 0.004873580764979124, 'logps/chosen': -95.35507202148438, 'logps/rejected': -103.48601531982422, 'logps/ref_chosen': -95.39530181884766, 'logps/ref_rejected': -103.47351837158203, 'logits/chosen': -0.9675413370132446, 'logits/rejected': -0.6590346693992615, 'kl/p_epsilon_steps': 0.546875, 'kl/n_epsilon_steps': 0.453125, 'kl/beta': 0.09937884658575058, 'kl/avg_steps': 0.09375, 'epoch': 0.03}
3%|███▋ | 22/681 [01:08<35:30, 3.23s/it] 3%|███▊ | 23/681 [01:11<35:40, 3.25s/it] {'loss': 1.3916, 'grad_norm': 31.29548454284668, 'learning_rate': 1.5942028985507245e-07, 'rewards/chosen': 0.00023264711489900947, 'rewards/rejected': 0.005144191440194845, 'rewards/accuracies': 0.4375, 'rewards/margins': -0.004911544732749462, 'logps/chosen': -90.63298034667969, 'logps/rejected': -86.54367065429688, 'logps/ref_chosen': -90.63751220703125, 'logps/ref_rejected': -86.59425354003906, 'logits/chosen': -1.156263828277588, 'logits/rejected': -0.3607323169708252, 'kl/p_epsilon_steps': 0.453125, 'kl/n_epsilon_steps': 0.546875, 'kl/beta': 0.09928576648235321, 'kl/avg_steps': -0.09375, 'epoch': 0.03}
3%|███▊ | 23/681 [01:11<35:40, 3.25s/it] 4%|████ | 24/681 [01:14<35:23, 3.23s/it] {'loss': 1.395, 'grad_norm': 44.2933464050293, 'learning_rate': 1.6666666666666665e-07, 'rewards/chosen': -0.006438862532377243, 'rewards/rejected': 0.0017685755155980587, 'rewards/accuracies': 0.453125, 'rewards/margins': -0.008207438513636589, 'logps/chosen': -69.98039245605469, 'logps/rejected': -106.59368133544922, 'logps/ref_chosen': -69.91728973388672, 'logps/ref_rejected': -106.60990142822266, 'logits/chosen': -0.8095067143440247, 'logits/rejected': -0.5244461297988892, 'kl/p_epsilon_steps': 0.484375, 'kl/n_epsilon_steps': 0.515625, 'kl/beta': 0.09937893599271774, 'kl/avg_steps': -0.03125, 'epoch': 0.04}
4%|████ | 24/681 [01:14<35:23, 3.23s/it] 4%|████▏ | 25/681 [01:18<35:05, 3.21s/it] {'loss': 1.3773, 'grad_norm': 36.52031326293945, 'learning_rate': 1.7391304347826085e-07, 'rewards/chosen': -0.0005520773120224476, 'rewards/rejected': -0.010130547918379307, 'rewards/accuracies': 0.65625, 'rewards/margins': 0.009578470140695572, 'logps/chosen': -80.82945251464844, 'logps/rejected': -96.06101989746094, 'logps/ref_chosen': -80.82548522949219, 'logps/ref_rejected': -95.95710754394531, 'logits/chosen': -1.2055686712265015, 'logits/rejected': -0.4221525490283966, 'kl/p_epsilon_steps': 0.625, 'kl/n_epsilon_steps': 0.375, 'kl/beta': 0.09941000491380692, 'kl/avg_steps': 0.25, 'epoch': 0.04}
4%|████▏ | 25/681 [01:18<35:05, 3.21s/it] 4%|████▎ | 26/681 [01:20<33:23, 3.06s/it] {'loss': 1.3825, 'grad_norm': 40.3887939453125, 'learning_rate': 1.8115942028985507e-07, 'rewards/chosen': 0.00035169871989637613, 'rewards/rejected': -0.0038559872191399336, 'rewards/accuracies': 0.53125, 'rewards/margins': 0.004207686521112919, 'logps/chosen': -88.89604949951172, 'logps/rejected': -109.86863708496094, 'logps/ref_chosen': -88.90116882324219, 'logps/ref_rejected': -109.82818603515625, 'logits/chosen': -1.4493508338928223, 'logits/rejected': -0.5392433404922485, 'kl/p_epsilon_steps': 0.578125, 'kl/n_epsilon_steps': 0.421875, 'kl/beta': 0.09916209429502487, 'kl/avg_steps': 0.15625, 'epoch': 0.04}
4%|████▎ | 26/681 [01:20<33:23, 3.06s/it] 4%|████▌ | 27/681 [01:23<32:57, 3.02s/it] {'loss': 1.3714, 'grad_norm': 43.48580551147461, 'learning_rate': 1.8840579710144927e-07, 'rewards/chosen': 0.0019225336145609617, 'rewards/rejected': -0.013401055708527565, 'rewards/accuracies': 0.65625, 'rewards/margins': 0.01532358955591917, 'logps/chosen': -77.5752182006836, 'logps/rejected': -104.07573699951172, 'logps/ref_chosen': -77.59600830078125, 'logps/ref_rejected': -103.93850708007812, 'logits/chosen': -1.6640090942382812, 'logits/rejected': -0.6294593811035156, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.09900739789009094, 'kl/avg_steps': 0.28125, 'epoch': 0.04}
4%|████▌ | 27/681 [01:23<32:57, 3.02s/it] 4%|████▋ | 28/681 [01:26<32:54, 3.02s/it] {'loss': 1.3723, 'grad_norm': 35.73523712158203, 'learning_rate': 1.9565217391304347e-07, 'rewards/chosen': 0.004990905057638884, 'rewards/rejected': -0.00980809610337019, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.014799000695347786, 'logps/chosen': -102.17597961425781, 'logps/rejected': -97.06103515625, 'logps/ref_chosen': -102.22856140136719, 'logps/ref_rejected': -96.9594955444336, 'logits/chosen': -0.956201434135437, 'logits/rejected': -0.39606085419654846, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.09872972220182419, 'kl/avg_steps': 0.4375, 'epoch': 0.04}
4%|████▋ | 28/681 [01:26<32:54, 3.02s/it] 4%|████▊ | 29/681 [01:29<31:39, 2.91s/it] {'loss': 1.3704, 'grad_norm': 41.57979965209961, 'learning_rate': 2.028985507246377e-07, 'rewards/chosen': 0.0016324977623298764, 'rewards/rejected': -0.014854353852570057, 'rewards/accuracies': 0.609375, 'rewards/margins': 0.01648685149848461, 'logps/chosen': -88.62876892089844, 'logps/rejected': -103.11316680908203, 'logps/ref_chosen': -88.64704895019531, 'logps/ref_rejected': -102.96011352539062, 'logits/chosen': -1.4259010553359985, 'logits/rejected': -0.6437522172927856, 'kl/p_epsilon_steps': 0.578125, 'kl/n_epsilon_steps': 0.421875, 'kl/beta': 0.09829965978860855, 'kl/avg_steps': 0.15625, 'epoch': 0.04}
4%|████▊ | 29/681 [01:29<31:39, 2.91s/it] 4%|█████ | 30/681 [01:32<33:42, 3.11s/it] {'loss': 1.3817, 'grad_norm': 38.55412292480469, 'learning_rate': 2.1014492753623187e-07, 'rewards/chosen': 6.297486834228039e-06, 'rewards/rejected': -0.005099880509078503, 'rewards/accuracies': 0.578125, 'rewards/margins': 0.0051061781123280525, 'logps/chosen': -88.3867416381836, 'logps/rejected': -102.37272644042969, 'logps/ref_chosen': -88.38838958740234, 'logps/ref_rejected': -102.31889343261719, 'logits/chosen': -0.9132494926452637, 'logits/rejected': -0.49612629413604736, 'kl/p_epsilon_steps': 0.53125, 'kl/n_epsilon_steps': 0.46875, 'kl/beta': 0.09814630448818207, 'kl/avg_steps': 0.0625, 'epoch': 0.04}
4%|█████ | 30/681 [01:33<33:42, 3.11s/it] 5%|█████▏ | 31/681 [01:36<34:06, 3.15s/it] {'loss': 1.3782, 'grad_norm': 30.63753890991211, 'learning_rate': 2.1739130434782607e-07, 'rewards/chosen': -0.0009215597528964281, 'rewards/rejected': -0.009414611384272575, 'rewards/accuracies': 0.578125, 'rewards/margins': 0.008493051864206791, 'logps/chosen': -101.13359832763672, 'logps/rejected': -79.95623779296875, 'logps/ref_chosen': -101.12565612792969, 'logps/ref_rejected': -79.85842895507812, 'logits/chosen': -0.8628894090652466, 'logits/rejected': -0.3569292426109314, 'kl/p_epsilon_steps': 0.546875, 'kl/n_epsilon_steps': 0.453125, 'kl/beta': 0.09808500111103058, 'kl/avg_steps': 0.09375, 'epoch': 0.05}
5%|█████▏ | 31/681 [01:36<34:06, 3.15s/it] 5%|█████▎ | 32/681 [01:39<34:00, 3.14s/it] {'loss': 1.3789, 'grad_norm': 34.43489456176758, 'learning_rate': 2.2463768115942027e-07, 'rewards/chosen': -0.006817132234573364, 'rewards/rejected': -0.015008427202701569, 'rewards/accuracies': 0.5625, 'rewards/margins': 0.008191294968128204, 'logps/chosen': -96.68499755859375, 'logps/rejected': -96.38025665283203, 'logps/ref_chosen': -96.61703491210938, 'logps/ref_rejected': -96.224365234375, 'logits/chosen': -0.9892777800559998, 'logits/rejected': -0.34080418944358826, 'kl/p_epsilon_steps': 0.5, 'kl/n_epsilon_steps': 0.5, 'kl/beta': 0.09799313545227051, 'kl/avg_steps': 0.0, 'epoch': 0.05}
5%|█████▎ | 32/681 [01:39<34:00, 3.14s/it] 5%|█████▌ | 33/681 [01:42<33:01, 3.06s/it] {'loss': 1.376, 'grad_norm': 32.965362548828125, 'learning_rate': 2.318840579710145e-07, 'rewards/chosen': 0.0003811400383710861, 'rewards/rejected': -0.01046024076640606, 'rewards/accuracies': 0.578125, 'rewards/margins': 0.010841380804777145, 'logps/chosen': -81.51568603515625, 'logps/rejected': -93.91485595703125, 'logps/ref_chosen': -81.5210189819336, 'logps/ref_rejected': -93.80595397949219, 'logits/chosen': -1.2965284585952759, 'logits/rejected': -0.3348070979118347, 'kl/p_epsilon_steps': 0.5625, 'kl/n_epsilon_steps': 0.421875, 'kl/beta': 0.09799313545227051, 'kl/avg_steps': 0.140625, 'epoch': 0.05}
5%|█████▌ | 33/681 [01:42<33:01, 3.06s/it] 5%|█████▋ | 34/681 [01:45<33:12, 3.08s/it] {'loss': 1.3705, 'grad_norm': 39.99357604980469, 'learning_rate': 2.391304347826087e-07, 'rewards/chosen': 0.004645414184778929, 'rewards/rejected': -0.011928501538932323, 'rewards/accuracies': 0.65625, 'rewards/margins': 0.016573915258049965, 'logps/chosen': -77.15280151367188, 'logps/rejected': -106.84297943115234, 'logps/ref_chosen': -77.20204162597656, 'logps/ref_rejected': -106.71875762939453, 'logits/chosen': -1.2087818384170532, 'logits/rejected': -0.19416889548301697, 'kl/p_epsilon_steps': 0.609375, 'kl/n_epsilon_steps': 0.375, 'kl/beta': 0.09785552322864532, 'kl/avg_steps': 0.234375, 'epoch': 0.05}
5%|█████▋ | 34/681 [01:45<33:12, 3.08s/it] 5%|█████▊ | 35/681 [01:48<32:53, 3.05s/it] {'loss': 1.3678, 'grad_norm': 41.544822692871094, 'learning_rate': 2.463768115942029e-07, 'rewards/chosen': 0.0007331545930355787, 'rewards/rejected': -0.018401240929961205, 'rewards/accuracies': 0.65625, 'rewards/margins': 0.019134394824504852, 'logps/chosen': -77.5616683959961, 'logps/rejected': -112.379638671875, 'logps/ref_chosen': -77.57035827636719, 'logps/ref_rejected': -112.18855285644531, 'logits/chosen': -1.575798749923706, 'logits/rejected': -0.5509282350540161, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.0976267158985138, 'kl/avg_steps': 0.34375, 'epoch': 0.05}
5%|█████▊ | 35/681 [01:48<32:53, 3.05s/it] 5%|██████ | 36/681 [01:51<33:03, 3.07s/it] {'loss': 1.3662, 'grad_norm': 31.387723922729492, 'learning_rate': 2.536231884057971e-07, 'rewards/chosen': -0.0043023210018873215, 'rewards/rejected': -0.025263587012887, 'rewards/accuracies': 0.625, 'rewards/margins': 0.02096126601099968, 'logps/chosen': -83.33061218261719, 'logps/rejected': -92.01959228515625, 'logps/ref_chosen': -83.28824615478516, 'logps/ref_rejected': -91.75741577148438, 'logits/chosen': -1.432613730430603, 'logits/rejected': -0.642814040184021, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.09729227423667908, 'kl/avg_steps': 0.28125, 'epoch': 0.05}
5%|██████ | 36/681 [01:51<33:03, 3.07s/it] 5%|██████▏ | 37/681 [01:54<33:52, 3.16s/it] {'loss': 1.368, 'grad_norm': 34.76736831665039, 'learning_rate': 2.6086956521739126e-07, 'rewards/chosen': -0.005295893643051386, 'rewards/rejected': -0.0246497243642807, 'rewards/accuracies': 0.53125, 'rewards/margins': 0.01935383304953575, 'logps/chosen': -94.82408142089844, 'logps/rejected': -85.97396087646484, 'logps/ref_chosen': -94.77108764648438, 'logps/ref_rejected': -85.7172622680664, 'logits/chosen': -0.9882210493087769, 'logits/rejected': -0.7192566990852356, 'kl/p_epsilon_steps': 0.484375, 'kl/n_epsilon_steps': 0.515625, 'kl/beta': 0.09701940417289734, 'kl/avg_steps': -0.03125, 'epoch': 0.05}
5%|██████▏ | 37/681 [01:54<33:52, 3.16s/it] 6%|██████▎ | 38/681 [01:57<33:08, 3.09s/it] {'loss': 1.3601, 'grad_norm': 35.520565032958984, 'learning_rate': 2.681159420289855e-07, 'rewards/chosen': 0.0053534312173724174, 'rewards/rejected': -0.02198958396911621, 'rewards/accuracies': 0.65625, 'rewards/margins': 0.027343016117811203, 'logps/chosen': -75.92189025878906, 'logps/rejected': -104.33356475830078, 'logps/ref_chosen': -75.97850799560547, 'logps/ref_rejected': -104.10401916503906, 'logits/chosen': -0.9708235263824463, 'logits/rejected': -0.36027270555496216, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.34375, 'kl/beta': 0.09704973548650742, 'kl/avg_steps': 0.3125, 'epoch': 0.06}
6%|██████▎ | 38/681 [01:57<33:08, 3.09s/it] 6%|██████▌ | 39/681 [02:00<32:48, 3.07s/it] {'loss': 1.3596, 'grad_norm': 34.98324966430664, 'learning_rate': 2.753623188405797e-07, 'rewards/chosen': -0.0006124734645709395, 'rewards/rejected': -0.027974674478173256, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.02736220322549343, 'logps/chosen': -81.19099426269531, 'logps/rejected': -84.48798370361328, 'logps/ref_chosen': -81.18577575683594, 'logps/ref_rejected': -84.1959228515625, 'logits/chosen': -1.2482174634933472, 'logits/rejected': -0.5844467878341675, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.09674739837646484, 'kl/avg_steps': 0.46875, 'epoch': 0.06}
6%|██████▌ | 39/681 [02:00<32:48, 3.07s/it] 6%|██████▋ | 40/681 [02:03<32:23, 3.03s/it] {'loss': 1.363, 'grad_norm': 27.852684020996094, 'learning_rate': 2.8260869565217386e-07, 'rewards/chosen': 0.007215453311800957, 'rewards/rejected': -0.016598014160990715, 'rewards/accuracies': 0.6875, 'rewards/margins': 0.023813467472791672, 'logps/chosen': -83.25595092773438, 'logps/rejected': -80.43058776855469, 'logps/ref_chosen': -83.33256530761719, 'logps/ref_rejected': -80.25591278076172, 'logits/chosen': -0.8650610446929932, 'logits/rejected': -0.20084291696548462, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.09629600495100021, 'kl/avg_steps': 0.40625, 'epoch': 0.06}
6%|██████▋ | 40/681 [02:03<32:23, 3.03s/it] 6%|██████▊ | 41/681 [02:06<32:26, 3.04s/it] {'loss': 1.3656, 'grad_norm': 31.773061752319336, 'learning_rate': 2.898550724637681e-07, 'rewards/chosen': -0.004216345027089119, 'rewards/rejected': -0.026013631373643875, 'rewards/accuracies': 0.671875, 'rewards/margins': 0.021797288209199905, 'logps/chosen': -93.19059753417969, 'logps/rejected': -102.35368347167969, 'logps/ref_chosen': -93.14866638183594, 'logps/ref_rejected': -102.07920837402344, 'logits/chosen': -1.026604413986206, 'logits/rejected': -0.37627846002578735, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.09590639173984528, 'kl/avg_steps': 0.34375, 'epoch': 0.06}
6%|██████▊ | 41/681 [02:06<32:26, 3.04s/it] 6%|███████ | 42/681 [02:09<32:23, 3.04s/it] {'loss': 1.3432, 'grad_norm': 38.71194839477539, 'learning_rate': 2.971014492753623e-07, 'rewards/chosen': 0.002619321458041668, 'rewards/rejected': -0.041931942105293274, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.044551268219947815, 'logps/chosen': -90.67225646972656, 'logps/rejected': -114.30716705322266, 'logps/ref_chosen': -90.70162200927734, 'logps/ref_rejected': -113.8646469116211, 'logits/chosen': -1.2612457275390625, 'logits/rejected': -0.5760804414749146, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.09557784348726273, 'kl/avg_steps': 0.40625, 'epoch': 0.06}
6%|███████ | 42/681 [02:09<32:23, 3.04s/it] 6%|███████▏ | 43/681 [02:12<32:27, 3.05s/it] {'loss': 1.3476, 'grad_norm': 35.74715805053711, 'learning_rate': 3.043478260869565e-07, 'rewards/chosen': 0.002490551210939884, 'rewards/rejected': -0.03793483227491379, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.0404253825545311, 'logps/chosen': -89.61614990234375, 'logps/rejected': -104.35449981689453, 'logps/ref_chosen': -89.64402770996094, 'logps/ref_rejected': -103.95185852050781, 'logits/chosen': -1.2501815557479858, 'logits/rejected': -0.3675554692745209, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.09519112855195999, 'kl/avg_steps': 0.4375, 'epoch': 0.06}
6%|███████▏ | 43/681 [02:12<32:27, 3.05s/it] 6%|███████▎ | 44/681 [02:15<32:17, 3.04s/it] {'loss': 1.3562, 'grad_norm': 32.69614028930664, 'learning_rate': 3.115942028985507e-07, 'rewards/chosen': -0.010416124947369099, 'rewards/rejected': -0.041503991931676865, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.03108786605298519, 'logps/chosen': -81.98731994628906, 'logps/rejected': -113.86365509033203, 'logps/ref_chosen': -81.8783187866211, 'logps/ref_rejected': -113.421630859375, 'logits/chosen': -1.5350993871688843, 'logits/rejected': -0.39478617906570435, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.09477648138999939, 'kl/avg_steps': 0.46875, 'epoch': 0.06}
6%|███████▎ | 44/681 [02:15<32:17, 3.04s/it] 7%|███████▌ | 45/681 [02:19<34:17, 3.24s/it] {'loss': 1.3645, 'grad_norm': 22.90825080871582, 'learning_rate': 3.188405797101449e-07, 'rewards/chosen': -0.009415511973202229, 'rewards/rejected': -0.031987544149160385, 'rewards/accuracies': 0.75, 'rewards/margins': 0.022572031244635582, 'logps/chosen': -77.44337463378906, 'logps/rejected': -84.58972930908203, 'logps/ref_chosen': -77.34459686279297, 'logps/ref_rejected': -84.24774169921875, 'logits/chosen': -0.9012535810470581, 'logits/rejected': -0.24215500056743622, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.09433428943157196, 'kl/avg_steps': 0.40625, 'epoch': 0.07}
7%|███████▌ | 45/681 [02:19<34:17, 3.24s/it] 7%|███████▋ | 46/681 [02:22<34:02, 3.22s/it] {'loss': 1.3528, 'grad_norm': 31.163087844848633, 'learning_rate': 3.260869565217391e-07, 'rewards/chosen': -0.005449555814266205, 'rewards/rejected': -0.039974093437194824, 'rewards/accuracies': 0.75, 'rewards/margins': 0.03452453762292862, 'logps/chosen': -90.39727020263672, 'logps/rejected': -101.10578918457031, 'logps/ref_chosen': -90.3408203125, 'logps/ref_rejected': -100.676513671875, 'logits/chosen': -1.383570909500122, 'logits/rejected': -0.5739269256591797, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.09395260363817215, 'kl/avg_steps': 0.546875, 'epoch': 0.07}
7%|███████▋ | 46/681 [02:22<34:02, 3.22s/it] 7%|███████▊ | 47/681 [02:25<33:51, 3.20s/it] {'loss': 1.3497, 'grad_norm': 31.565263748168945, 'learning_rate': 3.333333333333333e-07, 'rewards/chosen': -0.0006828177720308304, 'rewards/rejected': -0.03886501491069794, 'rewards/accuracies': 0.671875, 'rewards/margins': 0.03818219527602196, 'logps/chosen': -104.41678619384766, 'logps/rejected': -101.67523193359375, 'logps/ref_chosen': -104.41130065917969, 'logps/ref_rejected': -101.25489807128906, 'logits/chosen': -0.9797345995903015, 'logits/rejected': -0.36259299516677856, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.09344159811735153, 'kl/avg_steps': 0.375, 'epoch': 0.07}
7%|███████▊ | 47/681 [02:26<33:51, 3.20s/it] 7%|████████ | 48/681 [02:29<33:37, 3.19s/it] {'loss': 1.3375, 'grad_norm': 35.98372268676758, 'learning_rate': 3.4057971014492755e-07, 'rewards/chosen': -0.004128246568143368, 'rewards/rejected': -0.05539228022098541, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.05126403272151947, 'logps/chosen': -91.1424560546875, 'logps/rejected': -95.68135070800781, 'logps/ref_chosen': -91.10027313232422, 'logps/ref_rejected': -95.08057403564453, 'logits/chosen': -1.2902014255523682, 'logits/rejected': -0.7823787331581116, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.09309250116348267, 'kl/avg_steps': 0.53125, 'epoch': 0.07}
7%|████████ | 48/681 [02:29<33:37, 3.19s/it] 7%|████████▏ | 49/681 [02:32<33:08, 3.15s/it] {'loss': 1.3199, 'grad_norm': 36.75701904296875, 'learning_rate': 3.478260869565217e-07, 'rewards/chosen': 0.007629199419170618, 'rewards/rejected': -0.06168051436543465, 'rewards/accuracies': 0.75, 'rewards/margins': 0.06930971145629883, 'logps/chosen': -92.91861724853516, 'logps/rejected': -92.42121887207031, 'logps/ref_chosen': -93.00367736816406, 'logps/ref_rejected': -91.74899291992188, 'logits/chosen': -2.277641773223877, 'logits/rejected': -0.8717272281646729, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.09260056167840958, 'kl/avg_steps': 0.46875, 'epoch': 0.07}
7%|████████▏ | 49/681 [02:32<33:08, 3.15s/it] 7%|████████▎ | 50/681 [02:35<32:25, 3.08s/it] {'loss': 1.3264, 'grad_norm': 30.533233642578125, 'learning_rate': 3.5507246376811595e-07, 'rewards/chosen': -0.007727333344519138, 'rewards/rejected': -0.07026051729917526, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.0625331848859787, 'logps/chosen': -94.70893859863281, 'logps/rejected': -104.34310913085938, 'logps/ref_chosen': -94.62681579589844, 'logps/ref_rejected': -103.57435607910156, 'logits/chosen': -1.142214059829712, 'logits/rejected': -0.34731245040893555, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.09216851741075516, 'kl/avg_steps': 0.59375, 'epoch': 0.07}
7%|████████▎ | 50/681 [02:35<32:25, 3.08s/it] 7%|████████▌ | 51/681 [02:38<32:35, 3.10s/it] {'loss': 1.3355, 'grad_norm': 27.474159240722656, 'learning_rate': 3.6231884057971015e-07, 'rewards/chosen': -0.007823294959962368, 'rewards/rejected': -0.06161422282457352, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.053790926933288574, 'logps/chosen': -87.59027099609375, 'logps/rejected': -84.15083312988281, 'logps/ref_chosen': -87.50727844238281, 'logps/ref_rejected': -83.47235870361328, 'logits/chosen': -1.5295517444610596, 'logits/rejected': -1.121992826461792, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.09162449836730957, 'kl/avg_steps': 0.53125, 'epoch': 0.07}
7%|████████▌ | 51/681 [02:38<32:35, 3.10s/it] 8%|████████▋ | 52/681 [02:41<32:37, 3.11s/it] {'loss': 1.3116, 'grad_norm': 29.224390029907227, 'learning_rate': 3.695652173913043e-07, 'rewards/chosen': -0.004274226725101471, 'rewards/rejected': -0.08229520916938782, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.07802098244428635, 'logps/chosen': -90.67510986328125, 'logps/rejected': -87.94956970214844, 'logps/ref_chosen': -90.63026428222656, 'logps/ref_rejected': -87.0390625, 'logits/chosen': -1.5085875988006592, 'logits/rejected': -0.8081971406936646, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.09114031493663788, 'kl/avg_steps': 0.625, 'epoch': 0.08}
8%|████████▋ | 52/681 [02:41<32:37, 3.11s/it] 8%|████████▊ | 53/681 [02:44<33:11, 3.17s/it] {'loss': 1.2987, 'grad_norm': 35.840335845947266, 'learning_rate': 3.7681159420289855e-07, 'rewards/chosen': -0.004804985597729683, 'rewards/rejected': -0.09881868213415146, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.09401369839906693, 'logps/chosen': -81.63391876220703, 'logps/rejected': -96.76312255859375, 'logps/ref_chosen': -81.58306884765625, 'logps/ref_rejected': -95.66152954101562, 'logits/chosen': -1.548392415046692, 'logits/rejected': -1.1408261060714722, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0905742272734642, 'kl/avg_steps': 0.59375, 'epoch': 0.08}
8%|████████▊ | 53/681 [02:44<33:11, 3.17s/it] 8%|█████████ | 54/681 [02:47<32:05, 3.07s/it] {'loss': 1.2939, 'grad_norm': 35.994388580322266, 'learning_rate': 3.8405797101449274e-07, 'rewards/chosen': -0.002394177485257387, 'rewards/rejected': -0.10055913031101227, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.09816494584083557, 'logps/chosen': -88.93460083007812, 'logps/rejected': -100.24466705322266, 'logps/ref_chosen': -88.91016387939453, 'logps/ref_rejected': -99.1175537109375, 'logits/chosen': -1.3138582706451416, 'logits/rejected': -0.6569335460662842, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.09003961831331253, 'kl/avg_steps': 0.5, 'epoch': 0.08}
8%|█████████ | 54/681 [02:47<32:05, 3.07s/it] 8%|█████████▏ | 55/681 [02:50<30:49, 2.95s/it] {'loss': 1.2933, 'grad_norm': 31.913665771484375, 'learning_rate': 3.9130434782608694e-07, 'rewards/chosen': 0.010466434992849827, 'rewards/rejected': -0.0895768478512764, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.10004328191280365, 'logps/chosen': -92.33493041992188, 'logps/rejected': -93.97943115234375, 'logps/ref_chosen': -92.45592498779297, 'logps/ref_rejected': -92.97093963623047, 'logits/chosen': -1.6863343715667725, 'logits/rejected': -0.6388131976127625, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.08959165960550308, 'kl/avg_steps': 0.46875, 'epoch': 0.08}
8%|█████████▏ | 55/681 [02:50<30:49, 2.95s/it] 8%|█████████▎ | 56/681 [02:53<31:22, 3.01s/it] {'loss': 1.3033, 'grad_norm': 29.783920288085938, 'learning_rate': 3.9855072463768114e-07, 'rewards/chosen': -0.008556234650313854, 'rewards/rejected': -0.09898769855499268, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.09043145924806595, 'logps/chosen': -87.32936096191406, 'logps/rejected': -101.76547241210938, 'logps/ref_chosen': -87.23665618896484, 'logps/ref_rejected': -100.64553833007812, 'logits/chosen': -1.9710514545440674, 'logits/rejected': -0.6645182371139526, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.08917365968227386, 'kl/avg_steps': 0.5625, 'epoch': 0.08}
8%|█████████▎ | 56/681 [02:53<31:22, 3.01s/it] 8%|█████████▌ | 57/681 [02:56<31:09, 3.00s/it] {'loss': 1.2927, 'grad_norm': 30.23967933654785, 'learning_rate': 4.057971014492754e-07, 'rewards/chosen': -0.003269542008638382, 'rewards/rejected': -0.10384444147348404, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.10057489573955536, 'logps/chosen': -98.1844253540039, 'logps/rejected': -102.23455047607422, 'logps/ref_chosen': -98.15074157714844, 'logps/ref_rejected': -101.05284118652344, 'logits/chosen': -1.7121399641036987, 'logits/rejected': -0.9029750227928162, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.08867485821247101, 'kl/avg_steps': 0.5, 'epoch': 0.08}
8%|█████████▌ | 57/681 [02:56<31:09, 3.00s/it] 9%|█████████▋ | 58/681 [02:59<31:34, 3.04s/it] {'loss': 1.2623, 'grad_norm': 33.85087966918945, 'learning_rate': 4.1304347826086954e-07, 'rewards/chosen': 0.02414235845208168, 'rewards/rejected': -0.11232372373342514, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.13646608591079712, 'logps/chosen': -99.30267333984375, 'logps/rejected': -92.50682067871094, 'logps/ref_chosen': -99.58097076416016, 'logps/ref_rejected': -91.22227478027344, 'logits/chosen': -1.996552586555481, 'logits/rejected': -1.058452844619751, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.08823369443416595, 'kl/avg_steps': 0.65625, 'epoch': 0.09}
9%|█████████▋ | 58/681 [02:59<31:34, 3.04s/it] 9%|█████████▉ | 59/681 [03:02<31:30, 3.04s/it] {'loss': 1.2934, 'grad_norm': 30.440584182739258, 'learning_rate': 4.2028985507246374e-07, 'rewards/chosen': -0.0023305192589759827, 'rewards/rejected': -0.10135940462350845, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.09902888536453247, 'logps/chosen': -89.82688903808594, 'logps/rejected': -95.44361877441406, 'logps/ref_chosen': -89.80232238769531, 'logps/ref_rejected': -94.27667236328125, 'logits/chosen': -1.5328807830810547, 'logits/rejected': -0.9605180621147156, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.08765843510627747, 'kl/avg_steps': 0.625, 'epoch': 0.09}
9%|█████████▉ | 59/681 [03:02<31:30, 3.04s/it] 9%|██████████ | 60/681 [03:05<31:35, 3.05s/it] {'loss': 1.2904, 'grad_norm': 25.22243309020996, 'learning_rate': 4.2753623188405794e-07, 'rewards/chosen': 0.002980598248541355, 'rewards/rejected': -0.0993606448173523, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.10234124958515167, 'logps/chosen': -95.11837768554688, 'logps/rejected': -92.52310180664062, 'logps/ref_chosen': -95.15571594238281, 'logps/ref_rejected': -91.3724365234375, 'logits/chosen': -1.692917823791504, 'logits/rejected': -0.8258851170539856, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.08711396902799606, 'kl/avg_steps': 0.40625, 'epoch': 0.09}
9%|██████████ | 60/681 [03:05<31:35, 3.05s/it] 9%|██████████▏ | 61/681 [03:08<32:13, 3.12s/it] {'loss': 1.2933, 'grad_norm': 25.07730484008789, 'learning_rate': 4.3478260869565214e-07, 'rewards/chosen': -0.014695134945213795, 'rewards/rejected': -0.11627216637134552, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.101577028632164, 'logps/chosen': -85.89845275878906, 'logps/rejected': -99.31712341308594, 'logps/ref_chosen': -85.73231506347656, 'logps/ref_rejected': -97.96575927734375, 'logits/chosen': -1.4720783233642578, 'logits/rejected': -0.7469910979270935, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.08676150441169739, 'kl/avg_steps': 0.53125, 'epoch': 0.09}
9%|██████████▏ | 61/681 [03:08<32:13, 3.12s/it] 9%|██████████▍ | 62/681 [03:12<32:30, 3.15s/it] {'loss': 1.2896, 'grad_norm': 24.972980499267578, 'learning_rate': 4.420289855072464e-07, 'rewards/chosen': -0.008462773635983467, 'rewards/rejected': -0.11501070111989975, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.10654792189598083, 'logps/chosen': -81.72990417480469, 'logps/rejected': -85.38259887695312, 'logps/ref_chosen': -81.63538360595703, 'logps/ref_rejected': -84.03831481933594, 'logits/chosen': -1.7914103269577026, 'logits/rejected': -0.8692626953125, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.08630301803350449, 'kl/avg_steps': 0.65625, 'epoch': 0.09}
9%|██████████▍ | 62/681 [03:12<32:30, 3.15s/it] 9%|██████████▌ | 63/681 [03:15<31:50, 3.09s/it] {'loss': 1.2686, 'grad_norm': 30.612600326538086, 'learning_rate': 4.4927536231884053e-07, 'rewards/chosen': -0.002559835556894541, 'rewards/rejected': -0.12838411331176758, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.12582427263259888, 'logps/chosen': -103.65047454833984, 'logps/rejected': -104.9129638671875, 'logps/ref_chosen': -103.62405395507812, 'logps/ref_rejected': -103.40303039550781, 'logits/chosen': -1.5954476594924927, 'logits/rejected': -0.8784996271133423, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.0857403501868248, 'kl/avg_steps': 0.625, 'epoch': 0.09}
9%|██████████▌ | 63/681 [03:15<31:50, 3.09s/it] 9%|██████████▋ | 64/681 [03:18<31:25, 3.06s/it] {'loss': 1.2612, 'grad_norm': 30.650426864624023, 'learning_rate': 4.5652173913043473e-07, 'rewards/chosen': -0.014812503941357136, 'rewards/rejected': -0.15303777158260345, 'rewards/accuracies': 0.75, 'rewards/margins': 0.1382252722978592, 'logps/chosen': -87.17109680175781, 'logps/rejected': -102.39601135253906, 'logps/ref_chosen': -87.0015869140625, 'logps/ref_rejected': -100.5854721069336, 'logits/chosen': -1.671600580215454, 'logits/rejected': -0.9036776423454285, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.08520779758691788, 'kl/avg_steps': 0.5, 'epoch': 0.09}
9%|██████████▋ | 64/681 [03:18<31:25, 3.06s/it] 10%|██████████▉ | 65/681 [03:21<31:24, 3.06s/it] {'loss': 1.2492, 'grad_norm': 33.42892837524414, 'learning_rate': 4.63768115942029e-07, 'rewards/chosen': -0.006513871252536774, 'rewards/rejected': -0.15611997246742249, 'rewards/accuracies': 0.875, 'rewards/margins': 0.14960609376430511, 'logps/chosen': -91.29652404785156, 'logps/rejected': -117.1944351196289, 'logps/ref_chosen': -91.22191619873047, 'logps/ref_rejected': -115.33553314208984, 'logits/chosen': -1.976373314857483, 'logits/rejected': -1.0799182653427124, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.0847838819026947, 'kl/avg_steps': 0.75, 'epoch': 0.1}
10%|██████████▉ | 65/681 [03:21<31:24, 3.06s/it] 10%|███████████ | 66/681 [03:24<31:31, 3.08s/it] {'loss': 1.2937, 'grad_norm': 22.01787567138672, 'learning_rate': 4.7101449275362313e-07, 'rewards/chosen': -0.026944037526845932, 'rewards/rejected': -0.12860071659088135, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.10165668278932571, 'logps/chosen': -84.10005187988281, 'logps/rejected': -84.29136657714844, 'logps/ref_chosen': -83.78422546386719, 'logps/ref_rejected': -82.7520980834961, 'logits/chosen': -2.0008394718170166, 'logits/rejected': -1.0472722053527832, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.08415273576974869, 'kl/avg_steps': 0.5, 'epoch': 0.1}
10%|███████████ | 66/681 [03:24<31:31, 3.08s/it] 10%|███████████▏ | 67/681 [03:27<31:15, 3.05s/it] {'loss': 1.3088, 'grad_norm': 20.735403060913086, 'learning_rate': 4.782608695652174e-07, 'rewards/chosen': -0.07106294482946396, 'rewards/rejected': -0.16433589160442352, 'rewards/accuracies': 0.671875, 'rewards/margins': 0.09327295422554016, 'logps/chosen': -88.51578521728516, 'logps/rejected': -81.0645751953125, 'logps/ref_chosen': -87.67295837402344, 'logps/ref_rejected': -79.08674621582031, 'logits/chosen': -2.0153298377990723, 'logits/rejected': -1.0666594505310059, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.08373406529426575, 'kl/avg_steps': 0.34375, 'epoch': 0.1}
10%|███████████▏ | 67/681 [03:27<31:15, 3.05s/it] 10%|███████████▍ | 68/681 [03:30<31:06, 3.05s/it] {'loss': 1.2341, 'grad_norm': 24.614826202392578, 'learning_rate': 4.855072463768116e-07, 'rewards/chosen': -0.02329547144472599, 'rewards/rejected': -0.19710469245910645, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.1738092005252838, 'logps/chosen': -97.353759765625, 'logps/rejected': -86.50021362304688, 'logps/ref_chosen': -97.07884216308594, 'logps/ref_rejected': -84.11872863769531, 'logits/chosen': -1.7102564573287964, 'logits/rejected': -1.115820288658142, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.08344721049070358, 'kl/avg_steps': 0.5625, 'epoch': 0.1}
10%|███████████▍ | 68/681 [03:30<31:06, 3.05s/it] 10%|███████████▌ | 69/681 [03:33<32:10, 3.15s/it] {'loss': 1.2348, 'grad_norm': 26.68158531188965, 'learning_rate': 4.927536231884058e-07, 'rewards/chosen': -0.025998366996645927, 'rewards/rejected': -0.20006409287452698, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.1740657240152359, 'logps/chosen': -86.03010559082031, 'logps/rejected': -111.91256713867188, 'logps/ref_chosen': -85.71971130371094, 'logps/ref_rejected': -109.4802017211914, 'logits/chosen': -2.055922269821167, 'logits/rejected': -1.1354026794433594, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0829804465174675, 'kl/avg_steps': 0.5625, 'epoch': 0.1}
10%|███████████▌ | 69/681 [03:33<32:10, 3.15s/it] 10%|███████████▋ | 70/681 [03:36<31:18, 3.07s/it] {'loss': 1.201, 'grad_norm': 25.407556533813477, 'learning_rate': 5e-07, 'rewards/chosen': -0.05099921301007271, 'rewards/rejected': -0.2685357332229614, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.21753652393817902, 'logps/chosen': -95.62403869628906, 'logps/rejected': -99.49327087402344, 'logps/ref_chosen': -95.00994873046875, 'logps/ref_rejected': -96.21272277832031, 'logits/chosen': -1.9641715288162231, 'logits/rejected': -1.39949631690979, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.08251629024744034, 'kl/avg_steps': 0.53125, 'epoch': 0.1}
10%|███████████▋ | 70/681 [03:36<31:18, 3.07s/it] 10%|███████████▉ | 71/681 [03:39<31:03, 3.06s/it] {'loss': 1.14, 'grad_norm': 30.39597511291504, 'learning_rate': 4.999967061337492e-07, 'rewards/chosen': -0.00880364328622818, 'rewards/rejected': -0.3007497787475586, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.291946142911911, 'logps/chosen': -91.07780456542969, 'logps/rejected': -106.28335571289062, 'logps/ref_chosen': -90.97735595703125, 'logps/ref_rejected': -102.59103393554688, 'logits/chosen': -2.9308741092681885, 'logits/rejected': -1.5702245235443115, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.08208024501800537, 'kl/avg_steps': 0.71875, 'epoch': 0.1}
10%|███████████▉ | 71/681 [03:39<31:03, 3.06s/it] 11%|████████████ | 72/681 [03:42<30:32, 3.01s/it] {'loss': 1.1831, 'grad_norm': 25.40204620361328, 'learning_rate': 4.999868246217933e-07, 'rewards/chosen': -0.04744531959295273, 'rewards/rejected': -0.282520592212677, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.23507529497146606, 'logps/chosen': -98.47068786621094, 'logps/rejected': -103.68728637695312, 'logps/ref_chosen': -97.89152526855469, 'logps/ref_rejected': -100.19171142578125, 'logits/chosen': -2.533379077911377, 'logits/rejected': -1.5070923566818237, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.08149450272321701, 'kl/avg_steps': 0.5625, 'epoch': 0.11}
11%|████████████ | 72/681 [03:42<30:32, 3.01s/it] 11%|████████████▏ | 73/681 [03:45<30:40, 3.03s/it] {'loss': 1.1906, 'grad_norm': 24.58971405029297, 'learning_rate': 4.999703557245192e-07, 'rewards/chosen': -0.06719671189785004, 'rewards/rejected': -0.3144572973251343, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.24726057052612305, 'logps/chosen': -96.5920181274414, 'logps/rejected': -99.84471130371094, 'logps/ref_chosen': -95.7690200805664, 'logps/ref_rejected': -95.93243408203125, 'logits/chosen': -2.8086342811584473, 'logits/rejected': -1.789698600769043, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.08103866130113602, 'kl/avg_steps': 0.40625, 'epoch': 0.11}
11%|████████████▏ | 73/681 [03:45<30:40, 3.03s/it] 11%|████████████▍ | 74/681 [03:48<30:32, 3.02s/it] {'loss': 1.1518, 'grad_norm': 27.501953125, 'learning_rate': 4.999472998758977e-07, 'rewards/chosen': -0.08012489974498749, 'rewards/rejected': -0.3666760325431824, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.2865511476993561, 'logps/chosen': -79.79828643798828, 'logps/rejected': -106.22779083251953, 'logps/ref_chosen': -78.80839538574219, 'logps/ref_rejected': -101.64676666259766, 'logits/chosen': -2.912767171859741, 'logits/rejected': -1.9761888980865479, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.080710768699646, 'kl/avg_steps': 0.5, 'epoch': 0.11}
11%|████████████▍ | 74/681 [03:48<30:32, 3.02s/it] 11%|████████████▌ | 75/681 [03:51<31:13, 3.09s/it] {'loss': 1.1034, 'grad_norm': 29.259719848632812, 'learning_rate': 4.999176576834721e-07, 'rewards/chosen': -0.09254170209169388, 'rewards/rejected': -0.45137819647789, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.35883650183677673, 'logps/chosen': -79.43289947509766, 'logps/rejected': -121.06864929199219, 'logps/ref_chosen': -78.28185272216797, 'logps/ref_rejected': -115.40311431884766, 'logits/chosen': -2.8685462474823, 'logits/rejected': -1.5099756717681885, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.08030922710895538, 'kl/avg_steps': 0.53125, 'epoch': 0.11}
11%|████████████▌ | 75/681 [03:51<31:13, 3.09s/it] 11%|████████████▋ | 76/681 [03:55<31:51, 3.16s/it] {'loss': 1.1945, 'grad_norm': 19.229440689086914, 'learning_rate': 4.998814299283415e-07, 'rewards/chosen': -0.12006325274705887, 'rewards/rejected': -0.35829272866249084, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.23822948336601257, 'logps/chosen': -89.3749008178711, 'logps/rejected': -90.23574829101562, 'logps/ref_chosen': -87.87714385986328, 'logps/ref_rejected': -85.71968078613281, 'logits/chosen': -3.326892375946045, 'logits/rejected': -2.0114529132843018, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.079884834587574, 'kl/avg_steps': 0.375, 'epoch': 0.11}
11%|████████████▋ | 76/681 [03:55<31:51, 3.16s/it] 11%|████████████▉ | 77/681 [03:57<30:22, 3.02s/it] {'loss': 1.1752, 'grad_norm': 22.950105667114258, 'learning_rate': 4.998386175651409e-07, 'rewards/chosen': -0.1325981616973877, 'rewards/rejected': -0.4033338725566864, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.2707356810569763, 'logps/chosen': -101.35743713378906, 'logps/rejected': -103.30995178222656, 'logps/ref_chosen': -99.70034790039062, 'logps/ref_rejected': -98.20576477050781, 'logits/chosen': -2.7979376316070557, 'logits/rejected': -1.765808343887329, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.07958638668060303, 'kl/avg_steps': 0.4375, 'epoch': 0.11}
11%|████████████▉ | 77/681 [03:57<30:22, 3.02s/it] 11%|█████████████ | 78/681 [04:00<30:52, 3.07s/it] {'loss': 1.1464, 'grad_norm': 22.01249885559082, 'learning_rate': 4.997892217220159e-07, 'rewards/chosen': -0.05697726085782051, 'rewards/rejected': -0.34640786051750183, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.2894305884838104, 'logps/chosen': -91.00945281982422, 'logps/rejected': -95.54225158691406, 'logps/ref_chosen': -90.29670715332031, 'logps/ref_rejected': -91.13772583007812, 'logits/chosen': -2.7685134410858154, 'logits/rejected': -1.9960670471191406, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.07923971116542816, 'kl/avg_steps': 0.484375, 'epoch': 0.11}
11%|█████████████ | 78/681 [04:00<30:52, 3.07s/it] 12%|█████████████▏ | 79/681 [04:04<30:54, 3.08s/it] {'loss': 1.0944, 'grad_norm': 24.0369815826416, 'learning_rate': 4.997332437005931e-07, 'rewards/chosen': -0.0716433972120285, 'rewards/rejected': -0.44942277669906616, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.37777939438819885, 'logps/chosen': -87.27758026123047, 'logps/rejected': -99.85430908203125, 'logps/ref_chosen': -86.37832641601562, 'logps/ref_rejected': -94.10777282714844, 'logits/chosen': -3.548121452331543, 'logits/rejected': -2.20076847076416, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.07885774970054626, 'kl/avg_steps': 0.59375, 'epoch': 0.12}
12%|█████████████▏ | 79/681 [04:04<30:54, 3.08s/it] 12%|█████████████▍ | 80/681 [04:07<30:39, 3.06s/it] {'loss': 1.1543, 'grad_norm': 22.031478881835938, 'learning_rate': 4.996706849759452e-07, 'rewards/chosen': -0.12929072976112366, 'rewards/rejected': -0.424081027507782, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.29479026794433594, 'logps/chosen': -95.61634826660156, 'logps/rejected': -98.0230941772461, 'logps/ref_chosen': -93.97032165527344, 'logps/ref_rejected': -92.57441711425781, 'logits/chosen': -3.074854850769043, 'logits/rejected': -2.0579161643981934, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.07839228957891464, 'kl/avg_steps': 0.40625, 'epoch': 0.12}
12%|█████████████▍ | 80/681 [04:07<30:39, 3.06s/it] 12%|█████████████▌ | 81/681 [04:10<31:03, 3.11s/it] {'loss': 1.0766, 'grad_norm': 24.160234451293945, 'learning_rate': 4.996015471965529e-07, 'rewards/chosen': -0.0865008607506752, 'rewards/rejected': -0.509419322013855, 'rewards/accuracies': 0.875, 'rewards/margins': 0.42291849851608276, 'logps/chosen': -100.93592834472656, 'logps/rejected': -140.25442504882812, 'logps/ref_chosen': -99.83012390136719, 'logps/ref_rejected': -133.67245483398438, 'logits/chosen': -3.137648105621338, 'logits/rejected': -1.6095255613327026, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.078075110912323, 'kl/avg_steps': 0.53125, 'epoch': 0.12}
12%|█████████████▌ | 81/681 [04:10<31:03, 3.11s/it] 12%|█████████████▋ | 82/681 [04:13<30:37, 3.07s/it] {'loss': 1.1567, 'grad_norm': 21.8071346282959, 'learning_rate': 4.995258321842611e-07, 'rewards/chosen': -0.15859441459178925, 'rewards/rejected': -0.4496329426765442, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.29103851318359375, 'logps/chosen': -85.08958435058594, 'logps/rejected': -100.36309814453125, 'logps/ref_chosen': -83.04598236083984, 'logps/ref_rejected': -94.52595520019531, 'logits/chosen': -3.389235019683838, 'logits/rejected': -2.2665112018585205, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.07766252756118774, 'kl/avg_steps': 0.53125, 'epoch': 0.12}
12%|█████████████▋ | 82/681 [04:13<30:37, 3.07s/it] 12%|█████████████▉ | 83/681 [04:16<31:18, 3.14s/it] {'loss': 1.1507, 'grad_norm': 21.79849624633789, 'learning_rate': 4.994435419342304e-07, 'rewards/chosen': -0.19215711951255798, 'rewards/rejected': -0.5176993012428284, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.3255422115325928, 'logps/chosen': -94.64817810058594, 'logps/rejected': -114.48917388916016, 'logps/ref_chosen': -92.17621612548828, 'logps/ref_rejected': -107.74464416503906, 'logits/chosen': -3.9037327766418457, 'logits/rejected': -2.2527427673339844, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.07725212723016739, 'kl/avg_steps': 0.28125, 'epoch': 0.12}
12%|█████████████▉ | 83/681 [04:16<31:18, 3.14s/it] 12%|██████████████ | 84/681 [04:19<31:30, 3.17s/it] {'loss': 1.2405, 'grad_norm': 22.36842918395996, 'learning_rate': 4.993546786148857e-07, 'rewards/chosen': -0.24369969964027405, 'rewards/rejected': -0.4419947564601898, 'rewards/accuracies': 0.6875, 'rewards/margins': 0.19829508662223816, 'logps/chosen': -104.68441772460938, 'logps/rejected': -98.19878387451172, 'logps/ref_chosen': -101.5264892578125, 'logps/ref_rejected': -92.42608642578125, 'logits/chosen': -3.4791769981384277, 'logits/rejected': -2.393409490585327, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.34375, 'kl/beta': 0.07703546434640884, 'kl/avg_steps': 0.3125, 'epoch': 0.12}
12%|██████████████ | 84/681 [04:19<31:30, 3.17s/it] 12%|██████████████▏ | 85/681 [04:22<31:17, 3.15s/it] {'loss': 1.1934, 'grad_norm': 20.850772857666016, 'learning_rate': 4.992592445678582e-07, 'rewards/chosen': -0.21086883544921875, 'rewards/rejected': -0.4798212945461273, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.2689524292945862, 'logps/chosen': -98.8546371459961, 'logps/rejected': -91.33653259277344, 'logps/ref_chosen': -96.12738037109375, 'logps/ref_rejected': -85.05519104003906, 'logits/chosen': -3.586297035217285, 'logits/rejected': -2.9060792922973633, 'kl/p_epsilon_steps': 0.625, 'kl/n_epsilon_steps': 0.375, 'kl/beta': 0.07679548114538193, 'kl/avg_steps': 0.25, 'epoch': 0.12}
12%|██████████████▏ | 85/681 [04:22<31:17, 3.15s/it] 13%|██████████████▍ | 86/681 [04:25<30:43, 3.10s/it] {'loss': 1.1471, 'grad_norm': 18.46939468383789, 'learning_rate': 4.991572423079235e-07, 'rewards/chosen': -0.17061173915863037, 'rewards/rejected': -0.5097446441650391, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.3391328752040863, 'logps/chosen': -83.9306640625, 'logps/rejected': -100.36227416992188, 'logps/ref_chosen': -81.70426940917969, 'logps/ref_rejected': -93.6554946899414, 'logits/chosen': -3.4410600662231445, 'logits/rejected': -2.2581920623779297, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.0766039714217186, 'kl/avg_steps': 0.40625, 'epoch': 0.13}
13%|██████████████▍ | 86/681 [04:25<30:43, 3.10s/it] 13%|██████████████▌ | 87/681 [04:28<30:41, 3.10s/it] {'loss': 1.137, 'grad_norm': 21.282371520996094, 'learning_rate': 4.990486745229364e-07, 'rewards/chosen': -0.19558626413345337, 'rewards/rejected': -0.5641908049583435, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.36860454082489014, 'logps/chosen': -95.24873352050781, 'logps/rejected': -110.37586212158203, 'logps/ref_chosen': -92.68596649169922, 'logps/ref_rejected': -102.91818237304688, 'logits/chosen': -4.2626729011535645, 'logits/rejected': -2.455996513366699, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.07629402726888657, 'kl/avg_steps': 0.53125, 'epoch': 0.13}
13%|██████████████▌ | 87/681 [04:29<30:41, 3.10s/it] 13%|██████████████▋ | 88/681 [04:32<30:43, 3.11s/it] {'loss': 1.1828, 'grad_norm': 19.734506607055664, 'learning_rate': 4.989335440737586e-07, 'rewards/chosen': -0.239786297082901, 'rewards/rejected': -0.5274724960327148, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.28768616914749146, 'logps/chosen': -103.91253662109375, 'logps/rejected': -120.1460189819336, 'logps/ref_chosen': -100.76298522949219, 'logps/ref_rejected': -113.15037536621094, 'logits/chosen': -3.0765509605407715, 'logits/rejected': -2.2642664909362793, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.07589085400104523, 'kl/avg_steps': 0.34375, 'epoch': 0.13}
13%|██████████████▋ | 88/681 [04:32<30:43, 3.11s/it] 13%|██████████████▉ | 89/681 [04:35<30:05, 3.05s/it] {'loss': 1.1584, 'grad_norm': 20.043725967407227, 'learning_rate': 4.988118539941847e-07, 'rewards/chosen': -0.19024960696697235, 'rewards/rejected': -0.5089821815490723, 'rewards/accuracies': 0.75, 'rewards/margins': 0.3187325596809387, 'logps/chosen': -92.20272827148438, 'logps/rejected': -95.34589385986328, 'logps/ref_chosen': -89.69108581542969, 'logps/ref_rejected': -88.56832885742188, 'logits/chosen': -3.5043540000915527, 'logits/rejected': -2.6482343673706055, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.07563087344169617, 'kl/avg_steps': 0.34375, 'epoch': 0.13}
13%|██████████████▉ | 89/681 [04:35<30:05, 3.05s/it] 13%|███████████████ | 90/681 [04:38<30:13, 3.07s/it] {'loss': 1.0847, 'grad_norm': 22.877426147460938, 'learning_rate': 4.986836074908615e-07, 'rewards/chosen': -0.1728699803352356, 'rewards/rejected': -0.5974611043930054, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.4245910942554474, 'logps/chosen': -83.66722106933594, 'logps/rejected': -125.76116180419922, 'logps/ref_chosen': -81.38255310058594, 'logps/ref_rejected': -117.77714538574219, 'logits/chosen': -4.08919620513916, 'logits/rejected': -2.59071683883667, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.07537178695201874, 'kl/avg_steps': 0.53125, 'epoch': 0.13}
13%|███████████████ | 90/681 [04:38<30:13, 3.07s/it] 13%|███████████████▏ | 91/681 [04:41<31:05, 3.16s/it] {'loss': 1.1414, 'grad_norm': 21.175289154052734, 'learning_rate': 4.985488079432037e-07, 'rewards/chosen': -0.20845842361450195, 'rewards/rejected': -0.5571086406707764, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.3486502170562744, 'logps/chosen': -99.98194885253906, 'logps/rejected': -100.44671630859375, 'logps/ref_chosen': -97.22188568115234, 'logps/ref_rejected': -92.97674560546875, 'logits/chosen': -3.6395578384399414, 'logits/rejected': -2.8058741092681885, 'kl/p_epsilon_steps': 0.609375, 'kl/n_epsilon_steps': 0.390625, 'kl/beta': 0.07497348636388779, 'kl/avg_steps': 0.21875, 'epoch': 0.13}
13%|███████████████▏ | 91/681 [04:41<31:05, 3.16s/it] 14%|███████████████▍ | 92/681 [04:44<30:24, 3.10s/it] {'loss': 1.1718, 'grad_norm': 19.441743850708008, 'learning_rate': 4.984074589033043e-07, 'rewards/chosen': -0.22265848517417908, 'rewards/rejected': -0.5193834900856018, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.29672500491142273, 'logps/chosen': -87.49899291992188, 'logps/rejected': -91.48877716064453, 'logps/ref_chosen': -84.5302734375, 'logps/ref_rejected': -84.5013198852539, 'logits/chosen': -4.114226341247559, 'logits/rejected': -3.050344944000244, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.07480984181165695, 'kl/avg_steps': 0.28125, 'epoch': 0.14}
14%|███████████████▍ | 92/681 [04:44<30:24, 3.10s/it] 14%|███████████████▌ | 93/681 [04:47<28:42, 2.93s/it] {'loss': 1.192, 'grad_norm': 20.867734909057617, 'learning_rate': 4.982595640958425e-07, 'rewards/chosen': -0.24706237018108368, 'rewards/rejected': -0.535523533821106, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.2884610891342163, 'logps/chosen': -93.55166625976562, 'logps/rejected': -91.32198333740234, 'logps/ref_chosen': -90.25043487548828, 'logps/ref_rejected': -84.09422302246094, 'logits/chosen': -3.9874632358551025, 'logits/rejected': -2.371685266494751, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.07460002601146698, 'kl/avg_steps': 0.28125, 'epoch': 0.14}
14%|███████████████▌ | 93/681 [04:47<28:42, 2.93s/it] 14%|███████████████▋ | 94/681 [04:50<29:36, 3.03s/it] {'loss': 1.0808, 'grad_norm': 22.021520614624023, 'learning_rate': 4.98105127417984e-07, 'rewards/chosen': -0.25266966223716736, 'rewards/rejected': -0.6970977783203125, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.44442811608314514, 'logps/chosen': -95.84749603271484, 'logps/rejected': -114.69227600097656, 'logps/ref_chosen': -92.4542236328125, 'logps/ref_rejected': -105.24728393554688, 'logits/chosen': -4.104582786560059, 'logits/rejected': -2.78466796875, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.07439080625772476, 'kl/avg_steps': 0.53125, 'epoch': 0.14}
14%|███████████████▋ | 94/681 [04:50<29:36, 3.03s/it] 14%|███████████████▉ | 95/681 [04:53<29:10, 2.99s/it] {'loss': 1.1379, 'grad_norm': 18.556907653808594, 'learning_rate': 4.979441529392784e-07, 'rewards/chosen': -0.201766699552536, 'rewards/rejected': -0.5478700399398804, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.346103310585022, 'logps/chosen': -81.59260559082031, 'logps/rejected': -91.0462875366211, 'logps/ref_chosen': -78.87370300292969, 'logps/ref_rejected': -83.59121704101562, 'logits/chosen': -4.07137393951416, 'logits/rejected': -2.9404728412628174, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.07399769127368927, 'kl/avg_steps': 0.34375, 'epoch': 0.14}
14%|███████████████▉ | 95/681 [04:53<29:10, 2.99s/it] 14%|████████████████ | 96/681 [04:56<29:07, 2.99s/it] {'loss': 1.0757, 'grad_norm': 19.24999237060547, 'learning_rate': 4.977766449015534e-07, 'rewards/chosen': -0.18087545037269592, 'rewards/rejected': -0.6175007224082947, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.43662530183792114, 'logps/chosen': -109.04351806640625, 'logps/rejected': -110.20415496826172, 'logps/ref_chosen': -106.5921630859375, 'logps/ref_rejected': -101.76802062988281, 'logits/chosen': -3.7676031589508057, 'logits/rejected': -2.4514598846435547, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.07374419271945953, 'kl/avg_steps': 0.46875, 'epoch': 0.14}
14%|████████████████ | 96/681 [04:56<29:07, 2.99s/it] 14%|████████████████▏ | 97/681 [04:59<29:18, 3.01s/it] {'loss': 1.1006, 'grad_norm': 21.76212501525879, 'learning_rate': 4.976026077188012e-07, 'rewards/chosen': -0.26550614833831787, 'rewards/rejected': -0.6464561223983765, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.380950003862381, 'logps/chosen': -92.28569030761719, 'logps/rejected': -93.66389465332031, 'logps/ref_chosen': -88.67988586425781, 'logps/ref_rejected': -84.81229400634766, 'logits/chosen': -4.675760746002197, 'logits/rejected': -3.565962791442871, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.07340013235807419, 'kl/avg_steps': 0.40625, 'epoch': 0.14}
14%|████████████████▏ | 97/681 [04:59<29:18, 3.01s/it] 14%|████████████████▍ | 98/681 [05:02<29:33, 3.04s/it] {'loss': 1.0488, 'grad_norm': 19.99883460998535, 'learning_rate': 4.974220459770639e-07, 'rewards/chosen': -0.2048511505126953, 'rewards/rejected': -0.6484156250953674, 'rewards/accuracies': 0.875, 'rewards/margins': 0.4435645043849945, 'logps/chosen': -95.04853057861328, 'logps/rejected': -110.45492553710938, 'logps/ref_chosen': -92.24249267578125, 'logps/ref_rejected': -101.51948547363281, 'logits/chosen': -3.876922607421875, 'logits/rejected': -2.7262768745422363, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0731031522154808, 'kl/avg_steps': 0.53125, 'epoch': 0.14}
14%|████████████████▍ | 98/681 [05:02<29:33, 3.04s/it] 15%|████████████████▌ | 99/681 [05:05<28:56, 2.98s/it] {'loss': 1.0479, 'grad_norm': 20.251489639282227, 'learning_rate': 4.972349644343108e-07, 'rewards/chosen': -0.15243776142597198, 'rewards/rejected': -0.6443277597427368, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.49188998341560364, 'logps/chosen': -74.27732849121094, 'logps/rejected': -100.81095123291016, 'logps/ref_chosen': -72.18464660644531, 'logps/ref_rejected': -91.88131713867188, 'logits/chosen': -4.384429454803467, 'logits/rejected': -3.071768045425415, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0727168396115303, 'kl/avg_steps': 0.53125, 'epoch': 0.15}
15%|████████████████▌ | 99/681 [05:05<28:56, 2.98s/it] 15%|████████████████▌ | 100/681 [05:08<29:12, 3.02s/it] {'loss': 1.1171, 'grad_norm': 17.54705047607422, 'learning_rate': 4.970413680203148e-07, 'rewards/chosen': -0.18433162569999695, 'rewards/rejected': -0.5253919363021851, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.3410602807998657, 'logps/chosen': -92.06529235839844, 'logps/rejected': -88.53147888183594, 'logps/ref_chosen': -89.51382446289062, 'logps/ref_rejected': -81.21713256835938, 'logits/chosen': -4.1084113121032715, 'logits/rejected': -2.772047758102417, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.07233257591724396, 'kl/avg_steps': 0.40625, 'epoch': 0.15}
15%|████████████████▌ | 100/681 [05:08<29:12, 3.02s/it][INFO|trainer.py:4307] 2026-04-24 04:21:15,383 >>
***** Running Evaluation *****
[INFO|trainer.py:4309] 2026-04-24 04:21:15,383 >> Num examples = 2339
[INFO|trainer.py:4312] 2026-04-24 04:21:15,383 >> Batch size = 8
0%| | 0/73 [00:00<?, ?it/s]
3%|███▏ | 2/73 [00:00<00:23, 3.04it/s]
4%|████▊ | 3/73 [00:01<00:32, 2.14it/s]
5%|██████▎ | 4/73 [00:01<00:36, 1.87it/s]
7%|███████▉ | 5/73 [00:02<00:38, 1.75it/s]
8%|█████████▌ | 6/73 [00:03<00:40, 1.65it/s]
10%|███████████ | 7/73 [00:03<00:38, 1.71it/s]
11%|████████████▋ | 8/73 [00:04<00:40, 1.59it/s]
12%|██████████████▎ | 9/73 [00:05<00:41, 1.54it/s]
14%|███████████████▊ | 10/73 [00:05<00:41, 1.52it/s]
15%|█████████████████▎ | 11/73 [00:06<00:40, 1.52it/s]
16%|██████████████████▉ | 12/73 [00:07<00:40, 1.50it/s]
18%|████████████████████▍ | 13/73 [00:07<00:39, 1.52it/s]
19%|██████████████████████ | 14/73 [00:08<00:39, 1.50it/s]
21%|███████████████████████▋ | 15/73 [00:09<00:38, 1.51it/s]
22%|█████████████████████████▏ | 16/73 [00:09<00:39, 1.46it/s]
23%|██████████████████████████▊ | 17/73 [00:10<00:38, 1.45it/s]
25%|████████████████████████████▎ | 18/73 [00:11<00:38, 1.43it/s]
26%|█████████████████████████████▉ | 19/73 [00:12<00:38, 1.42it/s]
27%|███████████████████████████████▌ | 20/73 [00:12<00:37, 1.41it/s]
29%|█████████████████████████████████ | 21/73 [00:13<00:37, 1.40it/s]
30%|██████████████████████████████████▋ | 22/73 [00:14<00:36, 1.39it/s]
32%|████████████████████████████████████▏ | 23/73 [00:14<00:35, 1.42it/s]
33%|█████████████████████████████████████▊ | 24/73 [00:15<00:34, 1.43it/s]
34%|███████████████████████████████████████▍ | 25/73 [00:16<00:33, 1.44it/s]
36%|████████████████████████████████████████▉ | 26/73 [00:17<00:32, 1.45it/s]
37%|██████████████████████████████████████████▌ | 27/73 [00:17<00:28, 1.60it/s]
38%|████████████████████████████████████████████ | 28/73 [00:18<00:28, 1.56it/s]
40%|█████████████████████████████████████████████▋ | 29/73 [00:18<00:28, 1.57it/s]
41%|███████████████████████████████████████████████▎ | 30/73 [00:19<00:26, 1.60it/s]
42%|████████████████████████████████████████████████▊ | 31/73 [00:20<00:27, 1.52it/s]
44%|██████████████████████████████████████████████████▍ | 32/73 [00:20<00:26, 1.55it/s]
45%|███████████████████████████████████████████████████▉ | 33/73 [00:21<00:25, 1.58it/s]
47%|█████████████████████████████████████████████████████▌ | 34/73 [00:22<00:25, 1.54it/s]
48%|███████████████████████████████████████████████████████▏ | 35/73 [00:22<00:25, 1.48it/s]
49%|████████████████████████████████████████████████████████▋ | 36/73 [00:23<00:25, 1.48it/s]
51%|██████████████████████████████████████████████████████████▎ | 37/73 [00:24<00:24, 1.47it/s]
52%|███████████████████████████████████████████████████████████▊ | 38/73 [00:24<00:22, 1.55it/s]
53%|█████████████████████████████████████████████████████████████▍ | 39/73 [00:25<00:22, 1.49it/s]
55%|███████████████████████████████████████████████████████████████ | 40/73 [00:26<00:22, 1.50it/s]
56%|████████████████████████████████████████████████████████████████▌ | 41/73 [00:26<00:20, 1.55it/s]
58%|██████████████████████████████████████████████████████████████████▏ | 42/73 [00:27<00:20, 1.52it/s]
59%|███████████████████████████████████████████████████████████████████▋ | 43/73 [00:28<00:19, 1.54it/s]
60%|█████████████████████████████████████████████████████████████████████▎ | 44/73 [00:28<00:19, 1.52it/s]
62%|██████████████████████████████████████████████████████████████████████▉ | 45/73 [00:29<00:19, 1.47it/s]
63%|████████████████████████████████████████████████████████████████████████▍ | 46/73 [00:30<00:17, 1.53it/s]
64%|██████████████████████████████████████████████████████████████████████████ | 47/73 [00:30<00:17, 1.50it/s]
66%|███████████████████████████████████████████████████████████████████████████▌ | 48/73 [00:31<00:16, 1.49it/s]
67%|█████████████████████████████████████████████████████████████████████████████▏ | 49/73 [00:32<00:16, 1.47it/s]
68%|██████████████████████████████████████████████████████████████████████████████▊ | 50/73 [00:32<00:15, 1.49it/s]
70%|████████████████████████████████████████████████████████████████████████████████▎ | 51/73 [00:33<00:14, 1.47it/s]
71%|█████████████████████████████████████████████████████████████████████████████████▉ | 52/73 [00:34<00:14, 1.44it/s]
73%|███████████████████████████████████████████████████████████████████████████████████▍ | 53/73 [00:34<00:14, 1.42it/s]
74%|█████████████████████████████████████████████████████████████████████████████████████ | 54/73 [00:35<00:12, 1.50it/s]
75%|██████████████████████████████████████████████████████████████████████████████████████▋ | 55/73 [00:36<00:12, 1.49it/s]
77%|████████████████████████████████████████████████████████████████████████████████████████▏ | 56/73 [00:36<00:11, 1.53it/s]
78%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 57/73 [00:37<00:10, 1.47it/s]
79%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 58/73 [00:38<00:09, 1.53it/s]
81%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 59/73 [00:38<00:09, 1.53it/s]
82%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 60/73 [00:39<00:08, 1.50it/s]
84%|████████████████████████████████████████████████████████████████████████████████████████████████ | 61/73 [00:40<00:08, 1.49it/s]
85%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 62/73 [00:40<00:07, 1.50it/s]
86%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 63/73 [00:41<00:06, 1.58it/s]
88%|████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 64/73 [00:41<00:05, 1.60it/s]
89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 65/73 [00:42<00:05, 1.58it/s]
90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 66/73 [00:43<00:04, 1.52it/s]
92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 67/73 [00:43<00:03, 1.55it/s]
93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 68/73 [00:44<00:03, 1.51it/s]
95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 69/73 [00:45<00:02, 1.48it/s]
96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 70/73 [00:46<00:02, 1.47it/s]
97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 71/73 [00:46<00:01, 1.48it/s]
99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 72/73 [00:47<00:00, 1.48it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
{'eval_loss': 0.6201064586639404, 'eval_runtime': 48.5803, 'eval_samples_per_second': 48.147, 'eval_steps_per_second': 1.523, 'eval_rewards/chosen': -0.31309446692466736, 'eval_rewards/rejected': -0.5532342195510864, 'eval_rewards/accuracies': 0.6840753555297852, 'eval_rewards/margins': 0.24013973772525787, 'eval_logps/chosen': -104.82538604736328, 'eval_logps/rejected': -101.79056549072266, 'eval_logps/ref_chosen': -100.49356842041016, 'eval_logps/ref_rejected': -94.06775665283203, 'eval_logits/chosen': -3.108599901199341, 'eval_logits/rejected': -2.4245166778564453, 'eval_kl/p_epsilon_steps': 0.6134417653083801, 'eval_kl/n_epsilon_steps': 0.3857020437717438, 'epoch': 0.15}
15%|████████████████▌ | 100/681 [05:56<29:12, 3.02s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
 15%|████████████████▍ | 101/681 [05:59<2:49:32, 17.54s/it] {'loss': 1.1745, 'grad_norm': 21.208911895751953, 'learning_rate': 4.968412618365215e-07, 'rewards/chosen': -0.2888886034488678, 'rewards/rejected': -0.6014888286590576, 'rewards/accuracies': 0.75, 'rewards/margins': 0.3126002252101898, 'logps/chosen': -98.40846252441406, 'logps/rejected': -98.31871032714844, 'logps/ref_chosen': -94.4031982421875, 'logps/ref_rejected': -89.91001892089844, 'logits/chosen': -3.8228065967559814, 'logits/rejected': -3.2613918781280518, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.07203991711139679, 'kl/avg_steps': 0.34375, 'epoch': 0.15}
15%|████████████████▍ | 101/681 [05:59<2:49:32, 17.54s/it] 15%|████████████████▋ | 102/681 [06:02<2:06:52, 13.15s/it] {'loss': 1.2292, 'grad_norm': 26.08466148376465, 'learning_rate': 4.966346511559149e-07, 'rewards/chosen': -0.3448712229728699, 'rewards/rejected': -0.6022634506225586, 'rewards/accuracies': 0.75, 'rewards/margins': 0.2573922276496887, 'logps/chosen': -96.39299774169922, 'logps/rejected': -84.2006607055664, 'logps/ref_chosen': -91.60942840576172, 'logps/ref_rejected': -75.76891326904297, 'logits/chosen': -4.024471282958984, 'logits/rejected': -3.1433403491973877, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.07179312407970428, 'kl/avg_steps': 0.390625, 'epoch': 0.15}
15%|████████████████▋ | 102/681 [06:02<2:06:52, 13.15s/it] 15%|████████████████▊ | 103/681 [06:05<1:36:37, 10.03s/it] {'loss': 1.0673, 'grad_norm': 20.780960083007812, 'learning_rate': 4.964215414228785e-07, 'rewards/chosen': -0.26924505829811096, 'rewards/rejected': -0.7265477776527405, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.4573027193546295, 'logps/chosen': -101.16336059570312, 'logps/rejected': -109.21807861328125, 'logps/ref_chosen': -97.40238952636719, 'logps/ref_rejected': -98.98168182373047, 'logits/chosen': -4.443181037902832, 'logits/rejected': -3.242924928665161, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.07151377201080322, 'kl/avg_steps': 0.53125, 'epoch': 0.15}
15%|████████████████▊ | 103/681 [06:05<1:36:37, 10.03s/it] 15%|████████████████▉ | 104/681 [06:07<1:15:05, 7.81s/it] {'loss': 1.0609, 'grad_norm': 17.77613639831543, 'learning_rate': 4.96201938253052e-07, 'rewards/chosen': -0.2158360481262207, 'rewards/rejected': -0.6963146924972534, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.4804786741733551, 'logps/chosen': -97.89046478271484, 'logps/rejected': -106.888427734375, 'logps/ref_chosen': -94.85636138916016, 'logps/ref_rejected': -97.02668762207031, 'logits/chosen': -4.106165885925293, 'logits/rejected': -3.3227481842041016, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.07113586366176605, 'kl/avg_steps': 0.40625, 'epoch': 0.15}
15%|████████████████▉ | 104/681 [06:08<1:15:05, 7.81s/it] 15%|█████████████████ | 105/681 [06:11<1:01:57, 6.45s/it] {'loss': 1.0544, 'grad_norm': 20.64962387084961, 'learning_rate': 4.959758474331832e-07, 'rewards/chosen': -0.2757706940174103, 'rewards/rejected': -0.7616356611251831, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.4858649969100952, 'logps/chosen': -100.64320373535156, 'logps/rejected': -112.40327453613281, 'logps/ref_chosen': -96.75790405273438, 'logps/ref_rejected': -101.5774917602539, 'logits/chosen': -4.99745512008667, 'logits/rejected': -3.3221869468688965, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.07084804773330688, 'kl/avg_steps': 0.53125, 'epoch': 0.15}
15%|█████████████████ | 105/681 [06:11<1:01:57, 6.45s/it] 16%|█████████████████▌ | 106/681 [06:14<53:06, 5.54s/it] {'loss': 1.0813, 'grad_norm': 19.51259422302246, 'learning_rate': 4.957432749209755e-07, 'rewards/chosen': -0.27257949113845825, 'rewards/rejected': -0.7240015268325806, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.4514220058917999, 'logps/chosen': -96.30393981933594, 'logps/rejected': -101.36027526855469, 'logps/ref_chosen': -92.43791198730469, 'logps/ref_rejected': -91.00969696044922, 'logits/chosen': -4.652019500732422, 'logits/rejected': -3.1131434440612793, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.07047365605831146, 'kl/avg_steps': 0.46875, 'epoch': 0.16}
16%|█████████████████▌ | 106/681 [06:14<53:06, 5.54s/it] 16%|█████████████████▊ | 107/681 [06:17<46:04, 4.82s/it] {'loss': 1.0554, 'grad_norm': 19.533924102783203, 'learning_rate': 4.955042268449307e-07, 'rewards/chosen': -0.20865142345428467, 'rewards/rejected': -0.7428811192512512, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.5342296361923218, 'logps/chosen': -108.09788513183594, 'logps/rejected': -111.86402130126953, 'logps/ref_chosen': -105.12500762939453, 'logps/ref_rejected': -101.18250274658203, 'logits/chosen': -3.9555153846740723, 'logits/rejected': -2.785179615020752, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.07014484703540802, 'kl/avg_steps': 0.5, 'epoch': 0.16}
16%|█████████████████▊ | 107/681 [06:17<46:04, 4.82s/it] 16%|█████████████████▉ | 108/681 [06:20<40:15, 4.22s/it] {'loss': 1.0533, 'grad_norm': 19.818805694580078, 'learning_rate': 4.952587095041881e-07, 'rewards/chosen': -0.29724550247192383, 'rewards/rejected': -0.8035186529159546, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.5062732100486755, 'logps/chosen': -93.46691131591797, 'logps/rejected': -113.4327163696289, 'logps/ref_chosen': -89.20936584472656, 'logps/ref_rejected': -101.83784484863281, 'logits/chosen': -5.145913124084473, 'logits/rejected': -3.347729206085205, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.0697958692908287, 'kl/avg_steps': 0.46875, 'epoch': 0.16}
16%|█████████████████▉ | 108/681 [06:20<40:15, 4.22s/it] 16%|██████████████████ | 109/681 [06:23<37:14, 3.91s/it] {'loss': 0.9893, 'grad_norm': 18.633304595947266, 'learning_rate': 4.95006729368358e-07, 'rewards/chosen': -0.24741317331790924, 'rewards/rejected': -0.8620044589042664, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.6145913004875183, 'logps/chosen': -103.74415588378906, 'logps/rejected': -116.03982543945312, 'logps/ref_chosen': -100.18708801269531, 'logps/ref_rejected': -103.53739929199219, 'logits/chosen': -4.46284294128418, 'logits/rejected': -3.8937301635742188, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.06947022676467896, 'kl/avg_steps': 0.375, 'epoch': 0.16}
16%|██████████████████ | 109/681 [06:23<37:14, 3.91s/it] 16%|██████████████████▎ | 110/681 [06:26<35:04, 3.68s/it] {'loss': 1.1493, 'grad_norm': 22.954145431518555, 'learning_rate': 4.947482930773511e-07, 'rewards/chosen': -0.30703774094581604, 'rewards/rejected': -0.6591849327087402, 'rewards/accuracies': 0.75, 'rewards/margins': 0.3521472215652466, 'logps/chosen': -111.32714080810547, 'logps/rejected': -95.40037536621094, 'logps/ref_chosen': -106.89572143554688, 'logps/ref_rejected': -85.81120300292969, 'logits/chosen': -4.040766716003418, 'logits/rejected': -3.4033002853393555, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.34375, 'kl/beta': 0.06921068578958511, 'kl/avg_steps': 0.3125, 'epoch': 0.16}
16%|██████████████████▎ | 110/681 [06:27<35:04, 3.68s/it] 16%|██████████████████▍ | 111/681 [06:29<33:00, 3.48s/it] {'loss': 0.9406, 'grad_norm': 17.981557846069336, 'learning_rate': 4.944834074412042e-07, 'rewards/chosen': -0.2670831084251404, 'rewards/rejected': -0.9773316383361816, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.7102484703063965, 'logps/chosen': -97.20205688476562, 'logps/rejected': -118.46589660644531, 'logps/ref_chosen': -93.3293685913086, 'logps/ref_rejected': -104.177490234375, 'logits/chosen': -4.594253063201904, 'logits/rejected': -3.7376368045806885, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.06899508088827133, 'kl/avg_steps': 0.5625, 'epoch': 0.16}
16%|██████████████████▍ | 111/681 [06:29<33:00, 3.48s/it] 16%|██████████████████▌ | 112/681 [06:32<31:01, 3.27s/it] {'loss': 1.1448, 'grad_norm': 19.202064514160156, 'learning_rate': 4.942120794399002e-07, 'rewards/chosen': -0.3032221496105194, 'rewards/rejected': -0.6522951126098633, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.3490729331970215, 'logps/chosen': -87.14385986328125, 'logps/rejected': -81.11387634277344, 'logps/ref_chosen': -82.7470474243164, 'logps/ref_rejected': -71.55845642089844, 'logits/chosen': -5.043312072753906, 'logits/rejected': -4.008922576904297, 'kl/p_epsilon_steps': 0.59375, 'kl/n_epsilon_steps': 0.40625, 'kl/beta': 0.06860914826393127, 'kl/avg_steps': 0.1875, 'epoch': 0.16}
16%|██████████████████▌ | 112/681 [06:32<31:01, 3.27s/it] 17%|██████████████████▊ | 113/681 [06:35<30:48, 3.25s/it] {'loss': 1.1132, 'grad_norm': 22.44894790649414, 'learning_rate': 4.939343162231841e-07, 'rewards/chosen': -0.37001150846481323, 'rewards/rejected': -0.7502257823944092, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.38021430373191833, 'logps/chosen': -103.29286193847656, 'logps/rejected': -95.4296875, 'logps/ref_chosen': -97.89949798583984, 'logps/ref_rejected': -84.41960144042969, 'logits/chosen': -4.6388139724731445, 'logits/rejected': -3.630887031555176, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.06848075240850449, 'kl/avg_steps': 0.40625, 'epoch': 0.17}
17%|██████████████████▊ | 113/681 [06:36<30:48, 3.25s/it] 17%|██████████████████▉ | 114/681 [06:39<30:33, 3.23s/it] {'loss': 1.0716, 'grad_norm': 22.647991180419922, 'learning_rate': 4.936501251103751e-07, 'rewards/chosen': -0.34747976064682007, 'rewards/rejected': -0.880822479724884, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.533342719078064, 'logps/chosen': -101.57315063476562, 'logps/rejected': -105.01116180419922, 'logps/ref_chosen': -96.4951171875, 'logps/ref_rejected': -92.01536560058594, 'logits/chosen': -5.111181259155273, 'logits/rejected': -3.6170125007629395, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.06820367276668549, 'kl/avg_steps': 0.34375, 'epoch': 0.17}
17%|██████████████████▉ | 114/681 [06:39<30:33, 3.23s/it] 17%|███████████████████ | 115/681 [06:42<30:20, 3.22s/it] {'loss': 1.0422, 'grad_norm': 20.238996505737305, 'learning_rate': 4.933595135901732e-07, 'rewards/chosen': -0.3536038398742676, 'rewards/rejected': -0.8884330987930298, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.5348291993141174, 'logps/chosen': -108.37738800048828, 'logps/rejected': -118.87684631347656, 'logps/ref_chosen': -103.18013000488281, 'logps/ref_rejected': -105.71595764160156, 'logits/chosen': -4.847101211547852, 'logits/rejected': -3.5432467460632324, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.06797002255916595, 'kl/avg_steps': 0.4375, 'epoch': 0.17}
17%|███████████████████ | 115/681 [06:42<30:20, 3.22s/it] 17%|███████████████████▏ | 116/681 [06:45<29:03, 3.08s/it] {'loss': 1.064, 'grad_norm': 17.519254684448242, 'learning_rate': 4.930624893204624e-07, 'rewards/chosen': -0.300454318523407, 'rewards/rejected': -0.7515127062797546, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.45105841755867004, 'logps/chosen': -85.59718322753906, 'logps/rejected': -97.48081970214844, 'logps/ref_chosen': -81.15837097167969, 'logps/ref_rejected': -86.30586242675781, 'logits/chosen': -5.042938232421875, 'logits/rejected': -3.4571943283081055, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.0676739513874054, 'kl/avg_steps': 0.375, 'epoch': 0.17}
17%|███████████████████▏ | 116/681 [06:45<29:03, 3.08s/it] 17%|███████████████████▍ | 117/681 [06:48<28:48, 3.06s/it] {'loss': 1.1723, 'grad_norm': 29.290815353393555, 'learning_rate': 4.927590601281083e-07, 'rewards/chosen': -0.36769068241119385, 'rewards/rejected': -0.7117009162902832, 'rewards/accuracies': 0.75, 'rewards/margins': 0.34401029348373413, 'logps/chosen': -102.51126098632812, 'logps/rejected': -85.75031280517578, 'logps/ref_chosen': -97.0809555053711, 'logps/ref_rejected': -75.1399154663086, 'logits/chosen': -4.475282192230225, 'logits/rejected': -3.8478426933288574, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.34375, 'kl/beta': 0.06742112338542938, 'kl/avg_steps': 0.3125, 'epoch': 0.17}
17%|███████████████████▍ | 117/681 [06:48<28:48, 3.06s/it] 17%|███████████████████▌ | 118/681 [06:51<28:30, 3.04s/it] {'loss': 1.0216, 'grad_norm': 16.853479385375977, 'learning_rate': 4.924492340087524e-07, 'rewards/chosen': -0.2619098126888275, 'rewards/rejected': -0.7679451107978821, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.506035327911377, 'logps/chosen': -90.43417358398438, 'logps/rejected': -94.443603515625, 'logps/ref_chosen': -86.5362319946289, 'logps/ref_rejected': -82.93563842773438, 'logits/chosen': -4.661401748657227, 'logits/rejected': -3.8089747428894043, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.0672110915184021, 'kl/avg_steps': 0.484375, 'epoch': 0.17}
17%|███████████████████▌ | 118/681 [06:51<28:30, 3.04s/it] 17%|███████████████████▋ | 119/681 [06:54<28:49, 3.08s/it] {'loss': 1.0755, 'grad_norm': 20.742483139038086, 'learning_rate': 4.92133019126601e-07, 'rewards/chosen': -0.38021495938301086, 'rewards/rejected': -0.9111853837966919, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.5309704542160034, 'logps/chosen': -110.28329467773438, 'logps/rejected': -122.87940979003906, 'logps/ref_chosen': -104.60908508300781, 'logps/ref_rejected': -109.1578140258789, 'logits/chosen': -5.028365135192871, 'logits/rejected': -4.106209754943848, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.06688710302114487, 'kl/avg_steps': 0.5, 'epoch': 0.17}
17%|███████████████████▋ | 119/681 [06:54<28:49, 3.08s/it] 18%|███████████████████▉ | 120/681 [06:57<29:01, 3.10s/it] {'loss': 1.0054, 'grad_norm': 25.90955924987793, 'learning_rate': 4.918104238142103e-07, 'rewards/chosen': -0.30640918016433716, 'rewards/rejected': -0.9960211515426636, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6896119117736816, 'logps/chosen': -119.28671264648438, 'logps/rejected': -126.28087615966797, 'logps/ref_chosen': -114.68742370605469, 'logps/ref_rejected': -111.18586730957031, 'logits/chosen': -4.781673431396484, 'logits/rejected': -3.775351047515869, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.06655433028936386, 'kl/avg_steps': 0.59375, 'epoch': 0.18}
18%|███████████████████▉ | 120/681 [06:57<29:01, 3.10s/it] 18%|████████████████████ | 121/681 [07:00<29:11, 3.13s/it] {'loss': 0.9533, 'grad_norm': 18.882291793823242, 'learning_rate': 4.91481456572267e-07, 'rewards/chosen': -0.26827573776245117, 'rewards/rejected': -0.8953906297683716, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6271148920059204, 'logps/chosen': -96.17632293701172, 'logps/rejected': -128.61309814453125, 'logps/ref_chosen': -92.11666107177734, 'logps/ref_rejected': -114.972900390625, 'logits/chosen': -5.0012335777282715, 'logits/rejected': -3.5916976928710938, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.06616149842739105, 'kl/avg_steps': 0.53125, 'epoch': 0.18}
18%|████████████████████ | 121/681 [07:00<29:11, 3.13s/it] 18%|████████████████████▏ | 122/681 [07:03<28:25, 3.05s/it] {'loss': 0.9203, 'grad_norm': 18.61724281311035, 'learning_rate': 4.911461260693638e-07, 'rewards/chosen': -0.2842973470687866, 'rewards/rejected': -1.0009193420410156, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.716621994972229, 'logps/chosen': -89.19412994384766, 'logps/rejected': -126.41008758544922, 'logps/ref_chosen': -84.87757110595703, 'logps/ref_rejected': -111.0829849243164, 'logits/chosen': -4.714388847351074, 'logits/rejected': -3.9871459007263184, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.0658118724822998, 'kl/avg_steps': 0.5, 'epoch': 0.18}
18%|████████████████████▏ | 122/681 [07:03<28:25, 3.05s/it] 18%|████████████████████▍ | 123/681 [07:06<28:50, 3.10s/it] {'loss': 1.0567, 'grad_norm': 18.020599365234375, 'learning_rate': 4.908044411417711e-07, 'rewards/chosen': -0.3202221989631653, 'rewards/rejected': -0.8172218799591064, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.49699968099594116, 'logps/chosen': -103.35702514648438, 'logps/rejected': -106.42095947265625, 'logps/ref_chosen': -98.47874450683594, 'logps/ref_rejected': -93.85417175292969, 'logits/chosen': -4.689582824707031, 'logits/rejected': -3.9507789611816406, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.34375, 'kl/beta': 0.06548444926738739, 'kl/avg_steps': 0.3125, 'epoch': 0.18}
18%|████████████████████▍ | 123/681 [07:06<28:50, 3.10s/it] 18%|████████████████████▌ | 124/681 [07:09<28:45, 3.10s/it] {'loss': 1.0345, 'grad_norm': 18.883220672607422, 'learning_rate': 4.904564107932048e-07, 'rewards/chosen': -0.29794853925704956, 'rewards/rejected': -0.9165425300598145, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.6185939908027649, 'logps/chosen': -107.31734466552734, 'logps/rejected': -137.15272521972656, 'logps/ref_chosen': -102.76290130615234, 'logps/ref_rejected': -123.005615234375, 'logits/chosen': -4.445045471191406, 'logits/rejected': -3.482326030731201, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.06528045237064362, 'kl/avg_steps': 0.375, 'epoch': 0.18}
18%|████████████████████▌ | 124/681 [07:09<28:45, 3.10s/it] 18%|████████████████████▋ | 125/681 [07:12<28:11, 3.04s/it] {'loss': 1.1176, 'grad_norm': 21.531150817871094, 'learning_rate': 4.90102044194588e-07, 'rewards/chosen': -0.3898157477378845, 'rewards/rejected': -0.8257647752761841, 'rewards/accuracies': 0.65625, 'rewards/margins': 0.43594902753829956, 'logps/chosen': -96.40164947509766, 'logps/rejected': -102.02460479736328, 'logps/ref_chosen': -90.42627716064453, 'logps/ref_rejected': -89.24901580810547, 'logits/chosen': -4.837682247161865, 'logits/rejected': -4.35373592376709, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.06503656506538391, 'kl/avg_steps': 0.28125, 'epoch': 0.18}
18%|████████████████████▋ | 125/681 [07:12<28:11, 3.04s/it] 19%|████████████████████▉ | 126/681 [07:15<28:24, 3.07s/it] {'loss': 0.94, 'grad_norm': 18.599184036254883, 'learning_rate': 4.897413506838102e-07, 'rewards/chosen': -0.30715566873550415, 'rewards/rejected': -0.9453917741775513, 'rewards/accuracies': 0.875, 'rewards/margins': 0.6382361650466919, 'logps/chosen': -99.16655731201172, 'logps/rejected': -118.57122039794922, 'logps/ref_chosen': -94.42288208007812, 'logps/ref_rejected': -103.88156127929688, 'logits/chosen': -4.741695404052734, 'logits/rejected': -3.933988094329834, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.06485415995121002, 'kl/avg_steps': 0.59375, 'epoch': 0.19}
19%|████████████████████▉ | 126/681 [07:15<28:24, 3.07s/it] 19%|█████████████████████ | 127/681 [07:19<28:53, 3.13s/it] {'loss': 1.0846, 'grad_norm': 20.271377563476562, 'learning_rate': 4.89374339765481e-07, 'rewards/chosen': -0.3097778856754303, 'rewards/rejected': -0.7753020524978638, 'rewards/accuracies': 0.75, 'rewards/margins': 0.4655241370201111, 'logps/chosen': -89.53435516357422, 'logps/rejected': -95.63729858398438, 'logps/ref_chosen': -84.75105285644531, 'logps/ref_rejected': -83.54523468017578, 'logits/chosen': -5.095927715301514, 'logits/rejected': -3.9779820442199707, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.06447136402130127, 'kl/avg_steps': 0.28125, 'epoch': 0.19}
19%|█████████████████████ | 127/681 [07:19<28:53, 3.13s/it] 19%|█████████████████████▏ | 128/681 [07:22<29:16, 3.18s/it] {'loss': 1.1062, 'grad_norm': 22.813669204711914, 'learning_rate': 4.890010211106795e-07, 'rewards/chosen': -0.29941076040267944, 'rewards/rejected': -0.7819145321846008, 'rewards/accuracies': 0.75, 'rewards/margins': 0.482503741979599, 'logps/chosen': -95.26617431640625, 'logps/rejected': -95.3511734008789, 'logps/ref_chosen': -90.61964416503906, 'logps/ref_rejected': -83.10347747802734, 'logits/chosen': -4.714390754699707, 'logits/rejected': -3.6033260822296143, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.06429054588079453, 'kl/avg_steps': 0.328125, 'epoch': 0.19}
19%|█████████████████████▏ | 128/681 [07:22<29:16, 3.18s/it] 19%|█████████████████████▍ | 129/681 [07:25<29:10, 3.17s/it] {'loss': 1.0793, 'grad_norm': 18.353992462158203, 'learning_rate': 4.88621404556699e-07, 'rewards/chosen': -0.32816505432128906, 'rewards/rejected': -0.8270788192749023, 'rewards/accuracies': 0.6875, 'rewards/margins': 0.4989137649536133, 'logps/chosen': -104.02716064453125, 'logps/rejected': -116.28022003173828, 'logps/ref_chosen': -98.91870880126953, 'logps/ref_rejected': -103.28419494628906, 'logits/chosen': -4.422687530517578, 'logits/rejected': -3.368746757507324, 'kl/p_epsilon_steps': 0.625, 'kl/n_epsilon_steps': 0.375, 'kl/beta': 0.06408028304576874, 'kl/avg_steps': 0.25, 'epoch': 0.19}
19%|█████████████████████▍ | 129/681 [07:25<29:10, 3.17s/it] 19%|█████████████████████▌ | 130/681 [07:28<28:25, 3.10s/it] {'loss': 0.9753, 'grad_norm': 16.212928771972656, 'learning_rate': 4.882355001067891e-07, 'rewards/chosen': -0.2548287510871887, 'rewards/rejected': -0.8984166383743286, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.6435878276824951, 'logps/chosen': -77.16912841796875, 'logps/rejected': -102.19236755371094, 'logps/ref_chosen': -73.18368530273438, 'logps/ref_rejected': -88.02154541015625, 'logits/chosen': -5.384788513183594, 'logits/rejected': -4.525052070617676, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.06392048299312592, 'kl/avg_steps': 0.5625, 'epoch': 0.19}
19%|█████████████████████▌ | 130/681 [07:28<28:25, 3.10s/it] 19%|█████████████████████▋ | 131/681 [07:31<27:59, 3.05s/it] {'loss': 1.0071, 'grad_norm': 18.157773971557617, 'learning_rate': 4.878433179298909e-07, 'rewards/chosen': -0.21306927502155304, 'rewards/rejected': -0.7927961349487305, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.5797268748283386, 'logps/chosen': -77.2987289428711, 'logps/rejected': -107.02981567382812, 'logps/ref_chosen': -73.9564437866211, 'logps/ref_rejected': -94.45582580566406, 'logits/chosen': -5.078728675842285, 'logits/rejected': -3.907397747039795, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.06356293708086014, 'kl/avg_steps': 0.5625, 'epoch': 0.19}
19%|█████████████████████▋ | 131/681 [07:31<27:59, 3.05s/it] 19%|█████████████████████▉ | 132/681 [07:34<27:40, 3.02s/it] {'loss': 1.0126, 'grad_norm': 21.45513153076172, 'learning_rate': 4.874448683603694e-07, 'rewards/chosen': -0.3079445958137512, 'rewards/rejected': -0.8823122978210449, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.5743677020072937, 'logps/chosen': -97.30502319335938, 'logps/rejected': -108.19319152832031, 'logps/ref_chosen': -92.43948364257812, 'logps/ref_rejected': -94.13251495361328, 'logits/chosen': -5.31199836730957, 'logits/rejected': -4.291113376617432, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.06320739537477493, 'kl/avg_steps': 0.59375, 'epoch': 0.19}
19%|█████████████████████▉ | 132/681 [07:34<27:40, 3.02s/it] 20%|██████████████████████ | 133/681 [07:37<27:43, 3.04s/it] {'loss': 0.9767, 'grad_norm': 17.602924346923828, 'learning_rate': 4.870401618977415e-07, 'rewards/chosen': -0.24647247791290283, 'rewards/rejected': -0.843220591545105, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.5967481732368469, 'logps/chosen': -97.0290298461914, 'logps/rejected': -115.60842895507812, 'logps/ref_chosen': -93.11666107177734, 'logps/ref_rejected': -102.09491729736328, 'logits/chosen': -5.221474647521973, 'logits/rejected': -4.042792320251465, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.06283431500196457, 'kl/avg_steps': 0.5, 'epoch': 0.2}
20%|██████████████████████ | 133/681 [07:37<27:43, 3.04s/it] 20%|██████████████████████▏ | 134/681 [07:40<27:00, 2.96s/it] {'loss': 0.9528, 'grad_norm': 16.935972213745117, 'learning_rate': 4.866292092063986e-07, 'rewards/chosen': -0.26006680727005005, 'rewards/rejected': -0.8904794454574585, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6304125785827637, 'logps/chosen': -87.7016830444336, 'logps/rejected': -108.22308349609375, 'logps/ref_chosen': -83.55012512207031, 'logps/ref_rejected': -93.8815689086914, 'logits/chosen': -5.074636936187744, 'logits/rejected': -4.259199142456055, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.06252171099185944, 'kl/avg_steps': 0.4375, 'epoch': 0.2}
20%|██████████████████████▏ | 134/681 [07:40<27:00, 2.96s/it] 20%|██████████████████████▍ | 135/681 [07:43<27:13, 2.99s/it] {'loss': 0.9797, 'grad_norm': 21.895069122314453, 'learning_rate': 4.862120211153265e-07, 'rewards/chosen': -0.24222984910011292, 'rewards/rejected': -0.8613507747650146, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.6191209554672241, 'logps/chosen': -81.46526336669922, 'logps/rejected': -134.99432373046875, 'logps/ref_chosen': -77.58262634277344, 'logps/ref_rejected': -121.05648803710938, 'logits/chosen': -4.999515533447266, 'logits/rejected': -3.6168813705444336, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.06224936991930008, 'kl/avg_steps': 0.5625, 'epoch': 0.2}
20%|██████████████████████▍ | 135/681 [07:43<27:13, 2.99s/it] 20%|██████████████████████▌ | 136/681 [07:46<28:05, 3.09s/it] {'loss': 0.9543, 'grad_norm': 17.361907958984375, 'learning_rate': 4.857886086178193e-07, 'rewards/chosen': -0.2606146037578583, 'rewards/rejected': -0.8741957545280457, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.613581120967865, 'logps/chosen': -98.3825454711914, 'logps/rejected': -117.48695373535156, 'logps/ref_chosen': -94.1727294921875, 'logps/ref_rejected': -103.26428985595703, 'logits/chosen': -5.6198625564575195, 'logits/rejected': -4.077550888061523, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.06190117448568344, 'kl/avg_steps': 0.46875, 'epoch': 0.2}
20%|██████████████████████▌ | 136/681 [07:46<28:05, 3.09s/it] 20%|██████████████████████▋ | 137/681 [07:49<28:14, 3.12s/it] {'loss': 0.8065, 'grad_norm': 18.447507858276367, 'learning_rate': 4.853589828711902e-07, 'rewards/chosen': -0.20868739485740662, 'rewards/rejected': -1.1479192972183228, 'rewards/accuracies': 0.921875, 'rewards/margins': 0.9392319321632385, 'logps/chosen': -81.5108642578125, 'logps/rejected': -136.72076416015625, 'logps/ref_chosen': -78.11874389648438, 'logps/ref_rejected': -117.92237091064453, 'logits/chosen': -5.679508686065674, 'logits/rejected': -4.036385536193848, 'kl/p_epsilon_steps': 0.90625, 'kl/n_epsilon_steps': 0.09375, 'kl/beta': 0.06161236763000488, 'kl/avg_steps': 0.8125, 'epoch': 0.2}
20%|██████████████████████▋ | 137/681 [07:49<28:14, 3.12s/it] 20%|██████████████████████▉ | 138/681 [07:52<27:17, 3.01s/it] {'loss': 0.992, 'grad_norm': 16.60499382019043, 'learning_rate': 4.849231551964771e-07, 'rewards/chosen': -0.35308200120925903, 'rewards/rejected': -0.9304844737052917, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.5774024724960327, 'logps/chosen': -98.61029815673828, 'logps/rejected': -113.98179626464844, 'logps/ref_chosen': -92.82994079589844, 'logps/ref_rejected': -98.64839935302734, 'logits/chosen': -4.7099151611328125, 'logits/rejected': -4.268791198730469, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.061115801334381104, 'kl/avg_steps': 0.5625, 'epoch': 0.2}
20%|██████████████████████▉ | 138/681 [07:52<27:17, 3.01s/it] 20%|███████████████████████ | 139/681 [07:55<26:43, 2.96s/it] {'loss': 0.9595, 'grad_norm': 16.9788875579834, 'learning_rate': 4.844811370781446e-07, 'rewards/chosen': -0.23357324302196503, 'rewards/rejected': -0.9010961055755615, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.6675229072570801, 'logps/chosen': -82.84307861328125, 'logps/rejected': -100.77876281738281, 'logps/ref_chosen': -79.00301361083984, 'logps/ref_rejected': -85.82733154296875, 'logits/chosen': -5.217746734619141, 'logits/rejected': -4.438928127288818, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.06077394634485245, 'kl/avg_steps': 0.65625, 'epoch': 0.2}
20%|███████████████████████ | 139/681 [07:55<26:43, 2.96s/it] 21%|███████████████████████▏ | 140/681 [07:58<26:13, 2.91s/it] {'loss': 0.9655, 'grad_norm': 15.984687805175781, 'learning_rate': 4.840329401637809e-07, 'rewards/chosen': -0.24829351902008057, 'rewards/rejected': -0.8534231781959534, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.6051296591758728, 'logps/chosen': -92.46583557128906, 'logps/rejected': -104.81709289550781, 'logps/ref_chosen': -88.35244750976562, 'logps/ref_rejected': -90.57477569580078, 'logits/chosen': -5.573225975036621, 'logits/rejected': -4.482275485992432, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.06037771701812744, 'kl/avg_steps': 0.5, 'epoch': 0.21}
21%|███████████████████████▏ | 140/681 [07:58<26:13, 2.91s/it] 21%|███████████████████████▍ | 141/681 [08:01<26:46, 2.97s/it] {'loss': 0.9503, 'grad_norm': 22.27216911315918, 'learning_rate': 4.83578576263792e-07, 'rewards/chosen': -0.3238122761249542, 'rewards/rejected': -1.0087032318115234, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.6848909854888916, 'logps/chosen': -112.61961364746094, 'logps/rejected': -121.64073181152344, 'logps/ref_chosen': -107.22016906738281, 'logps/ref_rejected': -104.71223449707031, 'logits/chosen': -5.062009811401367, 'logits/rejected': -4.248960018157959, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.06007733196020126, 'kl/avg_steps': 0.65625, 'epoch': 0.21}
21%|███████████████████████▍ | 141/681 [08:01<26:46, 2.97s/it] 21%|███████████████████████▌ | 142/681 [08:04<27:27, 3.06s/it] {'loss': 0.9498, 'grad_norm': 16.56783676147461, 'learning_rate': 4.83118057351089e-07, 'rewards/chosen': -0.34996697306632996, 'rewards/rejected': -1.0082789659500122, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6583119630813599, 'logps/chosen': -94.61272430419922, 'logps/rejected': -116.76524353027344, 'logps/ref_chosen': -88.72894287109375, 'logps/ref_rejected': -99.73038482666016, 'logits/chosen': -5.5315375328063965, 'logits/rejected': -4.753322601318359, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.05968564376235008, 'kl/avg_steps': 0.5625, 'epoch': 0.21}
21%|███████████████████████▌ | 142/681 [08:04<27:27, 3.06s/it] 21%|███████████████████████▋ | 143/681 [08:07<27:21, 3.05s/it] {'loss': 1.0174, 'grad_norm': 16.937776565551758, 'learning_rate': 4.826513955607734e-07, 'rewards/chosen': -0.2356262505054474, 'rewards/rejected': -0.7835693359375, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.547943115234375, 'logps/chosen': -89.47615051269531, 'logps/rejected': -99.1048583984375, 'logps/ref_chosen': -85.50457763671875, 'logps/ref_rejected': -85.8056640625, 'logits/chosen': -5.914580345153809, 'logits/rejected': -4.424531936645508, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.05935179069638252, 'kl/avg_steps': 0.375, 'epoch': 0.21}
21%|███████████████████████▋ | 143/681 [08:07<27:21, 3.05s/it] 21%|███████████████████████▉ | 144/681 [08:10<27:59, 3.13s/it] {'loss': 1.0494, 'grad_norm': 22.844528198242188, 'learning_rate': 4.821786031898176e-07, 'rewards/chosen': -0.2811855673789978, 'rewards/rejected': -0.8360341787338257, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.5548486113548279, 'logps/chosen': -109.42718505859375, 'logps/rejected': -101.95695495605469, 'logps/ref_chosen': -104.68708801269531, 'logps/ref_rejected': -87.71908569335938, 'logits/chosen': -5.4832234382629395, 'logits/rejected': -4.616265296936035, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.05913005396723747, 'kl/avg_steps': 0.4375, 'epoch': 0.21}
21%|███████████████████████▉ | 144/681 [08:10<27:59, 3.13s/it] 21%|████████████████████████ | 145/681 [08:13<27:30, 3.08s/it] {'loss': 0.948, 'grad_norm': 17.371044158935547, 'learning_rate': 4.816996926967401e-07, 'rewards/chosen': -0.24210438132286072, 'rewards/rejected': -0.8907926678657532, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6486882567405701, 'logps/chosen': -92.62870788574219, 'logps/rejected': -99.01193237304688, 'logps/ref_chosen': -88.51988983154297, 'logps/ref_rejected': -83.76513671875, 'logits/chosen': -5.808310031890869, 'logits/rejected': -4.368202209472656, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.058872487396001816, 'kl/avg_steps': 0.5, 'epoch': 0.21}
21%|████████████████████████ | 145/681 [08:13<27:30, 3.08s/it] 21%|████████████████████████▏ | 146/681 [08:16<27:14, 3.05s/it] {'loss': 0.9083, 'grad_norm': 17.89594841003418, 'learning_rate': 4.812146767012779e-07, 'rewards/chosen': -0.2555050253868103, 'rewards/rejected': -0.9711916446685791, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.715686559677124, 'logps/chosen': -107.27189636230469, 'logps/rejected': -105.66189575195312, 'logps/ref_chosen': -102.9058609008789, 'logps/ref_rejected': -88.9493637084961, 'logits/chosen': -5.770585060119629, 'logits/rejected': -4.961735725402832, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.05857958644628525, 'kl/avg_steps': 0.59375, 'epoch': 0.21}
21%|████████████████████████▏ | 146/681 [08:16<27:14, 3.05s/it] 22%|████████████████████████▍ | 147/681 [08:19<27:02, 3.04s/it] {'loss': 0.8803, 'grad_norm': 16.04936408996582, 'learning_rate': 4.807235679840536e-07, 'rewards/chosen': -0.2465764731168747, 'rewards/rejected': -0.9858845472335815, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7393079996109009, 'logps/chosen': -93.50482177734375, 'logps/rejected': -93.5650405883789, 'logps/ref_chosen': -89.26219177246094, 'logps/ref_rejected': -76.50011444091797, 'logits/chosen': -6.048250198364258, 'logits/rejected': -5.216745853424072, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.058233823627233505, 'kl/avg_steps': 0.5625, 'epoch': 0.22}
22%|████████████████████████▍ | 147/681 [08:19<27:02, 3.04s/it] 22%|████████████████████████▌ | 148/681 [08:22<27:02, 3.04s/it] {'loss': 0.962, 'grad_norm': 17.10126304626465, 'learning_rate': 4.802263794862384e-07, 'rewards/chosen': -0.15071691572666168, 'rewards/rejected': -0.8026539087295532, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.6519370079040527, 'logps/chosen': -94.35877990722656, 'logps/rejected': -124.3576889038086, 'logps/ref_chosen': -91.76287841796875, 'logps/ref_rejected': -110.38552856445312, 'logits/chosen': -5.5003767013549805, 'logits/rejected': -3.983274459838867, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.05790809169411659, 'kl/avg_steps': 0.375, 'epoch': 0.22}
22%|████████████████████████▌ | 148/681 [08:22<27:02, 3.04s/it] 22%|████████████████████████▋ | 149/681 [08:26<27:18, 3.08s/it] {'loss': 0.9992, 'grad_norm': 17.85907745361328, 'learning_rate': 4.797231243092118e-07, 'rewards/chosen': -0.3102826178073883, 'rewards/rejected': -0.9375884532928467, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6273058652877808, 'logps/chosen': -98.8444595336914, 'logps/rejected': -121.99969482421875, 'logps/ref_chosen': -93.4840087890625, 'logps/ref_rejected': -105.64369201660156, 'logits/chosen': -5.346034049987793, 'logits/rejected': -4.409138202667236, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.057691749185323715, 'kl/avg_steps': 0.4375, 'epoch': 0.22}
22%|████████████████████████▋ | 149/681 [08:26<27:18, 3.08s/it] 22%|████████████████████████▉ | 150/681 [08:29<27:08, 3.07s/it] {'loss': 0.9614, 'grad_norm': 15.73752498626709, 'learning_rate': 4.792138157142157e-07, 'rewards/chosen': -0.24165549874305725, 'rewards/rejected': -0.907067060470581, 'rewards/accuracies': 0.875, 'rewards/margins': 0.6654115915298462, 'logps/chosen': -79.96965026855469, 'logps/rejected': -105.93204498291016, 'logps/ref_chosen': -75.7786865234375, 'logps/ref_rejected': -90.02676391601562, 'logits/chosen': -5.399765968322754, 'logits/rejected': -4.2668867111206055, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.05744044482707977, 'kl/avg_steps': 0.5625, 'epoch': 0.22}
22%|████████████████████████▉ | 150/681 [08:29<27:08, 3.07s/it] 22%|█████████████████████████ | 151/681 [08:32<26:46, 3.03s/it] {'loss': 0.9827, 'grad_norm': 19.85856056213379, 'learning_rate': 4.786984671220053e-07, 'rewards/chosen': -0.3297105133533478, 'rewards/rejected': -0.9636145234107971, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.633903980255127, 'logps/chosen': -108.49866485595703, 'logps/rejected': -123.03112030029297, 'logps/ref_chosen': -102.73422241210938, 'logps/ref_rejected': -106.03781127929688, 'logits/chosen': -5.871485710144043, 'logits/rejected': -4.4503984451293945, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.05711914971470833, 'kl/avg_steps': 0.5, 'epoch': 0.22}
22%|█████████████████████████ | 151/681 [08:32<26:46, 3.03s/it] 22%|█████████████████████████▏ | 152/681 [08:35<27:23, 3.11s/it] {'loss': 0.8941, 'grad_norm': 16.81842803955078, 'learning_rate': 4.78177092112495e-07, 'rewards/chosen': -0.1701817512512207, 'rewards/rejected': -0.9250707626342773, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7548890113830566, 'logps/chosen': -100.57263946533203, 'logps/rejected': -126.90365600585938, 'logps/ref_chosen': -97.59074401855469, 'logps/ref_rejected': -110.49913024902344, 'logits/chosen': -4.908111572265625, 'logits/rejected': -3.836758613586426, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.056834977120161057, 'kl/avg_steps': 0.59375, 'epoch': 0.22}
22%|█████████████████████████▏ | 152/681 [08:35<27:23, 3.11s/it] 22%|█████████████████████████▍ | 153/681 [08:38<27:09, 3.09s/it] {'loss': 1.0443, 'grad_norm': 17.356094360351562, 'learning_rate': 4.776497044244016e-07, 'rewards/chosen': -0.3063035011291504, 'rewards/rejected': -0.8451663255691528, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.5388628244400024, 'logps/chosen': -92.43611907958984, 'logps/rejected': -106.13377380371094, 'logps/ref_chosen': -87.02560424804688, 'logps/ref_rejected': -91.06853485107422, 'logits/chosen': -5.573543548583984, 'logits/rejected': -4.830375671386719, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.05649951100349426, 'kl/avg_steps': 0.4375, 'epoch': 0.22}
22%|█████████████████████████▍ | 153/681 [08:38<27:09, 3.09s/it] 23%|█████████████████████████▌ | 154/681 [08:41<26:51, 3.06s/it] {'loss': 0.8199, 'grad_norm': 15.954586029052734, 'learning_rate': 4.771163179548808e-07, 'rewards/chosen': -0.21082431077957153, 'rewards/rejected': -1.0753536224365234, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.8645293116569519, 'logps/chosen': -94.85427856445312, 'logps/rejected': -130.63352966308594, 'logps/ref_chosen': -91.10566711425781, 'logps/ref_rejected': -111.37238311767578, 'logits/chosen': -5.686001300811768, 'logits/rejected': -4.457275390625, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.056253399699926376, 'kl/avg_steps': 0.46875, 'epoch': 0.23}
23%|█████████████████████████▌ | 154/681 [08:41<26:51, 3.06s/it] 23%|█████████████████████████▋ | 155/681 [08:44<26:53, 3.07s/it] {'loss': 1.0092, 'grad_norm': 23.60336685180664, 'learning_rate': 4.7657694675916247e-07, 'rewards/chosen': -0.3092682957649231, 'rewards/rejected': -0.9719428420066833, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6626745462417603, 'logps/chosen': -100.05500793457031, 'logps/rejected': -102.24840545654297, 'logps/ref_chosen': -94.56097412109375, 'logps/ref_rejected': -84.77359008789062, 'logits/chosen': -5.547731399536133, 'logits/rejected': -5.115335464477539, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.05599094182252884, 'kl/avg_steps': 0.4375, 'epoch': 0.23}
23%|█████████████████████████▋ | 155/681 [08:44<26:53, 3.07s/it] 23%|█████████████████████████▉ | 156/681 [08:47<26:59, 3.08s/it] {'loss': 1.0161, 'grad_norm': 19.952051162719727, 'learning_rate': 4.7603160505017893e-07, 'rewards/chosen': -0.38445067405700684, 'rewards/rejected': -0.9737200736999512, 'rewards/accuracies': 0.75, 'rewards/margins': 0.5892693400382996, 'logps/chosen': -101.51638793945312, 'logps/rejected': -101.98014831542969, 'logps/ref_chosen': -94.65538787841797, 'logps/ref_rejected': -84.41934204101562, 'logits/chosen': -5.7792768478393555, 'logits/rejected': -4.890432357788086, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.055747050791978836, 'kl/avg_steps': 0.359375, 'epoch': 0.23}
23%|█████████████████████████▉ | 156/681 [08:47<26:59, 3.08s/it] 23%|██████████████████████████ | 157/681 [08:50<25:59, 2.98s/it] {'loss': 0.8564, 'grad_norm': 17.198551177978516, 'learning_rate': 4.7548030719819154e-07, 'rewards/chosen': -0.35640445351600647, 'rewards/rejected': -1.1438487768173218, 'rewards/accuracies': 0.875, 'rewards/margins': 0.7874443531036377, 'logps/chosen': -103.44956970214844, 'logps/rejected': -132.464111328125, 'logps/ref_chosen': -97.01239013671875, 'logps/ref_rejected': -111.70133972167969, 'logits/chosen': -6.013505458831787, 'logits/rejected': -4.734508037567139, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.055547427386045456, 'kl/avg_steps': 0.6875, 'epoch': 0.23}
23%|██████████████████████████ | 157/681 [08:50<25:59, 2.98s/it] 23%|██████████████████████████▏ | 158/681 [08:53<26:36, 3.05s/it] {'loss': 0.8512, 'grad_norm': 16.71400260925293, 'learning_rate': 4.7492306773041136e-07, 'rewards/chosen': -0.316554993391037, 'rewards/rejected': -1.1312541961669922, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.814699113368988, 'logps/chosen': -100.4566650390625, 'logps/rejected': -141.5926971435547, 'logps/ref_chosen': -94.71295928955078, 'logps/ref_rejected': -120.93733978271484, 'logits/chosen': -5.91513204574585, 'logits/rejected': -4.732603073120117, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.05516814440488815, 'kl/avg_steps': 0.59375, 'epoch': 0.23}
23%|██████████████████████████▏ | 158/681 [08:53<26:36, 3.05s/it] 23%|██████████████████████████▍ | 159/681 [08:56<27:15, 3.13s/it] {'loss': 0.9148, 'grad_norm': 15.837252616882324, 'learning_rate': 4.743599013306165e-07, 'rewards/chosen': -0.2735292613506317, 'rewards/rejected': -1.0276559591293335, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7541267275810242, 'logps/chosen': -113.19328308105469, 'logps/rejected': -113.71253204345703, 'logps/ref_chosen': -108.2182388305664, 'logps/ref_rejected': -94.83901977539062, 'logits/chosen': -6.067984104156494, 'logits/rejected': -5.724274158477783, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.0548425167798996, 'kl/avg_steps': 0.484375, 'epoch': 0.23}
23%|██████████████████████████▍ | 159/681 [08:56<27:15, 3.13s/it] 23%|██████████████████████████▌ | 160/681 [09:00<27:16, 3.14s/it] {'loss': 0.9092, 'grad_norm': 15.891210556030273, 'learning_rate': 4.737908228387656e-07, 'rewards/chosen': -0.22743016481399536, 'rewards/rejected': -0.9378353953361511, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7104052305221558, 'logps/chosen': -102.32695770263672, 'logps/rejected': -119.59290313720703, 'logps/ref_chosen': -98.16340637207031, 'logps/ref_rejected': -102.2809829711914, 'logits/chosen': -6.017996788024902, 'logits/rejected': -4.622193336486816, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.054578155279159546, 'kl/avg_steps': 0.53125, 'epoch': 0.23}
23%|██████████████████████████▌ | 160/681 [09:00<27:16, 3.14s/it] 24%|██████████████████████████▋ | 161/681 [09:02<25:54, 2.99s/it] {'loss': 0.9714, 'grad_norm': 17.31453514099121, 'learning_rate': 4.7321584725060594e-07, 'rewards/chosen': -0.32883331179618835, 'rewards/rejected': -0.9861065149307251, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.6572731733322144, 'logps/chosen': -92.63033294677734, 'logps/rejected': -107.30801391601562, 'logps/ref_chosen': -86.59187316894531, 'logps/ref_rejected': -89.01643371582031, 'logits/chosen': -6.470645904541016, 'logits/rejected': -5.285548210144043, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.054289739578962326, 'kl/avg_steps': 0.4375, 'epoch': 0.24}
24%|██████████████████████████▋ | 161/681 [09:02<25:54, 2.99s/it] 24%|██████████████████████████▉ | 162/681 [09:05<26:33, 3.07s/it] {'loss': 0.9167, 'grad_norm': 15.398876190185547, 'learning_rate': 4.7263498971727905e-07, 'rewards/chosen': -0.278072714805603, 'rewards/rejected': -0.9862060546875, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.7081333994865417, 'logps/chosen': -102.532470703125, 'logps/rejected': -114.22633361816406, 'logps/ref_chosen': -97.40288543701172, 'logps/ref_rejected': -95.86923217773438, 'logits/chosen': -5.194693088531494, 'logits/rejected': -4.573456764221191, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.054053258150815964, 'kl/avg_steps': 0.5625, 'epoch': 0.24}
24%|██████████████████████████▉ | 162/681 [09:05<26:33, 3.07s/it] 24%|███████████████████████████ | 163/681 [09:08<26:19, 3.05s/it] {'loss': 0.8482, 'grad_norm': 13.848814964294434, 'learning_rate': 4.720482655449212e-07, 'rewards/chosen': -0.26623615622520447, 'rewards/rejected': -1.1262279748916626, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8599917888641357, 'logps/chosen': -87.13270568847656, 'logps/rejected': -103.97894287109375, 'logps/ref_chosen': -82.1753921508789, 'logps/ref_rejected': -82.86473846435547, 'logits/chosen': -6.277469635009766, 'logits/rejected': -5.564417839050293, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.053750909864902496, 'kl/avg_steps': 0.5625, 'epoch': 0.24}
24%|███████████████████████████ | 163/681 [09:08<26:19, 3.05s/it] 24%|███████████████████████████▏ | 164/681 [09:11<26:01, 3.02s/it] {'loss': 1.0038, 'grad_norm': 17.024089813232422, 'learning_rate': 4.714556901942599e-07, 'rewards/chosen': -0.4347628355026245, 'rewards/rejected': -1.013280987739563, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.5785181522369385, 'logps/chosen': -98.60881805419922, 'logps/rejected': -106.09263610839844, 'logps/ref_chosen': -90.4742660522461, 'logps/ref_rejected': -87.02809143066406, 'logits/chosen': -5.856521129608154, 'logits/rejected': -4.963005542755127, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.34375, 'kl/beta': 0.05345025286078453, 'kl/avg_steps': 0.3125, 'epoch': 0.24}
24%|███████████████████████████▏ | 164/681 [09:11<26:01, 3.02s/it] 24%|███████████████████████████▍ | 165/681 [09:14<25:53, 3.01s/it] {'loss': 0.857, 'grad_norm': 14.025198936462402, 'learning_rate': 4.708572792802069e-07, 'rewards/chosen': -0.27433332800865173, 'rewards/rejected': -1.1087267398834229, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8343935012817383, 'logps/chosen': -92.47659301757812, 'logps/rejected': -102.87800598144531, 'logps/ref_chosen': -87.313232421875, 'logps/ref_rejected': -81.902587890625, 'logits/chosen': -5.403741359710693, 'logits/rejected': -5.085051536560059, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.05328373983502388, 'kl/avg_steps': 0.625, 'epoch': 0.24}
24%|███████████████████████████▍ | 165/681 [09:14<25:53, 3.01s/it] 24%|███████████████████████████▌ | 166/681 [09:17<25:30, 2.97s/it] {'loss': 0.9066, 'grad_norm': 15.63122844696045, 'learning_rate': 4.702530485714461e-07, 'rewards/chosen': -0.49418699741363525, 'rewards/rejected': -1.2831027507781982, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7889156341552734, 'logps/chosen': -90.94509887695312, 'logps/rejected': -127.57855224609375, 'logps/ref_chosen': -81.5860595703125, 'logps/ref_rejected': -103.16517639160156, 'logits/chosen': -6.5487165451049805, 'logits/rejected': -5.4730329513549805, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.052952785044908524, 'kl/avg_steps': 0.59375, 'epoch': 0.24}
24%|███████████████████████████▌ | 166/681 [09:17<25:30, 2.97s/it] 25%|███████████████████████████▋ | 167/681 [09:21<26:30, 3.09s/it] {'loss': 0.8129, 'grad_norm': 14.752199172973633, 'learning_rate': 4.6964301399001877e-07, 'rewards/chosen': -0.4190889000892639, 'rewards/rejected': -1.282209873199463, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.8631209135055542, 'logps/chosen': -91.57254028320312, 'logps/rejected': -123.61196899414062, 'logps/ref_chosen': -83.56167602539062, 'logps/ref_rejected': -99.05305480957031, 'logits/chosen': -6.093404769897461, 'logits/rejected': -5.437514305114746, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.05264023318886757, 'kl/avg_steps': 0.59375, 'epoch': 0.25}
25%|███████████████████████████▋ | 167/681 [09:21<26:30, 3.09s/it] 25%|███████████████████████████▉ | 168/681 [09:24<26:24, 3.09s/it] {'loss': 0.9032, 'grad_norm': 16.830984115600586, 'learning_rate': 4.690271916109034e-07, 'rewards/chosen': -0.4084423780441284, 'rewards/rejected': -1.1381680965423584, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.7297256588935852, 'logps/chosen': -87.75483703613281, 'logps/rejected': -102.02044677734375, 'logps/ref_chosen': -79.93124389648438, 'logps/ref_rejected': -80.10216522216797, 'logits/chosen': -6.080756664276123, 'logits/rejected': -5.517510414123535, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.05232952535152435, 'kl/avg_steps': 0.5625, 'epoch': 0.25}
25%|███████████████████████████▉ | 168/681 [09:24<26:24, 3.09s/it] 25%|████████████████████████████ | 169/681 [09:27<26:49, 3.14s/it] {'loss': 0.8642, 'grad_norm': 15.615654945373535, 'learning_rate': 4.6840559766159235e-07, 'rewards/chosen': -0.33170753717422485, 'rewards/rejected': -1.149538516998291, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8178310394287109, 'logps/chosen': -89.30410766601562, 'logps/rejected': -112.40554809570312, 'logps/ref_chosen': -82.90370178222656, 'logps/ref_rejected': -90.12191009521484, 'logits/chosen': -5.397192001342773, 'logits/rejected': -5.175437927246094, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.052036818116903305, 'kl/avg_steps': 0.625, 'epoch': 0.25}
25%|████████████████████████████ | 169/681 [09:27<26:49, 3.14s/it] 25%|████████████████████████████▏ | 170/681 [09:30<26:42, 3.14s/it] {'loss': 0.8871, 'grad_norm': 16.472681045532227, 'learning_rate': 4.6777824852166437e-07, 'rewards/chosen': -0.4205918610095978, 'rewards/rejected': -1.146958589553833, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7263666987419128, 'logps/chosen': -83.11984252929688, 'logps/rejected': -104.49899291992188, 'logps/ref_chosen': -74.98197174072266, 'logps/ref_rejected': -82.16560363769531, 'logits/chosen': -5.991590976715088, 'logits/rejected': -5.608396053314209, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.05171360820531845, 'kl/avg_steps': 0.5625, 'epoch': 0.25}
25%|████████████████████████████▏ | 170/681 [09:30<26:42, 3.14s/it] 25%|████████████████████████████▎ | 171/681 [09:33<25:40, 3.02s/it] {'loss': 0.9253, 'grad_norm': 19.757944107055664, 'learning_rate': 4.6714516072235273e-07, 'rewards/chosen': -0.5009539127349854, 'rewards/rejected': -1.3104692697525024, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8095153570175171, 'logps/chosen': -128.7462158203125, 'logps/rejected': -141.45156860351562, 'logps/ref_chosen': -118.97853088378906, 'logps/ref_rejected': -115.7693099975586, 'logits/chosen': -5.451948165893555, 'logits/rejected': -4.568760871887207, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.05142434686422348, 'kl/avg_steps': 0.46875, 'epoch': 0.25}
25%|████████████████████████████▎ | 171/681 [09:33<25:40, 3.02s/it] 25%|████████████████████████████▌ | 172/681 [09:36<25:11, 2.97s/it] {'loss': 0.8464, 'grad_norm': 14.202917098999023, 'learning_rate': 4.6650635094610966e-07, 'rewards/chosen': -0.5407711267471313, 'rewards/rejected': -1.2872977256774902, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.7465265989303589, 'logps/chosen': -111.73493957519531, 'logps/rejected': -116.28710174560547, 'logps/ref_chosen': -101.15379333496094, 'logps/ref_rejected': -91.00567626953125, 'logits/chosen': -6.403740882873535, 'logits/rejected': -5.487905025482178, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.05118441954255104, 'kl/avg_steps': 0.46875, 'epoch': 0.25}
25%|████████████████████████████▌ | 172/681 [09:36<25:11, 2.97s/it] 25%|████████████████████████████▋ | 173/681 [09:39<25:15, 2.98s/it] {'loss': 0.9611, 'grad_norm': 14.85136604309082, 'learning_rate': 4.6586183602616687e-07, 'rewards/chosen': -0.5136967897415161, 'rewards/rejected': -1.1345391273498535, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.620842456817627, 'logps/chosen': -109.14393615722656, 'logps/rejected': -106.2504653930664, 'logps/ref_chosen': -99.05668640136719, 'logps/ref_rejected': -83.8507308959961, 'logits/chosen': -5.471677780151367, 'logits/rejected': -5.009877681732178, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.05094561353325844, 'kl/avg_steps': 0.46875, 'epoch': 0.25}
25%|████████████████████████████▋ | 173/681 [09:39<25:15, 2.98s/it] 26%|████████████████████████████▊ | 174/681 [09:42<25:29, 3.02s/it] {'loss': 0.8609, 'grad_norm': 16.939958572387695, 'learning_rate': 4.652116329460919e-07, 'rewards/chosen': -0.49034202098846436, 'rewards/rejected': -1.3025699853897095, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8122279644012451, 'logps/chosen': -90.09979248046875, 'logps/rejected': -134.08053588867188, 'logps/ref_chosen': -80.40890502929688, 'logps/ref_rejected': -108.20171356201172, 'logits/chosen': -6.058362007141113, 'logits/rejected': -5.3862104415893555, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.050707921385765076, 'kl/avg_steps': 0.625, 'epoch': 0.26}
26%|████████████████████████████▊ | 174/681 [09:42<25:29, 3.02s/it] 26%|█████████████████████████████ | 175/681 [09:45<26:04, 3.09s/it] {'loss': 0.8423, 'grad_norm': 16.927989959716797, 'learning_rate': 4.645557588393406e-07, 'rewards/chosen': -0.6450769305229187, 'rewards/rejected': -1.4552021026611328, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.8101252317428589, 'logps/chosen': -95.58826446533203, 'logps/rejected': -125.78546142578125, 'logps/ref_chosen': -82.71649169921875, 'logps/ref_rejected': -96.66712951660156, 'logits/chosen': -6.3116278648376465, 'logits/rejected': -5.949856758117676, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.050392962992191315, 'kl/avg_steps': 0.78125, 'epoch': 0.26}
26%|█████████████████████████████ | 175/681 [09:45<26:04, 3.09s/it] 26%|█████████████████████████████▏ | 176/681 [09:48<25:03, 2.98s/it] {'loss': 0.9033, 'grad_norm': 15.526463508605957, 'learning_rate': 4.638942309888058e-07, 'rewards/chosen': -0.5223788619041443, 'rewards/rejected': -1.3372724056243896, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8148936629295349, 'logps/chosen': -90.09957885742188, 'logps/rejected': -129.64431762695312, 'logps/ref_chosen': -79.62930297851562, 'logps/ref_rejected': -102.6885986328125, 'logits/chosen': -6.382523059844971, 'logits/rejected': -5.265742778778076, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.050002321600914, 'kl/avg_steps': 0.53125, 'epoch': 0.26}
26%|█████████████████████████████▏ | 176/681 [09:48<25:03, 2.98s/it] 26%|█████████████████████████████▎ | 177/681 [09:51<25:15, 3.01s/it] {'loss': 0.807, 'grad_norm': 14.051129341125488, 'learning_rate': 4.6322706682636137e-07, 'rewards/chosen': -0.5626887083053589, 'rewards/rejected': -1.5128169059753418, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.9501281380653381, 'logps/chosen': -103.93224334716797, 'logps/rejected': -133.99789428710938, 'logps/ref_chosen': -92.56109619140625, 'logps/ref_rejected': -103.33838653564453, 'logits/chosen': -5.991172790527344, 'logits/rejected': -5.463325500488281, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.0497380867600441, 'kl/avg_steps': 0.65625, 'epoch': 0.26}
26%|█████████████████████████████▎ | 177/681 [09:51<25:15, 3.01s/it] 26%|█████████████████████████████▌ | 178/681 [09:54<25:00, 2.98s/it] {'loss': 0.7237, 'grad_norm': 16.428043365478516, 'learning_rate': 4.6255428393240354e-07, 'rewards/chosen': -0.5214236974716187, 'rewards/rejected': -1.6175870895385742, 'rewards/accuracies': 0.9375, 'rewards/margins': 1.096163272857666, 'logps/chosen': -94.86000061035156, 'logps/rejected': -143.80020141601562, 'logps/ref_chosen': -84.26579284667969, 'logps/ref_rejected': -110.79730987548828, 'logits/chosen': -6.9524335861206055, 'logits/rejected': -6.058289527893066, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.049413807690143585, 'kl/avg_steps': 0.75, 'epoch': 0.26}
26%|█████████████████████████████▌ | 178/681 [09:54<25:00, 2.98s/it] 26%|█████████████████████████████▋ | 179/681 [09:57<25:32, 3.05s/it] {'loss': 0.9762, 'grad_norm': 18.81154441833496, 'learning_rate': 4.6187590003538724e-07, 'rewards/chosen': -0.6620521545410156, 'rewards/rejected': -1.3974061012268066, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.735353946685791, 'logps/chosen': -106.93254089355469, 'logps/rejected': -122.918212890625, 'logps/ref_chosen': -93.4479751586914, 'logps/ref_rejected': -94.25540924072266, 'logits/chosen': -6.702276229858398, 'logits/rejected': -5.42424201965332, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.04904596507549286, 'kl/avg_steps': 0.4375, 'epoch': 0.26}
26%|█████████████████████████████▋ | 179/681 [09:57<25:32, 3.05s/it] 26%|█████████████████████████████▊ | 180/681 [10:00<24:55, 2.99s/it] {'loss': 0.8081, 'grad_norm': 19.117578506469727, 'learning_rate': 4.611919330113591e-07, 'rewards/chosen': -0.5459901094436646, 'rewards/rejected': -1.4474952220916748, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.901505172252655, 'logps/chosen': -100.06759643554688, 'logps/rejected': -133.78744506835938, 'logps/ref_chosen': -88.83554077148438, 'logps/ref_rejected': -103.90696716308594, 'logits/chosen': -5.92899751663208, 'logits/rejected': -5.633153915405273, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.04883232340216637, 'kl/avg_steps': 0.6875, 'epoch': 0.26}
26%|█████████████████████████████▊ | 180/681 [10:00<24:55, 2.99s/it] 27%|██████████████████████████████ | 181/681 [10:03<25:22, 3.04s/it] {'loss': 0.9401, 'grad_norm': 16.675270080566406, 'learning_rate': 4.605024008834863e-07, 'rewards/chosen': -0.5707411766052246, 'rewards/rejected': -1.266374111175537, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6956329345703125, 'logps/chosen': -91.0196762084961, 'logps/rejected': -95.4429931640625, 'logps/ref_chosen': -79.23396301269531, 'logps/ref_rejected': -69.17442321777344, 'logits/chosen': -6.083570957183838, 'logits/rejected': -5.588318824768066, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.04849889129400253, 'kl/avg_steps': 0.453125, 'epoch': 0.27}
27%|██████████████████████████████ | 181/681 [10:03<25:22, 3.04s/it] 27%|██████████████████████████████▏ | 182/681 [10:06<25:20, 3.05s/it] {'loss': 0.8622, 'grad_norm': 15.468544006347656, 'learning_rate': 4.598073218215817e-07, 'rewards/chosen': -0.7034212350845337, 'rewards/rejected': -1.6029670238494873, 'rewards/accuracies': 0.875, 'rewards/margins': 0.8995457887649536, 'logps/chosen': -90.71196746826172, 'logps/rejected': -128.43321228027344, 'logps/ref_chosen': -76.08218383789062, 'logps/ref_rejected': -94.9686050415039, 'logits/chosen': -6.867135047912598, 'logits/rejected': -6.089150428771973, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.04828012362122536, 'kl/avg_steps': 0.5625, 'epoch': 0.27}
27%|██████████████████████████████▏ | 182/681 [10:06<25:20, 3.05s/it] 27%|██████████████████████████████▎ | 183/681 [10:09<25:04, 3.02s/it] {'loss': 0.8151, 'grad_norm': 16.670425415039062, 'learning_rate': 4.5910671414162484e-07, 'rewards/chosen': -0.6983296871185303, 'rewards/rejected': -1.6060365438461304, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.9077068567276001, 'logps/chosen': -108.50236511230469, 'logps/rejected': -117.24481201171875, 'logps/ref_chosen': -93.90790557861328, 'logps/ref_rejected': -83.56061553955078, 'logits/chosen': -7.112674713134766, 'logits/rejected': -6.326689720153809, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.04801006615161896, 'kl/avg_steps': 0.59375, 'epoch': 0.27}
27%|██████████████████████████████▎ | 183/681 [10:09<25:04, 3.02s/it] 27%|██████████████████████████████▌ | 184/681 [10:12<24:59, 3.02s/it] {'loss': 1.0505, 'grad_norm': 17.39129066467285, 'learning_rate': 4.5840059630527985e-07, 'rewards/chosen': -0.6346135139465332, 'rewards/rejected': -1.2299377918243408, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.5953242778778076, 'logps/chosen': -100.19184875488281, 'logps/rejected': -110.23704528808594, 'logps/ref_chosen': -86.86790466308594, 'logps/ref_rejected': -84.28456115722656, 'logits/chosen': -5.867307662963867, 'logits/rejected': -5.439145565032959, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.04772669076919556, 'kl/avg_steps': 0.375, 'epoch': 0.27}
27%|██████████████████████████████▌ | 184/681 [10:12<24:59, 3.02s/it] 27%|██████████████████████████████▋ | 185/681 [10:15<24:39, 2.98s/it] {'loss': 0.9684, 'grad_norm': 19.491111755371094, 'learning_rate': 4.5768898691940836e-07, 'rewards/chosen': -0.6773566007614136, 'rewards/rejected': -1.4021331071853638, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7247765064239502, 'logps/chosen': -108.25900268554688, 'logps/rejected': -111.11482238769531, 'logps/ref_chosen': -94.02142333984375, 'logps/ref_rejected': -81.45668029785156, 'logits/chosen': -6.329668045043945, 'logits/rejected': -5.343301773071289, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.047548383474349976, 'kl/avg_steps': 0.4375, 'epoch': 0.27}
27%|██████████████████████████████▋ | 185/681 [10:15<24:39, 2.98s/it] 27%|██████████████████████████████▊ | 186/681 [10:18<24:33, 2.98s/it] {'loss': 0.8852, 'grad_norm': 18.766021728515625, 'learning_rate': 4.5697190473557947e-07, 'rewards/chosen': -0.7987364530563354, 'rewards/rejected': -1.5978922843933105, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7991558313369751, 'logps/chosen': -131.09014892578125, 'logps/rejected': -127.2985610961914, 'logps/ref_chosen': -114.19168090820312, 'logps/ref_rejected': -93.33912658691406, 'logits/chosen': -6.936692237854004, 'logits/rejected': -6.2103986740112305, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.04734126478433609, 'kl/avg_steps': 0.46875, 'epoch': 0.27}
27%|██████████████████████████████▊ | 186/681 [10:18<24:33, 2.98s/it] 27%|███████████████████████████████ | 187/681 [10:21<23:49, 2.89s/it] {'loss': 0.968, 'grad_norm': 16.06877899169922, 'learning_rate': 4.5624936864957555e-07, 'rewards/chosen': -0.705847978591919, 'rewards/rejected': -1.4218103885650635, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7159624099731445, 'logps/chosen': -91.86463928222656, 'logps/rejected': -118.53182983398438, 'logps/ref_chosen': -76.86955261230469, 'logps/ref_rejected': -88.16494750976562, 'logits/chosen': -6.3317413330078125, 'logits/rejected': -5.483089447021484, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.047120388597249985, 'kl/avg_steps': 0.53125, 'epoch': 0.27}
27%|███████████████████████████████ | 187/681 [10:21<23:49, 2.89s/it] 28%|███████████████████████████████▏ | 188/681 [10:24<24:20, 2.96s/it] {'loss': 0.8615, 'grad_norm': 15.785407066345215, 'learning_rate': 4.5552139770089454e-07, 'rewards/chosen': -0.5177306532859802, 'rewards/rejected': -1.349142074584961, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8314113616943359, 'logps/chosen': -90.27525329589844, 'logps/rejected': -125.84371948242188, 'logps/ref_chosen': -79.19479370117188, 'logps/ref_rejected': -96.82890319824219, 'logits/chosen': -6.618991851806641, 'logits/rejected': -5.915453910827637, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.046871382743120193, 'kl/avg_steps': 0.625, 'epoch': 0.28}
28%|███████████████████████████████▏ | 188/681 [10:24<24:20, 2.96s/it] 28%|███████████████████████████████▎ | 189/681 [10:27<24:14, 2.96s/it] {'loss': 0.8845, 'grad_norm': 14.651037216186523, 'learning_rate': 4.5478801107224794e-07, 'rewards/chosen': -0.6822503805160522, 'rewards/rejected': -1.5371246337890625, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.8548742532730103, 'logps/chosen': -98.51889038085938, 'logps/rejected': -112.61798095703125, 'logps/ref_chosen': -83.8927993774414, 'logps/ref_rejected': -79.46589660644531, 'logits/chosen': -6.843825817108154, 'logits/rejected': -6.071965217590332, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.04658025875687599, 'kl/avg_steps': 0.34375, 'epoch': 0.28}
28%|███████████████████████████████▎ | 189/681 [10:27<24:14, 2.96s/it] 28%|███████████████████████████████▌ | 190/681 [10:29<23:42, 2.90s/it] {'loss': 0.7663, 'grad_norm': 15.543427467346191, 'learning_rate': 4.5404922808905543e-07, 'rewards/chosen': -0.5097974538803101, 'rewards/rejected': -1.501394510269165, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.991597056388855, 'logps/chosen': -114.46714782714844, 'logps/rejected': -141.81443786621094, 'logps/ref_chosen': -103.44146728515625, 'logps/ref_rejected': -109.21503448486328, 'logits/chosen': -6.244542121887207, 'logits/rejected': -5.1248064041137695, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.046420685946941376, 'kl/avg_steps': 0.65625, 'epoch': 0.28}
28%|███████████████████████████████▌ | 190/681 [10:29<23:42, 2.90s/it] 28%|███████████████████████████████▋ | 191/681 [10:33<24:49, 3.04s/it] {'loss': 0.7528, 'grad_norm': 16.704288482666016, 'learning_rate': 4.5330506821893565e-07, 'rewards/chosen': -0.47380760312080383, 'rewards/rejected': -1.5543913841247559, 'rewards/accuracies': 0.890625, 'rewards/margins': 1.0805836915969849, 'logps/chosen': -110.06816864013672, 'logps/rejected': -149.94467163085938, 'logps/ref_chosen': -99.78551483154297, 'logps/ref_rejected': -116.00028228759766, 'logits/chosen': -5.9573469161987305, 'logits/rejected': -5.739363193511963, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.04611803591251373, 'kl/avg_steps': 0.59375, 'epoch': 0.28}
28%|███████████████████████████████▋ | 191/681 [10:33<24:49, 3.04s/it] 28%|███████████████████████████████▊ | 192/681 [10:36<24:27, 3.00s/it] {'loss': 0.9386, 'grad_norm': 19.49271583557129, 'learning_rate': 4.5255555107119336e-07, 'rewards/chosen': -0.8361877202987671, 'rewards/rejected': -1.5933136940002441, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.757125973701477, 'logps/chosen': -119.15298461914062, 'logps/rejected': -144.4326934814453, 'logps/ref_chosen': -100.86032104492188, 'logps/ref_rejected': -109.44500732421875, 'logits/chosen': -6.447579860687256, 'logits/rejected': -5.825846195220947, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.04584582895040512, 'kl/avg_steps': 0.53125, 'epoch': 0.28}
28%|███████████████████████████████▊ | 192/681 [10:36<24:27, 3.00s/it] 28%|████████████████████████████████ | 193/681 [10:39<23:58, 2.95s/it] {'loss': 1.05, 'grad_norm': 19.705093383789062, 'learning_rate': 4.5180069639630236e-07, 'rewards/chosen': -0.6607075929641724, 'rewards/rejected': -1.304963231086731, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.6442556381225586, 'logps/chosen': -107.38533020019531, 'logps/rejected': -111.87471771240234, 'logps/ref_chosen': -92.93831634521484, 'logps/ref_rejected': -83.10917663574219, 'logits/chosen': -6.4181036949157715, 'logits/rejected': -5.755351543426514, 'kl/p_epsilon_steps': 0.625, 'kl/n_epsilon_steps': 0.375, 'kl/beta': 0.04560355842113495, 'kl/avg_steps': 0.25, 'epoch': 0.28}
28%|████████████████████████████████ | 193/681 [10:39<23:58, 2.95s/it] 28%|████████████████████████████████▏ | 194/681 [10:41<23:20, 2.87s/it] {'loss': 0.9448, 'grad_norm': 14.911462783813477, 'learning_rate': 4.510405240853854e-07, 'rewards/chosen': -0.6952458620071411, 'rewards/rejected': -1.4378409385681152, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7425950765609741, 'logps/chosen': -79.15022277832031, 'logps/rejected': -100.59823608398438, 'logps/ref_chosen': -63.79157257080078, 'logps/ref_rejected': -68.72665405273438, 'logits/chosen': -6.82811164855957, 'logits/rejected': -6.5324788093566895, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0454898327589035, 'kl/avg_steps': 0.5625, 'epoch': 0.28}
28%|████████████████████████████████▏ | 194/681 [10:41<23:20, 2.87s/it] 29%|████████████████████████████████▎ | 195/681 [10:44<23:46, 2.94s/it] {'loss': 0.847, 'grad_norm': 15.496999740600586, 'learning_rate': 4.5027505416968985e-07, 'rewards/chosen': -0.6492324471473694, 'rewards/rejected': -1.583635687828064, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.9344033002853394, 'logps/chosen': -98.27217102050781, 'logps/rejected': -139.27413940429688, 'logps/ref_chosen': -83.88545989990234, 'logps/ref_rejected': -104.00125122070312, 'logits/chosen': -6.802285194396973, 'logits/rejected': -5.815672874450684, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.04523538425564766, 'kl/avg_steps': 0.625, 'epoch': 0.29}
29%|████████████████████████████████▎ | 195/681 [10:44<23:46, 2.94s/it] 29%|████████████████████████████████▌ | 196/681 [10:47<24:11, 2.99s/it] {'loss': 0.8099, 'grad_norm': 13.717185974121094, 'learning_rate': 4.495043068200599e-07, 'rewards/chosen': -0.7399119138717651, 'rewards/rejected': -1.6481503248214722, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.908238410949707, 'logps/chosen': -105.92664337158203, 'logps/rejected': -120.28113555908203, 'logps/ref_chosen': -89.4013671875, 'logps/ref_rejected': -83.34735107421875, 'logits/chosen': -6.458456993103027, 'logits/rejected': -5.85124397277832, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.04495441913604736, 'kl/avg_steps': 0.5625, 'epoch': 0.29}
29%|████████████████████████████████▌ | 196/681 [10:48<24:11, 2.99s/it] 29%|████████████████████████████████▋ | 197/681 [10:51<24:33, 3.04s/it] {'loss': 1.0239, 'grad_norm': 16.101377487182617, 'learning_rate': 4.4872830234640493e-07, 'rewards/chosen': -0.7603064179420471, 'rewards/rejected': -1.3241004943847656, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.5637940168380737, 'logps/chosen': -103.89794921875, 'logps/rejected': -112.60846710205078, 'logps/ref_chosen': -86.8406982421875, 'logps/ref_rejected': -82.79289245605469, 'logits/chosen': -6.566864013671875, 'logits/rejected': -5.954580783843994, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.04470296576619148, 'kl/avg_steps': 0.5625, 'epoch': 0.29}
29%|████████████████████████████████▋ | 197/681 [10:51<24:33, 3.04s/it] 29%|████████████████████████████████▊ | 198/681 [10:54<24:23, 3.03s/it] {'loss': 0.8573, 'grad_norm': 18.52680206298828, 'learning_rate': 4.479470611971645e-07, 'rewards/chosen': -0.8809682130813599, 'rewards/rejected': -1.730950117111206, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.8499820232391357, 'logps/chosen': -104.97913360595703, 'logps/rejected': -143.3640594482422, 'logps/ref_chosen': -85.04824829101562, 'logps/ref_rejected': -104.10365295410156, 'logits/chosen': -6.684327125549316, 'logits/rejected': -5.88149881362915, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.04445291683077812, 'kl/avg_steps': 0.71875, 'epoch': 0.29}
29%|████████████████████████████████▊ | 198/681 [10:54<24:23, 3.03s/it] 29%|█████████████████████████████████ | 199/681 [10:57<25:04, 3.12s/it] {'loss': 0.9335, 'grad_norm': 15.971412658691406, 'learning_rate': 4.471606039587695e-07, 'rewards/chosen': -0.8515444993972778, 'rewards/rejected': -1.5823307037353516, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7307862043380737, 'logps/chosen': -101.11582946777344, 'logps/rejected': -126.85987091064453, 'logps/ref_chosen': -81.78669738769531, 'logps/ref_rejected': -90.790771484375, 'logits/chosen': -6.563417434692383, 'logits/rejected': -6.149653434753418, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.044135693460702896, 'kl/avg_steps': 0.4375, 'epoch': 0.29}
29%|█████████████████████████████████ | 199/681 [10:57<25:04, 3.12s/it] 29%|█████████████████████████████████▏ | 200/681 [11:00<24:55, 3.11s/it] {'loss': 0.9014, 'grad_norm': 14.888577461242676, 'learning_rate': 4.4636895135509966e-07, 'rewards/chosen': -0.8119258284568787, 'rewards/rejected': -1.6711798906326294, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8592541813850403, 'logps/chosen': -102.80973052978516, 'logps/rejected': -125.29328918457031, 'logps/ref_chosen': -84.26809692382812, 'logps/ref_rejected': -86.96786499023438, 'logits/chosen': -6.618537425994873, 'logits/rejected': -5.955951690673828, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.043943438678979874, 'kl/avg_steps': 0.59375, 'epoch': 0.29}
29%|█████████████████████████████████▏ | 200/681 [11:00<24:55, 3.11s/it][INFO|trainer.py:4307] 2026-04-24 04:27:07,658 >>
***** Running Evaluation *****
[INFO|trainer.py:4309] 2026-04-24 04:27:07,658 >> Num examples = 2339
[INFO|trainer.py:4312] 2026-04-24 04:27:07,658 >> Batch size = 8
0%| | 0/73 [00:00<?, ?it/s]
3%|███▏ | 2/73 [00:00<00:23, 3.02it/s]
4%|████▊ | 3/73 [00:01<00:32, 2.13it/s]
5%|██████▎ | 4/73 [00:01<00:36, 1.87it/s]
7%|███████▉ | 5/73 [00:02<00:38, 1.75it/s]
8%|█████████▌ | 6/73 [00:03<00:40, 1.65it/s]
10%|███████████ | 7/73 [00:03<00:38, 1.71it/s]
11%|████████████▋ | 8/73 [00:04<00:40, 1.59it/s]
12%|██████████████▎ | 9/73 [00:05<00:41, 1.54it/s]
14%|███████████████▊ | 10/73 [00:05<00:41, 1.52it/s]
15%|█████████████████▎ | 11/73 [00:06<00:40, 1.52it/s]
16%|██████████████████▉ | 12/73 [00:07<00:40, 1.49it/s]
18%|████████████████████▍ | 13/73 [00:07<00:39, 1.53it/s]
19%|██████████████████████ | 14/73 [00:08<00:39, 1.51it/s]
21%|███████████████████████▋ | 15/73 [00:09<00:38, 1.51it/s]
22%|█████████████████████████▏ | 16/73 [00:09<00:39, 1.46it/s]
23%|██████████████████████████▊ | 17/73 [00:10<00:38, 1.45it/s]
25%|████████████████████████████▎ | 18/73 [00:11<00:38, 1.43it/s]
26%|█████████████████████████████▉ | 19/73 [00:12<00:38, 1.42it/s]
27%|███████████████████████████████▌ | 20/73 [00:12<00:37, 1.40it/s]
29%|█████████████████████████████████ | 21/73 [00:13<00:37, 1.40it/s]
30%|██████████████████████████████████▋ | 22/73 [00:14<00:36, 1.39it/s]
32%|████████████████████████████████████▏ | 23/73 [00:14<00:35, 1.42it/s]
33%|█████████████████████████████████████▊ | 24/73 [00:15<00:34, 1.43it/s]
34%|███████████████████████████████████████▍ | 25/73 [00:16<00:33, 1.44it/s]
36%|████████████████████████████████████████▉ | 26/73 [00:17<00:32, 1.45it/s]
37%|██████████████████████████████████████████▌ | 27/73 [00:17<00:28, 1.61it/s]
38%|████████████████████████████████████████████ | 28/73 [00:18<00:28, 1.56it/s]
40%|█████████████████████████████████████████████▋ | 29/73 [00:18<00:27, 1.57it/s]
41%|███████████████████████████████████████████████▎ | 30/73 [00:19<00:26, 1.60it/s]
42%|████████████████████████████████████████████████▊ | 31/73 [00:20<00:27, 1.53it/s]
44%|██████████████████████████████████████████████████▍ | 32/73 [00:20<00:26, 1.56it/s]
45%|███████████████████████████████████████████████████▉ | 33/73 [00:21<00:25, 1.58it/s]
47%|█████████████████████████████████████████████████████▌ | 34/73 [00:22<00:25, 1.55it/s]
48%|███████████████████████████████████████████████████████▏ | 35/73 [00:22<00:25, 1.49it/s]
49%|████████████████████████████████████████████████████████▋ | 36/73 [00:23<00:24, 1.49it/s]
51%|██████████████████████████████████████████████████████████▎ | 37/73 [00:24<00:24, 1.48it/s]
52%|███████████████████████████████████████████████████████████▊ | 38/73 [00:24<00:22, 1.56it/s]
53%|█████████████████████████████████████████████████████████████▍ | 39/73 [00:25<00:22, 1.49it/s]
55%|███████████████████████████████████████████████████████████████ | 40/73 [00:26<00:21, 1.50it/s]
56%|████████████████████████████████████████████████████████████████▌ | 41/73 [00:26<00:20, 1.56it/s]
58%|██████████████████████████████████████████████████████████████████▏ | 42/73 [00:27<00:20, 1.52it/s]
59%|███████████████████████████████████████████████████████████████████▋ | 43/73 [00:27<00:19, 1.54it/s]
60%|█████████████████████████████████████████████████████████████████████▎ | 44/73 [00:28<00:19, 1.52it/s]
62%|██████████████████████████████████████████████████████████████████████▉ | 45/73 [00:29<00:19, 1.47it/s]
63%|████████████████████████████████████████████████████████████████████████▍ | 46/73 [00:29<00:17, 1.53it/s]
64%|██████████████████████████████████████████████████████████████████████████ | 47/73 [00:30<00:17, 1.50it/s]
66%|███████████████████████████████████████████████████████████████████████████▌ | 48/73 [00:31<00:16, 1.49it/s]
67%|█████████████████████████████████████████████████████████████████████████████▏ | 49/73 [00:32<00:16, 1.47it/s]
68%|██████████████████████████████████████████████████████████████████████████████▊ | 50/73 [00:32<00:15, 1.49it/s]
70%|████████████████████████████████████████████████████████████████████████████████▎ | 51/73 [00:33<00:15, 1.47it/s]
71%|█████████████████████████████████████████████████████████████████████████████████▉ | 52/73 [00:34<00:14, 1.44it/s]
73%|███████████████████████████████████████████████████████████████████████████████████▍ | 53/73 [00:34<00:14, 1.41it/s]
74%|█████████████████████████████████████████████████████████████████████████████████████ | 54/73 [00:35<00:12, 1.50it/s]
75%|██████████████████████████████████████████████████████████████████████████████████████▋ | 55/73 [00:36<00:12, 1.49it/s]
77%|████████████████████████████████████████████████████████████████████████████████████████▏ | 56/73 [00:36<00:11, 1.53it/s]
78%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 57/73 [00:37<00:10, 1.47it/s]
79%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 58/73 [00:38<00:09, 1.53it/s]
81%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 59/73 [00:38<00:09, 1.53it/s]
82%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 60/73 [00:39<00:08, 1.50it/s]
84%|████████████████████████████████████████████████████████████████████████████████████████████████ | 61/73 [00:40<00:08, 1.49it/s]
85%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 62/73 [00:40<00:07, 1.50it/s]
86%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 63/73 [00:41<00:06, 1.58it/s]
88%|████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 64/73 [00:41<00:05, 1.60it/s]
89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 65/73 [00:42<00:05, 1.58it/s]
90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 66/73 [00:43<00:04, 1.52it/s]
92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 67/73 [00:43<00:03, 1.55it/s]
93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 68/73 [00:44<00:03, 1.52it/s]
95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 69/73 [00:45<00:02, 1.49it/s]
96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 70/73 [00:45<00:02, 1.48it/s]
97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 71/73 [00:46<00:01, 1.48it/s]
99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 72/73 [00:47<00:00, 1.48it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
{'eval_loss': 0.5126060247421265, 'eval_runtime': 48.5322, 'eval_samples_per_second': 48.195, 'eval_steps_per_second': 1.525, 'eval_rewards/chosen': -0.9392141103744507, 'eval_rewards/rejected': -1.5651392936706543, 'eval_rewards/accuracies': 0.7619863152503967, 'eval_rewards/margins': 0.6259253621101379, 'eval_logps/chosen': -122.01994323730469, 'eval_logps/rejected': -130.100341796875, 'eval_logps/ref_chosen': -100.49356842041016, 'eval_logps/ref_rejected': -94.06775665283203, 'eval_logits/chosen': -5.910821914672852, 'eval_logits/rejected': -5.565565586090088, 'eval_kl/p_epsilon_steps': 0.690496563911438, 'eval_kl/n_epsilon_steps': 0.3069349229335785, 'epoch': 0.29}
29%|█████████████████████████████████▏ | 200/681 [11:49<24:55, 3.11s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
[INFO|trainer.py:3984] 2026-04-24 04:28:10,709 >> Saving model checkpoint to /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-200
[INFO|configuration_utils.py:419] 2026-04-24 04:28:10,716 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-200/config.json
[INFO|configuration_utils.py:911] 2026-04-24 04:28:10,720 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-200/generation_config.json
[INFO|modeling_utils.py:3580] 2026-04-24 04:28:50,086 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-200/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2510] 2026-04-24 04:28:50,089 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-200/tokenizer_config.json
[INFO|tokenization_utils_base.py:2519] 2026-04-24 04:28:50,092 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-200/special_tokens_map.json
30%|████████████████████████████████▍ | 201/681 [15:34<11:15:55, 84.49s/it] {'loss': 0.9275, 'grad_norm': 16.643579483032227, 'learning_rate': 4.455721242469372e-07, 'rewards/chosen': -0.8622275590896606, 'rewards/rejected': -1.6865991353988647, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.8243715763092041, 'logps/chosen': -127.57327270507812, 'logps/rejected': -158.6796875, 'logps/ref_chosen': -107.77249145507812, 'logps/ref_rejected': -119.79248046875, 'logits/chosen': -6.567927360534668, 'logits/rejected': -5.76133394241333, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.04368406534194946, 'kl/avg_steps': 0.5, 'epoch': 0.3}
30%|████████████████████████████████▍ | 201/681 [15:34<11:15:55, 84.49s/it] 30%|████████████████████████████████▉ | 202/681 [15:37<7:59:32, 60.07s/it] {'loss': 0.947, 'grad_norm': 15.5465669631958, 'learning_rate': 4.4477014363141755e-07, 'rewards/chosen': -0.8463935256004333, 'rewards/rejected': -1.568117618560791, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.7217241525650024, 'logps/chosen': -95.47408294677734, 'logps/rejected': -130.75558471679688, 'logps/ref_chosen': -75.97245025634766, 'logps/ref_rejected': -94.4599838256836, 'logits/chosen': -6.6011576652526855, 'logits/rejected': -5.9799299240112305, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.043466731905937195, 'kl/avg_steps': 0.5, 'epoch': 0.3}
30%|████████████████████████████████▉ | 202/681 [15:38<7:59:32, 60.07s/it] 30%|█████████████████████████████████ | 203/681 [15:41<5:42:40, 43.01s/it] {'loss': 0.8472, 'grad_norm': 14.089229583740234, 'learning_rate': 4.439630306414758e-07, 'rewards/chosen': -0.7273061275482178, 'rewards/rejected': -1.5748505592346191, 'rewards/accuracies': 0.875, 'rewards/margins': 0.8475444316864014, 'logps/chosen': -111.82502746582031, 'logps/rejected': -129.54229736328125, 'logps/ref_chosen': -94.96715545654297, 'logps/ref_rejected': -92.8876724243164, 'logits/chosen': -6.752559661865234, 'logits/rejected': -5.879279136657715, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.04325047880411148, 'kl/avg_steps': 0.5625, 'epoch': 0.3}
30%|█████████████████████████████████ | 203/681 [15:41<5:42:40, 43.01s/it] 30%|█████████████████████████████████▎ | 204/681 [15:44<4:07:12, 31.10s/it] {'loss': 0.9609, 'grad_norm': 17.907960891723633, 'learning_rate': 4.431508065452897e-07, 'rewards/chosen': -0.9145333170890808, 'rewards/rejected': -1.6298069953918457, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.7152736186981201, 'logps/chosen': -137.6444091796875, 'logps/rejected': -131.52774047851562, 'logps/ref_chosen': -116.35719299316406, 'logps/ref_rejected': -93.39759063720703, 'logits/chosen': -6.687747955322266, 'logits/rejected': -5.982107162475586, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.043008554726839066, 'kl/avg_steps': 0.40625, 'epoch': 0.3}
30%|█████████████████████████████████▎ | 204/681 [15:44<4:07:12, 31.10s/it] 30%|█████████████████████████████████▍ | 205/681 [15:47<3:00:14, 22.72s/it] {'loss': 0.7808, 'grad_norm': 12.563446998596191, 'learning_rate': 4.4233349274571974e-07, 'rewards/chosen': -0.8293547630310059, 'rewards/rejected': -1.7540370225906372, 'rewards/accuracies': 0.875, 'rewards/margins': 0.9246822595596313, 'logps/chosen': -108.34004211425781, 'logps/rejected': -133.15499877929688, 'logps/ref_chosen': -88.85934448242188, 'logps/ref_rejected': -91.8544921875, 'logits/chosen': -6.747779846191406, 'logits/rejected': -6.664907932281494, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.04283454269170761, 'kl/avg_steps': 0.75, 'epoch': 0.3}
30%|█████████████████████████████████▍ | 205/681 [15:47<3:00:14, 22.72s/it] 30%|█████████████████████████████████▌ | 206/681 [15:50<2:13:03, 16.81s/it] {'loss': 0.8253, 'grad_norm': 17.306621551513672, 'learning_rate': 4.415111107797445e-07, 'rewards/chosen': -0.8388286828994751, 'rewards/rejected': -1.6743868589401245, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.8355581164360046, 'logps/chosen': -96.40692138671875, 'logps/rejected': -142.6748809814453, 'logps/ref_chosen': -76.54634857177734, 'logps/ref_rejected': -102.95314025878906, 'logits/chosen': -6.731910705566406, 'logits/rejected': -5.665759086608887, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.042515672743320465, 'kl/avg_steps': 0.78125, 'epoch': 0.3}
30%|█████████████████████████████████▌ | 206/681 [15:50<2:13:03, 16.81s/it] 30%|█████████████████████████████████▋ | 207/681 [15:53<1:39:59, 12.66s/it] {'loss': 0.9093, 'grad_norm': 19.9998722076416, 'learning_rate': 4.4068368231789365e-07, 'rewards/chosen': -0.9145298600196838, 'rewards/rejected': -1.8097199201583862, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8951901197433472, 'logps/chosen': -107.92840576171875, 'logps/rejected': -133.81866455078125, 'logps/ref_chosen': -86.23164367675781, 'logps/ref_rejected': -90.65512084960938, 'logits/chosen': -7.154547214508057, 'logits/rejected': -6.694057464599609, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.04218609631061554, 'kl/avg_steps': 0.5, 'epoch': 0.3}
30%|█████████████████████████████████▋ | 207/681 [15:53<1:39:59, 12.66s/it] 31%|█████████████████████████████████▉ | 208/681 [15:56<1:17:01, 9.77s/it] {'loss': 0.7998, 'grad_norm': 15.674067497253418, 'learning_rate': 4.398512291636768e-07, 'rewards/chosen': -0.8975973725318909, 'rewards/rejected': -1.838052749633789, 'rewards/accuracies': 0.921875, 'rewards/margins': 0.9404553771018982, 'logps/chosen': -115.6429672241211, 'logps/rejected': -145.07806396484375, 'logps/ref_chosen': -94.1595458984375, 'logps/ref_rejected': -100.96233367919922, 'logits/chosen': -7.054866313934326, 'logits/rejected': -6.2353129386901855, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.041976213455200195, 'kl/avg_steps': 0.65625, 'epoch': 0.31}
31%|█████████████████████████████████▉ | 208/681 [15:56<1:17:01, 9.77s/it] 31%|██████████████████████████████████ | 209/681 [15:59<1:00:23, 7.68s/it] {'loss': 0.8985, 'grad_norm': 15.056414604187012, 'learning_rate': 4.3901377325300857e-07, 'rewards/chosen': -0.7734057903289795, 'rewards/rejected': -1.5872459411621094, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8138402104377747, 'logps/chosen': -102.76752471923828, 'logps/rejected': -125.9391860961914, 'logps/ref_chosen': -84.17056274414062, 'logps/ref_rejected': -87.61955261230469, 'logits/chosen': -7.251114845275879, 'logits/rejected': -6.478391647338867, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.04170254245400429, 'kl/avg_steps': 0.5, 'epoch': 0.31}
31%|██████████████████████████████████ | 209/681 [15:59<1:00:23, 7.68s/it] 31%|██████████████████████████████████▊ | 210/681 [16:02<48:55, 6.23s/it] {'loss': 0.8996, 'grad_norm': 13.82249641418457, 'learning_rate': 4.381713366536311e-07, 'rewards/chosen': -0.8534192442893982, 'rewards/rejected': -1.6337693929672241, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7803501486778259, 'logps/chosen': -101.83394622802734, 'logps/rejected': -123.829833984375, 'logps/ref_chosen': -81.17117309570312, 'logps/ref_rejected': -84.17478942871094, 'logits/chosen': -7.367747783660889, 'logits/rejected': -6.7037177085876465, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.04149506613612175, 'kl/avg_steps': 0.625, 'epoch': 0.31}
31%|██████████████████████████████████▊ | 210/681 [16:02<48:55, 6.23s/it] 31%|███████████████████████████████████ | 211/681 [16:05<40:39, 5.19s/it] {'loss': 0.9141, 'grad_norm': 16.27776527404785, 'learning_rate': 4.373239415645323e-07, 'rewards/chosen': -0.9316354990005493, 'rewards/rejected': -1.7147372961044312, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.7831017971038818, 'logps/chosen': -131.35140991210938, 'logps/rejected': -135.39334106445312, 'logps/ref_chosen': -108.71271514892578, 'logps/ref_rejected': -93.55564880371094, 'logits/chosen': -6.783376216888428, 'logits/rejected': -6.2379889488220215, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.04123733192682266, 'kl/avg_steps': 0.46875, 'epoch': 0.31}
31%|███████████████████████████████████ | 211/681 [16:05<40:39, 5.19s/it] 31%|███████████████████████████████████▏ | 212/681 [16:08<35:53, 4.59s/it] {'loss': 0.7363, 'grad_norm': 13.126346588134766, 'learning_rate': 4.3647161031536086e-07, 'rewards/chosen': -0.6175893545150757, 'rewards/rejected': -1.706433653831482, 'rewards/accuracies': 0.90625, 'rewards/margins': 1.0888441801071167, 'logps/chosen': -113.453125, 'logps/rejected': -151.75265502929688, 'logps/ref_chosen': -98.36194610595703, 'logps/ref_rejected': -109.88999938964844, 'logits/chosen': -6.987194061279297, 'logits/rejected': -6.6716461181640625, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0410449355840683, 'kl/avg_steps': 0.53125, 'epoch': 0.31}
31%|███████████████████████████████████▏ | 212/681 [16:08<35:53, 4.59s/it] 31%|███████████████████████████████████▎ | 213/681 [16:11<32:38, 4.19s/it] {'loss': 0.8605, 'grad_norm': 19.004343032836914, 'learning_rate': 4.3561436536583774e-07, 'rewards/chosen': -0.7606232166290283, 'rewards/rejected': -1.6314427852630615, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8708195686340332, 'logps/chosen': -126.75518798828125, 'logps/rejected': -140.40818786621094, 'logps/ref_chosen': -108.05531311035156, 'logps/ref_rejected': -100.14414978027344, 'logits/chosen': -6.994483947753906, 'logits/rejected': -6.408557891845703, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.04082803428173065, 'kl/avg_steps': 0.5625, 'epoch': 0.31}
31%|███████████████████████████████████▎ | 213/681 [16:11<32:38, 4.19s/it] 31%|███████████████████████████████████▌ | 214/681 [16:14<29:19, 3.77s/it] {'loss': 0.9841, 'grad_norm': 15.15538501739502, 'learning_rate': 4.3475222930516473e-07, 'rewards/chosen': -0.8206988573074341, 'rewards/rejected': -1.570618987083435, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.749920129776001, 'logps/chosen': -98.05276489257812, 'logps/rejected': -123.21199798583984, 'logps/ref_chosen': -77.80473327636719, 'logps/ref_rejected': -84.27578735351562, 'logits/chosen': -6.964527130126953, 'logits/rejected': -6.679478645324707, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.04059966281056404, 'kl/avg_steps': 0.4375, 'epoch': 0.31}
31%|███████████████████████████████████▌ | 214/681 [16:14<29:19, 3.77s/it] 32%|███████████████████████████████████▋ | 215/681 [16:17<27:47, 3.58s/it] {'loss': 0.7734, 'grad_norm': 13.474177360534668, 'learning_rate': 4.3388522485142885e-07, 'rewards/chosen': -0.6723982095718384, 'rewards/rejected': -1.663226842880249, 'rewards/accuracies': 0.875, 'rewards/margins': 0.9908286333084106, 'logps/chosen': -101.85633087158203, 'logps/rejected': -138.35052490234375, 'logps/ref_chosen': -85.1138916015625, 'logps/ref_rejected': -96.86151885986328, 'logits/chosen': -7.422246932983398, 'logits/rejected': -6.752878189086914, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.04042281210422516, 'kl/avg_steps': 0.6875, 'epoch': 0.32}
32%|███████████████████████████████████▋ | 215/681 [16:17<27:47, 3.58s/it] 32%|███████████████████████████████████▊ | 216/681 [16:20<26:52, 3.47s/it] {'loss': 0.8502, 'grad_norm': 13.187141418457031, 'learning_rate': 4.330133748510036e-07, 'rewards/chosen': -0.7613588571548462, 'rewards/rejected': -1.5942199230194092, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.832861065864563, 'logps/chosen': -99.6475830078125, 'logps/rejected': -121.41978454589844, 'logps/ref_chosen': -80.5923080444336, 'logps/ref_rejected': -81.41983795166016, 'logits/chosen': -6.856760501861572, 'logits/rejected': -6.676458835601807, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.04014680162072182, 'kl/avg_steps': 0.53125, 'epoch': 0.32}
32%|███████████████████████████████████▊ | 216/681 [16:20<26:52, 3.47s/it] 32%|████████████████████████████████████ | 217/681 [16:23<25:46, 3.33s/it] {'loss': 0.9483, 'grad_norm': 17.514558792114258, 'learning_rate': 4.3213670227794757e-07, 'rewards/chosen': -0.8944262266159058, 'rewards/rejected': -1.6273518800735474, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7329256534576416, 'logps/chosen': -115.89637756347656, 'logps/rejected': -144.48422241210938, 'logps/ref_chosen': -93.47257995605469, 'logps/ref_rejected': -103.488525390625, 'logits/chosen': -7.025016784667969, 'logits/rejected': -6.148534774780273, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.03993465006351471, 'kl/avg_steps': 0.46875, 'epoch': 0.32}
32%|████████████████████████████████████ | 217/681 [16:23<25:46, 3.33s/it] 32%|████████████████████████████████████▏ | 218/681 [16:26<25:07, 3.26s/it] {'loss': 0.8984, 'grad_norm': 15.729193687438965, 'learning_rate': 4.3125523023339815e-07, 'rewards/chosen': -0.8316381573677063, 'rewards/rejected': -1.6821269989013672, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8504889607429504, 'logps/chosen': -110.00785827636719, 'logps/rejected': -136.88497924804688, 'logps/ref_chosen': -89.05883026123047, 'logps/ref_rejected': -94.30680847167969, 'logits/chosen': -7.055582046508789, 'logits/rejected': -6.02012300491333, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.03974832966923714, 'kl/avg_steps': 0.453125, 'epoch': 0.32}
32%|████████████████████████████████████▏ | 218/681 [16:26<25:07, 3.26s/it] 32%|████████████████████████████████████▎ | 219/681 [16:30<25:00, 3.25s/it] {'loss': 0.9475, 'grad_norm': 19.58609390258789, 'learning_rate': 4.303689819449636e-07, 'rewards/chosen': -0.7558479309082031, 'rewards/rejected': -1.5479310750961304, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.7920831441879272, 'logps/chosen': -120.11994171142578, 'logps/rejected': -131.8204803466797, 'logps/ref_chosen': -101.00733947753906, 'logps/ref_rejected': -92.46794128417969, 'logits/chosen': -7.102532386779785, 'logits/rejected': -6.606064796447754, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.039569031447172165, 'kl/avg_steps': 0.46875, 'epoch': 0.32}
32%|████████████████████████████████████▎ | 219/681 [16:30<25:00, 3.25s/it] 32%|████████████████████████████████████▌ | 220/681 [16:33<24:52, 3.24s/it] {'loss': 0.9298, 'grad_norm': 16.06678581237793, 'learning_rate': 4.2947798076611047e-07, 'rewards/chosen': -0.7732564210891724, 'rewards/rejected': -1.4757641553878784, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.702507734298706, 'logps/chosen': -115.22142028808594, 'logps/rejected': -132.008544921875, 'logps/ref_chosen': -95.53721618652344, 'logps/ref_rejected': -94.30703735351562, 'logits/chosen': -6.876911163330078, 'logits/rejected': -6.016563415527344, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.03938441723585129, 'kl/avg_steps': 0.46875, 'epoch': 0.32}
32%|████████████████████████████████████▌ | 220/681 [16:33<24:52, 3.24s/it] 32%|████████████████████████████████████▋ | 221/681 [16:36<24:47, 3.23s/it] {'loss': 0.7544, 'grad_norm': 14.65071964263916, 'learning_rate': 4.285822501755485e-07, 'rewards/chosen': -0.7377252578735352, 'rewards/rejected': -1.7821805477142334, 'rewards/accuracies': 0.875, 'rewards/margins': 1.0444551706314087, 'logps/chosen': -101.76954650878906, 'logps/rejected': -156.64260864257812, 'logps/ref_chosen': -82.84486389160156, 'logps/ref_rejected': -110.81179809570312, 'logits/chosen': -6.791396141052246, 'logits/rejected': -5.949629783630371, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.039200663566589355, 'kl/avg_steps': 0.6875, 'epoch': 0.32}
32%|████████████████████████████████████▋ | 221/681 [16:36<24:47, 3.23s/it] 33%|████████████████████████████████████▊ | 222/681 [16:39<24:16, 3.17s/it] {'loss': 0.9338, 'grad_norm': 20.610990524291992, 'learning_rate': 4.276818137766118e-07, 'rewards/chosen': -0.9227837920188904, 'rewards/rejected': -1.7836687564849854, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.8608850240707397, 'logps/chosen': -118.88861083984375, 'logps/rejected': -152.92138671875, 'logps/ref_chosen': -95.14198303222656, 'logps/ref_rejected': -106.80441284179688, 'logits/chosen': -7.152358055114746, 'logits/rejected': -6.381607532501221, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.03893300145864487, 'kl/avg_steps': 0.53125, 'epoch': 0.33}
33%|████████████████████████████████████▊ | 222/681 [16:39<24:16, 3.17s/it] 33%|█████████████████████████████████████ | 223/681 [16:42<22:50, 2.99s/it] {'loss': 0.98, 'grad_norm': 15.403647422790527, 'learning_rate': 4.2677669529663686e-07, 'rewards/chosen': -0.8735491037368774, 'rewards/rejected': -1.556617021560669, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.6830679178237915, 'logps/chosen': -108.16079711914062, 'logps/rejected': -126.88802337646484, 'logps/ref_chosen': -85.57511138916016, 'logps/ref_rejected': -86.45238494873047, 'logits/chosen': -7.370273590087891, 'logits/rejected': -6.7063889503479, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.03872726112604141, 'kl/avg_steps': 0.5, 'epoch': 0.33}
33%|█████████████████████████████████████ | 223/681 [16:42<22:50, 2.99s/it] 33%|█████████████████████████████████████▏ | 224/681 [16:44<21:49, 2.87s/it] {'loss': 0.867, 'grad_norm': 15.034847259521484, 'learning_rate': 4.2586691858633747e-07, 'rewards/chosen': -0.820472002029419, 'rewards/rejected': -1.6667978763580322, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8463258147239685, 'logps/chosen': -104.06519317626953, 'logps/rejected': -126.1408462524414, 'logps/ref_chosen': -82.72380065917969, 'logps/ref_rejected': -82.59538269042969, 'logits/chosen': -7.058864593505859, 'logits/rejected': -6.6526618003845215, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.03853458911180496, 'kl/avg_steps': 0.5, 'epoch': 0.33}
33%|█████████████████████████████████████▏ | 224/681 [16:44<21:49, 2.87s/it] 33%|█████████████████████████████████████▎ | 225/681 [16:47<21:28, 2.83s/it] {'loss': 0.8512, 'grad_norm': 15.859506607055664, 'learning_rate': 4.249525076191759e-07, 'rewards/chosen': -0.8980911374092102, 'rewards/rejected': -1.83120858669281, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.9331176280975342, 'logps/chosen': -119.15274810791016, 'logps/rejected': -153.17977905273438, 'logps/ref_chosen': -95.67768096923828, 'logps/ref_rejected': -105.09687805175781, 'logits/chosen': -7.314513206481934, 'logits/rejected': -6.8365983963012695, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.03834287449717522, 'kl/avg_steps': 0.53125, 'epoch': 0.33}
33%|█████████████████████████████████████▎ | 225/681 [16:47<21:28, 2.83s/it] 33%|█████████████████████████████████████▌ | 226/681 [16:50<22:06, 2.91s/it] {'loss': 0.8457, 'grad_norm': 13.422561645507812, 'learning_rate': 4.2403348649073167e-07, 'rewards/chosen': -0.7408524751663208, 'rewards/rejected': -1.5925018787384033, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8516495227813721, 'logps/chosen': -112.96669006347656, 'logps/rejected': -128.74534606933594, 'logps/ref_chosen': -93.46092987060547, 'logps/ref_rejected': -86.7017593383789, 'logits/chosen': -7.025399684906006, 'logits/rejected': -6.684216499328613, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.03814025595784187, 'kl/avg_steps': 0.625, 'epoch': 0.33}
33%|█████████████████████████████████████▌ | 226/681 [16:50<22:06, 2.91s/it] 33%|█████████████████████████████████████▋ | 227/681 [16:53<22:11, 2.93s/it] {'loss': 0.906, 'grad_norm': 15.427087783813477, 'learning_rate': 4.2310987941806615e-07, 'rewards/chosen': -1.0186035633087158, 'rewards/rejected': -1.8651710748672485, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8465676307678223, 'logps/chosen': -119.78524017333984, 'logps/rejected': -154.31568908691406, 'logps/ref_chosen': -92.81427001953125, 'logps/ref_rejected': -104.73692321777344, 'logits/chosen': -7.346820831298828, 'logits/rejected': -6.7165141105651855, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.03790335729718208, 'kl/avg_steps': 0.5625, 'epoch': 0.33}
33%|█████████████████████████████████████▋ | 227/681 [16:53<22:11, 2.93s/it] 33%|█████████████████████████████████████▊ | 228/681 [16:56<22:31, 2.98s/it] {'loss': 0.962, 'grad_norm': 15.720854759216309, 'learning_rate': 4.2218171073908463e-07, 'rewards/chosen': -0.9233343601226807, 'rewards/rejected': -1.5975819826126099, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6742476224899292, 'logps/chosen': -118.57168579101562, 'logps/rejected': -138.6416015625, 'logps/ref_chosen': -94.03712463378906, 'logps/ref_rejected': -96.02151489257812, 'logits/chosen': -7.082067489624023, 'logits/rejected': -6.396081924438477, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.03769134357571602, 'kl/avg_steps': 0.40625, 'epoch': 0.33}
33%|█████████████████████████████████████▊ | 228/681 [16:56<22:31, 2.98s/it] 34%|█████████████████████████████████████▉ | 229/681 [16:59<22:57, 3.05s/it] {'loss': 0.7638, 'grad_norm': 13.65562915802002, 'learning_rate': 4.212490049118951e-07, 'rewards/chosen': -0.7962503433227539, 'rewards/rejected': -1.879028558731079, 'rewards/accuracies': 0.859375, 'rewards/margins': 1.0827782154083252, 'logps/chosen': -116.85358428955078, 'logps/rejected': -139.57473754882812, 'logps/ref_chosen': -95.57766723632812, 'logps/ref_rejected': -89.17379760742188, 'logits/chosen': -7.236542224884033, 'logits/rejected': -6.673727989196777, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.03753884509205818, 'kl/avg_steps': 0.5625, 'epoch': 0.34}
34%|█████████████████████████████████████▉ | 229/681 [16:59<22:57, 3.05s/it] 34%|██████████████████████████████████████▏ | 230/681 [17:02<22:19, 2.97s/it] {'loss': 0.7439, 'grad_norm': 15.202421188354492, 'learning_rate': 4.203117865141635e-07, 'rewards/chosen': -0.8491979241371155, 'rewards/rejected': -1.8918962478637695, 'rewards/accuracies': 0.90625, 'rewards/margins': 1.0426983833312988, 'logps/chosen': -86.61280822753906, 'logps/rejected': -143.0418243408203, 'logps/ref_chosen': -63.713626861572266, 'logps/ref_rejected': -91.9087142944336, 'logits/chosen': -7.399197578430176, 'logits/rejected': -6.820221900939941, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.037328869104385376, 'kl/avg_steps': 0.78125, 'epoch': 0.34}
34%|██████████████████████████████████████▏ | 230/681 [17:02<22:19, 2.97s/it] 34%|██████████████████████████████████████▎ | 231/681 [17:05<22:32, 3.01s/it] {'loss': 0.8274, 'grad_norm': 13.081562995910645, 'learning_rate': 4.1937008024246625e-07, 'rewards/chosen': -0.957029402256012, 'rewards/rejected': -1.8156838417053223, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8586543798446655, 'logps/chosen': -121.447998046875, 'logps/rejected': -130.36236572265625, 'logps/ref_chosen': -95.45567321777344, 'logps/ref_rejected': -80.95568084716797, 'logits/chosen': -6.668990135192871, 'logits/rejected': -6.165530681610107, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.03703949600458145, 'kl/avg_steps': 0.671875, 'epoch': 0.34}
34%|██████████████████████████████████████▎ | 231/681 [17:05<22:32, 3.01s/it] 34%|██████████████████████████████████████▍ | 232/681 [17:08<22:40, 3.03s/it] {'loss': 0.9199, 'grad_norm': 15.00186538696289, 'learning_rate': 4.1842391091163933e-07, 'rewards/chosen': -0.9383091926574707, 'rewards/rejected': -1.7088375091552734, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7705283164978027, 'logps/chosen': -122.46676635742188, 'logps/rejected': -136.52182006835938, 'logps/ref_chosen': -96.89726257324219, 'logps/ref_rejected': -89.76461791992188, 'logits/chosen': -6.989046096801758, 'logits/rejected': -6.184296607971191, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.03679230064153671, 'kl/avg_steps': 0.5, 'epoch': 0.34}
34%|██████████████████████████████████████▍ | 232/681 [17:08<22:40, 3.03s/it] 34%|██████████████████████████████████████▋ | 233/681 [17:11<23:07, 3.10s/it] {'loss': 0.7772, 'grad_norm': 14.02349853515625, 'learning_rate': 4.174733034541245e-07, 'rewards/chosen': -1.0092403888702393, 'rewards/rejected': -2.045924663543701, 'rewards/accuracies': 0.828125, 'rewards/margins': 1.036684513092041, 'logps/chosen': -116.73382568359375, 'logps/rejected': -169.069580078125, 'logps/ref_chosen': -89.05032348632812, 'logps/ref_rejected': -112.75917053222656, 'logits/chosen': -7.088686943054199, 'logits/rejected': -6.224997520446777, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.03660925105214119, 'kl/avg_steps': 0.625, 'epoch': 0.34}
34%|██████████████████████████████████████▋ | 233/681 [17:12<23:07, 3.10s/it] 34%|██████████████████████████████████████▊ | 234/681 [17:15<23:39, 3.18s/it] {'loss': 0.7907, 'grad_norm': 13.894185066223145, 'learning_rate': 4.165182829193126e-07, 'rewards/chosen': -0.9510073661804199, 'rewards/rejected': -2.022789239883423, 'rewards/accuracies': 0.875, 'rewards/margins': 1.071781873703003, 'logps/chosen': -100.55630493164062, 'logps/rejected': -162.41140747070312, 'logps/ref_chosen': -74.318359375, 'logps/ref_rejected': -106.38758850097656, 'logits/chosen': -7.294958114624023, 'logits/rejected': -6.297677040100098, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.03638186678290367, 'kl/avg_steps': 0.625, 'epoch': 0.34}
34%|██████████████████████████████████████▊ | 234/681 [17:15<23:39, 3.18s/it] 35%|██████████████████████████████████████▉ | 235/681 [17:18<23:15, 3.13s/it] {'loss': 0.8641, 'grad_norm': 13.863997459411621, 'learning_rate': 4.1555887447288255e-07, 'rewards/chosen': -1.1314321756362915, 'rewards/rejected': -1.9421896934509277, 'rewards/accuracies': 0.875, 'rewards/margins': 0.810757577419281, 'logps/chosen': -129.61538696289062, 'logps/rejected': -151.29678344726562, 'logps/ref_chosen': -98.217041015625, 'logps/ref_rejected': -97.24677276611328, 'logits/chosen': -6.75037956237793, 'logits/rejected': -6.180291175842285, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.03615589067339897, 'kl/avg_steps': 0.53125, 'epoch': 0.35}
35%|██████████████████████████████████████▉ | 235/681 [17:18<23:15, 3.13s/it] 35%|███████████████████████████████████████▏ | 236/681 [17:21<23:37, 3.19s/it] {'loss': 0.8264, 'grad_norm': 14.032482147216797, 'learning_rate': 4.1459510339613946e-07, 'rewards/chosen': -0.9487680196762085, 'rewards/rejected': -1.9531795978546143, 'rewards/accuracies': 0.921875, 'rewards/margins': 1.0044116973876953, 'logps/chosen': -105.3929672241211, 'logps/rejected': -163.83856201171875, 'logps/ref_chosen': -78.83773040771484, 'logps/ref_rejected': -109.06343078613281, 'logits/chosen': -6.762874126434326, 'logits/rejected': -6.526078224182129, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.03596482798457146, 'kl/avg_steps': 0.71875, 'epoch': 0.35}
35%|███████████████████████████████████████▏ | 236/681 [17:21<23:37, 3.19s/it] 35%|███████████████████████████████████████▎ | 237/681 [17:24<23:39, 3.20s/it] {'loss': 0.9932, 'grad_norm': 15.7308931350708, 'learning_rate': 4.136269950853473e-07, 'rewards/chosen': -1.0902290344238281, 'rewards/rejected': -1.89679753780365, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8065685033798218, 'logps/chosen': -115.78558349609375, 'logps/rejected': -153.35638427734375, 'logps/ref_chosen': -85.21128845214844, 'logps/ref_rejected': -99.90999603271484, 'logits/chosen': -7.501216888427734, 'logits/rejected': -6.719086647033691, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.03570817783474922, 'kl/avg_steps': 0.46875, 'epoch': 0.35}
35%|███████████████████████████████████████▎ | 237/681 [17:24<23:39, 3.20s/it] 35%|███████████████████████████████████████▍ | 238/681 [17:28<23:39, 3.20s/it] {'loss': 0.8772, 'grad_norm': 13.518217086791992, 'learning_rate': 4.126545750510605e-07, 'rewards/chosen': -1.190969705581665, 'rewards/rejected': -2.0659611225128174, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.8749915361404419, 'logps/chosen': -112.39120483398438, 'logps/rejected': -153.98170471191406, 'logps/ref_chosen': -78.73123168945312, 'logps/ref_rejected': -95.41840362548828, 'logits/chosen': -6.873219013214111, 'logits/rejected': -5.906381607055664, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.03554157540202141, 'kl/avg_steps': 0.625, 'epoch': 0.35}
35%|███████████████████████████████████████▍ | 238/681 [17:28<23:39, 3.20s/it] 35%|███████████████████████████████████████▋ | 239/681 [17:30<22:38, 3.07s/it] {'loss': 0.9094, 'grad_norm': 14.817635536193848, 'learning_rate': 4.116778689174514e-07, 'rewards/chosen': -1.117043375968933, 'rewards/rejected': -1.9282580614089966, 'rewards/accuracies': 0.75, 'rewards/margins': 0.8112146854400635, 'logps/chosen': -124.30506134033203, 'logps/rejected': -155.46278381347656, 'logps/ref_chosen': -92.60093688964844, 'logps/ref_rejected': -100.51769256591797, 'logits/chosen': -7.087869644165039, 'logits/rejected': -6.191803932189941, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.035320818424224854, 'kl/avg_steps': 0.5, 'epoch': 0.35}
35%|███████████████████████████████████████▋ | 239/681 [17:30<22:38, 3.07s/it] 35%|███████████████████████████████████████▊ | 240/681 [17:33<22:33, 3.07s/it] {'loss': 0.9385, 'grad_norm': 17.138147354125977, 'learning_rate': 4.106969024216348e-07, 'rewards/chosen': -1.1331913471221924, 'rewards/rejected': -1.8612143993377686, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.7280229926109314, 'logps/chosen': -118.43206787109375, 'logps/rejected': -133.68617248535156, 'logps/ref_chosen': -86.15977478027344, 'logps/ref_rejected': -80.45567321777344, 'logits/chosen': -7.254019260406494, 'logits/rejected': -7.064877986907959, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.03514509275555611, 'kl/avg_steps': 0.375, 'epoch': 0.35}
35%|███████████████████████████████████████▊ | 240/681 [17:33<22:33, 3.07s/it] 35%|███████████████████████████████████████▉ | 241/681 [17:36<22:04, 3.01s/it] {'loss': 0.6851, 'grad_norm': 13.10658073425293, 'learning_rate': 4.097117014129903e-07, 'rewards/chosen': -0.9694840312004089, 'rewards/rejected': -2.225541353225708, 'rewards/accuracies': 0.859375, 'rewards/margins': 1.2560572624206543, 'logps/chosen': -128.86328125, 'logps/rejected': -158.0999298095703, 'logps/ref_chosen': -101.04594421386719, 'logps/ref_rejected': -94.04934692382812, 'logits/chosen': -7.0530242919921875, 'logits/rejected': -6.54688835144043, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.03501379117369652, 'kl/avg_steps': 0.6875, 'epoch': 0.35}
35%|███████████████████████████████████████▉ | 241/681 [17:36<22:04, 3.01s/it] 36%|████████████████████████████████████████▏ | 242/681 [17:39<22:04, 3.02s/it] {'loss': 0.8351, 'grad_norm': 14.712688446044922, 'learning_rate': 4.087222918524807e-07, 'rewards/chosen': -1.184476613998413, 'rewards/rejected': -2.078831434249878, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8943547606468201, 'logps/chosen': -129.89794921875, 'logps/rejected': -150.88523864746094, 'logps/ref_chosen': -95.67266082763672, 'logps/ref_rejected': -90.65454864501953, 'logits/chosen': -7.671010494232178, 'logits/rejected': -7.0201215744018555, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.034774716943502426, 'kl/avg_steps': 0.6875, 'epoch': 0.36}
36%|████████████████████████████████████████▏ | 242/681 [17:39<22:04, 3.02s/it] 36%|████████████████████████████████████████▎ | 243/681 [17:42<22:11, 3.04s/it] {'loss': 0.8947, 'grad_norm': 14.254800796508789, 'learning_rate': 4.07728699811968e-07, 'rewards/chosen': -1.1293984651565552, 'rewards/rejected': -2.0041677951812744, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.8747692704200745, 'logps/chosen': -130.82611083984375, 'logps/rejected': -141.62124633789062, 'logps/ref_chosen': -98.03140258789062, 'logps/ref_rejected': -83.18806457519531, 'logits/chosen': -7.436939239501953, 'logits/rejected': -7.066817283630371, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.03453727439045906, 'kl/avg_steps': 0.5625, 'epoch': 0.36}
36%|████████████████████████████████████████▎ | 243/681 [17:42<22:11, 3.04s/it] 36%|████████████████████████████████████████▍ | 244/681 [17:46<22:17, 3.06s/it] {'loss': 0.8616, 'grad_norm': 14.085189819335938, 'learning_rate': 4.067309514735267e-07, 'rewards/chosen': -1.187152624130249, 'rewards/rejected': -2.082256317138672, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8951038122177124, 'logps/chosen': -123.65585327148438, 'logps/rejected': -163.68592834472656, 'logps/ref_chosen': -88.89391326904297, 'logps/ref_rejected': -102.57278442382812, 'logits/chosen': -7.297283172607422, 'logits/rejected': -6.48216438293457, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.03434408828616142, 'kl/avg_steps': 0.65625, 'epoch': 0.36}
36%|████████████████████████████████████████▍ | 244/681 [17:46<22:17, 3.06s/it] 36%|████████████████████████████████████████▋ | 245/681 [17:49<22:43, 3.13s/it] {'loss': 0.777, 'grad_norm': 13.635170936584473, 'learning_rate': 4.057290731287531e-07, 'rewards/chosen': -1.0023081302642822, 'rewards/rejected': -1.9891510009765625, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.9868428707122803, 'logps/chosen': -133.73660278320312, 'logps/rejected': -151.42526245117188, 'logps/ref_chosen': -104.19400024414062, 'logps/ref_rejected': -92.65645599365234, 'logits/chosen': -7.1143999099731445, 'logits/rejected': -6.517737865447998, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.03412017226219177, 'kl/avg_steps': 0.78125, 'epoch': 0.36}
36%|████████████████████████████████████████▋ | 245/681 [17:49<22:43, 3.13s/it] 36%|████████████████████████████████████████▊ | 246/681 [17:52<22:35, 3.12s/it] {'loss': 0.9225, 'grad_norm': 15.270983695983887, 'learning_rate': 4.047230911780736e-07, 'rewards/chosen': -1.277728796005249, 'rewards/rejected': -2.0303685665130615, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.7526397705078125, 'logps/chosen': -141.06942749023438, 'logps/rejected': -151.3291015625, 'logps/ref_chosen': -103.21904754638672, 'logps/ref_rejected': -90.9922103881836, 'logits/chosen': -6.8056464195251465, 'logits/rejected': -6.480011940002441, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.03385567665100098, 'kl/avg_steps': 0.53125, 'epoch': 0.36}
36%|████████████████████████████████████████▊ | 246/681 [17:52<22:35, 3.12s/it] 36%|████████████████████████████████████████▉ | 247/681 [17:55<22:10, 3.07s/it] {'loss': 0.7459, 'grad_norm': 12.317371368408203, 'learning_rate': 4.0371303213004814e-07, 'rewards/chosen': -1.1581556797027588, 'rewards/rejected': -2.265951156616211, 'rewards/accuracies': 0.875, 'rewards/margins': 1.1077954769134521, 'logps/chosen': -121.5998306274414, 'logps/rejected': -179.18167114257812, 'logps/ref_chosen': -86.99436950683594, 'logps/ref_rejected': -111.33802795410156, 'logits/chosen': -7.767994403839111, 'logits/rejected': -7.044826507568359, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.03367676958441734, 'kl/avg_steps': 0.78125, 'epoch': 0.36}
36%|████████████████████████████████████████▉ | 247/681 [17:55<22:10, 3.07s/it] 36%|█████████████████████████████████████████▏ | 248/681 [17:58<22:03, 3.06s/it] {'loss': 0.9066, 'grad_norm': 15.562211990356445, 'learning_rate': 4.0269892260067197e-07, 'rewards/chosen': -1.2984592914581299, 'rewards/rejected': -2.0758614540100098, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.7774021625518799, 'logps/chosen': -113.81005859375, 'logps/rejected': -160.82879638671875, 'logps/ref_chosen': -74.7855224609375, 'logps/ref_rejected': -98.27689361572266, 'logits/chosen': -7.3634748458862305, 'logits/rejected': -6.64450740814209, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.03341570869088173, 'kl/avg_steps': 0.5625, 'epoch': 0.36}
36%|█████████████████████████████████████████▏ | 248/681 [17:58<22:03, 3.06s/it] 37%|█████████████████████████████████████████▎ | 249/681 [18:01<21:29, 2.99s/it] {'loss': 0.9861, 'grad_norm': 17.970985412597656, 'learning_rate': 4.0168078931267426e-07, 'rewards/chosen': -1.3996614217758179, 'rewards/rejected': -2.111713409423828, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.7120518684387207, 'logps/chosen': -137.87208557128906, 'logps/rejected': -151.0142059326172, 'logps/ref_chosen': -95.70379638671875, 'logps/ref_rejected': -87.14646911621094, 'logits/chosen': -7.139558792114258, 'logits/rejected': -6.51041316986084, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.0332287959754467, 'kl/avg_steps': 0.4375, 'epoch': 0.37}
37%|█████████████████████████████████████████▎ | 249/681 [18:01<21:29, 2.99s/it] 37%|█████████████████████████████████████████▍ | 250/681 [18:04<21:35, 3.01s/it] {'loss': 0.9481, 'grad_norm': 18.755630493164062, 'learning_rate': 4.006586590948141e-07, 'rewards/chosen': -1.287900686264038, 'rewards/rejected': -2.0340938568115234, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.7461932897567749, 'logps/chosen': -153.10398864746094, 'logps/rejected': -142.95217895507812, 'logps/ref_chosen': -114.05220794677734, 'logps/ref_rejected': -81.08768463134766, 'logits/chosen': -7.073131561279297, 'logits/rejected': -6.734502792358398, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.03308405354619026, 'kl/avg_steps': 0.5625, 'epoch': 0.37}
37%|█████████████████████████████████████████▍ | 250/681 [18:04<21:35, 3.01s/it] 37%|█████████████████████████████████████████▋ | 251/681 [18:07<21:20, 2.98s/it] {'loss': 0.9872, 'grad_norm': 15.883668899536133, 'learning_rate': 3.9963255888117325e-07, 'rewards/chosen': -1.2341444492340088, 'rewards/rejected': -1.9472013711929321, 'rewards/accuracies': 0.75, 'rewards/margins': 0.7130569219589233, 'logps/chosen': -135.24278259277344, 'logps/rejected': -142.99111938476562, 'logps/ref_chosen': -97.71128845214844, 'logps/ref_rejected': -83.52742004394531, 'logits/chosen': -7.458626747131348, 'logits/rejected': -6.964792251586914, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.03289899602532387, 'kl/avg_steps': 0.28125, 'epoch': 0.37}
37%|█████████████████████████████████████████▋ | 251/681 [18:07<21:20, 2.98s/it] 37%|█████████████████████████████████████████▊ | 252/681 [18:10<21:50, 3.05s/it] {'loss': 0.93, 'grad_norm': 16.21406364440918, 'learning_rate': 3.9860251571044666e-07, 'rewards/chosen': -1.2076178789138794, 'rewards/rejected': -1.948510766029358, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7408928871154785, 'logps/chosen': -145.88766479492188, 'logps/rejected': -151.3180389404297, 'logps/ref_chosen': -108.9861068725586, 'logps/ref_rejected': -91.56424713134766, 'logits/chosen': -7.022328853607178, 'logits/rejected': -6.755680084228516, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.03280672803521156, 'kl/avg_steps': 0.5, 'epoch': 0.37}
37%|█████████████████████████████████████████▊ | 252/681 [18:10<21:50, 3.05s/it] 37%|█████████████████████████████████████████▉ | 253/681 [18:13<22:00, 3.09s/it] {'loss': 0.9468, 'grad_norm': 15.019343376159668, 'learning_rate': 3.9756855672522986e-07, 'rewards/chosen': -1.3202366828918457, 'rewards/rejected': -2.0234005451202393, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7031639218330383, 'logps/chosen': -140.8556365966797, 'logps/rejected': -168.1231689453125, 'logps/ref_chosen': -100.21630859375, 'logps/ref_rejected': -105.67670440673828, 'logits/chosen': -6.749828815460205, 'logits/rejected': -6.487143516540527, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.03264351189136505, 'kl/avg_steps': 0.625, 'epoch': 0.37}
37%|█████████████████████████████████████████▉ | 253/681 [18:13<22:00, 3.09s/it] 37%|██████████████████████████████████████████▏ | 254/681 [18:16<22:10, 3.12s/it] {'loss': 0.971, 'grad_norm': 14.38194751739502, 'learning_rate': 3.965307091713037e-07, 'rewards/chosen': -1.2845338582992554, 'rewards/rejected': -1.9614899158477783, 'rewards/accuracies': 0.75, 'rewards/margins': 0.676956057548523, 'logps/chosen': -138.41323852539062, 'logps/rejected': -154.52410888671875, 'logps/ref_chosen': -98.73518371582031, 'logps/ref_rejected': -93.73825073242188, 'logits/chosen': -7.187813758850098, 'logits/rejected': -6.099554061889648, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.032440755516290665, 'kl/avg_steps': 0.359375, 'epoch': 0.37}
37%|██████████████████████████████████████████▏ | 254/681 [18:16<22:10, 3.12s/it] 37%|██████████████████████████████████████████▎ | 255/681 [18:19<21:41, 3.06s/it] {'loss': 0.8306, 'grad_norm': 14.393223762512207, 'learning_rate': 3.954890003969163e-07, 'rewards/chosen': -1.3301056623458862, 'rewards/rejected': -2.233675003051758, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9035694599151611, 'logps/chosen': -131.79598999023438, 'logps/rejected': -166.75144958496094, 'logps/ref_chosen': -90.382568359375, 'logps/ref_rejected': -97.07625579833984, 'logits/chosen': -7.347094535827637, 'logits/rejected': -6.871660232543945, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.03232458978891373, 'kl/avg_steps': 0.71875, 'epoch': 0.37}
37%|██████████████████████████████████████████▎ | 255/681 [18:19<21:41, 3.06s/it] 38%|██████████████████████████████████████████▍ | 256/681 [18:22<21:20, 3.01s/it] {'loss': 0.8817, 'grad_norm': 14.8760986328125, 'learning_rate': 3.944434578520628e-07, 'rewards/chosen': -1.1998153924942017, 'rewards/rejected': -2.102677822113037, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.9028624296188354, 'logps/chosen': -126.28778076171875, 'logps/rejected': -164.5001220703125, 'logps/ref_chosen': -88.7528076171875, 'logps/ref_rejected': -98.49382781982422, 'logits/chosen': -7.483323097229004, 'logits/rejected': -6.505797386169434, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.03209391236305237, 'kl/avg_steps': 0.65625, 'epoch': 0.38}
38%|██████████████████████████████████████████▍ | 256/681 [18:22<21:20, 3.01s/it] 38%|██████████████████████████████████████████▋ | 257/681 [18:25<21:30, 3.04s/it] {'loss': 0.8809, 'grad_norm': 15.880992889404297, 'learning_rate': 3.933941090877615e-07, 'rewards/chosen': -1.2686723470687866, 'rewards/rejected': -2.1270086765289307, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8583362698554993, 'logps/chosen': -122.74734497070312, 'logps/rejected': -153.03878784179688, 'logps/ref_chosen': -82.80352783203125, 'logps/ref_rejected': -85.8677978515625, 'logits/chosen': -7.26761531829834, 'logits/rejected': -6.992372512817383, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.03188467025756836, 'kl/avg_steps': 0.5625, 'epoch': 0.38}
38%|██████████████████████████████████████████▋ | 257/681 [18:25<21:30, 3.04s/it] 38%|██████████████████████████████████████████▊ | 258/681 [18:28<21:10, 3.00s/it] {'loss': 0.8631, 'grad_norm': 14.805852890014648, 'learning_rate': 3.923409817553284e-07, 'rewards/chosen': -1.2512186765670776, 'rewards/rejected': -2.0762269496917725, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8250081539154053, 'logps/chosen': -129.7963409423828, 'logps/rejected': -169.39410400390625, 'logps/ref_chosen': -90.187744140625, 'logps/ref_rejected': -103.47068786621094, 'logits/chosen': -7.284233093261719, 'logits/rejected': -6.5939788818359375, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.031706321984529495, 'kl/avg_steps': 0.5625, 'epoch': 0.38}
38%|██████████████████████████████████████████▊ | 258/681 [18:28<21:10, 3.00s/it] 38%|██████████████████████████████████████████▉ | 259/681 [18:31<21:05, 3.00s/it] {'loss': 0.9968, 'grad_norm': 15.601424217224121, 'learning_rate': 3.9128410360564793e-07, 'rewards/chosen': -1.3569505214691162, 'rewards/rejected': -2.0308425426483154, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6738921403884888, 'logps/chosen': -133.95770263671875, 'logps/rejected': -159.41954040527344, 'logps/ref_chosen': -90.77254486083984, 'logps/ref_rejected': -94.58816528320312, 'logits/chosen': -6.8948163986206055, 'logits/rejected': -6.679216384887695, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.03152897208929062, 'kl/avg_steps': 0.5625, 'epoch': 0.38}
38%|██████████████████████████████████████████▉ | 259/681 [18:31<21:05, 3.00s/it] 38%|███████████████████████████████████████████▏ | 260/681 [18:35<21:53, 3.12s/it] {'loss': 1.0562, 'grad_norm': 16.810916900634766, 'learning_rate': 3.9022350248844246e-07, 'rewards/chosen': -1.3429489135742188, 'rewards/rejected': -1.9885942935943604, 'rewards/accuracies': 0.75, 'rewards/margins': 0.6456452012062073, 'logps/chosen': -118.55705261230469, 'logps/rejected': -164.71981811523438, 'logps/ref_chosen': -75.59269714355469, 'logps/ref_rejected': -100.84554290771484, 'logits/chosen': -7.622888088226318, 'logits/rejected': -6.941500186920166, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.03135261312127113, 'kl/avg_steps': 0.5, 'epoch': 0.38}
38%|███████████████████████████████████████████▏ | 260/681 [18:35<21:53, 3.12s/it] 38%|███████████████████████████████████████████▎ | 261/681 [18:37<20:45, 2.96s/it] {'loss': 0.8049, 'grad_norm': 13.099295616149902, 'learning_rate': 3.891592063515376e-07, 'rewards/chosen': -1.0897305011749268, 'rewards/rejected': -2.0056087970733643, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9158782958984375, 'logps/chosen': -129.3551025390625, 'logps/rejected': -158.41415405273438, 'logps/ref_chosen': -94.25491333007812, 'logps/ref_rejected': -93.65699768066406, 'logits/chosen': -7.312009811401367, 'logits/rejected': -7.067169666290283, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.031196629628539085, 'kl/avg_steps': 0.65625, 'epoch': 0.38}
38%|███████████████████████████████████████████▎ | 261/681 [18:37<20:45, 2.96s/it] 38%|███████████████████████████████████████████▍ | 262/681 [18:40<20:24, 2.92s/it] {'loss': 0.8587, 'grad_norm': 13.834671020507812, 'learning_rate': 3.880912432401264e-07, 'rewards/chosen': -1.1732381582260132, 'rewards/rejected': -2.021843194961548, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8486050367355347, 'logps/chosen': -123.33172607421875, 'logps/rejected': -156.57522583007812, 'logps/ref_chosen': -85.26730346679688, 'logps/ref_rejected': -90.82609558105469, 'logits/chosen': -7.694709777832031, 'logits/rejected': -6.795375823974609, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.03099323809146881, 'kl/avg_steps': 0.6875, 'epoch': 0.38}
38%|███████████████████████████████████████████▍ | 262/681 [18:40<20:24, 2.92s/it] 39%|███████████████████████████████████████████▋ | 263/681 [18:43<20:22, 2.93s/it] {'loss': 0.7435, 'grad_norm': 13.023946762084961, 'learning_rate': 3.870196412960302e-07, 'rewards/chosen': -0.9609699845314026, 'rewards/rejected': -2.075908660888672, 'rewards/accuracies': 0.859375, 'rewards/margins': 1.114938497543335, 'logps/chosen': -127.14653015136719, 'logps/rejected': -169.84298706054688, 'logps/ref_chosen': -95.75790405273438, 'logps/ref_rejected': -101.83377075195312, 'logits/chosen': -6.901250839233398, 'logits/rejected': -6.722340106964111, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.030781613662838936, 'kl/avg_steps': 0.75, 'epoch': 0.39}
39%|███████████████████████████████████████████▋ | 263/681 [18:43<20:22, 2.93s/it] 39%|███████████████████████████████████████████▊ | 264/681 [18:46<20:53, 3.01s/it] {'loss': 0.8463, 'grad_norm': 13.047411918640137, 'learning_rate': 3.8594442875695665e-07, 'rewards/chosen': -1.0397236347198486, 'rewards/rejected': -1.9546151161193848, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9148914217948914, 'logps/chosen': -124.76637268066406, 'logps/rejected': -164.76083374023438, 'logps/ref_chosen': -90.6226577758789, 'logps/ref_rejected': -100.32554626464844, 'logits/chosen': -7.175429344177246, 'logits/rejected': -6.558530330657959, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.030552471056580544, 'kl/avg_steps': 0.5625, 'epoch': 0.39}
39%|███████████████████████████████████████████▊ | 264/681 [18:46<20:53, 3.01s/it] 39%|███████████████████████████████████████████▉ | 265/681 [18:49<20:48, 3.00s/it] {'loss': 0.8792, 'grad_norm': 12.617591857910156, 'learning_rate': 3.848656339557562e-07, 'rewards/chosen': -1.1058483123779297, 'rewards/rejected': -1.9895697832107544, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.8837213516235352, 'logps/chosen': -128.9173583984375, 'logps/rejected': -160.58474731445312, 'logps/ref_chosen': -92.37232971191406, 'logps/ref_rejected': -94.62757110595703, 'logits/chosen': -6.967649936676025, 'logits/rejected': -6.795099258422852, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.030381573364138603, 'kl/avg_steps': 0.5625, 'epoch': 0.39}
39%|███████████████████████████████████████████▉ | 265/681 [18:49<20:48, 3.00s/it] 39%|████████████████████████████████████████████▏ | 266/681 [18:52<20:54, 3.02s/it] {'loss': 0.9138, 'grad_norm': 15.880788803100586, 'learning_rate': 3.8378328531967507e-07, 'rewards/chosen': -1.2558131217956543, 'rewards/rejected': -2.005937099456787, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7501237392425537, 'logps/chosen': -143.80422973632812, 'logps/rejected': -141.06625366210938, 'logps/ref_chosen': -102.20002746582031, 'logps/ref_rejected': -74.36642456054688, 'logits/chosen': -7.422516345977783, 'logits/rejected': -7.111500263214111, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.030211633071303368, 'kl/avg_steps': 0.34375, 'epoch': 0.39}
39%|████████████████████████████████████████████▏ | 266/681 [18:52<20:54, 3.02s/it] 39%|████████████████████████████████████████████▎ | 267/681 [18:55<21:07, 3.06s/it] {'loss': 0.7978, 'grad_norm': 13.670655250549316, 'learning_rate': 3.8269741136960646e-07, 'rewards/chosen': -1.04282808303833, 'rewards/rejected': -1.9617595672607422, 'rewards/accuracies': 0.875, 'rewards/margins': 0.9189317226409912, 'logps/chosen': -139.10177612304688, 'logps/rejected': -161.63311767578125, 'logps/ref_chosen': -104.28599548339844, 'logps/ref_rejected': -95.98719024658203, 'logits/chosen': -7.119365692138672, 'logits/rejected': -6.709759712219238, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.0301081370562315, 'kl/avg_steps': 0.65625, 'epoch': 0.39}
39%|████████████████████████████████████████████▎ | 267/681 [18:55<21:07, 3.06s/it] 39%|████████████████████████████████████████████▍ | 268/681 [18:58<21:13, 3.08s/it] {'loss': 0.7894, 'grad_norm': 13.244879722595215, 'learning_rate': 3.8160804071933894e-07, 'rewards/chosen': -1.030745029449463, 'rewards/rejected': -1.9844391345977783, 'rewards/accuracies': 0.875, 'rewards/margins': 0.9536941051483154, 'logps/chosen': -121.36919403076172, 'logps/rejected': -176.08946228027344, 'logps/ref_chosen': -86.69622039794922, 'logps/ref_rejected': -109.19183349609375, 'logits/chosen': -7.436177730560303, 'logits/rejected': -6.697609901428223, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.029911840334534645, 'kl/avg_steps': 0.75, 'epoch': 0.39}
39%|████████████████████████████████████████████▍ | 268/681 [18:58<21:13, 3.08s/it] 40%|████████████████████████████████████████████▋ | 269/681 [19:01<20:51, 3.04s/it] {'loss': 0.8031, 'grad_norm': 13.288008689880371, 'learning_rate': 3.8051520207480204e-07, 'rewards/chosen': -1.0468546152114868, 'rewards/rejected': -2.0211329460144043, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9742782115936279, 'logps/chosen': -140.38177490234375, 'logps/rejected': -181.06021118164062, 'logps/ref_chosen': -104.97181701660156, 'logps/ref_rejected': -112.4764633178711, 'logits/chosen': -7.45705509185791, 'logits/rejected': -6.695377826690674, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.02968917228281498, 'kl/avg_steps': 0.5625, 'epoch': 0.4}
40%|████████████████████████████████████████████▋ | 269/681 [19:01<20:51, 3.04s/it] 40%|████████████████████████████████████████████▊ | 270/681 [19:04<20:59, 3.07s/it] {'loss': 0.9174, 'grad_norm': 13.801987648010254, 'learning_rate': 3.794189242333106e-07, 'rewards/chosen': -1.1459414958953857, 'rewards/rejected': -2.0066967010498047, 'rewards/accuracies': 0.75, 'rewards/margins': 0.8607551455497742, 'logps/chosen': -139.99703979492188, 'logps/rejected': -186.19866943359375, 'logps/ref_chosen': -101.07383728027344, 'logps/ref_rejected': -117.75289916992188, 'logits/chosen': -7.553506851196289, 'logits/rejected': -6.664368152618408, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.029523104429244995, 'kl/avg_steps': 0.5625, 'epoch': 0.4}
40%|████████████████████████████████████████████▊ | 270/681 [19:05<20:59, 3.07s/it] 40%|████████████████████████████████████████████▉ | 271/681 [19:07<20:27, 2.99s/it] {'loss': 0.8123, 'grad_norm': 12.916444778442383, 'learning_rate': 3.7831923608280514e-07, 'rewards/chosen': -0.9474867582321167, 'rewards/rejected': -1.9652299880981445, 'rewards/accuracies': 0.890625, 'rewards/margins': 1.0177432298660278, 'logps/chosen': -129.179443359375, 'logps/rejected': -166.06207275390625, 'logps/ref_chosen': -96.72459411621094, 'logps/ref_rejected': -98.5244140625, 'logits/chosen': -6.938437461853027, 'logits/rejected': -6.5058183670043945, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.029357966035604477, 'kl/avg_steps': 0.75, 'epoch': 0.4}
40%|████████████████████████████████████████████▉ | 271/681 [19:07<20:27, 2.99s/it] 40%|█████████████████████████████████████████████▏ | 272/681 [19:10<20:39, 3.03s/it] {'loss': 0.8282, 'grad_norm': 14.018041610717773, 'learning_rate': 3.772161666010912e-07, 'rewards/chosen': -0.7937983870506287, 'rewards/rejected': -1.7605557441711426, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.9667574167251587, 'logps/chosen': -108.3265380859375, 'logps/rejected': -169.4478759765625, 'logps/ref_chosen': -80.97721862792969, 'logps/ref_rejected': -108.55535888671875, 'logits/chosen': -7.514509677886963, 'logits/rejected': -7.041294574737549, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.02913942001760006, 'kl/avg_steps': 0.625, 'epoch': 0.4}
40%|█████████████████████████████████████████████▏ | 272/681 [19:10<20:39, 3.03s/it] 40%|█████████████████████████████████████████████▎ | 273/681 [19:13<20:27, 3.01s/it] {'loss': 0.8615, 'grad_norm': 13.580171585083008, 'learning_rate': 3.761097448550755e-07, 'rewards/chosen': -1.0409189462661743, 'rewards/rejected': -2.039475440979004, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9985566139221191, 'logps/chosen': -128.26251220703125, 'logps/rejected': -168.29635620117188, 'logps/ref_chosen': -92.22460174560547, 'logps/ref_rejected': -97.3630599975586, 'logits/chosen': -7.261774063110352, 'logits/rejected': -6.644618988037109, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.028958430513739586, 'kl/avg_steps': 0.59375, 'epoch': 0.4}
40%|█████████████████████████████████████████████▎ | 273/681 [19:13<20:27, 3.01s/it] 40%|█████████████████████████████████████████████▍ | 274/681 [19:16<20:20, 3.00s/it] {'loss': 0.8819, 'grad_norm': 15.386717796325684, 'learning_rate': 3.75e-07, 'rewards/chosen': -1.0782254934310913, 'rewards/rejected': -1.8726218938827515, 'rewards/accuracies': 0.875, 'rewards/margins': 0.7943964004516602, 'logps/chosen': -119.94572448730469, 'logps/rejected': -149.46990966796875, 'logps/ref_chosen': -82.2608871459961, 'logps/ref_rejected': -83.87699127197266, 'logits/chosen': -7.7138991355896, 'logits/rejected': -6.974665641784668, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.028787503018975258, 'kl/avg_steps': 0.75, 'epoch': 0.4}
40%|█████████████████████████████████████████████▍ | 274/681 [19:16<20:20, 3.00s/it] 40%|█████████████████████████████████████████████▋ | 275/681 [19:20<21:08, 3.12s/it] {'loss': 0.9047, 'grad_norm': 15.320517539978027, 'learning_rate': 3.738869612786737e-07, 'rewards/chosen': -0.8999394178390503, 'rewards/rejected': -1.7487740516662598, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.8488346338272095, 'logps/chosen': -111.30049896240234, 'logps/rejected': -160.444580078125, 'logps/ref_chosen': -79.68695831298828, 'logps/ref_rejected': -98.7509765625, 'logits/chosen': -7.628604888916016, 'logits/rejected': -6.829320907592773, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.028573205694556236, 'kl/avg_steps': 0.71875, 'epoch': 0.4}
40%|█████████████████████████████████████████████▋ | 275/681 [19:20<21:08, 3.12s/it] 41%|█████████████████████████████████████████████▊ | 276/681 [19:23<20:25, 3.03s/it] {'loss': 0.9138, 'grad_norm': 15.482325553894043, 'learning_rate': 3.7277065802070204e-07, 'rewards/chosen': -1.1226071119308472, 'rewards/rejected': -1.9377222061157227, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8151148557662964, 'logps/chosen': -126.2287826538086, 'logps/rejected': -146.60427856445312, 'logps/ref_chosen': -86.53970336914062, 'logps/ref_rejected': -77.85394287109375, 'logits/chosen': -7.228045463562012, 'logits/rejected': -6.846656799316406, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.028369300067424774, 'kl/avg_steps': 0.53125, 'epoch': 0.41}
41%|█████████████████████████████████████████████▊ | 276/681 [19:23<20:25, 3.03s/it] 41%|█████████████████████████████████████████████▉ | 277/681 [19:25<19:39, 2.92s/it] {'loss': 0.898, 'grad_norm': 15.835052490234375, 'learning_rate': 3.71651119641714e-07, 'rewards/chosen': -1.0951545238494873, 'rewards/rejected': -1.8525817394256592, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.7574270963668823, 'logps/chosen': -123.2606430053711, 'logps/rejected': -164.99749755859375, 'logps/ref_chosen': -84.24411010742188, 'logps/ref_rejected': -98.83421325683594, 'logits/chosen': -7.30094051361084, 'logits/rejected': -6.272882461547852, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.02821938507258892, 'kl/avg_steps': 0.65625, 'epoch': 0.41}
41%|█████████████████████████████████████████████▉ | 277/681 [19:25<19:39, 2.92s/it] 41%|██████████████████████████████████████████████▏ | 278/681 [19:28<19:56, 2.97s/it] {'loss': 0.9088, 'grad_norm': 13.882027626037598, 'learning_rate': 3.705283756425872e-07, 'rewards/chosen': -1.0521965026855469, 'rewards/rejected': -1.8657093048095703, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.8135128021240234, 'logps/chosen': -120.13731384277344, 'logps/rejected': -164.93141174316406, 'logps/ref_chosen': -82.431884765625, 'logps/ref_rejected': -97.85691833496094, 'logits/chosen': -7.213944435119629, 'logits/rejected': -6.768559455871582, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.028035402297973633, 'kl/avg_steps': 0.65625, 'epoch': 0.41}
41%|██████████████████████████████████████████████▏ | 278/681 [19:28<19:56, 2.97s/it] 41%|██████████████████████████████████████████████▎ | 279/681 [19:31<19:54, 2.97s/it] {'loss': 0.8719, 'grad_norm': 13.638445854187012, 'learning_rate': 3.6940245560867e-07, 'rewards/chosen': -1.119484305381775, 'rewards/rejected': -1.9745573997497559, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.8550732135772705, 'logps/chosen': -125.54033660888672, 'logps/rejected': -165.55284118652344, 'logps/ref_chosen': -85.16799926757812, 'logps/ref_rejected': -94.12664794921875, 'logits/chosen': -7.466989517211914, 'logits/rejected': -7.357805252075195, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.02785261906683445, 'kl/avg_steps': 0.59375, 'epoch': 0.41}
41%|██████████████████████████████████████████████▎ | 279/681 [19:31<19:54, 2.97s/it] 41%|██████████████████████████████████████████████▍ | 280/681 [19:35<20:29, 3.06s/it] {'loss': 0.7947, 'grad_norm': 13.675751686096191, 'learning_rate': 3.6827338920900253e-07, 'rewards/chosen': -1.0486705303192139, 'rewards/rejected': -2.0240159034729004, 'rewards/accuracies': 0.875, 'rewards/margins': 0.9753453731536865, 'logps/chosen': -123.91621398925781, 'logps/rejected': -177.7848663330078, 'logps/ref_chosen': -85.85641479492188, 'logps/ref_rejected': -104.11859130859375, 'logits/chosen': -7.190787315368652, 'logits/rejected': -6.597271919250488, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.027688222005963326, 'kl/avg_steps': 0.625, 'epoch': 0.41}
41%|██████████████████████████████████████████████▍ | 280/681 [19:35<20:29, 3.06s/it] 41%|██████████████████████████████████████████████▋ | 281/681 [19:38<20:49, 3.12s/it] {'loss': 0.8278, 'grad_norm': 11.921332359313965, 'learning_rate': 3.6714120619553435e-07, 'rewards/chosen': -1.0626280307769775, 'rewards/rejected': -2.0622613430023193, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9996333718299866, 'logps/chosen': -125.37873840332031, 'logps/rejected': -164.1964874267578, 'logps/ref_chosen': -86.55081939697266, 'logps/ref_rejected': -88.62866973876953, 'logits/chosen': -7.7915472984313965, 'logits/rejected': -6.586763381958008, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.02751624397933483, 'kl/avg_steps': 0.6875, 'epoch': 0.41}
41%|██████████████████████████████████████████████▋ | 281/681 [19:38<20:49, 3.12s/it] 41%|██████████████████████████████████████████████▊ | 282/681 [19:41<20:45, 3.12s/it] {'loss': 0.6931, 'grad_norm': 11.661370277404785, 'learning_rate': 3.660059364023408e-07, 'rewards/chosen': -1.037402868270874, 'rewards/rejected': -2.1895809173583984, 'rewards/accuracies': 0.953125, 'rewards/margins': 1.152178168296814, 'logps/chosen': -143.371826171875, 'logps/rejected': -183.70306396484375, 'logps/ref_chosen': -105.10511016845703, 'logps/ref_rejected': -102.85336303710938, 'logits/chosen': -7.430984973907471, 'logits/rejected': -6.962902545928955, 'kl/p_epsilon_steps': 0.90625, 'kl/n_epsilon_steps': 0.09375, 'kl/beta': 0.027328362688422203, 'kl/avg_steps': 0.8125, 'epoch': 0.41}
41%|██████████████████████████████████████████████▊ | 282/681 [19:41<20:45, 3.12s/it] 42%|██████████████████████████████████████████████▉ | 283/681 [19:44<20:32, 3.10s/it] {'loss': 0.9734, 'grad_norm': 17.020105361938477, 'learning_rate': 3.6486760974483685e-07, 'rewards/chosen': -1.1950119733810425, 'rewards/rejected': -1.9943175315856934, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.7993054389953613, 'logps/chosen': -139.34780883789062, 'logps/rejected': -177.7033233642578, 'logps/ref_chosen': -95.05259704589844, 'logps/ref_rejected': -103.54454803466797, 'logits/chosen': -7.325111389160156, 'logits/rejected': -6.981871604919434, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.02710810862481594, 'kl/avg_steps': 0.65625, 'epoch': 0.42}
42%|██████████████████████████████████████████████▉ | 283/681 [19:44<20:32, 3.10s/it] 42%|███████████████████████████████████████████████ | 284/681 [19:47<20:48, 3.14s/it] {'loss': 0.6944, 'grad_norm': 11.532405853271484, 'learning_rate': 3.6372625621898863e-07, 'rewards/chosen': -0.9900509119033813, 'rewards/rejected': -2.1134815216064453, 'rewards/accuracies': 0.921875, 'rewards/margins': 1.1234304904937744, 'logps/chosen': -124.68060302734375, 'logps/rejected': -177.91339111328125, 'logps/ref_chosen': -87.6664810180664, 'logps/ref_rejected': -98.75103759765625, 'logits/chosen': -7.086446762084961, 'logits/rejected': -6.687787055969238, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.02693137153983116, 'kl/avg_steps': 0.78125, 'epoch': 0.42}
42%|███████████████████████████████████████████████ | 284/681 [19:47<20:48, 3.14s/it] 42%|███████████████████████████████████████████████▎ | 285/681 [19:50<20:30, 3.11s/it] {'loss': 0.8825, 'grad_norm': 18.86406898498535, 'learning_rate': 3.625819059005228e-07, 'rewards/chosen': -1.1481690406799316, 'rewards/rejected': -1.9326767921447754, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7845076322555542, 'logps/chosen': -137.629150390625, 'logps/rejected': -176.97714233398438, 'logps/ref_chosen': -94.43303680419922, 'logps/ref_rejected': -104.07194519042969, 'logits/chosen': -7.379509449005127, 'logits/rejected': -7.0911407470703125, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.026722600683569908, 'kl/avg_steps': 0.6875, 'epoch': 0.42}
42%|███████████████████████████████████████████████▎ | 285/681 [19:50<20:30, 3.11s/it] 42%|███████████████████████████████████████████████▍ | 286/681 [19:53<20:17, 3.08s/it] {'loss': 0.8014, 'grad_norm': 14.356172561645508, 'learning_rate': 3.614345889441346e-07, 'rewards/chosen': -1.2212910652160645, 'rewards/rejected': -2.1978063583374023, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9765151143074036, 'logps/chosen': -149.8736572265625, 'logps/rejected': -179.59671020507812, 'logps/ref_chosen': -103.72039794921875, 'logps/ref_rejected': -96.25775909423828, 'logits/chosen': -7.472691535949707, 'logits/rejected': -7.013616561889648, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.026540137827396393, 'kl/avg_steps': 0.5625, 'epoch': 0.42}
42%|███████████████████████████████████████████████▍ | 286/681 [19:53<20:17, 3.08s/it] 42%|███████████████████████████████████████████████▌ | 287/681 [19:56<19:26, 2.96s/it] {'loss': 1.0332, 'grad_norm': 15.217528343200684, 'learning_rate': 3.6028433558269275e-07, 'rewards/chosen': -1.4231257438659668, 'rewards/rejected': -2.0668678283691406, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.6437419652938843, 'logps/chosen': -147.86685180664062, 'logps/rejected': -162.0262451171875, 'logps/ref_chosen': -93.88988494873047, 'logps/ref_rejected': -83.33365631103516, 'logits/chosen': -7.295309066772461, 'logits/rejected': -6.699875831604004, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.026391685009002686, 'kl/avg_steps': 0.34375, 'epoch': 0.42}
42%|███████████████████████████████████████████████▌ | 287/681 [19:56<19:26, 2.96s/it] 42%|███████████████████████████████████████████████▊ | 288/681 [19:59<19:43, 3.01s/it] {'loss': 0.816, 'grad_norm': 14.122811317443848, 'learning_rate': 3.5913117612644327e-07, 'rewards/chosen': -1.3554046154022217, 'rewards/rejected': -2.3520970344543457, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9966922998428345, 'logps/chosen': -139.82644653320312, 'logps/rejected': -183.26959228515625, 'logps/ref_chosen': -88.15602111816406, 'logps/ref_rejected': -93.28195190429688, 'logits/chosen': -7.697492599487305, 'logits/rejected': -7.256779670715332, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.026301274076104164, 'kl/avg_steps': 0.53125, 'epoch': 0.42}
42%|███████████████████████████████████████████████▊ | 288/681 [19:59<19:43, 3.01s/it] 42%|███████████████████████████████████████████████▉ | 289/681 [20:02<19:45, 3.03s/it] {'loss': 0.8297, 'grad_norm': 13.091045379638672, 'learning_rate': 3.5797514096221024e-07, 'rewards/chosen': -1.3352906703948975, 'rewards/rejected': -2.3961217403411865, 'rewards/accuracies': 0.796875, 'rewards/margins': 1.0608309507369995, 'logps/chosen': -126.64739227294922, 'logps/rejected': -185.3885498046875, 'logps/ref_chosen': -75.39292907714844, 'logps/ref_rejected': -93.15428161621094, 'logits/chosen': -7.527945518493652, 'logits/rejected': -7.149833679199219, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.02616228722035885, 'kl/avg_steps': 0.59375, 'epoch': 0.42}
42%|███████████████████████████████████████████████▉ | 289/681 [20:02<19:45, 3.03s/it] 43%|████████████████████████████████████████████████ | 290/681 [20:05<19:32, 3.00s/it] {'loss': 0.8033, 'grad_norm': 14.055940628051758, 'learning_rate': 3.568162605525952e-07, 'rewards/chosen': -1.4665530920028687, 'rewards/rejected': -2.53206205368042, 'rewards/accuracies': 0.859375, 'rewards/margins': 1.0655090808868408, 'logps/chosen': -144.79820251464844, 'logps/rejected': -221.4034423828125, 'logps/ref_chosen': -88.0419692993164, 'logps/ref_rejected': -123.21215057373047, 'logits/chosen': -7.388311386108398, 'logits/rejected': -7.0829925537109375, 'kl/p_epsilon_steps': 0.90625, 'kl/n_epsilon_steps': 0.09375, 'kl/beta': 0.026007864624261856, 'kl/avg_steps': 0.8125, 'epoch': 0.43}
43%|████████████████████████████████████████████████ | 290/681 [20:05<19:32, 3.00s/it] 43%|████████████████████████████████████████████████▎ | 291/681 [20:08<19:50, 3.05s/it] {'loss': 0.9127, 'grad_norm': 14.552366256713867, 'learning_rate': 3.5565456543517485e-07, 'rewards/chosen': -1.408342719078064, 'rewards/rejected': -2.232980251312256, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8246374130249023, 'logps/chosen': -148.91311645507812, 'logps/rejected': -183.22817993164062, 'logps/ref_chosen': -94.09524536132812, 'logps/ref_rejected': -96.05006408691406, 'logits/chosen': -7.654292106628418, 'logits/rejected': -6.923820972442627, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.0257982537150383, 'kl/avg_steps': 0.625, 'epoch': 0.43}
43%|████████████████████████████████████████████████▎ | 291/681 [20:08<19:50, 3.05s/it] 43%|████████████████████████████████████████████████▍ | 292/681 [20:11<19:20, 2.98s/it] {'loss': 0.7522, 'grad_norm': 12.810431480407715, 'learning_rate': 3.5449008622169583e-07, 'rewards/chosen': -1.3947566747665405, 'rewards/rejected': -2.3737521171569824, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.9789954423904419, 'logps/chosen': -142.94708251953125, 'logps/rejected': -189.70559692382812, 'logps/ref_chosen': -88.25041198730469, 'logps/ref_rejected': -96.41764068603516, 'logits/chosen': -7.4952850341796875, 'logits/rejected': -7.096066951751709, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.025638015940785408, 'kl/avg_steps': 0.6875, 'epoch': 0.43}
43%|████████████████████████████████████████████████▍ | 292/681 [20:11<19:20, 2.98s/it] 43%|████████████████████████████████████████████████▌ | 293/681 [20:14<19:27, 3.01s/it] {'loss': 0.9942, 'grad_norm': 13.038102149963379, 'learning_rate': 3.5332285359726846e-07, 'rewards/chosen': -1.4899965524673462, 'rewards/rejected': -2.205336570739746, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.7153399586677551, 'logps/chosen': -146.07388305664062, 'logps/rejected': -172.91517639160156, 'logps/ref_chosen': -87.37654876708984, 'logps/ref_rejected': -85.75579833984375, 'logits/chosen': -7.2822585105896, 'logits/rejected': -6.593290328979492, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.025462958961725235, 'kl/avg_steps': 0.53125, 'epoch': 0.43}
43%|████████████████████████████████████████████████▌ | 293/681 [20:14<19:27, 3.01s/it] 43%|████████████████████████████████████████████████▊ | 294/681 [20:17<19:20, 3.00s/it] {'loss': 1.0529, 'grad_norm': 14.321362495422363, 'learning_rate': 3.5215289831955786e-07, 'rewards/chosen': -1.5407367944717407, 'rewards/rejected': -2.1650938987731934, 'rewards/accuracies': 0.75, 'rewards/margins': 0.6243571043014526, 'logps/chosen': -134.58004760742188, 'logps/rejected': -174.16912841796875, 'logps/ref_chosen': -73.5079574584961, 'logps/ref_rejected': -88.08877563476562, 'logits/chosen': -6.959033966064453, 'logits/rejected': -6.0898590087890625, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.02532840147614479, 'kl/avg_steps': 0.53125, 'epoch': 0.43}
43%|████████████████████████████████████████████████▊ | 294/681 [20:17<19:20, 3.00s/it] 43%|████████████████████████████████████████████████▉ | 295/681 [20:20<19:01, 2.96s/it] {'loss': 0.8606, 'grad_norm': 13.053278923034668, 'learning_rate': 3.509802512179737e-07, 'rewards/chosen': -1.4831101894378662, 'rewards/rejected': -2.3872218132019043, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9041118025779724, 'logps/chosen': -136.8726043701172, 'logps/rejected': -189.65672302246094, 'logps/ref_chosen': -77.76548767089844, 'logps/ref_rejected': -94.24726867675781, 'logits/chosen': -7.271920680999756, 'logits/rejected': -7.100852012634277, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.025194555521011353, 'kl/avg_steps': 0.59375, 'epoch': 0.43}
43%|████████████████████████████████████████████████▉ | 295/681 [20:20<19:01, 2.96s/it] 43%|█████████████████████████████████████████████████ | 296/681 [20:23<19:19, 3.01s/it] {'loss': 0.8194, 'grad_norm': 13.810120582580566, 'learning_rate': 3.498049431928577e-07, 'rewards/chosen': -1.586834192276001, 'rewards/rejected': -2.5582542419433594, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9714199304580688, 'logps/chosen': -161.45822143554688, 'logps/rejected': -203.63685607910156, 'logps/ref_chosen': -97.85641479492188, 'logps/ref_rejected': -100.81631469726562, 'logits/chosen': -7.517086505889893, 'logits/rejected': -7.0501275062561035, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.02504584565758705, 'kl/avg_steps': 0.5625, 'epoch': 0.43}
43%|█████████████████████████████████████████████████ | 296/681 [20:23<19:19, 3.01s/it] 44%|█████████████████████████████████████████████████▎ | 297/681 [20:26<19:25, 3.03s/it] {'loss': 0.8867, 'grad_norm': 14.663503646850586, 'learning_rate': 3.486270052146694e-07, 'rewards/chosen': -1.6361867189407349, 'rewards/rejected': -2.453274726867676, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8170878887176514, 'logps/chosen': -154.54638671875, 'logps/rejected': -200.7189178466797, 'logps/ref_chosen': -88.56583404541016, 'logps/ref_rejected': -101.55656433105469, 'logits/chosen': -7.523492813110352, 'logits/rejected': -6.788856506347656, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.024905750527977943, 'kl/avg_steps': 0.5625, 'epoch': 0.44}
44%|█████████████████████████████████████████████████▎ | 297/681 [20:26<19:25, 3.03s/it] 44%|█████████████████████████████████████████████████▍ | 298/681 [20:30<19:58, 3.13s/it] {'loss': 0.8289, 'grad_norm': 16.382911682128906, 'learning_rate': 3.474464683231698e-07, 'rewards/chosen': -1.460301160812378, 'rewards/rejected': -2.431180000305176, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9708787202835083, 'logps/chosen': -154.0912322998047, 'logps/rejected': -221.15391540527344, 'logps/ref_chosen': -94.88043975830078, 'logps/ref_rejected': -122.31101989746094, 'logits/chosen': -7.475191593170166, 'logits/rejected': -6.408700942993164, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.02476643957197666, 'kl/avg_steps': 0.59375, 'epoch': 0.44}
44%|█████████████████████████████████████████████████▍ | 298/681 [20:30<19:58, 3.13s/it] 44%|█████████████████████████████████████████████████▌ | 299/681 [20:33<19:41, 3.09s/it] {'loss': 0.785, 'grad_norm': 12.90149974822998, 'learning_rate': 3.462633636266041e-07, 'rewards/chosen': -1.5161972045898438, 'rewards/rejected': -2.5289955139160156, 'rewards/accuracies': 0.859375, 'rewards/margins': 1.0127980709075928, 'logps/chosen': -142.339111328125, 'logps/rejected': -193.05116271972656, 'logps/ref_chosen': -80.40835571289062, 'logps/ref_rejected': -89.53716278076172, 'logits/chosen': -7.417859077453613, 'logits/rejected': -6.741024971008301, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.024620257318019867, 'kl/avg_steps': 0.71875, 'epoch': 0.44}
44%|█████████████████████████████████████████████████▌ | 299/681 [20:33<19:41, 3.09s/it] 44%|█████████████████████████████████████████████████▊ | 300/681 [20:36<19:44, 3.11s/it] {'loss': 0.912, 'grad_norm': 14.46535587310791, 'learning_rate': 3.4507772230088147e-07, 'rewards/chosen': -1.6017869710922241, 'rewards/rejected': -2.454439640045166, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8526525497436523, 'logps/chosen': -153.97088623046875, 'logps/rejected': -202.05059814453125, 'logps/ref_chosen': -88.15890502929688, 'logps/ref_rejected': -100.93919372558594, 'logits/chosen': -7.5312700271606445, 'logits/rejected': -7.016364097595215, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.024444561451673508, 'kl/avg_steps': 0.625, 'epoch': 0.44}
44%|█████████████████████████████████████████████████▊ | 300/681 [20:36<19:44, 3.11s/it][INFO|trainer.py:4307] 2026-04-24 04:36:43,351 >>
***** Running Evaluation *****
[INFO|trainer.py:4309] 2026-04-24 04:36:43,352 >> Num examples = 2339
[INFO|trainer.py:4312] 2026-04-24 04:36:43,352 >> Batch size = 8
0%| | 0/73 [00:00<?, ?it/s]
3%|███▏ | 2/73 [00:00<00:23, 3.03it/s]
4%|████▊ | 3/73 [00:01<00:32, 2.14it/s]
5%|██████▎ | 4/73 [00:01<00:36, 1.88it/s]
7%|███████▉ | 5/73 [00:02<00:38, 1.76it/s]
8%|█████████▌ | 6/73 [00:03<00:40, 1.65it/s]
10%|███████████ | 7/73 [00:03<00:38, 1.71it/s]
11%|████████████▋ | 8/73 [00:04<00:40, 1.59it/s]
12%|██████████████▎ | 9/73 [00:05<00:41, 1.54it/s]
14%|███████████████▊ | 10/73 [00:05<00:41, 1.51it/s]
15%|█████████████████▎ | 11/73 [00:06<00:40, 1.52it/s]
16%|██████████████████▉ | 12/73 [00:07<00:40, 1.49it/s]
18%|████████████████████▍ | 13/73 [00:07<00:39, 1.52it/s]
19%|██████████████████████ | 14/73 [00:08<00:39, 1.51it/s]
21%|███████████████████████▋ | 15/73 [00:09<00:38, 1.51it/s]
22%|█████████████████████████▏ | 16/73 [00:09<00:39, 1.46it/s]
23%|██████████████████████████▊ | 17/73 [00:10<00:38, 1.44it/s]
25%|████████████████████████████▎ | 18/73 [00:11<00:38, 1.43it/s]
26%|█████████████████████████████▉ | 19/73 [00:12<00:38, 1.42it/s]
27%|███████████████████████████████▌ | 20/73 [00:12<00:37, 1.41it/s]
29%|█████████████████████████████████ | 21/73 [00:13<00:37, 1.40it/s]
30%|██████████████████████████████████▋ | 22/73 [00:14<00:36, 1.39it/s]
32%|████████████████████████████████████▏ | 23/73 [00:14<00:35, 1.42it/s]
33%|█████████████████████████████████████▊ | 24/73 [00:15<00:34, 1.43it/s]
34%|███████████████████████████████████████▍ | 25/73 [00:16<00:33, 1.44it/s]
36%|████████████████████████████████████████▉ | 26/73 [00:17<00:32, 1.45it/s]
37%|██████████████████████████████████████████▌ | 27/73 [00:17<00:28, 1.60it/s]
38%|████████████████████████████████████████████ | 28/73 [00:18<00:28, 1.56it/s]
40%|█████████████████████████████████████████████▋ | 29/73 [00:18<00:28, 1.57it/s]
41%|███████████████████████████████████████████████▎ | 30/73 [00:19<00:27, 1.59it/s]
42%|████████████████████████████████████████████████▊ | 31/73 [00:20<00:27, 1.53it/s]
44%|██████████████████████████████████████████████████▍ | 32/73 [00:20<00:26, 1.56it/s]
45%|███████████████████████████████████████████████████▉ | 33/73 [00:21<00:25, 1.58it/s]
47%|█████████████████████████████████████████████████████▌ | 34/73 [00:22<00:25, 1.55it/s]
48%|███████████████████████████████████████████████████████▏ | 35/73 [00:22<00:25, 1.49it/s]
49%|████████████████████████████████████████████████████████▋ | 36/73 [00:23<00:24, 1.49it/s]
51%|██████████████████████████████████████████████████████████▎ | 37/73 [00:24<00:24, 1.48it/s]
52%|███████████████████████████████████████████████████████████▊ | 38/73 [00:24<00:22, 1.56it/s]
53%|█████████████████████████████████████████████████████████████▍ | 39/73 [00:25<00:22, 1.50it/s]
55%|███████████████████████████████████████████████████████████████ | 40/73 [00:26<00:21, 1.51it/s]
56%|████████████████████████████████████████████████████████████████▌ | 41/73 [00:26<00:20, 1.56it/s]
58%|██████████████████████████████████████████████████████████████████▏ | 42/73 [00:27<00:20, 1.52it/s]
59%|███████████████████████████████████████████████████████████████████▋ | 43/73 [00:27<00:19, 1.54it/s]
60%|█████████████████████████████████████████████████████████████████████▎ | 44/73 [00:28<00:19, 1.52it/s]
62%|██████████████████████████████████████████████████████████████████████▉ | 45/73 [00:29<00:19, 1.47it/s]
63%|████████████████████████████████████████████████████████████████████████▍ | 46/73 [00:29<00:17, 1.53it/s]
64%|██████████████████████████████████████████████████████████████████████████ | 47/73 [00:30<00:17, 1.50it/s]
66%|███████████████████████████████████████████████████████████████████████████▌ | 48/73 [00:31<00:16, 1.49it/s]
67%|█████████████████████████████████████████████████████████████████████████████▏ | 49/73 [00:32<00:16, 1.47it/s]
68%|██████████████████████████████████████████████████████████████████████████████▊ | 50/73 [00:32<00:15, 1.49it/s]
70%|████████████████████████████████████████████████████████████████████████████████▎ | 51/73 [00:33<00:15, 1.46it/s]
71%|█████████████████████████████████████████████████████████████████████████████████▉ | 52/73 [00:34<00:14, 1.43it/s]
73%|███████████████████████████████████████████████████████████████████████████████████▍ | 53/73 [00:34<00:14, 1.41it/s]
74%|█████████████████████████████████████████████████████████████████████████████████████ | 54/73 [00:35<00:12, 1.50it/s]
75%|██████████████████████████████████████████████████████████████████████████████████████▋ | 55/73 [00:36<00:12, 1.49it/s]
77%|████████████████████████████████████████████████████████████████████████████████████████▏ | 56/73 [00:36<00:11, 1.53it/s]
78%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 57/73 [00:37<00:10, 1.48it/s]
79%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 58/73 [00:38<00:09, 1.53it/s]
81%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 59/73 [00:38<00:09, 1.52it/s]
82%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 60/73 [00:39<00:08, 1.50it/s]
84%|████████████████████████████████████████████████████████████████████████████████████████████████ | 61/73 [00:40<00:08, 1.48it/s]
85%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 62/73 [00:40<00:07, 1.50it/s]
86%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 63/73 [00:41<00:06, 1.58it/s]
88%|████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 64/73 [00:41<00:05, 1.60it/s]
89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 65/73 [00:42<00:05, 1.58it/s]
90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 66/73 [00:43<00:04, 1.52it/s]
92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 67/73 [00:43<00:03, 1.55it/s]
93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 68/73 [00:44<00:03, 1.52it/s]
95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 69/73 [00:45<00:02, 1.48it/s]
96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 70/73 [00:46<00:02, 1.47it/s]
97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 71/73 [00:46<00:01, 1.48it/s]
99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 72/73 [00:47<00:00, 1.48it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
{'eval_loss': 0.4874841272830963, 'eval_runtime': 48.5566, 'eval_samples_per_second': 48.171, 'eval_steps_per_second': 1.524, 'eval_rewards/chosen': -1.6289423704147339, 'eval_rewards/rejected': -2.347970724105835, 'eval_rewards/accuracies': 0.7786815166473389, 'eval_rewards/margins': 0.7190282940864563, 'eval_logps/chosen': -167.77142333984375, 'eval_logps/rejected': -191.32177734375, 'eval_logps/ref_chosen': -100.49356842041016, 'eval_logps/ref_rejected': -94.06775665283203, 'eval_logits/chosen': -7.007012367248535, 'eval_logits/rejected': -6.721932888031006, 'eval_kl/p_epsilon_steps': 0.7508561611175537, 'eval_kl/n_epsilon_steps': 0.24700342118740082, 'epoch': 0.44}
44%|█████████████████████████████████████████████████▊ | 300/681 [21:24<19:44, 3.11s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
 44%|█████████████████████████████████████████████████ | 301/681 [21:27<1:51:13, 17.56s/it] {'loss': 0.7859, 'grad_norm': 13.402172088623047, 'learning_rate': 3.4388957558875316e-07, 'rewards/chosen': -1.4873508214950562, 'rewards/rejected': -2.530000925064087, 'rewards/accuracies': 0.828125, 'rewards/margins': 1.0426502227783203, 'logps/chosen': -146.97164916992188, 'logps/rejected': -203.77188110351562, 'logps/ref_chosen': -85.50491333007812, 'logps/ref_rejected': -98.9168472290039, 'logits/chosen': -7.663355827331543, 'logits/rejected': -6.919793128967285, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.024292731657624245, 'kl/avg_steps': 0.609375, 'epoch': 0.44}
44%|█████████████████████████████████████████████████ | 301/681 [21:27<1:51:13, 17.56s/it] 44%|█████████████████████████████████████████████████▏ | 302/681 [21:30<1:23:32, 13.23s/it] {'loss': 0.9129, 'grad_norm': 15.27524185180664, 'learning_rate': 3.426989547989902e-07, 'rewards/chosen': -1.5182812213897705, 'rewards/rejected': -2.352370262145996, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8340890407562256, 'logps/chosen': -142.61880493164062, 'logps/rejected': -201.1916046142578, 'logps/ref_chosen': -79.45040893554688, 'logps/ref_rejected': -103.05909729003906, 'logits/chosen': -7.2973504066467285, 'logits/rejected': -6.5177412033081055, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.024145595729351044, 'kl/avg_steps': 0.625, 'epoch': 0.44}
44%|█████████████████████████████████████████████████▏ | 302/681 [21:30<1:23:32, 13.23s/it] 44%|█████████████████████████████████████████████████▍ | 303/681 [21:33<1:04:15, 10.20s/it] {'loss': 0.8158, 'grad_norm': 12.639718055725098, 'learning_rate': 3.4150589130555773e-07, 'rewards/chosen': -1.356937289237976, 'rewards/rejected': -2.3115692138671875, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.9546318650245667, 'logps/chosen': -152.356689453125, 'logps/rejected': -189.04776000976562, 'logps/ref_chosen': -95.59829711914062, 'logps/ref_rejected': -92.0645751953125, 'logits/chosen': -7.532470703125, 'logits/rejected': -7.15334939956665, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.023995622992515564, 'kl/avg_steps': 0.5625, 'epoch': 0.44}
44%|█████████████████████████████████████████████████▍ | 303/681 [21:33<1:04:15, 10.20s/it] 45%|██████████████████████████████████████████████████▍ | 304/681 [21:37<51:28, 8.19s/it] {'loss': 0.7929, 'grad_norm': 13.857175827026367, 'learning_rate': 3.403104165467883e-07, 'rewards/chosen': -1.365452766418457, 'rewards/rejected': -2.362959146499634, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9975064396858215, 'logps/chosen': -156.93124389648438, 'logps/rejected': -197.8302001953125, 'logps/ref_chosen': -99.4531478881836, 'logps/ref_rejected': -98.08226013183594, 'logits/chosen': -7.706064224243164, 'logits/rejected': -7.303454399108887, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.02386140264570713, 'kl/avg_steps': 0.59375, 'epoch': 0.45}
45%|██████████████████████████████████████████████████▍ | 304/681 [21:37<51:28, 8.19s/it] 45%|██████████████████████████████████████████████████▌ | 305/681 [21:40<42:05, 6.72s/it] {'loss': 0.8001, 'grad_norm': 13.020018577575684, 'learning_rate': 3.391125620245535e-07, 'rewards/chosen': -1.4017119407653809, 'rewards/rejected': -2.38907790184021, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9873659610748291, 'logps/chosen': -162.6031494140625, 'logps/rejected': -199.2515869140625, 'logps/ref_chosen': -103.26339721679688, 'logps/ref_rejected': -97.82897186279297, 'logits/chosen': -7.775443077087402, 'logits/rejected': -6.882397174835205, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.02372056059539318, 'kl/avg_steps': 0.59375, 'epoch': 0.45}
45%|██████████████████████████████████████████████████▌ | 305/681 [21:40<42:05, 6.72s/it] 45%|██████████████████████████████████████████████████▊ | 306/681 [21:43<35:35, 5.69s/it] {'loss': 0.6474, 'grad_norm': 12.147577285766602, 'learning_rate': 3.3791235930343417e-07, 'rewards/chosen': -1.2215442657470703, 'rewards/rejected': -2.4097707271575928, 'rewards/accuracies': 0.9375, 'rewards/margins': 1.1882264614105225, 'logps/chosen': -152.4990234375, 'logps/rejected': -195.33135986328125, 'logps/ref_chosen': -100.3038330078125, 'logps/ref_rejected': -92.23086547851562, 'logits/chosen': -7.674917697906494, 'logits/rejected': -7.210516452789307, 'kl/p_epsilon_steps': 0.921875, 'kl/n_epsilon_steps': 0.078125, 'kl/beta': 0.023580551147460938, 'kl/avg_steps': 0.84375, 'epoch': 0.45}
45%|██████████████████████████████████████████████████▊ | 306/681 [21:43<35:35, 5.69s/it] 45%|██████████████████████████████████████████████████▉ | 307/681 [21:47<30:58, 4.97s/it] {'loss': 0.9218, 'grad_norm': 14.133451461791992, 'learning_rate': 3.367098400098881e-07, 'rewards/chosen': -1.3819317817687988, 'rewards/rejected': -2.2242679595947266, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8423360586166382, 'logps/chosen': -155.960205078125, 'logps/rejected': -189.1453857421875, 'logps/ref_chosen': -96.64976501464844, 'logps/ref_rejected': -93.34860229492188, 'logits/chosen': -7.329036235809326, 'logits/rejected': -7.163274765014648, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.023383256047964096, 'kl/avg_steps': 0.53125, 'epoch': 0.45}
45%|██████████████████████████████████████████████████▉ | 307/681 [21:47<30:58, 4.97s/it] 45%|███████████████████████████████████████████████████ | 308/681 [21:50<27:21, 4.40s/it] {'loss': 0.9237, 'grad_norm': 15.209975242614746, 'learning_rate': 3.355050358314172e-07, 'rewards/chosen': -1.399101734161377, 'rewards/rejected': -2.237349033355713, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.8382473587989807, 'logps/chosen': -146.19422912597656, 'logps/rejected': -183.3817596435547, 'logps/ref_chosen': -85.72467041015625, 'logps/ref_rejected': -86.42318725585938, 'logits/chosen': -7.033696174621582, 'logits/rejected': -7.267401218414307, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.023259688168764114, 'kl/avg_steps': 0.625, 'epoch': 0.45}
45%|███████████████████████████████████████████████████ | 308/681 [21:50<27:21, 4.40s/it] 45%|███████████████████████████████████████████████████▎ | 309/681 [21:53<25:00, 4.03s/it] {'loss': 0.8829, 'grad_norm': 14.02563762664795, 'learning_rate': 3.3429797851573183e-07, 'rewards/chosen': -1.5072109699249268, 'rewards/rejected': -2.375739097595215, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8685282468795776, 'logps/chosen': -148.60951232910156, 'logps/rejected': -190.06564331054688, 'logps/ref_chosen': -83.15145874023438, 'logps/ref_rejected': -86.58602142333984, 'logits/chosen': -7.613210201263428, 'logits/rejected': -7.04583215713501, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.023115217685699463, 'kl/avg_steps': 0.59375, 'epoch': 0.45}
45%|███████████████████████████████████████████████████▎ | 309/681 [21:53<25:00, 4.03s/it] 46%|███████████████████████████████████████████████████▍ | 310/681 [21:56<23:16, 3.77s/it] {'loss': 0.9143, 'grad_norm': 13.904105186462402, 'learning_rate': 3.3308869986991487e-07, 'rewards/chosen': -1.3350245952606201, 'rewards/rejected': -2.1611814498901367, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.826156735420227, 'logps/chosen': -148.03106689453125, 'logps/rejected': -181.75921630859375, 'logps/ref_chosen': -89.73799133300781, 'logps/ref_rejected': -87.07083129882812, 'logits/chosen': -7.28669548034668, 'logits/rejected': -7.22123908996582, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.022978780791163445, 'kl/avg_steps': 0.53125, 'epoch': 0.46}
46%|███████████████████████████████████████████████████▍ | 310/681 [21:56<23:16, 3.77s/it] 46%|███████████████████████████████████████████████████▌ | 311/681 [21:59<21:54, 3.55s/it] {'loss': 0.7578, 'grad_norm': 13.777924537658691, 'learning_rate': 3.3187723175958346e-07, 'rewards/chosen': -1.6187620162963867, 'rewards/rejected': -2.665005683898926, 'rewards/accuracies': 0.890625, 'rewards/margins': 1.046243667602539, 'logps/chosen': -159.7617645263672, 'logps/rejected': -199.7523651123047, 'logps/ref_chosen': -88.52644348144531, 'logps/ref_rejected': -82.26608276367188, 'logits/chosen': -7.765219688415527, 'logits/rejected': -6.731596946716309, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.02285735122859478, 'kl/avg_steps': 0.65625, 'epoch': 0.46}
46%|███████████████████████████████████████████████████▌ | 311/681 [21:59<21:54, 3.55s/it] 46%|███████████████████████████████████████████████████▊ | 312/681 [22:02<20:37, 3.35s/it] {'loss': 0.8414, 'grad_norm': 13.674013137817383, 'learning_rate': 3.306636061080487e-07, 'rewards/chosen': -1.5027642250061035, 'rewards/rejected': -2.474343776702881, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.9715795516967773, 'logps/chosen': -146.90805053710938, 'logps/rejected': -192.24771118164062, 'logps/ref_chosen': -80.40069580078125, 'logps/ref_rejected': -82.4471435546875, 'logits/chosen': -7.522071838378906, 'logits/rejected': -6.754973411560059, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.022708328440785408, 'kl/avg_steps': 0.6875, 'epoch': 0.46}
46%|███████████████████████████████████████████████████▊ | 312/681 [22:02<20:37, 3.35s/it] 46%|███████████████████████████████████████████████████▉ | 313/681 [22:05<19:37, 3.20s/it] {'loss': 0.9267, 'grad_norm': 12.186444282531738, 'learning_rate': 3.2944785489547537e-07, 'rewards/chosen': -1.4026517868041992, 'rewards/rejected': -2.2639105319976807, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8612587451934814, 'logps/chosen': -146.6564483642578, 'logps/rejected': -194.61727905273438, 'logps/ref_chosen': -84.14190673828125, 'logps/ref_rejected': -93.42684936523438, 'logits/chosen': -7.390935897827148, 'logits/rejected': -6.827203750610352, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.022553274407982826, 'kl/avg_steps': 0.5625, 'epoch': 0.46}
46%|███████████████████████████████████████████████████▉ | 313/681 [22:05<19:37, 3.20s/it] 46%|████████████████████████████████████████████████████ | 314/681 [22:08<19:31, 3.19s/it] {'loss': 0.872, 'grad_norm': 13.97519588470459, 'learning_rate': 3.2823001015803857e-07, 'rewards/chosen': -1.3944957256317139, 'rewards/rejected': -2.2951788902282715, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.9006831049919128, 'logps/chosen': -151.27520751953125, 'logps/rejected': -207.26962280273438, 'logps/ref_chosen': -88.80972290039062, 'logps/ref_rejected': -104.15892028808594, 'logits/chosen': -7.827272415161133, 'logits/rejected': -7.375405311584473, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.02242712303996086, 'kl/avg_steps': 0.625, 'epoch': 0.46}
46%|████████████████████████████████████████████████████ | 314/681 [22:08<19:31, 3.19s/it] 46%|████████████████████████████████████████████████████▎ | 315/681 [22:11<19:01, 3.12s/it] {'loss': 0.9216, 'grad_norm': 14.756415367126465, 'learning_rate': 3.270101039870797e-07, 'rewards/chosen': -1.391645073890686, 'rewards/rejected': -2.288558006286621, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8969128131866455, 'logps/chosen': -143.4265594482422, 'logps/rejected': -195.54171752929688, 'logps/ref_chosen': -80.793701171875, 'logps/ref_rejected': -92.13711547851562, 'logits/chosen': -7.049970626831055, 'logits/rejected': -6.526283264160156, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.022287823259830475, 'kl/avg_steps': 0.5625, 'epoch': 0.46}
46%|████████████████████████████████████████████████████▎ | 315/681 [22:11<19:01, 3.12s/it] 46%|████████████████████████████████████████████████████▍ | 316/681 [22:14<18:45, 3.08s/it] {'loss': 0.8837, 'grad_norm': 14.08152961730957, 'learning_rate': 3.2578816852826086e-07, 'rewards/chosen': -1.467881679534912, 'rewards/rejected': -2.358112096786499, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8902303576469421, 'logps/chosen': -150.78274536132812, 'logps/rejected': -212.91693115234375, 'logps/ref_chosen': -84.18425750732422, 'logps/ref_rejected': -105.6908187866211, 'logits/chosen': -7.215094089508057, 'logits/rejected': -7.132869720458984, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.022163156419992447, 'kl/avg_steps': 0.59375, 'epoch': 0.46}
46%|████████████████████████████████████████████████████▍ | 316/681 [22:14<18:45, 3.08s/it] 47%|████████████████████████████████████████████████████▌ | 317/681 [22:17<18:45, 3.09s/it] {'loss': 0.7381, 'grad_norm': 14.905621528625488, 'learning_rate': 3.2456423598071783e-07, 'rewards/chosen': -1.4228007793426514, 'rewards/rejected': -2.6340909004211426, 'rewards/accuracies': 0.875, 'rewards/margins': 1.2112901210784912, 'logps/chosen': -152.81094360351562, 'logps/rejected': -228.77639770507812, 'logps/ref_chosen': -87.87348937988281, 'logps/ref_rejected': -108.26519775390625, 'logits/chosen': -7.734880447387695, 'logits/rejected': -7.02924919128418, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.02203233912587166, 'kl/avg_steps': 0.65625, 'epoch': 0.47}
47%|████████████████████████████████████████████████████▌ | 317/681 [22:17<18:45, 3.09s/it] 47%|████████████████████████████████████████████████████▊ | 318/681 [22:20<18:17, 3.02s/it] {'loss': 0.7892, 'grad_norm': 11.828646659851074, 'learning_rate': 3.233383385962115e-07, 'rewards/chosen': -1.3737818002700806, 'rewards/rejected': -2.409672498703003, 'rewards/accuracies': 0.828125, 'rewards/margins': 1.0358906984329224, 'logps/chosen': -161.36456298828125, 'logps/rejected': -199.14776611328125, 'logps/ref_chosen': -98.20553588867188, 'logps/ref_rejected': -88.13629150390625, 'logits/chosen': -7.722089767456055, 'logits/rejected': -7.081110000610352, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.021888693794608116, 'kl/avg_steps': 0.71875, 'epoch': 0.47}
47%|████████████████████████████████████████████████████▊ | 318/681 [22:20<18:17, 3.02s/it] 47%|████████████████████████████████████████████████████▉ | 319/681 [22:23<18:55, 3.14s/it] {'loss': 0.7808, 'grad_norm': 13.124372482299805, 'learning_rate': 3.2211050867827805e-07, 'rewards/chosen': -1.1925673484802246, 'rewards/rejected': -2.2754123210906982, 'rewards/accuracies': 0.875, 'rewards/margins': 1.082844853401184, 'logps/chosen': -137.73611450195312, 'logps/rejected': -224.25790405273438, 'logps/ref_chosen': -82.50337219238281, 'logps/ref_rejected': -118.65068054199219, 'logits/chosen': -7.879185199737549, 'logits/rejected': -7.01790714263916, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.021732492372393608, 'kl/avg_steps': 0.78125, 'epoch': 0.47}
47%|████████████████████████████████████████████████████▉ | 319/681 [22:23<18:55, 3.14s/it] 47%|█████████████████████████████████████████████████████ | 320/681 [22:27<19:07, 3.18s/it] {'loss': 0.8435, 'grad_norm': 12.478285789489746, 'learning_rate': 3.208807785813777e-07, 'rewards/chosen': -1.2286326885223389, 'rewards/rejected': -2.149698495864868, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9210659861564636, 'logps/chosen': -144.46971130371094, 'logps/rejected': -203.762451171875, 'logps/ref_chosen': -87.1372299194336, 'logps/ref_rejected': -103.22412109375, 'logits/chosen': -7.674609184265137, 'logits/rejected': -7.378990173339844, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.021564023569226265, 'kl/avg_steps': 0.6875, 'epoch': 0.47}
47%|█████████████████████████████████████████████████████ | 320/681 [22:27<19:07, 3.18s/it] 47%|█████████████████████████████████████████████████████▎ | 321/681 [22:30<18:39, 3.11s/it] {'loss': 0.8056, 'grad_norm': 13.07551097869873, 'learning_rate': 3.1964918071004217e-07, 'rewards/chosen': -1.3688414096832275, 'rewards/rejected': -2.398127555847168, 'rewards/accuracies': 0.828125, 'rewards/margins': 1.0292859077453613, 'logps/chosen': -157.41146850585938, 'logps/rejected': -210.389892578125, 'logps/ref_chosen': -93.10466766357422, 'logps/ref_rejected': -97.4681396484375, 'logits/chosen': -7.491329193115234, 'logits/rejected': -7.139033317565918, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.021416783332824707, 'kl/avg_steps': 0.78125, 'epoch': 0.47}
47%|█████████████████████████████████████████████████████▎ | 321/681 [22:30<18:39, 3.11s/it] 47%|█████████████████████████████████████████████████████▍ | 322/681 [22:33<18:47, 3.14s/it] {'loss': 0.7797, 'grad_norm': 15.268094062805176, 'learning_rate': 3.184157475180207e-07, 'rewards/chosen': -1.2838683128356934, 'rewards/rejected': -2.4268603324890137, 'rewards/accuracies': 0.84375, 'rewards/margins': 1.1429917812347412, 'logps/chosen': -152.91873168945312, 'logps/rejected': -216.01556396484375, 'logps/ref_chosen': -92.19291687011719, 'logps/ref_rejected': -100.89321899414062, 'logits/chosen': -7.795929908752441, 'logits/rejected': -7.272746562957764, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.021250760182738304, 'kl/avg_steps': 0.65625, 'epoch': 0.47}
47%|█████████████████████████████████████████████████████▍ | 322/681 [22:33<18:47, 3.14s/it] 47%|█████████████████████████████████████████████████████▌ | 323/681 [22:36<18:46, 3.15s/it] {'loss': 0.7955, 'grad_norm': 12.563841819763184, 'learning_rate': 3.171805115074251e-07, 'rewards/chosen': -1.440096139907837, 'rewards/rejected': -2.449195623397827, 'rewards/accuracies': 0.890625, 'rewards/margins': 1.0090997219085693, 'logps/chosen': -148.90322875976562, 'logps/rejected': -197.24813842773438, 'logps/ref_chosen': -80.27210235595703, 'logps/ref_rejected': -80.30335235595703, 'logits/chosen': -7.563092231750488, 'logits/rejected': -7.294485092163086, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.021112211048603058, 'kl/avg_steps': 0.71875, 'epoch': 0.47}
47%|█████████████████████████████████████████████████████▌ | 323/681 [22:36<18:46, 3.15s/it] 48%|█████████████████████████████████████████████████████▊ | 324/681 [22:39<18:18, 3.08s/it] {'loss': 0.732, 'grad_norm': 13.184640884399414, 'learning_rate': 3.1594350522787295e-07, 'rewards/chosen': -1.4592244625091553, 'rewards/rejected': -2.543428897857666, 'rewards/accuracies': 0.859375, 'rewards/margins': 1.0842043161392212, 'logps/chosen': -165.7332763671875, 'logps/rejected': -216.46131896972656, 'logps/ref_chosen': -95.72935485839844, 'logps/ref_rejected': -94.16924285888672, 'logits/chosen': -7.81708288192749, 'logits/rejected': -6.640934944152832, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.020961550995707512, 'kl/avg_steps': 0.6875, 'epoch': 0.48}
48%|█████████████████████████████████████████████████████▊ | 324/681 [22:39<18:18, 3.08s/it] 48%|█████████████████████████████████████████████████████▉ | 325/681 [22:42<18:32, 3.12s/it] {'loss': 0.7957, 'grad_norm': 11.3136625289917, 'learning_rate': 3.147047612756302e-07, 'rewards/chosen': -1.37131929397583, 'rewards/rejected': -2.38810396194458, 'rewards/accuracies': 0.84375, 'rewards/margins': 1.0167847871780396, 'logps/chosen': -164.93191528320312, 'logps/rejected': -192.03103637695312, 'logps/ref_chosen': -98.70687103271484, 'logps/ref_rejected': -76.40809631347656, 'logits/chosen': -7.779555320739746, 'logits/rejected': -7.204484939575195, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.020818423479795456, 'kl/avg_steps': 0.65625, 'epoch': 0.48}
48%|█████████████████████████████████████████████████████▉ | 325/681 [22:42<18:32, 3.12s/it] 48%|██████████████████████████████████████████████████████ | 326/681 [22:45<18:36, 3.14s/it] {'loss': 0.8558, 'grad_norm': 13.992250442504883, 'learning_rate': 3.134643122927519e-07, 'rewards/chosen': -1.567757487297058, 'rewards/rejected': -2.4942867755889893, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9265294671058655, 'logps/chosen': -180.78939819335938, 'logps/rejected': -208.27098083496094, 'logps/ref_chosen': -104.66130065917969, 'logps/ref_rejected': -86.82537841796875, 'logits/chosen': -7.7820024490356445, 'logits/rejected': -7.323631286621094, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.020682694390416145, 'kl/avg_steps': 0.625, 'epoch': 0.48}
48%|██████████████████████████████████████████████████████ | 326/681 [22:45<18:36, 3.14s/it] 48%|██████████████████████████████████████████████████████▎ | 327/681 [22:49<18:50, 3.19s/it] {'loss': 0.8317, 'grad_norm': 13.753683090209961, 'learning_rate': 3.1222219096622264e-07, 'rewards/chosen': -1.491244912147522, 'rewards/rejected': -2.4625027179718018, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9712578058242798, 'logps/chosen': -180.28341674804688, 'logps/rejected': -237.6690216064453, 'logps/ref_chosen': -107.44024658203125, 'logps/ref_rejected': -117.05323028564453, 'logits/chosen': -7.905759811401367, 'logits/rejected': -7.24397611618042, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.020554229617118835, 'kl/avg_steps': 0.5625, 'epoch': 0.48}
48%|██████████████████████████████████████████████████████▎ | 327/681 [22:49<18:50, 3.19s/it] 48%|██████████████████████████████████████████████████████▍ | 328/681 [22:52<18:48, 3.20s/it] {'loss': 0.7407, 'grad_norm': 13.748778343200684, 'learning_rate': 3.1097843002709427e-07, 'rewards/chosen': -1.5384535789489746, 'rewards/rejected': -2.6775364875793457, 'rewards/accuracies': 0.875, 'rewards/margins': 1.1390830278396606, 'logps/chosen': -162.29519653320312, 'logps/rejected': -229.42196655273438, 'logps/ref_chosen': -86.6106948852539, 'logps/ref_rejected': -97.40188598632812, 'logits/chosen': -7.3603315353393555, 'logits/rejected': -6.873122215270996, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.020439259707927704, 'kl/avg_steps': 0.6875, 'epoch': 0.48}
48%|██████████████████████████████████████████████████████▍ | 328/681 [22:52<18:48, 3.20s/it] 48%|██████████████████████████████████████████████████████▌ | 329/681 [22:55<18:19, 3.12s/it] {'loss': 0.8164, 'grad_norm': 14.800250053405762, 'learning_rate': 3.0973306224962437e-07, 'rewards/chosen': -1.564449667930603, 'rewards/rejected': -2.521632194519043, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9571825861930847, 'logps/chosen': -178.1090850830078, 'logps/rejected': -230.13726806640625, 'logps/ref_chosen': -100.61308288574219, 'logps/ref_rejected': -104.95370483398438, 'logits/chosen': -7.575889587402344, 'logits/rejected': -7.213668346405029, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.020299699157476425, 'kl/avg_steps': 0.6875, 'epoch': 0.48}
48%|██████████████████████████████████████████████████████▌ | 329/681 [22:55<18:19, 3.12s/it] 48%|██████████████████████████████████████████████████████▊ | 330/681 [22:58<18:14, 3.12s/it] {'loss': 0.7539, 'grad_norm': 12.028066635131836, 'learning_rate': 3.084861204504122e-07, 'rewards/chosen': -1.4205009937286377, 'rewards/rejected': -2.431969165802002, 'rewards/accuracies': 0.875, 'rewards/margins': 1.0114681720733643, 'logps/chosen': -147.8223876953125, 'logps/rejected': -214.4041290283203, 'logps/ref_chosen': -76.95843505859375, 'logps/ref_rejected': -92.81494140625, 'logits/chosen': -7.624053955078125, 'logits/rejected': -7.438431739807129, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.020161090418696404, 'kl/avg_steps': 0.6875, 'epoch': 0.48}
48%|██████████████████████████████████████████████████████▊ | 330/681 [22:58<18:14, 3.12s/it] 49%|██████████████████████████████████████████████████████▉ | 331/681 [23:01<17:56, 3.08s/it] {'loss': 0.7566, 'grad_norm': 16.355430603027344, 'learning_rate': 3.072376374875335e-07, 'rewards/chosen': -1.4940705299377441, 'rewards/rejected': -2.540346622467041, 'rewards/accuracies': 0.875, 'rewards/margins': 1.0462760925292969, 'logps/chosen': -158.1719512939453, 'logps/rejected': -222.07827758789062, 'logps/ref_chosen': -83.0594253540039, 'logps/ref_rejected': -94.13113403320312, 'logits/chosen': -7.7047834396362305, 'logits/rejected': -7.323886394500732, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.02002342976629734, 'kl/avg_steps': 0.75, 'epoch': 0.49}
49%|██████████████████████████████████████████████████████▉ | 331/681 [23:01<17:56, 3.08s/it] 49%|███████████████████████████████████████████████████████ | 332/681 [23:04<17:35, 3.02s/it] {'loss': 0.9013, 'grad_norm': 13.534795761108398, 'learning_rate': 3.059876462596758e-07, 'rewards/chosen': -1.5361754894256592, 'rewards/rejected': -2.3551878929138184, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8190126419067383, 'logps/chosen': -159.60678100585938, 'logps/rejected': -202.23422241210938, 'logps/ref_chosen': -81.93089294433594, 'logps/ref_rejected': -82.84590148925781, 'logits/chosen': -7.760369300842285, 'logits/rejected': -7.029866695404053, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.019874371588230133, 'kl/avg_steps': 0.625, 'epoch': 0.49}
49%|███████████████████████████████████████████████████████ | 332/681 [23:04<17:35, 3.02s/it] 49%|███████████████████████████████████████████████████████▎ | 333/681 [23:06<17:08, 2.96s/it] {'loss': 0.8778, 'grad_norm': 13.899153709411621, 'learning_rate': 3.0473617970527015e-07, 'rewards/chosen': -1.7536228895187378, 'rewards/rejected': -2.7848353385925293, 'rewards/accuracies': 0.75, 'rewards/margins': 1.0312126874923706, 'logps/chosen': -177.69625854492188, 'logps/rejected': -244.46804809570312, 'logps/ref_chosen': -88.53257751464844, 'logps/ref_rejected': -102.46601867675781, 'logits/chosen': -7.564597129821777, 'logits/rejected': -7.067813396453857, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.01975092850625515, 'kl/avg_steps': 0.5625, 'epoch': 0.49}
49%|███████████████████████████████████████████████████████▎ | 333/681 [23:07<17:08, 2.96s/it] 49%|███████████████████████████████████████████████████████▍ | 334/681 [23:10<17:23, 3.01s/it] {'loss': 0.9017, 'grad_norm': 16.076087951660156, 'learning_rate': 3.034832708016243e-07, 'rewards/chosen': -1.6136884689331055, 'rewards/rejected': -2.5371832847595215, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9234945774078369, 'logps/chosen': -180.3138427734375, 'logps/rejected': -230.16831970214844, 'logps/ref_chosen': -97.75778198242188, 'logps/ref_rejected': -100.00584411621094, 'logits/chosen': -7.2526021003723145, 'logits/rejected': -7.094786643981934, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.019640451297163963, 'kl/avg_steps': 0.6875, 'epoch': 0.49}
49%|███████████████████████████████████████████████████████▍ | 334/681 [23:10<17:23, 3.01s/it] 49%|███████████████████████████████████████████████████████▌ | 335/681 [23:13<17:32, 3.04s/it] {'loss': 0.9078, 'grad_norm': 15.462376594543457, 'learning_rate': 3.022289525640531e-07, 'rewards/chosen': -1.618840217590332, 'rewards/rejected': -2.4653477668762207, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.8465075492858887, 'logps/chosen': -179.6153106689453, 'logps/rejected': -222.1100616455078, 'logps/ref_chosen': -96.37603759765625, 'logps/ref_rejected': -94.92401123046875, 'logits/chosen': -7.538198471069336, 'logits/rejected': -6.875842571258545, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.019506344571709633, 'kl/avg_steps': 0.53125, 'epoch': 0.49}
49%|███████████████████████████████████████████████████████▌ | 335/681 [23:13<17:32, 3.04s/it] 49%|███████████████████████████████████████████████████████▊ | 336/681 [23:16<17:42, 3.08s/it] {'loss': 0.7966, 'grad_norm': 14.706995010375977, 'learning_rate': 3.009732580450086e-07, 'rewards/chosen': -1.5775220394134521, 'rewards/rejected': -2.6134791374206543, 'rewards/accuracies': 0.890625, 'rewards/margins': 1.0359567403793335, 'logps/chosen': -170.34237670898438, 'logps/rejected': -245.36219787597656, 'logps/ref_chosen': -88.52952575683594, 'logps/ref_rejected': -109.55577850341797, 'logits/chosen': -7.663290500640869, 'logits/rejected': -7.155807971954346, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.01940326578915119, 'kl/avg_steps': 0.75, 'epoch': 0.49}
49%|███████████████████████████████████████████████████████▊ | 336/681 [23:16<17:42, 3.08s/it] 49%|███████████████████████████████████████████████████████▉ | 337/681 [23:19<17:16, 3.01s/it] {'loss': 0.8903, 'grad_norm': 15.293198585510254, 'learning_rate': 2.9971622033320914e-07, 'rewards/chosen': -1.5696876049041748, 'rewards/rejected': -2.5103797912597656, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9406921863555908, 'logps/chosen': -179.1387939453125, 'logps/rejected': -241.39308166503906, 'logps/ref_chosen': -97.31238555908203, 'logps/ref_rejected': -110.13134765625, 'logits/chosen': -7.815160274505615, 'logits/rejected': -7.3971452713012695, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.019258825108408928, 'kl/avg_steps': 0.59375, 'epoch': 0.49}
49%|███████████████████████████████████████████████████████▉ | 337/681 [23:19<17:16, 3.01s/it] 50%|████████████████████████████████████████████████████████ | 338/681 [23:22<17:14, 3.01s/it] {'loss': 0.8439, 'grad_norm': 15.433863639831543, 'learning_rate': 2.984578725527675e-07, 'rewards/chosen': -1.5996140241622925, 'rewards/rejected': -2.5144004821777344, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9147864580154419, 'logps/chosen': -167.595947265625, 'logps/rejected': -226.01394653320312, 'logps/ref_chosen': -83.63011932373047, 'logps/ref_rejected': -93.69235229492188, 'logits/chosen': -7.460289478302002, 'logits/rejected': -7.160378456115723, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.019145149737596512, 'kl/avg_steps': 0.625, 'epoch': 0.5}
50%|████████████████████████████████████████████████████████ | 338/681 [23:22<17:14, 3.01s/it] 50%|████████████████████████████████████████████████████████▎ | 339/681 [23:24<16:36, 2.91s/it] {'loss': 0.8648, 'grad_norm': 14.1244535446167, 'learning_rate': 2.9719824786231796e-07, 'rewards/chosen': -1.609981656074524, 'rewards/rejected': -2.5959229469299316, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.985941469669342, 'logps/chosen': -184.80462646484375, 'logps/rejected': -246.49473571777344, 'logps/ref_chosen': -99.85182189941406, 'logps/ref_rejected': -109.0743637084961, 'logits/chosen': -7.783293724060059, 'logits/rejected': -7.273715496063232, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.01902623660862446, 'kl/avg_steps': 0.59375, 'epoch': 0.5}
50%|████████████████████████████████████████████████████████▎ | 339/681 [23:25<16:36, 2.91s/it] 50%|████████████████████████████████████████████████████████▍ | 340/681 [23:28<16:47, 2.96s/it] {'loss': 0.8763, 'grad_norm': 17.34671974182129, 'learning_rate': 2.959373794541426e-07, 'rewards/chosen': -1.739735722541809, 'rewards/rejected': -2.6294469833374023, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8897112607955933, 'logps/chosen': -177.3800811767578, 'logps/rejected': -218.79473876953125, 'logps/ref_chosen': -84.97396087646484, 'logps/ref_rejected': -78.80033874511719, 'logits/chosen': -7.614503383636475, 'logits/rejected': -7.28403377532959, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.018913934007287025, 'kl/avg_steps': 0.625, 'epoch': 0.5}
50%|████████████████████████████████████████████████████████▍ | 340/681 [23:28<16:47, 2.96s/it] 50%|████████████████████████████████████████████████████████▌ | 341/681 [23:30<16:17, 2.88s/it] {'loss': 0.9302, 'grad_norm': 16.43499755859375, 'learning_rate': 2.946753005532965e-07, 'rewards/chosen': -1.850089430809021, 'rewards/rejected': -2.694669723510742, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.8445804119110107, 'logps/chosen': -184.0854949951172, 'logps/rejected': -250.41880798339844, 'logps/ref_chosen': -85.26399230957031, 'logps/ref_rejected': -106.08595275878906, 'logits/chosen': -7.702646732330322, 'logits/rejected': -6.986645698547363, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.01879645697772503, 'kl/avg_steps': 0.53125, 'epoch': 0.5}
50%|████████████████████████████████████████████████████████▌ | 341/681 [23:30<16:17, 2.88s/it] 50%|████████████████████████████████████████████████████████▋ | 342/681 [23:33<16:56, 3.00s/it] {'loss': 0.7283, 'grad_norm': 14.287416458129883, 'learning_rate': 2.934120444167326e-07, 'rewards/chosen': -1.4495775699615479, 'rewards/rejected': -2.629033088684082, 'rewards/accuracies': 0.875, 'rewards/margins': 1.179455280303955, 'logps/chosen': -178.91830444335938, 'logps/rejected': -240.55776977539062, 'logps/ref_chosen': -101.03860473632812, 'logps/ref_rejected': -98.89851379394531, 'logits/chosen': -7.695643424987793, 'logits/rejected': -6.729315280914307, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.01869712769985199, 'kl/avg_steps': 0.625, 'epoch': 0.5}
50%|████████████████████████████████████████████████████████▋ | 342/681 [23:34<16:56, 3.00s/it] 50%|████████████████████████████████████████████████████████▉ | 343/681 [23:36<16:44, 2.97s/it] {'loss': 0.8041, 'grad_norm': 15.677153587341309, 'learning_rate': 2.9214764433242476e-07, 'rewards/chosen': -1.5235764980316162, 'rewards/rejected': -2.5198564529418945, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.996279776096344, 'logps/chosen': -166.2335205078125, 'logps/rejected': -253.29689025878906, 'logps/ref_chosen': -83.74736785888672, 'logps/ref_rejected': -116.57908630371094, 'logits/chosen': -7.71726131439209, 'logits/rejected': -7.144956588745117, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.018580997362732887, 'kl/avg_steps': 0.71875, 'epoch': 0.5}
50%|████████████████████████████████████████████████████████▉ | 343/681 [23:36<16:44, 2.97s/it] 51%|█████████████████████████████████████████████████████████ | 344/681 [23:39<16:50, 3.00s/it] {'loss': 0.8489, 'grad_norm': 14.323365211486816, 'learning_rate': 2.9088213361849126e-07, 'rewards/chosen': -1.6444833278656006, 'rewards/rejected': -2.604991912841797, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9605084657669067, 'logps/chosen': -167.88778686523438, 'logps/rejected': -237.73025512695312, 'logps/ref_chosen': -78.25396728515625, 'logps/ref_rejected': -95.4287109375, 'logits/chosen': -7.541003227233887, 'logits/rejected': -7.094578266143799, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.018448399379849434, 'kl/avg_steps': 0.640625, 'epoch': 0.51}
51%|█████████████████████████████████████████████████████████ | 344/681 [23:40<16:50, 3.00s/it] 51%|█████████████████████████████████████████████████████████▏ | 345/681 [23:43<17:04, 3.05s/it] {'loss': 0.7793, 'grad_norm': 14.38456916809082, 'learning_rate': 2.896155456223163e-07, 'rewards/chosen': -1.5716257095336914, 'rewards/rejected': -2.634444236755371, 'rewards/accuracies': 0.890625, 'rewards/margins': 1.0628182888031006, 'logps/chosen': -184.28465270996094, 'logps/rejected': -248.8415985107422, 'logps/ref_chosen': -98.0794906616211, 'logps/ref_rejected': -104.03477478027344, 'logits/chosen': -7.9548492431640625, 'logits/rejected': -7.6490678787231445, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.01833096705377102, 'kl/avg_steps': 0.71875, 'epoch': 0.51}
51%|█████████████████████████████████████████████████████████▏ | 345/681 [23:43<17:04, 3.05s/it] 51%|█████████████████████████████████████████████████████████▍ | 346/681 [23:46<16:43, 3.00s/it] {'loss': 0.9413, 'grad_norm': 13.954184532165527, 'learning_rate': 2.883479137196714e-07, 'rewards/chosen': -1.788151502609253, 'rewards/rejected': -2.6421780586242676, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8540264368057251, 'logps/chosen': -179.28921508789062, 'logps/rejected': -232.02542114257812, 'logps/ref_chosen': -80.68348693847656, 'logps/ref_rejected': -85.89260864257812, 'logits/chosen': -7.322576522827148, 'logits/rejected': -6.97154426574707, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.01820015348494053, 'kl/avg_steps': 0.53125, 'epoch': 0.51}
51%|█████████████████████████████████████████████████████████▍ | 346/681 [23:46<16:43, 3.00s/it] 51%|█████████████████████████████████████████████████████████▌ | 347/681 [23:48<16:00, 2.88s/it] {'loss': 0.9517, 'grad_norm': 14.986981391906738, 'learning_rate': 2.8707927131383614e-07, 'rewards/chosen': -1.686805009841919, 'rewards/rejected': -2.5116419792175293, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.8248369693756104, 'logps/chosen': -187.2706298828125, 'logps/rejected': -237.28759765625, 'logps/ref_chosen': -93.67938232421875, 'logps/ref_rejected': -97.54251861572266, 'logits/chosen': -7.665702819824219, 'logits/rejected': -7.370832920074463, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.01810397580265999, 'kl/avg_steps': 0.5625, 'epoch': 0.51}
51%|█████████████████████████████████████████████████████████▌ | 347/681 [23:48<16:00, 2.88s/it] 51%|█████████████████████████████████████████████████████████▋ | 348/681 [23:51<15:42, 2.83s/it] {'loss': 0.888, 'grad_norm': 13.239946365356445, 'learning_rate': 2.858096518347179e-07, 'rewards/chosen': -1.4998319149017334, 'rewards/rejected': -2.376772403717041, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8769406676292419, 'logps/chosen': -173.94375610351562, 'logps/rejected': -228.76617431640625, 'logps/ref_chosen': -90.25224304199219, 'logps/ref_rejected': -95.77027893066406, 'logits/chosen': -7.554695129394531, 'logits/rejected': -7.127823829650879, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.01800270937383175, 'kl/avg_steps': 0.6875, 'epoch': 0.51}
51%|█████████████████████████████████████████████████████████▋ | 348/681 [23:51<15:42, 2.83s/it] 51%|█████████████████████████████████████████████████████████▉ | 349/681 [23:54<16:16, 2.94s/it] {'loss': 0.906, 'grad_norm': 13.889335632324219, 'learning_rate': 2.845390887379706e-07, 'rewards/chosen': -1.5828135013580322, 'rewards/rejected': -2.4346094131469727, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.8517956137657166, 'logps/chosen': -180.9906463623047, 'logps/rejected': -241.1156463623047, 'logps/ref_chosen': -92.05809020996094, 'logps/ref_rejected': -103.96527099609375, 'logits/chosen': -7.729681968688965, 'logits/rejected': -7.328579902648926, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0178797859698534, 'kl/avg_steps': 0.59375, 'epoch': 0.51}
51%|█████████████████████████████████████████████████████████▉ | 349/681 [23:54<16:16, 2.94s/it] 51%|██████████████████████████████████████████████████████████ | 350/681 [23:57<16:41, 3.03s/it] {'loss': 0.9483, 'grad_norm': 15.606201171875, 'learning_rate': 2.8326761550411346e-07, 'rewards/chosen': -1.7016650438308716, 'rewards/rejected': -2.411245822906494, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7095808982849121, 'logps/chosen': -183.2895050048828, 'logps/rejected': -232.35379028320312, 'logps/ref_chosen': -87.05419921875, 'logps/ref_rejected': -95.72193145751953, 'logits/chosen': -7.431205749511719, 'logits/rejected': -7.113109588623047, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.017774252220988274, 'kl/avg_steps': 0.59375, 'epoch': 0.51}
51%|██████████████████████████████████████████████████████████ | 350/681 [23:57<16:41, 3.03s/it] 52%|██████████████████████████████████████████████████████████▏ | 351/681 [24:00<16:53, 3.07s/it] {'loss': 0.8124, 'grad_norm': 13.189620018005371, 'learning_rate': 2.819952656376487e-07, 'rewards/chosen': -1.6149550676345825, 'rewards/rejected': -2.64827036857605, 'rewards/accuracies': 0.84375, 'rewards/margins': 1.0333154201507568, 'logps/chosen': -181.0399169921875, 'logps/rejected': -259.3538818359375, 'logps/ref_chosen': -89.16903686523438, 'logps/ref_rejected': -108.29801940917969, 'logits/chosen': -8.007095336914062, 'logits/rejected': -7.268853187561035, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.01766934059560299, 'kl/avg_steps': 0.65625, 'epoch': 0.52}
52%|██████████████████████████████████████████████████████████▏ | 351/681 [24:00<16:53, 3.07s/it] 52%|██████████████████████████████████████████████████████████▍ | 352/681 [24:03<16:50, 3.07s/it] {'loss': 0.8255, 'grad_norm': 14.201725959777832, 'learning_rate': 2.8072207266617854e-07, 'rewards/chosen': -1.509531021118164, 'rewards/rejected': -2.4648194313049316, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.955288290977478, 'logps/chosen': -183.04286193847656, 'logps/rejected': -225.66738891601562, 'logps/ref_chosen': -96.81147766113281, 'logps/ref_rejected': -84.40482330322266, 'logits/chosen': -7.791421890258789, 'logits/rejected': -7.136171340942383, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.017554141581058502, 'kl/avg_steps': 0.5625, 'epoch': 0.52}
52%|██████████████████████████████████████████████████████████▍ | 352/681 [24:04<16:50, 3.07s/it] 52%|██████████████████████████████████████████████████████████▌ | 353/681 [24:07<16:54, 3.09s/it] {'loss': 1.0508, 'grad_norm': 15.365931510925293, 'learning_rate': 2.794480701395219e-07, 'rewards/chosen': -1.7041988372802734, 'rewards/rejected': -2.301272392272949, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.5970733761787415, 'logps/chosen': -183.22384643554688, 'logps/rejected': -219.9366455078125, 'logps/ref_chosen': -85.36248016357422, 'logps/ref_rejected': -87.36732482910156, 'logits/chosen': -7.713462829589844, 'logits/rejected': -7.477531433105469, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.017455952242016792, 'kl/avg_steps': 0.40625, 'epoch': 0.52}
52%|██████████████████████████████████████████████████████████▌ | 353/681 [24:07<16:54, 3.09s/it] 52%|██████████████████████████████████████████████████████████▋ | 354/681 [24:10<16:59, 3.12s/it] {'loss': 0.7853, 'grad_norm': 12.326685905456543, 'learning_rate': 2.781732916288303e-07, 'rewards/chosen': -1.5137109756469727, 'rewards/rejected': -2.4479401111602783, 'rewards/accuracies': 0.875, 'rewards/margins': 0.9342291951179504, 'logps/chosen': -183.48825073242188, 'logps/rejected': -236.8675537109375, 'logps/ref_chosen': -95.8011703491211, 'logps/ref_rejected': -94.86614227294922, 'logits/chosen': -7.71798038482666, 'logits/rejected': -7.087685585021973, 'kl/p_epsilon_steps': 0.90625, 'kl/n_epsilon_steps': 0.09375, 'kl/beta': 0.017385324463248253, 'kl/avg_steps': 0.8125, 'epoch': 0.52}
52%|██████████████████████████████████████████████████████████▋ | 354/681 [24:10<16:59, 3.12s/it] 52%|██████████████████████████████████████████████████████████▉ | 355/681 [24:13<16:25, 3.02s/it] {'loss': 0.8456, 'grad_norm': 12.55382251739502, 'learning_rate': 2.7689777072570284e-07, 'rewards/chosen': -1.473813772201538, 'rewards/rejected': -2.3779821395874023, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.904168426990509, 'logps/chosen': -172.11546325683594, 'logps/rejected': -227.43336486816406, 'logps/ref_chosen': -86.185546875, 'logps/ref_rejected': -88.4615707397461, 'logits/chosen': -7.723712921142578, 'logits/rejected': -7.181628704071045, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.017245206981897354, 'kl/avg_steps': 0.6875, 'epoch': 0.52}
52%|██████████████████████████████████████████████████████████▉ | 355/681 [24:13<16:25, 3.02s/it] 52%|███████████████████████████████████████████████████████████ | 356/681 [24:16<16:43, 3.09s/it] {'loss': 1.0336, 'grad_norm': 15.903926849365234, 'learning_rate': 2.7562154104130176e-07, 'rewards/chosen': -1.7288204431533813, 'rewards/rejected': -2.390979290008545, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.6621589660644531, 'logps/chosen': -191.98709106445312, 'logps/rejected': -222.52052307128906, 'logps/ref_chosen': -90.818115234375, 'logps/ref_rejected': -82.1273193359375, 'logits/chosen': -7.718483924865723, 'logits/rejected': -7.080849647521973, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.01712745428085327, 'kl/avg_steps': 0.40625, 'epoch': 0.52}
52%|███████████████████████████████████████████████████████████ | 356/681 [24:16<16:43, 3.09s/it] 52%|███████████████████████████████████████████████████████████▏ | 357/681 [24:19<16:58, 3.14s/it] {'loss': 0.7488, 'grad_norm': 12.080792427062988, 'learning_rate': 2.7434463620546594e-07, 'rewards/chosen': -1.4316484928131104, 'rewards/rejected': -2.4867324829101562, 'rewards/accuracies': 0.875, 'rewards/margins': 1.055083990097046, 'logps/chosen': -171.85879516601562, 'logps/rejected': -239.12313842773438, 'logps/ref_chosen': -87.50475311279297, 'logps/ref_rejected': -92.24937438964844, 'logits/chosen': -7.890803337097168, 'logits/rejected': -7.257773399353027, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.017058156430721283, 'kl/avg_steps': 0.65625, 'epoch': 0.52}
52%|███████████████████████████████████████████████████████████▏ | 357/681 [24:19<16:58, 3.14s/it] 53%|███████████████████████████████████████████████████████████▍ | 358/681 [24:22<17:17, 3.21s/it] {'loss': 0.9431, 'grad_norm': 13.29261589050293, 'learning_rate': 2.730670898658255e-07, 'rewards/chosen': -1.6134233474731445, 'rewards/rejected': -2.3688549995422363, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.7554316520690918, 'logps/chosen': -190.7314453125, 'logps/rejected': -237.2618408203125, 'logps/ref_chosen': -95.05546569824219, 'logps/ref_rejected': -96.47611999511719, 'logits/chosen': -7.43376350402832, 'logits/rejected': -6.966874122619629, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.016946941614151, 'kl/avg_steps': 0.546875, 'epoch': 0.53}
53%|███████████████████████████████████████████████████████████▍ | 358/681 [24:23<17:17, 3.21s/it] 53%|███████████████████████████████████████████████████████████▌ | 359/681 [24:26<17:07, 3.19s/it] {'loss': 1.0136, 'grad_norm': 15.180391311645508, 'learning_rate': 2.717889356869146e-07, 'rewards/chosen': -1.762831687927246, 'rewards/rejected': -2.476144313812256, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.7133128046989441, 'logps/chosen': -192.33929443359375, 'logps/rejected': -236.61764526367188, 'logps/ref_chosen': -87.37332916259766, 'logps/ref_rejected': -88.77188110351562, 'logits/chosen': -7.312591552734375, 'logits/rejected': -6.956157684326172, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.016854766756296158, 'kl/avg_steps': 0.5, 'epoch': 0.53}
53%|███████████████████████████████████████████████████████████▌ | 359/681 [24:26<17:07, 3.19s/it] 53%|███████████████████████████████████████████████████████████▋ | 360/681 [24:29<16:43, 3.13s/it] {'loss': 0.8437, 'grad_norm': 14.518013954162598, 'learning_rate': 2.7051020734928443e-07, 'rewards/chosen': -1.6761372089385986, 'rewards/rejected': -2.5581398010253906, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8820022344589233, 'logps/chosen': -175.69329833984375, 'logps/rejected': -229.7479248046875, 'logps/ref_chosen': -75.22344970703125, 'logps/ref_rejected': -76.093994140625, 'logits/chosen': -7.531591415405273, 'logits/rejected': -6.866611480712891, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.01677091233432293, 'kl/avg_steps': 0.65625, 'epoch': 0.53}
53%|███████████████████████████████████████████████████████████▋ | 360/681 [24:29<16:43, 3.13s/it] 53%|███████████████████████████████████████████████████████████▉ | 361/681 [24:32<17:01, 3.19s/it] {'loss': 0.9577, 'grad_norm': 14.980257987976074, 'learning_rate': 2.6923093854861593e-07, 'rewards/chosen': -1.684104561805725, 'rewards/rejected': -2.467985153198242, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.7838806509971619, 'logps/chosen': -183.44650268554688, 'logps/rejected': -245.79881286621094, 'logps/ref_chosen': -81.95333099365234, 'logps/ref_rejected': -96.67919921875, 'logits/chosen': -7.7249908447265625, 'logits/rejected': -7.067028999328613, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.016661571338772774, 'kl/avg_steps': 0.5625, 'epoch': 0.53}
53%|███████████████████████████████████████████████████████████▉ | 361/681 [24:32<17:01, 3.19s/it] 53%|████████████████████████████████████████████████████████████ | 362/681 [24:35<16:17, 3.06s/it] {'loss': 0.8735, 'grad_norm': 12.824585914611816, 'learning_rate': 2.679511629948319e-07, 'rewards/chosen': -1.6923720836639404, 'rewards/rejected': -2.5505309104919434, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8581586480140686, 'logps/chosen': -192.10179138183594, 'logps/rejected': -265.67584228515625, 'logps/ref_chosen': -89.33971405029297, 'logps/ref_rejected': -110.5019760131836, 'logits/chosen': -7.730887413024902, 'logits/rejected': -7.379080772399902, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.016568373888731003, 'kl/avg_steps': 0.6875, 'epoch': 0.53}
53%|████████████████████████████████████████████████████████████ | 362/681 [24:35<16:17, 3.06s/it] 53%|████████████████████████████████████████████████████████████▏ | 363/681 [24:37<15:39, 2.95s/it] {'loss': 0.7656, 'grad_norm': 12.172687530517578, 'learning_rate': 2.6667091441120816e-07, 'rewards/chosen': -1.5798821449279785, 'rewards/rejected': -2.6781134605407715, 'rewards/accuracies': 0.8125, 'rewards/margins': 1.0982314348220825, 'logps/chosen': -177.34544372558594, 'logps/rejected': -243.47921752929688, 'logps/ref_chosen': -80.7750015258789, 'logps/ref_rejected': -79.43267822265625, 'logits/chosen': -7.627842903137207, 'logits/rejected': -7.226386070251465, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.01645524427294731, 'kl/avg_steps': 0.6875, 'epoch': 0.53}
53%|████████████████████████████████████████████████████████████▏ | 363/681 [24:37<15:39, 2.95s/it] 53%|████████████████████████████████████████████████████████████▍ | 364/681 [24:41<15:48, 2.99s/it] {'loss': 0.8462, 'grad_norm': 14.342798233032227, 'learning_rate': 2.6539022653348575e-07, 'rewards/chosen': -1.5991710424423218, 'rewards/rejected': -2.4389328956604004, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.8397619724273682, 'logps/chosen': -174.86444091796875, 'logps/rejected': -247.8596649169922, 'logps/ref_chosen': -76.52960205078125, 'logps/ref_rejected': -97.56173706054688, 'logits/chosen': -7.7837934494018555, 'logits/rejected': -6.654484748840332, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.01634288765490055, 'kl/avg_steps': 0.625, 'epoch': 0.53}
53%|████████████████████████████████████████████████████████████▍ | 364/681 [24:41<15:48, 2.99s/it] 54%|████████████████████████████████████████████████████████████▌ | 365/681 [24:44<16:06, 3.06s/it] {'loss': 0.8603, 'grad_norm': 13.283844947814941, 'learning_rate': 2.641091331089811e-07, 'rewards/chosen': -1.5808205604553223, 'rewards/rejected': -2.5037970542907715, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9229766130447388, 'logps/chosen': -178.31759643554688, 'logps/rejected': -254.62168884277344, 'logps/ref_chosen': -80.54489135742188, 'logps/ref_rejected': -99.32316589355469, 'logits/chosen': -7.504325866699219, 'logits/rejected': -7.238739967346191, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.01624137908220291, 'kl/avg_steps': 0.65625, 'epoch': 0.54}
54%|████████████████████████████████████████████████████████████▌ | 365/681 [24:44<16:06, 3.06s/it] 54%|████████████████████████████████████████████████████████████▋ | 366/681 [24:47<16:02, 3.05s/it] {'loss': 0.86, 'grad_norm': 12.054483413696289, 'learning_rate': 2.6282766789569736e-07, 'rewards/chosen': -1.5466276407241821, 'rewards/rejected': -2.407994031906128, 'rewards/accuracies': 0.921875, 'rewards/margins': 0.8613663911819458, 'logps/chosen': -168.7039337158203, 'logps/rejected': -242.06005859375, 'logps/ref_chosen': -72.23806762695312, 'logps/ref_rejected': -91.67182922363281, 'logits/chosen': -7.318140983581543, 'logits/rejected': -6.966005325317383, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.01613548956811428, 'kl/avg_steps': 0.71875, 'epoch': 0.54}
54%|████████████████████████████████████████████████████████████▋ | 366/681 [24:47<16:02, 3.05s/it] 54%|████████████████████████████████████████████████████████████▉ | 367/681 [24:50<16:11, 3.10s/it] {'loss': 0.8254, 'grad_norm': 12.747227668762207, 'learning_rate': 2.615458646614349e-07, 'rewards/chosen': -1.4997714757919312, 'rewards/rejected': -2.396198272705078, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.896426796913147, 'logps/chosen': -179.722412109375, 'logps/rejected': -234.3914031982422, 'logps/ref_chosen': -85.42201232910156, 'logps/ref_rejected': -83.51779174804688, 'logits/chosen': -7.577718734741211, 'logits/rejected': -7.4753642082214355, 'kl/p_epsilon_steps': 0.90625, 'kl/n_epsilon_steps': 0.078125, 'kl/beta': 0.016020342707633972, 'kl/avg_steps': 0.828125, 'epoch': 0.54}
54%|████████████████████████████████████████████████████████████▉ | 367/681 [24:50<16:11, 3.10s/it] 54%|█████████████████████████████████████████████████████████████ | 368/681 [24:53<16:06, 3.09s/it] {'loss': 0.8403, 'grad_norm': 14.037300109863281, 'learning_rate': 2.6026375718290083e-07, 'rewards/chosen': -1.4564285278320312, 'rewards/rejected': -2.388618230819702, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9321895241737366, 'logps/chosen': -165.94821166992188, 'logps/rejected': -256.14404296875, 'logps/ref_chosen': -73.84112548828125, 'logps/ref_rejected': -104.69528198242188, 'logits/chosen': -7.514585494995117, 'logits/rejected': -7.210457801818848, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.015888763591647148, 'kl/avg_steps': 0.625, 'epoch': 0.54}
54%|█████████████████████████████████████████████████████████████ | 368/681 [24:53<16:06, 3.09s/it] 54%|█████████████████████████████████████████████████████████████▏ | 369/681 [24:56<16:19, 3.14s/it] {'loss': 0.7803, 'grad_norm': 13.097122192382812, 'learning_rate': 2.589813792448196e-07, 'rewards/chosen': -1.545214056968689, 'rewards/rejected': -2.4702415466308594, 'rewards/accuracies': 0.875, 'rewards/margins': 0.9250272512435913, 'logps/chosen': -196.92771911621094, 'logps/rejected': -256.7950439453125, 'logps/ref_chosen': -98.46196746826172, 'logps/ref_rejected': -99.15672302246094, 'logits/chosen': -7.481395721435547, 'logits/rejected': -6.605921745300293, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.015790076926350594, 'kl/avg_steps': 0.75, 'epoch': 0.54}
54%|█████████████████████████████████████████████████████████████▏ | 369/681 [24:56<16:19, 3.14s/it] 54%|█████████████████████████████████████████████████████████████▍ | 370/681 [25:00<16:25, 3.17s/it] {'loss': 1.002, 'grad_norm': 13.142091751098633, 'learning_rate': 2.5769876463904263e-07, 'rewards/chosen': -1.687301754951477, 'rewards/rejected': -2.3558273315429688, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.6685256361961365, 'logps/chosen': -204.67718505859375, 'logps/rejected': -253.85340881347656, 'logps/ref_chosen': -96.77574157714844, 'logps/ref_rejected': -102.74411010742188, 'logits/chosen': -7.735294342041016, 'logits/rejected': -7.384430885314941, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.015672532841563225, 'kl/avg_steps': 0.40625, 'epoch': 0.54}
54%|█████████████████████████████████████████████████████████████▍ | 370/681 [25:00<16:25, 3.17s/it] 54%|█████████████████████████████████████████████████████████████▌ | 371/681 [25:03<16:23, 3.17s/it] {'loss': 1.0035, 'grad_norm': 14.304271697998047, 'learning_rate': 2.5641594716365744e-07, 'rewards/chosen': -1.6160762310028076, 'rewards/rejected': -2.2831013202667236, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.6670252084732056, 'logps/chosen': -206.90415954589844, 'logps/rejected': -252.56182861328125, 'logps/ref_chosen': -102.99020385742188, 'logps/ref_rejected': -105.35063171386719, 'logits/chosen': -7.651013374328613, 'logits/rejected': -7.241464614868164, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.015609120018780231, 'kl/avg_steps': 0.46875, 'epoch': 0.54}
54%|█████████████████████████████████████████████████████████████▌ | 371/681 [25:03<16:23, 3.17s/it] 55%|█████████████████████████████████████████████████████████████▋ | 372/681 [25:06<15:46, 3.06s/it] {'loss': 0.7449, 'grad_norm': 11.53231430053711, 'learning_rate': 2.551329606220976e-07, 'rewards/chosen': -1.3880116939544678, 'rewards/rejected': -2.452024221420288, 'rewards/accuracies': 0.890625, 'rewards/margins': 1.0640125274658203, 'logps/chosen': -181.00729370117188, 'logps/rejected': -244.81317138671875, 'logps/ref_chosen': -91.02623748779297, 'logps/ref_rejected': -85.6102294921875, 'logits/chosen': -7.6374592781066895, 'logits/rejected': -7.359759330749512, 'kl/p_epsilon_steps': 0.90625, 'kl/n_epsilon_steps': 0.09375, 'kl/beta': 0.0155362943187356, 'kl/avg_steps': 0.8125, 'epoch': 0.55}
55%|█████████████████████████████████████████████████████████████▋ | 372/681 [25:06<15:46, 3.06s/it] 55%|█████████████████████████████████████████████████████████████▉ | 373/681 [25:09<15:53, 3.10s/it] {'loss': 0.8783, 'grad_norm': 14.019618034362793, 'learning_rate': 2.538498388222517e-07, 'rewards/chosen': -1.5517187118530273, 'rewards/rejected': -2.3930416107177734, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8413230180740356, 'logps/chosen': -201.16326904296875, 'logps/rejected': -247.42393493652344, 'logps/ref_chosen': -100.07307434082031, 'logps/ref_rejected': -91.11634826660156, 'logits/chosen': -7.477275848388672, 'logits/rejected': -7.463842868804932, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.015411078929901123, 'kl/avg_steps': 0.625, 'epoch': 0.55}
55%|█████████████████████████████████████████████████████████████▉ | 373/681 [25:09<15:53, 3.10s/it] 55%|██████████████████████████████████████████████████████████████ | 374/681 [25:12<16:09, 3.16s/it] {'loss': 0.8132, 'grad_norm': 11.548641204833984, 'learning_rate': 2.525666155755725e-07, 'rewards/chosen': -1.3845758438110352, 'rewards/rejected': -2.258387327194214, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8738114833831787, 'logps/chosen': -188.9845733642578, 'logps/rejected': -249.3828582763672, 'logps/ref_chosen': -98.01234436035156, 'logps/ref_rejected': -100.75224304199219, 'logits/chosen': -7.55961799621582, 'logits/rejected': -7.308844089508057, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.015315357595682144, 'kl/avg_steps': 0.6875, 'epoch': 0.55}
55%|██████████████████████████████████████████████████████████████ | 374/681 [25:12<16:09, 3.16s/it] 55%|██████████████████████████████████████████████████████████████▏ | 375/681 [25:15<16:04, 3.15s/it] {'loss': 0.8142, 'grad_norm': 12.984651565551758, 'learning_rate': 2.512833246961859e-07, 'rewards/chosen': -1.4533175230026245, 'rewards/rejected': -2.398698091506958, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9453805685043335, 'logps/chosen': -180.4878387451172, 'logps/rejected': -254.66061401367188, 'logps/ref_chosen': -84.39627838134766, 'logps/ref_rejected': -95.7188491821289, 'logits/chosen': -7.9140520095825195, 'logits/rejected': -7.584393501281738, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.015210784040391445, 'kl/avg_steps': 0.71875, 'epoch': 0.55}
55%|██████████████████████████████████████████████████████████████▏ | 375/681 [25:15<16:04, 3.15s/it] 55%|██████████████████████████████████████████████████████████████▍ | 376/681 [25:18<16:16, 3.20s/it] {'loss': 0.7789, 'grad_norm': 12.451373100280762, 'learning_rate': 2.5e-07, 'rewards/chosen': -1.3709272146224976, 'rewards/rejected': -2.3846511840820312, 'rewards/accuracies': 0.890625, 'rewards/margins': 1.0137239694595337, 'logps/chosen': -187.0894775390625, 'logps/rejected': -271.1904296875, 'logps/ref_chosen': -95.73770141601562, 'logps/ref_rejected': -111.97331237792969, 'logits/chosen': -7.593746662139893, 'logits/rejected': -7.532279014587402, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.015102236531674862, 'kl/avg_steps': 0.75, 'epoch': 0.55}
55%|██████████████████████████████████████████████████████████████▍ | 376/681 [25:18<16:16, 3.20s/it] 55%|██████████████████████████████████████████████████████████████▌ | 377/681 [25:21<15:49, 3.12s/it] {'loss': 0.7926, 'grad_norm': 14.200995445251465, 'learning_rate': 2.487166753038141e-07, 'rewards/chosen': -1.4772100448608398, 'rewards/rejected': -2.5180411338806152, 'rewards/accuracies': 0.84375, 'rewards/margins': 1.0408310890197754, 'logps/chosen': -178.06936645507812, 'logps/rejected': -275.2508544921875, 'logps/ref_chosen': -78.92694854736328, 'logps/ref_rejected': -105.89892578125, 'logits/chosen': -7.635054588317871, 'logits/rejected': -7.0941009521484375, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.014989812858402729, 'kl/avg_steps': 0.6875, 'epoch': 0.55}
55%|██████████████████████████████████████████████████████████████▌ | 377/681 [25:21<15:49, 3.12s/it] 56%|██████████████████████████████████████████████████████████████▋ | 378/681 [25:24<14:53, 2.95s/it] {'loss': 0.8581, 'grad_norm': 13.540968894958496, 'learning_rate': 2.4743338442442754e-07, 'rewards/chosen': -1.4412063360214233, 'rewards/rejected': -2.3105268478393555, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.869320273399353, 'logps/chosen': -172.66668701171875, 'logps/rejected': -250.10272216796875, 'logps/ref_chosen': -75.37240600585938, 'logps/ref_rejected': -93.70409393310547, 'logits/chosen': -7.783210754394531, 'logits/rejected': -7.034327507019043, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.014887461438775063, 'kl/avg_steps': 0.6875, 'epoch': 0.56}
56%|██████████████████████████████████████████████████████████████▋ | 378/681 [25:24<14:53, 2.95s/it] 56%|██████████████████████████████████████████████████████████████▉ | 379/681 [25:27<14:46, 2.94s/it] {'loss': 0.9442, 'grad_norm': 12.696690559387207, 'learning_rate': 2.461501611777483e-07, 'rewards/chosen': -1.5753943920135498, 'rewards/rejected': -2.396908760070801, 'rewards/accuracies': 0.75, 'rewards/margins': 0.8215144276618958, 'logps/chosen': -195.7200927734375, 'logps/rejected': -284.5279541015625, 'logps/ref_chosen': -88.85568237304688, 'logps/ref_rejected': -121.38114929199219, 'logits/chosen': -7.8199920654296875, 'logits/rejected': -6.8525214195251465, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.014785809442400932, 'kl/avg_steps': 0.53125, 'epoch': 0.56}
56%|██████████████████████████████████████████████████████████████▉ | 379/681 [25:27<14:46, 2.94s/it] 56%|███████████████████████████████████████████████████████████████ | 380/681 [25:30<14:54, 2.97s/it] {'loss': 0.8692, 'grad_norm': 14.155634880065918, 'learning_rate': 2.4486703937790243e-07, 'rewards/chosen': -1.7239048480987549, 'rewards/rejected': -2.6357314586639404, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.9118264317512512, 'logps/chosen': -197.644775390625, 'logps/rejected': -291.2830810546875, 'logps/ref_chosen': -79.83321380615234, 'logps/ref_rejected': -110.74475860595703, 'logits/chosen': -7.855627059936523, 'logits/rejected': -7.288210868835449, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.014707674272358418, 'kl/avg_steps': 0.65625, 'epoch': 0.56}
56%|███████████████████████████████████████████████████████████████ | 380/681 [25:30<14:54, 2.97s/it] 56%|███████████████████████████████████████████████████████████████▏ | 381/681 [25:33<15:04, 3.01s/it] {'loss': 0.8887, 'grad_norm': 14.164735794067383, 'learning_rate': 2.435840528363426e-07, 'rewards/chosen': -1.5055629014968872, 'rewards/rejected': -2.312436580657959, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8068733811378479, 'logps/chosen': -190.10198974609375, 'logps/rejected': -243.78826904296875, 'logps/ref_chosen': -86.56683349609375, 'logps/ref_rejected': -84.416748046875, 'logits/chosen': -7.805541038513184, 'logits/rejected': -6.962313652038574, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.014611784368753433, 'kl/avg_steps': 0.625, 'epoch': 0.56}
56%|███████████████████████████████████████████████████████████████▏ | 381/681 [25:33<15:04, 3.01s/it] 56%|███████████████████████████████████████████████████████████████▍ | 382/681 [25:36<15:03, 3.02s/it] {'loss': 0.8502, 'grad_norm': 12.716059684753418, 'learning_rate': 2.4230123536095745e-07, 'rewards/chosen': -1.4875770807266235, 'rewards/rejected': -2.364816427230835, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8772394061088562, 'logps/chosen': -207.37265014648438, 'logps/rejected': -278.6050109863281, 'logps/ref_chosen': -104.46180725097656, 'logps/ref_rejected': -114.60223388671875, 'logits/chosen': -7.854497909545898, 'logits/rejected': -7.399716377258301, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.01452102791517973, 'kl/avg_steps': 0.59375, 'epoch': 0.56}
56%|███████████████████████████████████████████████████████████████▍ | 382/681 [25:36<15:03, 3.02s/it] 56%|███████████████████████████████████████████████████████████████▌ | 383/681 [25:39<15:03, 3.03s/it] {'loss': 0.927, 'grad_norm': 13.573335647583008, 'learning_rate': 2.4101862075518037e-07, 'rewards/chosen': -1.593533992767334, 'rewards/rejected': -2.3465826511383057, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.7530485987663269, 'logps/chosen': -192.78128051757812, 'logps/rejected': -262.2416076660156, 'logps/ref_chosen': -82.11607360839844, 'logps/ref_rejected': -98.77716064453125, 'logits/chosen': -7.561490058898926, 'logits/rejected': -7.388404369354248, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.014435318298637867, 'kl/avg_steps': 0.4375, 'epoch': 0.56}
56%|███████████████████████████████████████████████████████████████▌ | 383/681 [25:39<15:03, 3.03s/it] 56%|███████████████████████████████████████████████████████████████▋ | 384/681 [25:42<15:14, 3.08s/it] {'loss': 0.9637, 'grad_norm': 14.46293830871582, 'learning_rate': 2.397362428170992e-07, 'rewards/chosen': -1.6679697036743164, 'rewards/rejected': -2.444589853286743, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7766201496124268, 'logps/chosen': -197.14675903320312, 'logps/rejected': -262.77398681640625, 'logps/ref_chosen': -80.58525085449219, 'logps/ref_rejected': -91.44789123535156, 'logits/chosen': -7.834491729736328, 'logits/rejected': -7.562644004821777, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.014372439123690128, 'kl/avg_steps': 0.625, 'epoch': 0.56}
56%|███████████████████████████████████████████████████████████████▋ | 384/681 [25:42<15:14, 3.08s/it] 57%|███████████████████████████████████████████████████████████████▉ | 385/681 [25:45<14:56, 3.03s/it] {'loss': 0.8706, 'grad_norm': 13.538987159729004, 'learning_rate': 2.3845413533856514e-07, 'rewards/chosen': -1.4141170978546143, 'rewards/rejected': -2.3140358924865723, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8999189138412476, 'logps/chosen': -192.1517333984375, 'logps/rejected': -247.5975341796875, 'logps/ref_chosen': -92.76016235351562, 'logps/ref_rejected': -84.45997619628906, 'logits/chosen': -7.669012069702148, 'logits/rejected': -7.240923881530762, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.01428316906094551, 'kl/avg_steps': 0.546875, 'epoch': 0.57}
57%|███████████████████████████████████████████████████████████████▉ | 385/681 [25:45<14:56, 3.03s/it] 57%|████████████████████████████████████████████████████████████████ | 386/681 [25:48<14:54, 3.03s/it] {'loss': 0.8706, 'grad_norm': 11.718225479125977, 'learning_rate': 2.3717233210430254e-07, 'rewards/chosen': -1.461796522140503, 'rewards/rejected': -2.3275704383850098, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8657737970352173, 'logps/chosen': -188.76788330078125, 'logps/rejected': -262.10107421875, 'logps/ref_chosen': -85.35896301269531, 'logps/ref_rejected': -96.99065399169922, 'logits/chosen': -7.931042671203613, 'logits/rejected': -7.277120590209961, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.01420548278838396, 'kl/avg_steps': 0.65625, 'epoch': 0.57}
57%|████████████████████████████████████████████████████████████████ | 386/681 [25:48<14:54, 3.03s/it] 57%|████████████████████████████████████████████████████████████████▏ | 387/681 [25:51<14:45, 3.01s/it] {'loss': 0.8604, 'grad_norm': 12.556093215942383, 'learning_rate': 2.3589086689101889e-07, 'rewards/chosen': -1.6170012950897217, 'rewards/rejected': -2.446988582611084, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8299874067306519, 'logps/chosen': -217.92800903320312, 'logps/rejected': -274.15191650390625, 'logps/ref_chosen': -102.68814086914062, 'logps/ref_rejected': -99.42858123779297, 'logits/chosen': -7.733986854553223, 'logits/rejected': -6.640727519989014, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.014112867414951324, 'kl/avg_steps': 0.71875, 'epoch': 0.57}
57%|████████████████████████████████████████████████████████████████▏ | 387/681 [25:51<14:45, 3.01s/it] 57%|████████████████████████████████████████████████████████████████▍ | 388/681 [25:54<14:39, 3.00s/it] {'loss': 0.7893, 'grad_norm': 12.432695388793945, 'learning_rate': 2.3460977346651428e-07, 'rewards/chosen': -1.5865106582641602, 'rewards/rejected': -2.5361099243164062, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.9495990872383118, 'logps/chosen': -197.45010375976562, 'logps/rejected': -292.4563293457031, 'logps/ref_chosen': -83.58523559570312, 'logps/ref_rejected': -110.10037231445312, 'logits/chosen': -7.944313049316406, 'logits/rejected': -7.454949855804443, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.014012155123054981, 'kl/avg_steps': 0.6875, 'epoch': 0.57}
57%|████████████████████████████████████████████████████████████████▍ | 388/681 [25:54<14:39, 3.00s/it] 57%|████████████████████████████████████████████████████████████████▌ | 389/681 [25:57<14:37, 3.00s/it] {'loss': 0.9298, 'grad_norm': 13.698887825012207, 'learning_rate': 2.3332908558879177e-07, 'rewards/chosen': -1.663960337638855, 'rewards/rejected': -2.489985704421997, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.8260251879692078, 'logps/chosen': -209.8504180908203, 'logps/rejected': -264.9825134277344, 'logps/ref_chosen': -89.822265625, 'logps/ref_rejected': -84.85377502441406, 'logits/chosen': -7.64278507232666, 'logits/rejected': -7.23593807220459, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.013916479423642159, 'kl/avg_steps': 0.53125, 'epoch': 0.57}
57%|████████████████████████████████████████████████████████████████▌ | 389/681 [25:57<14:37, 3.00s/it] 57%|████████████████████████████████████████████████████████████████▋ | 390/681 [26:00<14:32, 3.00s/it] {'loss': 0.971, 'grad_norm': 12.986654281616211, 'learning_rate': 2.320488370051681e-07, 'rewards/chosen': -1.6698600053787231, 'rewards/rejected': -2.424213171005249, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.7543532252311707, 'logps/chosen': -198.42169189453125, 'logps/rejected': -268.6641540527344, 'logps/ref_chosen': -77.53244018554688, 'logps/ref_rejected': -92.58392333984375, 'logits/chosen': -7.480320453643799, 'logits/rejected': -7.105579853057861, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.013842938467860222, 'kl/avg_steps': 0.46875, 'epoch': 0.57}
57%|████████████████████████████████████████████████████████████████▋ | 390/681 [26:00<14:32, 3.00s/it] 57%|████████████████████████████████████████████████████████████████▉ | 391/681 [26:03<14:20, 2.97s/it] {'loss': 0.9043, 'grad_norm': 13.51944351196289, 'learning_rate': 2.3076906145138405e-07, 'rewards/chosen': -1.5192664861679077, 'rewards/rejected': -2.3522844314575195, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8330180048942566, 'logps/chosen': -198.9241943359375, 'logps/rejected': -264.8871765136719, 'logps/ref_chosen': -88.2329330444336, 'logps/ref_rejected': -93.00756072998047, 'logits/chosen': -7.788546085357666, 'logits/rejected': -6.940990447998047, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.013778353109955788, 'kl/avg_steps': 0.59375, 'epoch': 0.57}
57%|████████████████████████████████████████████████████████████████▉ | 391/681 [26:03<14:20, 2.97s/it] 58%|█████████████████████████████████████████████████████████████████ | 392/681 [26:06<14:51, 3.09s/it] {'loss': 0.8565, 'grad_norm': 13.640396118164062, 'learning_rate': 2.294897926507156e-07, 'rewards/chosen': -1.5143227577209473, 'rewards/rejected': -2.435004711151123, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.9206820726394653, 'logps/chosen': -198.85385131835938, 'logps/rejected': -287.69525146484375, 'logps/ref_chosen': -87.74441528320312, 'logps/ref_rejected': -108.57071685791016, 'logits/chosen': -7.636374473571777, 'logits/rejected': -7.049764156341553, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.013697026297450066, 'kl/avg_steps': 0.65625, 'epoch': 0.58}
58%|█████████████████████████████████████████████████████████████████ | 392/681 [26:06<14:51, 3.09s/it] 58%|█████████████████████████████████████████████████████████████████▏ | 393/681 [26:09<14:28, 3.01s/it] {'loss': 0.8682, 'grad_norm': 11.787603378295898, 'learning_rate': 2.2821106431308543e-07, 'rewards/chosen': -1.3515516519546509, 'rewards/rejected': -2.1951210498809814, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.8435693979263306, 'logps/chosen': -176.33932495117188, 'logps/rejected': -242.21641540527344, 'logps/ref_chosen': -76.56381225585938, 'logps/ref_rejected': -79.72618865966797, 'logits/chosen': -7.288888931274414, 'logits/rejected': -6.870842933654785, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.013607725501060486, 'kl/avg_steps': 0.625, 'epoch': 0.58}
58%|█████████████████████████████████████████████████████████████████▏ | 393/681 [26:09<14:28, 3.01s/it] 58%|█████████████████████████████████████████████████████████████████▍ | 394/681 [26:12<14:33, 3.04s/it] {'loss': 0.8111, 'grad_norm': 12.555571556091309, 'learning_rate': 2.2693291013417452e-07, 'rewards/chosen': -1.3974303007125854, 'rewards/rejected': -2.2949118614196777, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8974814414978027, 'logps/chosen': -188.681640625, 'logps/rejected': -268.5348815917969, 'logps/ref_chosen': -84.69866180419922, 'logps/ref_rejected': -97.457763671875, 'logits/chosen': -7.870112895965576, 'logits/rejected': -7.220252513885498, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.013523206114768982, 'kl/avg_steps': 0.703125, 'epoch': 0.58}
58%|█████████████████████████████████████████████████████████████████▍ | 394/681 [26:12<14:33, 3.04s/it] 58%|█████████████████████████████████████████████████████████████████▌ | 395/681 [26:15<14:18, 3.00s/it] {'loss': 1.0664, 'grad_norm': 14.517809867858887, 'learning_rate': 2.2565536379453404e-07, 'rewards/chosen': -1.4323430061340332, 'rewards/rejected': -2.1769344806671143, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.7445914149284363, 'logps/chosen': -195.88299560546875, 'logps/rejected': -254.3079376220703, 'logps/ref_chosen': -88.88763427734375, 'logps/ref_rejected': -91.0455322265625, 'logits/chosen': -7.730129718780518, 'logits/rejected': -7.53689432144165, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.013428784906864166, 'kl/avg_steps': 0.5625, 'epoch': 0.58}
58%|█████████████████████████████████████████████████████████████████▌ | 395/681 [26:15<14:18, 3.00s/it] 58%|█████████████████████████████████████████████████████████████████▋ | 396/681 [26:18<14:29, 3.05s/it] {'loss': 1.0318, 'grad_norm': 16.962797164916992, 'learning_rate': 2.2437845895869825e-07, 'rewards/chosen': -1.542496919631958, 'rewards/rejected': -2.142852783203125, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.6003559827804565, 'logps/chosen': -218.27194213867188, 'logps/rejected': -254.623046875, 'logps/ref_chosen': -102.47132873535156, 'logps/ref_rejected': -93.23478698730469, 'logits/chosen': -7.620447158813477, 'logits/rejected': -7.031801223754883, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.01335367001593113, 'kl/avg_steps': 0.40625, 'epoch': 0.58}
58%|█████████████████████████████████████████████████████████████████▋ | 396/681 [26:18<14:29, 3.05s/it] 58%|█████████████████████████████████████████████████████████████████▉ | 397/681 [26:22<14:28, 3.06s/it] {'loss': 0.794, 'grad_norm': 13.833749771118164, 'learning_rate': 2.2310222927429716e-07, 'rewards/chosen': -1.3734837770462036, 'rewards/rejected': -2.479825496673584, 'rewards/accuracies': 0.828125, 'rewards/margins': 1.1063416004180908, 'logps/chosen': -204.63792419433594, 'logps/rejected': -296.58837890625, 'logps/ref_chosen': -101.04441833496094, 'logps/ref_rejected': -108.8511734008789, 'logits/chosen': -8.157541275024414, 'logits/rejected': -7.536031246185303, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.013299640268087387, 'kl/avg_steps': 0.578125, 'epoch': 0.58}
58%|█████████████████████████████████████████████████████████████████▉ | 397/681 [26:22<14:28, 3.06s/it] 58%|██████████████████████████████████████████████████████████████████ | 398/681 [26:24<13:58, 2.96s/it] {'loss': 0.9283, 'grad_norm': 13.908939361572266, 'learning_rate': 2.2182670837116972e-07, 'rewards/chosen': -1.4388189315795898, 'rewards/rejected': -2.2673499584198, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.82853102684021, 'logps/chosen': -204.48220825195312, 'logps/rejected': -286.8526611328125, 'logps/ref_chosen': -95.04922485351562, 'logps/ref_rejected': -114.00563049316406, 'logits/chosen': -7.908001899719238, 'logits/rejected': -7.516959190368652, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.01322319358587265, 'kl/avg_steps': 0.65625, 'epoch': 0.58}
58%|██████████████████████████████████████████████████████████████████ | 398/681 [26:24<13:58, 2.96s/it] 59%|██████████████████████████████████████████████████████████████████▏ | 399/681 [26:27<13:34, 2.89s/it] {'loss': 0.7418, 'grad_norm': 12.327967643737793, 'learning_rate': 2.2055192986047804e-07, 'rewards/chosen': -1.3026535511016846, 'rewards/rejected': -2.3269894123077393, 'rewards/accuracies': 0.921875, 'rewards/margins': 1.0243358612060547, 'logps/chosen': -196.3619384765625, 'logps/rejected': -262.048828125, 'logps/ref_chosen': -96.63853454589844, 'logps/ref_rejected': -83.53645324707031, 'logits/chosen': -7.728894233703613, 'logits/rejected': -7.285490989685059, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.013136982917785645, 'kl/avg_steps': 0.71875, 'epoch': 0.59}
59%|██████████████████████████████████████████████████████████████████▏ | 399/681 [26:27<13:34, 2.89s/it] 59%|██████████████████████████████████████████████████████████████████▎ | 400/681 [26:30<13:48, 2.95s/it] {'loss': 0.7868, 'grad_norm': 14.875067710876465, 'learning_rate': 2.192779273338215e-07, 'rewards/chosen': -1.2911700010299683, 'rewards/rejected': -2.3648228645324707, 'rewards/accuracies': 0.859375, 'rewards/margins': 1.073652744293213, 'logps/chosen': -204.7100067138672, 'logps/rejected': -291.34063720703125, 'logps/ref_chosen': -105.30314636230469, 'logps/ref_rejected': -108.6555404663086, 'logits/chosen': -8.027582168579102, 'logits/rejected': -7.236846923828125, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.013043234124779701, 'kl/avg_steps': 0.6875, 'epoch': 0.59}
59%|██████████████████████████████████████████████████████████████████▎ | 400/681 [26:30<13:48, 2.95s/it][INFO|trainer.py:4307] 2026-04-24 04:42:37,685 >>
***** Running Evaluation *****
[INFO|trainer.py:4309] 2026-04-24 04:42:37,685 >> Num examples = 2339
[INFO|trainer.py:4312] 2026-04-24 04:42:37,685 >> Batch size = 8
0%| | 0/73 [00:00<?, ?it/s]
3%|███▏ | 2/73 [00:00<00:23, 3.03it/s]
4%|████▊ | 3/73 [00:01<00:32, 2.14it/s]
5%|██████▎ | 4/73 [00:01<00:36, 1.88it/s]
7%|███████▉ | 5/73 [00:02<00:38, 1.76it/s]
8%|█████████▌ | 6/73 [00:03<00:40, 1.66it/s]
10%|███████████ | 7/73 [00:03<00:38, 1.71it/s]
11%|████████████▋ | 8/73 [00:04<00:40, 1.59it/s]
12%|██████████████▎ | 9/73 [00:05<00:41, 1.54it/s]
14%|███████████████▊ | 10/73 [00:05<00:41, 1.52it/s]
15%|█████████████████▎ | 11/73 [00:06<00:40, 1.52it/s]
16%|██████████████████▉ | 12/73 [00:07<00:40, 1.49it/s]
18%|████████████████████▍ | 13/73 [00:07<00:39, 1.53it/s]
19%|██████████████████████ | 14/73 [00:08<00:39, 1.51it/s]
21%|███████████████████████▋ | 15/73 [00:09<00:38, 1.51it/s]
22%|█████████████████████████▏ | 16/73 [00:09<00:39, 1.46it/s]
23%|██████████████████████████▊ | 17/73 [00:10<00:38, 1.45it/s]
25%|████████████████████████████▎ | 18/73 [00:11<00:38, 1.43it/s]
26%|█████████████████████████████▉ | 19/73 [00:12<00:38, 1.41it/s]
27%|███████████████████████████████▌ | 20/73 [00:12<00:37, 1.40it/s]
29%|█████████████████████████████████ | 21/73 [00:13<00:37, 1.40it/s]
30%|██████████████████████████████████▋ | 22/73 [00:14<00:36, 1.39it/s]
32%|████████████████████████████████████▏ | 23/73 [00:14<00:35, 1.42it/s]
33%|█████████████████████████████████████▊ | 24/73 [00:15<00:34, 1.43it/s]
34%|███████████████████████████████████████▍ | 25/73 [00:16<00:33, 1.44it/s]
36%|████████████████████████████████████████▉ | 26/73 [00:17<00:32, 1.45it/s]
37%|██████████████████████████████████████████▌ | 27/73 [00:17<00:28, 1.60it/s]
38%|████████████████████████████████████████████ | 28/73 [00:18<00:28, 1.56it/s]
40%|█████████████████████████████████████████████▋ | 29/73 [00:18<00:28, 1.57it/s]
41%|███████████████████████████████████████████████▎ | 30/73 [00:19<00:27, 1.59it/s]
42%|████████████████████████████████████████████████▊ | 31/73 [00:20<00:27, 1.52it/s]
44%|██████████████████████████████████████████████████▍ | 32/73 [00:20<00:26, 1.55it/s]
45%|███████████████████████████████████████████████████▉ | 33/73 [00:21<00:25, 1.58it/s]
47%|█████████████████████████████████████████████████████▌ | 34/73 [00:22<00:25, 1.55it/s]
48%|███████████████████████████████████████████████████████▏ | 35/73 [00:22<00:25, 1.49it/s]
49%|████████████████████████████████████████████████████████▋ | 36/73 [00:23<00:24, 1.49it/s]
51%|██████████████████████████████████████████████████████████▎ | 37/73 [00:24<00:24, 1.48it/s]
52%|███████████████████████████████████████████████████████████▊ | 38/73 [00:24<00:22, 1.57it/s]
53%|█████████████████████████████████████████████████████████████▍ | 39/73 [00:25<00:22, 1.50it/s]
55%|███████████████████████████████████████████████████████████████ | 40/73 [00:26<00:21, 1.50it/s]
56%|████████████████████████████████████████████████████████████████▌ | 41/73 [00:26<00:20, 1.56it/s]
58%|██████████████████████████████████████████████████████████████████▏ | 42/73 [00:27<00:20, 1.52it/s]
59%|███████████████████████████████████████████████████████████████████▋ | 43/73 [00:27<00:19, 1.54it/s]
60%|█████████████████████████████████████████████████████████████████████▎ | 44/73 [00:28<00:19, 1.52it/s]
62%|██████████████████████████████████████████████████████████████████████▉ | 45/73 [00:29<00:19, 1.47it/s]
63%|████████████████████████████████████████████████████████████████████████▍ | 46/73 [00:29<00:17, 1.53it/s]
64%|██████████████████████████████████████████████████████████████████████████ | 47/73 [00:30<00:17, 1.50it/s]
66%|███████████████████████████████████████████████████████████████████████████▌ | 48/73 [00:31<00:16, 1.49it/s]
67%|█████████████████████████████████████████████████████████████████████████████▏ | 49/73 [00:32<00:16, 1.47it/s]
68%|██████████████████████████████████████████████████████████████████████████████▊ | 50/73 [00:32<00:15, 1.49it/s]
70%|████████████████████████████████████████████████████████████████████████████████▎ | 51/73 [00:33<00:15, 1.46it/s]
71%|█████████████████████████████████████████████████████████████████████████████████▉ | 52/73 [00:34<00:14, 1.43it/s]
73%|███████████████████████████████████████████████████████████████████████████████████▍ | 53/73 [00:34<00:14, 1.41it/s]
74%|█████████████████████████████████████████████████████████████████████████████████████ | 54/73 [00:35<00:12, 1.50it/s]
75%|██████████████████████████████████████████████████████████████████████████████████████▋ | 55/73 [00:36<00:12, 1.50it/s]
77%|████████████████████████████████████████████████████████████████████████████████████████▏ | 56/73 [00:36<00:11, 1.53it/s]
78%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 57/73 [00:37<00:10, 1.48it/s]
79%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 58/73 [00:38<00:09, 1.54it/s]
81%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 59/73 [00:38<00:09, 1.53it/s]
82%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 60/73 [00:39<00:08, 1.50it/s]
84%|████████████████████████████████████████████████████████████████████████████████████████████████ | 61/73 [00:40<00:08, 1.49it/s]
85%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 62/73 [00:40<00:07, 1.50it/s]
86%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 63/73 [00:41<00:06, 1.59it/s]
88%|████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 64/73 [00:41<00:05, 1.60it/s]
89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 65/73 [00:42<00:05, 1.59it/s]
90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 66/73 [00:43<00:04, 1.52it/s]
92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 67/73 [00:43<00:03, 1.55it/s]
93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 68/73 [00:44<00:03, 1.52it/s]
95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 69/73 [00:45<00:02, 1.48it/s]
96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 70/73 [00:45<00:02, 1.48it/s]
97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 71/73 [00:46<00:01, 1.48it/s]
99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 72/73 [00:47<00:00, 1.48it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
{'eval_loss': 0.49379390478134155, 'eval_runtime': 48.5073, 'eval_samples_per_second': 48.22, 'eval_steps_per_second': 1.526, 'eval_rewards/chosen': -1.5799260139465332, 'eval_rewards/rejected': -2.2533888816833496, 'eval_rewards/accuracies': 0.7791095972061157, 'eval_rewards/margins': 0.673462986946106, 'eval_logps/chosen': -222.89894104003906, 'eval_logps/rejected': -269.13983154296875, 'eval_logps/ref_chosen': -100.49356842041016, 'eval_logps/ref_rejected': -94.06775665283203, 'eval_logits/chosen': -7.477985382080078, 'eval_logits/rejected': -7.118831157684326, 'eval_kl/p_epsilon_steps': 0.7645547986030579, 'eval_kl/n_epsilon_steps': 0.23416095972061157, 'epoch': 0.59}
59%|██████████████████████████████████████████████████████████████████▎ | 400/681 [27:19<13:48, 2.95s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
[INFO|trainer.py:3984] 2026-04-24 04:43:40,465 >> Saving model checkpoint to /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-400
[INFO|configuration_utils.py:419] 2026-04-24 04:43:40,471 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-400/config.json
[INFO|configuration_utils.py:911] 2026-04-24 04:43:40,476 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-400/generation_config.json
[INFO|modeling_utils.py:3580] 2026-04-24 04:44:19,679 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-400/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2510] 2026-04-24 04:44:19,683 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-400/tokenizer_config.json
[INFO|tokenization_utils_base.py:2519] 2026-04-24 04:44:19,689 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-400/special_tokens_map.json
59%|█████████████████████████████████████████████████████████████████▎ | 401/681 [31:15<6:48:08, 87.46s/it] {'loss': 1.0429, 'grad_norm': 15.558944702148438, 'learning_rate': 2.1800473436235136e-07, 'rewards/chosen': -1.4610803127288818, 'rewards/rejected': -2.123840808868408, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.6627606749534607, 'logps/chosen': -197.28976440429688, 'logps/rejected': -255.7905731201172, 'logps/ref_chosen': -84.28916931152344, 'logps/ref_rejected': -90.943115234375, 'logits/chosen': -7.620119094848633, 'logits/rejected': -7.406888961791992, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.012954174540936947, 'kl/avg_steps': 0.375, 'epoch': 0.59}
59%|█████████████████████████████████████████████████████████████████▎ | 401/681 [31:15<6:48:08, 87.46s/it] 59%|█████████████████████████████████████████████████████████████████▌ | 402/681 [31:17<4:48:32, 62.05s/it] {'loss': 0.6486, 'grad_norm': 12.638237953186035, 'learning_rate': 2.1673238449588665e-07, 'rewards/chosen': -1.3018509149551392, 'rewards/rejected': -2.5145905017852783, 'rewards/accuracies': 0.90625, 'rewards/margins': 1.2127395868301392, 'logps/chosen': -185.138916015625, 'logps/rejected': -284.30706787109375, 'logps/ref_chosen': -83.59312438964844, 'logps/ref_rejected': -87.81027221679688, 'logits/chosen': -7.716352939605713, 'logits/rejected': -7.249260902404785, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.012905777432024479, 'kl/avg_steps': 0.78125, 'epoch': 0.59}
59%|█████████████████████████████████████████████████████████████████▌ | 402/681 [31:18<4:48:32, 62.05s/it] 59%|█████████████████████████████████████████████████████████████████▋ | 403/681 [31:21<3:25:34, 44.37s/it] {'loss': 0.8488, 'grad_norm': 11.707761764526367, 'learning_rate': 2.154609112620295e-07, 'rewards/chosen': -1.3442716598510742, 'rewards/rejected': -2.2409329414367676, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.8966612219810486, 'logps/chosen': -179.35401916503906, 'logps/rejected': -260.35247802734375, 'logps/ref_chosen': -73.75308227539062, 'logps/ref_rejected': -83.92012786865234, 'logits/chosen': -7.694974899291992, 'logits/rejected': -6.938326835632324, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.012805732898414135, 'kl/avg_steps': 0.71875, 'epoch': 0.59}
59%|█████████████████████████████████████████████████████████████████▋ | 403/681 [31:21<3:25:34, 44.37s/it] 59%|█████████████████████████████████████████████████████████████████▊ | 404/681 [31:24<2:27:45, 32.01s/it] {'loss': 0.9664, 'grad_norm': 17.042625427246094, 'learning_rate': 2.1419034816528218e-07, 'rewards/chosen': -1.567918300628662, 'rewards/rejected': -2.3647704124450684, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7968522906303406, 'logps/chosen': -203.4947509765625, 'logps/rejected': -271.59332275390625, 'logps/ref_chosen': -79.67617797851562, 'logps/ref_rejected': -84.280517578125, 'logits/chosen': -7.690559387207031, 'logits/rejected': -7.393105983734131, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.01271434873342514, 'kl/avg_steps': 0.59375, 'epoch': 0.59}
59%|█████████████████████████████████████████████████████████████████▊ | 404/681 [31:24<2:27:45, 32.01s/it] 59%|██████████████████████████████████████████████████████████████████ | 405/681 [31:27<1:47:06, 23.28s/it] {'loss': 0.8803, 'grad_norm': 11.477922439575195, 'learning_rate': 2.129207286861638e-07, 'rewards/chosen': -1.6104973554611206, 'rewards/rejected': -2.3875961303710938, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7770987749099731, 'logps/chosen': -224.55743408203125, 'logps/rejected': -283.11083984375, 'logps/ref_chosen': -96.46195220947266, 'logps/ref_rejected': -92.87071228027344, 'logits/chosen': -7.6790571212768555, 'logits/rejected': -7.126956462860107, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.012639302760362625, 'kl/avg_steps': 0.65625, 'epoch': 0.59}
59%|██████████████████████████████████████████████████████████████████ | 405/681 [31:27<1:47:06, 23.28s/it] 60%|██████████████████████████████████████████████████████████████████▏ | 406/681 [31:30<1:18:36, 17.15s/it] {'loss': 0.9891, 'grad_norm': 16.791955947875977, 'learning_rate': 2.1165208628032861e-07, 'rewards/chosen': -1.5131518840789795, 'rewards/rejected': -2.2241034507751465, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.7109516859054565, 'logps/chosen': -199.13026428222656, 'logps/rejected': -276.6565246582031, 'logps/ref_chosen': -78.13396453857422, 'logps/ref_rejected': -98.28359985351562, 'logits/chosen': -7.518543720245361, 'logits/rejected': -7.182583808898926, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.012556898407638073, 'kl/avg_steps': 0.59375, 'epoch': 0.6}
60%|██████████████████████████████████████████████████████████████████▏ | 406/681 [31:30<1:18:36, 17.15s/it] 60%|███████████████████████████████████████████████████████████████████▌ | 407/681 [31:33<59:11, 12.96s/it] {'loss': 0.9939, 'grad_norm': 14.285913467407227, 'learning_rate': 2.1038445437768375e-07, 'rewards/chosen': -1.5919959545135498, 'rewards/rejected': -2.248244285583496, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.6562482118606567, 'logps/chosen': -212.05181884765625, 'logps/rejected': -264.0316162109375, 'logps/ref_chosen': -84.01283264160156, 'logps/ref_rejected': -82.78103637695312, 'logits/chosen': -7.507756233215332, 'logits/rejected': -6.921146392822266, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.012482781894505024, 'kl/avg_steps': 0.5625, 'epoch': 0.6}
60%|███████████████████████████████████████████████████████████████████▌ | 407/681 [31:33<59:11, 12.96s/it] 60%|███████████████████████████████████████████████████████████████████▋ | 408/681 [31:36<45:32, 10.01s/it] {'loss': 1.011, 'grad_norm': 13.731730461120605, 'learning_rate': 2.0911786638150872e-07, 'rewards/chosen': -1.480905294418335, 'rewards/rejected': -2.146219253540039, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.6653140783309937, 'logps/chosen': -224.00033569335938, 'logps/rejected': -270.20428466796875, 'logps/ref_chosen': -104.46175384521484, 'logps/ref_rejected': -96.37218475341797, 'logits/chosen': -7.530098915100098, 'logits/rejected': -7.0652008056640625, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.012412958778440952, 'kl/avg_steps': 0.421875, 'epoch': 0.6}
60%|███████████████████████████████████████████████████████████████████▋ | 408/681 [31:36<45:32, 10.01s/it] 60%|███████████████████████████████████████████████████████████████████▊ | 409/681 [31:39<36:12, 7.99s/it] {'loss': 1.0303, 'grad_norm': 12.978145599365234, 'learning_rate': 2.0785235566757517e-07, 'rewards/chosen': -1.6262993812561035, 'rewards/rejected': -2.277886152267456, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6515868306159973, 'logps/chosen': -229.72506713867188, 'logps/rejected': -275.54852294921875, 'logps/ref_chosen': -97.66830444335938, 'logps/ref_rejected': -90.04584503173828, 'logits/chosen': -7.490418434143066, 'logits/rejected': -7.061441898345947, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.012360811233520508, 'kl/avg_steps': 0.5625, 'epoch': 0.6}
60%|███████████████████████████████████████████████████████████████████▊ | 409/681 [31:39<36:12, 7.99s/it] 60%|████████████████████████████████████████████████████████████████████ | 410/681 [31:42<29:37, 6.56s/it] {'loss': 0.9507, 'grad_norm': 12.724774360656738, 'learning_rate': 2.065879555832674e-07, 'rewards/chosen': -1.5810291767120361, 'rewards/rejected': -2.278393268585205, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.697364091873169, 'logps/chosen': -205.697265625, 'logps/rejected': -275.3063049316406, 'logps/ref_chosen': -76.46923828125, 'logps/ref_rejected': -88.64064025878906, 'logits/chosen': -7.461033821105957, 'logits/rejected': -7.218528747558594, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.01229167077690363, 'kl/avg_steps': 0.578125, 'epoch': 0.6}
60%|████████████████████████████████████████████████████████████████████ | 410/681 [31:42<29:37, 6.56s/it] 60%|████████████████████████████████████████████████████████████████████▏ | 411/681 [31:45<24:16, 5.39s/it] {'loss': 0.9201, 'grad_norm': 11.978557586669922, 'learning_rate': 2.0532469944670343e-07, 'rewards/chosen': -1.6353148221969604, 'rewards/rejected': -2.3577518463134766, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7224369049072266, 'logps/chosen': -221.50474548339844, 'logps/rejected': -281.2148742675781, 'logps/ref_chosen': -87.16630554199219, 'logps/ref_rejected': -87.09603118896484, 'logits/chosen': -7.684117317199707, 'logits/rejected': -7.11949348449707, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.012221017852425575, 'kl/avg_steps': 0.53125, 'epoch': 0.6}
60%|████████████████████████████████████████████████████████████████████▏ | 411/681 [31:45<24:16, 5.39s/it] 60%|████████████████████████████████████████████████████████████████████▎ | 412/681 [31:48<21:05, 4.70s/it] {'loss': 1.0053, 'grad_norm': 13.545778274536133, 'learning_rate': 2.0406262054585738e-07, 'rewards/chosen': -1.5305845737457275, 'rewards/rejected': -2.215937614440918, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.6853532791137695, 'logps/chosen': -205.3089599609375, 'logps/rejected': -289.611572265625, 'logps/ref_chosen': -78.94734191894531, 'logps/ref_rejected': -106.10554504394531, 'logits/chosen': -7.71500301361084, 'logits/rejected': -6.960906982421875, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.012156437151134014, 'kl/avg_steps': 0.46875, 'epoch': 0.6}
60%|████████████████████████████████████████████████████████████████████▎ | 412/681 [31:48<21:05, 4.70s/it] 61%|████████████████████████████████████████████████████████████████████▌ | 413/681 [31:51<18:46, 4.20s/it] {'loss': 0.9401, 'grad_norm': 12.726061820983887, 'learning_rate': 2.0280175213768205e-07, 'rewards/chosen': -1.650810956954956, 'rewards/rejected': -2.4081544876098633, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7573432922363281, 'logps/chosen': -232.97862243652344, 'logps/rejected': -308.6137390136719, 'logps/ref_chosen': -95.69471740722656, 'logps/ref_rejected': -107.96085357666016, 'logits/chosen': -7.195650100708008, 'logits/rejected': -7.105502128601074, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.012099719606339931, 'kl/avg_steps': 0.71875, 'epoch': 0.61}
61%|████████████████████████████████████████████████████████████████████▌ | 413/681 [31:51<18:46, 4.20s/it] 61%|████████████████████████████████████████████████████████████████████▋ | 414/681 [31:54<17:14, 3.87s/it] {'loss': 0.9978, 'grad_norm': 16.81880760192871, 'learning_rate': 2.0154212744723247e-07, 'rewards/chosen': -1.5135799646377563, 'rewards/rejected': -2.220407009124756, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.7068269848823547, 'logps/chosen': -214.6531524658203, 'logps/rejected': -278.8741455078125, 'logps/ref_chosen': -88.27667236328125, 'logps/ref_rejected': -92.87004089355469, 'logits/chosen': -7.222269058227539, 'logits/rejected': -6.936489105224609, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.012013372965157032, 'kl/avg_steps': 0.53125, 'epoch': 0.61}
61%|████████████████████████████████████████████████████████████████████▋ | 414/681 [31:54<17:14, 3.87s/it] 61%|████████████████████████████████████████████████████████████████████▊ | 415/681 [31:58<16:39, 3.76s/it] {'loss': 1.0108, 'grad_norm': 13.74764633178711, 'learning_rate': 2.002837796667909e-07, 'rewards/chosen': -1.5901520252227783, 'rewards/rejected': -2.227059841156006, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.6369077563285828, 'logps/chosen': -242.48207092285156, 'logps/rejected': -295.0639953613281, 'logps/ref_chosen': -108.91590118408203, 'logps/ref_rejected': -107.47135925292969, 'logits/chosen': -7.991650104522705, 'logits/rejected': -7.442322731018066, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.011949889361858368, 'kl/avg_steps': 0.5625, 'epoch': 0.61}
61%|████████████████████████████████████████████████████████████████████▊ | 415/681 [31:58<16:39, 3.76s/it] 61%|█████████████████████████████████████████████████████████████████████ | 416/681 [32:01<15:41, 3.55s/it] {'loss': 0.875, 'grad_norm': 11.695809364318848, 'learning_rate': 1.990267419549914e-07, 'rewards/chosen': -1.556352138519287, 'rewards/rejected': -2.4217967987060547, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.8654446601867676, 'logps/chosen': -224.93905639648438, 'logps/rejected': -302.91412353515625, 'logps/ref_chosen': -93.39888000488281, 'logps/ref_rejected': -97.6729736328125, 'logits/chosen': -7.476058006286621, 'logits/rejected': -7.21858024597168, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.01188304740935564, 'kl/avg_steps': 0.5625, 'epoch': 0.61}
61%|█████████████████████████████████████████████████████████████████████ | 416/681 [32:01<15:41, 3.55s/it] 61%|█████████████████████████████████████████████████████████████████████▏ | 417/681 [32:04<14:43, 3.34s/it] {'loss': 0.8896, 'grad_norm': 12.070055961608887, 'learning_rate': 1.9777104743594686e-07, 'rewards/chosen': -1.4651763439178467, 'rewards/rejected': -2.2487990856170654, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7836226224899292, 'logps/chosen': -208.16903686523438, 'logps/rejected': -266.1756591796875, 'logps/ref_chosen': -83.53533172607422, 'logps/ref_rejected': -74.44184112548828, 'logits/chosen': -7.844758033752441, 'logits/rejected': -6.70277214050293, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.011816578917205334, 'kl/avg_steps': 0.6875, 'epoch': 0.61}
61%|█████████████████████████████████████████████████████████████████████▏ | 417/681 [32:04<14:43, 3.34s/it] 61%|█████████████████████████████████████████████████████████████████████▎ | 418/681 [32:07<14:28, 3.30s/it] {'loss': 0.7851, 'grad_norm': 13.669163703918457, 'learning_rate': 1.965167291983757e-07, 'rewards/chosen': -1.3652478456497192, 'rewards/rejected': -2.3257837295532227, 'rewards/accuracies': 0.921875, 'rewards/margins': 0.9605357646942139, 'logps/chosen': -225.50465393066406, 'logps/rejected': -311.86407470703125, 'logps/ref_chosen': -108.22152709960938, 'logps/ref_rejected': -111.8646469116211, 'logits/chosen': -7.78868293762207, 'logits/rejected': -7.488645553588867, 'kl/p_epsilon_steps': 0.921875, 'kl/n_epsilon_steps': 0.078125, 'kl/beta': 0.011735894717276096, 'kl/avg_steps': 0.84375, 'epoch': 0.61}
61%|█████████████████████████████████████████████████████████████████████▎ | 418/681 [32:07<14:28, 3.30s/it] 62%|█████████████████████████████████████████████████████████████████████▌ | 419/681 [32:10<14:24, 3.30s/it] {'loss': 0.8759, 'grad_norm': 11.460182189941406, 'learning_rate': 1.9526382029472988e-07, 'rewards/chosen': -1.4567546844482422, 'rewards/rejected': -2.2854669094085693, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8287121057510376, 'logps/chosen': -223.0901641845703, 'logps/rejected': -296.13623046875, 'logps/ref_chosen': -97.18328094482422, 'logps/ref_rejected': -98.18531799316406, 'logits/chosen': -7.678118705749512, 'logits/rejected': -7.43233585357666, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.011637701652944088, 'kl/avg_steps': 0.71875, 'epoch': 0.62}
62%|█████████████████████████████████████████████████████████████████████▌ | 419/681 [32:10<14:24, 3.30s/it] 62%|█████████████████████████████████████████████████████████████████████▋ | 420/681 [32:13<13:59, 3.22s/it] {'loss': 0.9212, 'grad_norm': 14.639280319213867, 'learning_rate': 1.9401235374032425e-07, 'rewards/chosen': -1.4852039813995361, 'rewards/rejected': -2.24064040184021, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.7554365396499634, 'logps/chosen': -243.3070831298828, 'logps/rejected': -270.84930419921875, 'logps/ref_chosen': -114.30847930908203, 'logps/ref_rejected': -75.68356323242188, 'logits/chosen': -7.911190509796143, 'logits/rejected': -6.9983086585998535, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.011554652824997902, 'kl/avg_steps': 0.5625, 'epoch': 0.62}
62%|█████████████████████████████████████████████████████████████████████▋ | 420/681 [32:13<13:59, 3.22s/it] 62%|█████████████████████████████████████████████████████████████████████▊ | 421/681 [32:16<13:54, 3.21s/it] {'loss': 0.8958, 'grad_norm': 11.352572441101074, 'learning_rate': 1.9276236251246653e-07, 'rewards/chosen': -1.4818620681762695, 'rewards/rejected': -2.2694497108459473, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7875877618789673, 'logps/chosen': -215.33929443359375, 'logps/rejected': -295.12469482421875, 'logps/ref_chosen': -85.87985229492188, 'logps/ref_rejected': -96.33648681640625, 'logits/chosen': -7.578913688659668, 'logits/rejected': -6.917729377746582, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.011490020900964737, 'kl/avg_steps': 0.5625, 'epoch': 0.62}
62%|█████████████████████████████████████████████████████████████████████▊ | 421/681 [32:16<13:54, 3.21s/it] 62%|██████████████████████████████████████████████████████████████████████ | 422/681 [32:20<13:48, 3.20s/it] {'loss': 0.8898, 'grad_norm': 11.291938781738281, 'learning_rate': 1.9151387954958792e-07, 'rewards/chosen': -1.409227728843689, 'rewards/rejected': -2.1822290420532227, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7730013132095337, 'logps/chosen': -224.50631713867188, 'logps/rejected': -286.848388671875, 'logps/ref_chosen': -100.48060607910156, 'logps/ref_rejected': -94.40821838378906, 'logits/chosen': -7.927703857421875, 'logits/rejected': -7.495624542236328, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.011425751261413097, 'kl/avg_steps': 0.6875, 'epoch': 0.62}
62%|██████████████████████████████████████████████████████████████████████ | 422/681 [32:20<13:48, 3.20s/it] 62%|██████████████████████████████████████████████████████████████████████▏ | 423/681 [32:22<13:19, 3.10s/it] {'loss': 0.9664, 'grad_norm': 12.197092056274414, 'learning_rate': 1.902669377503756e-07, 'rewards/chosen': -1.364768624305725, 'rewards/rejected': -2.0834240913391113, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.718655526638031, 'logps/chosen': -198.95823669433594, 'logps/rejected': -276.6893310546875, 'logps/ref_chosen': -78.44993591308594, 'logps/ref_rejected': -92.04652404785156, 'logits/chosen': -7.66268253326416, 'logits/rejected': -7.711108207702637, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.011347736231982708, 'kl/avg_steps': 0.4375, 'epoch': 0.62}
62%|██████████████████████████████████████████████████████████████████████▏ | 423/681 [32:22<13:19, 3.10s/it] 62%|██████████████████████████████████████████████████████████████████████▎ | 424/681 [32:26<13:23, 3.13s/it] {'loss': 0.9078, 'grad_norm': 11.235798835754395, 'learning_rate': 1.890215699729057e-07, 'rewards/chosen': -1.4883897304534912, 'rewards/rejected': -2.245741367340088, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.7573517560958862, 'logps/chosen': -220.01161193847656, 'logps/rejected': -272.5384521484375, 'logps/ref_chosen': -87.6423568725586, 'logps/ref_rejected': -72.36566162109375, 'logits/chosen': -7.801157474517822, 'logits/rejected': -6.635551452636719, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.011298305355012417, 'kl/avg_steps': 0.59375, 'epoch': 0.62}
62%|██████████████████████████████████████████████████████████████████████▎ | 424/681 [32:26<13:23, 3.13s/it] 62%|██████████████████████████████████████████████████████████████████████▌ | 425/681 [32:29<13:03, 3.06s/it] {'loss': 1.1393, 'grad_norm': 13.440584182739258, 'learning_rate': 1.8777780903377732e-07, 'rewards/chosen': -1.5395094156265259, 'rewards/rejected': -2.0465922355651855, 'rewards/accuracies': 0.625, 'rewards/margins': 0.5070829391479492, 'logps/chosen': -215.4922332763672, 'logps/rejected': -285.57012939453125, 'logps/ref_chosen': -78.51979064941406, 'logps/ref_rejected': -102.74864196777344, 'logits/chosen': -7.429314613342285, 'logits/rejected': -7.256626605987549, 'kl/p_epsilon_steps': 0.59375, 'kl/n_epsilon_steps': 0.40625, 'kl/beta': 0.011231618002057076, 'kl/avg_steps': 0.1875, 'epoch': 0.62}
62%|██████████████████████████████████████████████████████████████████████▌ | 425/681 [32:29<13:03, 3.06s/it] 63%|██████████████████████████████████████████████████████████████████████▋ | 426/681 [32:32<13:20, 3.14s/it] {'loss': 0.8943, 'grad_norm': 13.104548454284668, 'learning_rate': 1.8653568770724803e-07, 'rewards/chosen': -1.3247995376586914, 'rewards/rejected': -2.0728135108947754, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7480140924453735, 'logps/chosen': -227.1387939453125, 'logps/rejected': -274.4301452636719, 'logps/ref_chosen': -108.50582885742188, 'logps/ref_rejected': -88.300048828125, 'logits/chosen': -7.60335636138916, 'logits/rejected': -7.2415266036987305, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.011210598051548004, 'kl/avg_steps': 0.5625, 'epoch': 0.63}
63%|██████████████████████████████████████████████████████████████████████▋ | 426/681 [32:32<13:20, 3.14s/it] 63%|██████████████████████████████████████████████████████████████████████▊ | 427/681 [32:35<13:38, 3.22s/it] {'loss': 0.9039, 'grad_norm': 10.724087715148926, 'learning_rate': 1.8529523872436977e-07, 'rewards/chosen': -1.119832158088684, 'rewards/rejected': -1.820410966873169, 'rewards/accuracies': 0.875, 'rewards/margins': 0.7005788087844849, 'logps/chosen': -200.22837829589844, 'logps/rejected': -250.37628173828125, 'logps/ref_chosen': -99.12046813964844, 'logps/ref_rejected': -85.724609375, 'logits/chosen': -7.953921794891357, 'logits/rejected': -7.36940860748291, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.01114789117127657, 'kl/avg_steps': 0.75, 'epoch': 0.63}
63%|██████████████████████████████████████████████████████████████████████▊ | 427/681 [32:35<13:38, 3.22s/it] 63%|███████████████████████████████████████████████████████████████████████ | 428/681 [32:38<13:37, 3.23s/it] {'loss': 1.0721, 'grad_norm': 12.104276657104492, 'learning_rate': 1.8405649477212697e-07, 'rewards/chosen': -1.520464301109314, 'rewards/rejected': -2.0730321407318115, 'rewards/accuracies': 0.75, 'rewards/margins': 0.5525679588317871, 'logps/chosen': -243.97320556640625, 'logps/rejected': -297.7687683105469, 'logps/ref_chosen': -105.96925354003906, 'logps/ref_rejected': -109.1021728515625, 'logits/chosen': -7.528921604156494, 'logits/rejected': -7.260353088378906, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.011064904741942883, 'kl/avg_steps': 0.5625, 'epoch': 0.63}
63%|███████████████████████████████████████████████████████████████████████ | 428/681 [32:39<13:37, 3.23s/it] 63%|███████████████████████████████████████████████████████████████████████▏ | 429/681 [32:42<13:31, 3.22s/it] {'loss': 0.9249, 'grad_norm': 13.0979642868042, 'learning_rate': 1.828194884925749e-07, 'rewards/chosen': -1.363885760307312, 'rewards/rejected': -2.127699851989746, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7638142108917236, 'logps/chosen': -237.86270141601562, 'logps/rejected': -292.80511474609375, 'logps/ref_chosen': -113.54486846923828, 'logps/ref_rejected': -98.24201965332031, 'logits/chosen': -7.794681072235107, 'logits/rejected': -7.093453407287598, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.011003012768924236, 'kl/avg_steps': 0.53125, 'epoch': 0.63}
63%|███████████████████████████████████████████████████████████████████████▏ | 429/681 [32:42<13:31, 3.22s/it] 63%|███████████████████████████████████████████████████████████████████████▎ | 430/681 [32:45<13:34, 3.24s/it] {'loss': 1.0248, 'grad_norm': 12.785799980163574, 'learning_rate': 1.8158425248197928e-07, 'rewards/chosen': -1.361114501953125, 'rewards/rejected': -1.9355595111846924, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.5744451284408569, 'logps/chosen': -216.0718994140625, 'logps/rejected': -288.0528564453125, 'logps/ref_chosen': -91.31936645507812, 'logps/ref_rejected': -110.1096420288086, 'logits/chosen': -7.379518032073975, 'logits/rejected': -7.428651809692383, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.010944867506623268, 'kl/avg_steps': 0.4375, 'epoch': 0.63}
63%|███████████████████████████████████████████████████████████████████████▎ | 430/681 [32:45<13:34, 3.24s/it] 63%|███████████████████████████████████████████████████████████████████████▌ | 431/681 [32:48<13:23, 3.22s/it] {'loss': 0.9558, 'grad_norm': 11.655879974365234, 'learning_rate': 1.8035081928995788e-07, 'rewards/chosen': -1.3294377326965332, 'rewards/rejected': -2.0129597187042236, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.6835219860076904, 'logps/chosen': -215.51998901367188, 'logps/rejected': -283.9945983886719, 'logps/ref_chosen': -93.18122100830078, 'logps/ref_rejected': -98.13226318359375, 'logits/chosen': -7.7428998947143555, 'logits/rejected': -7.222956657409668, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.010897193104028702, 'kl/avg_steps': 0.5, 'epoch': 0.63}
63%|███████████████████████████████████████████████████████████████████████▌ | 431/681 [32:48<13:23, 3.22s/it] 63%|███████████████████████████████████████████████████████████████████████▋ | 432/681 [32:52<13:32, 3.26s/it] {'loss': 0.8684, 'grad_norm': 10.997950553894043, 'learning_rate': 1.791192214186223e-07, 'rewards/chosen': -1.2002313137054443, 'rewards/rejected': -2.012035846710205, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.8118045330047607, 'logps/chosen': -215.7032928466797, 'logps/rejected': -292.071533203125, 'logps/ref_chosen': -104.43478393554688, 'logps/ref_rejected': -105.08955383300781, 'logits/chosen': -7.891312122344971, 'logits/rejected': -7.409452438354492, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.010842978022992611, 'kl/avg_steps': 0.625, 'epoch': 0.63}
63%|███████████████████████████████████████████████████████████████████████▋ | 432/681 [32:52<13:32, 3.26s/it] 64%|███████████████████████████████████████████████████████████████████████▊ | 433/681 [32:55<13:17, 3.22s/it] {'loss': 1.001, 'grad_norm': 11.87765884399414, 'learning_rate': 1.7788949132172193e-07, 'rewards/chosen': -1.4064010381698608, 'rewards/rejected': -2.044281482696533, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.6378804445266724, 'logps/chosen': -220.77711486816406, 'logps/rejected': -292.6389465332031, 'logps/ref_chosen': -89.84322357177734, 'logps/ref_rejected': -101.73345947265625, 'logits/chosen': -7.532526969909668, 'logits/rejected': -7.0036725997924805, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.010775630362331867, 'kl/avg_steps': 0.46875, 'epoch': 0.64}
64%|███████████████████████████████████████████████████████████████████████▊ | 433/681 [32:55<13:17, 3.22s/it] 64%|████████████████████████████████████████████████████████████████████████ | 434/681 [32:58<13:11, 3.20s/it] {'loss': 0.984, 'grad_norm': 11.362030982971191, 'learning_rate': 1.7666166140378853e-07, 'rewards/chosen': -1.3009238243103027, 'rewards/rejected': -1.9657368659973145, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.6648129820823669, 'logps/chosen': -219.51129150390625, 'logps/rejected': -268.685302734375, 'logps/ref_chosen': -97.6925277709961, 'logps/ref_rejected': -84.09130096435547, 'logits/chosen': -7.701730728149414, 'logits/rejected': -7.350008010864258, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.0107253547757864, 'kl/avg_steps': 0.5, 'epoch': 0.64}
64%|████████████████████████████████████████████████████████████████████████ | 434/681 [32:58<13:11, 3.20s/it] 64%|████████████████████████████████████████████████████████████████████████▏ | 435/681 [33:01<12:40, 3.09s/it] {'loss': 0.9314, 'grad_norm': 11.840472221374512, 'learning_rate': 1.7543576401928218e-07, 'rewards/chosen': -1.2436943054199219, 'rewards/rejected': -1.9595654010772705, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7158711552619934, 'logps/chosen': -203.26406860351562, 'logps/rejected': -278.71820068359375, 'logps/ref_chosen': -86.17192077636719, 'logps/ref_rejected': -93.751708984375, 'logits/chosen': -7.7169880867004395, 'logits/rejected': -7.244067192077637, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.010671994648873806, 'kl/avg_steps': 0.65625, 'epoch': 0.64}
64%|████████████████████████████████████████████████████████████████████████▏ | 435/681 [33:01<12:40, 3.09s/it] 64%|████████████████████████████████████████████████████████████████████████▎ | 436/681 [33:04<12:43, 3.12s/it] {'loss': 0.8215, 'grad_norm': 11.141986846923828, 'learning_rate': 1.742118314717391e-07, 'rewards/chosen': -1.0024278163909912, 'rewards/rejected': -1.8351895809173584, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.8327617645263672, 'logps/chosen': -200.92079162597656, 'logps/rejected': -263.154296875, 'logps/ref_chosen': -105.78710174560547, 'logps/ref_rejected': -88.62471008300781, 'logits/chosen': -8.073395729064941, 'logits/rejected': -6.879586219787598, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.010602416470646858, 'kl/avg_steps': 0.78125, 'epoch': 0.64}
64%|████████████████████████████████████████████████████████████████████████▎ | 436/681 [33:04<12:43, 3.12s/it] 64%|████████████████████████████████████████████████████████████████████████▌ | 437/681 [33:07<12:52, 3.17s/it] {'loss': 0.9969, 'grad_norm': 12.68950366973877, 'learning_rate': 1.7298989601292036e-07, 'rewards/chosen': -1.2023859024047852, 'rewards/rejected': -1.770288109779358, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.5679023265838623, 'logps/chosen': -210.84364318847656, 'logps/rejected': -258.4656677246094, 'logps/ref_chosen': -96.06204223632812, 'logps/ref_rejected': -89.01220703125, 'logits/chosen': -7.771686553955078, 'logits/rejected': -7.039131164550781, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.010520227253437042, 'kl/avg_steps': 0.53125, 'epoch': 0.64}
64%|████████████████████████████████████████████████████████████████████████▌ | 437/681 [33:07<12:52, 3.17s/it] 64%|████████████████████████████████████████████████████████████████████████▋ | 438/681 [33:10<12:37, 3.12s/it] {'loss': 0.949, 'grad_norm': 11.616890907287598, 'learning_rate': 1.7176998984196144e-07, 'rewards/chosen': -1.149721622467041, 'rewards/rejected': -1.8044536113739014, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.6547319889068604, 'logps/chosen': -212.05523681640625, 'logps/rejected': -262.99102783203125, 'logps/ref_chosen': -101.85537719726562, 'logps/ref_rejected': -89.4476547241211, 'logits/chosen': -7.707596302032471, 'logits/rejected': -7.402861595153809, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.01046463381499052, 'kl/avg_steps': 0.53125, 'epoch': 0.64}
64%|████████████████████████████████████████████████████████████████████████▋ | 438/681 [33:10<12:37, 3.12s/it] 64%|████████████████████████████████████████████████████████████████████████▊ | 439/681 [33:13<12:19, 3.06s/it] {'loss': 0.9918, 'grad_norm': 12.508461952209473, 'learning_rate': 1.7055214510452458e-07, 'rewards/chosen': -1.2794837951660156, 'rewards/rejected': -1.8945040702819824, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6150201559066772, 'logps/chosen': -205.33200073242188, 'logps/rejected': -273.9890441894531, 'logps/ref_chosen': -81.75563049316406, 'logps/ref_rejected': -90.58635711669922, 'logits/chosen': -7.787407875061035, 'logits/rejected': -7.194408416748047, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.010409334674477577, 'kl/avg_steps': 0.625, 'epoch': 0.64}
64%|████████████████████████████████████████████████████████████████████████▊ | 439/681 [33:13<12:19, 3.06s/it] 65%|█████████████████████████████████████████████████████████████████████████ | 440/681 [33:16<12:05, 3.01s/it] {'loss': 0.923, 'grad_norm': 10.161545753479004, 'learning_rate': 1.6933639389195134e-07, 'rewards/chosen': -1.1412960290908813, 'rewards/rejected': -1.8606973886489868, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.7194013595581055, 'logps/chosen': -216.48208618164062, 'logps/rejected': -284.5960388183594, 'logps/ref_chosen': -105.64108276367188, 'logps/ref_rejected': -103.40100860595703, 'logits/chosen': -7.840778827667236, 'logits/rejected': -7.435771465301514, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.010344680398702621, 'kl/avg_steps': 0.625, 'epoch': 0.65}
65%|█████████████████████████████████████████████████████████████████████████ | 440/681 [33:16<12:05, 3.01s/it] 65%|█████████████████████████████████████████████████████████████████████████▏ | 441/681 [33:19<12:23, 3.10s/it] {'loss': 0.8764, 'grad_norm': 10.330644607543945, 'learning_rate': 1.681227682404166e-07, 'rewards/chosen': -1.3216805458068848, 'rewards/rejected': -2.063693046569824, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.7420125007629395, 'logps/chosen': -221.08547973632812, 'logps/rejected': -306.0940856933594, 'logps/ref_chosen': -91.529541015625, 'logps/ref_rejected': -103.619384765625, 'logits/chosen': -7.877140522003174, 'logits/rejected': -6.723335266113281, 'kl/p_epsilon_steps': 0.921875, 'kl/n_epsilon_steps': 0.078125, 'kl/beta': 0.010280427522957325, 'kl/avg_steps': 0.84375, 'epoch': 0.65}
65%|█████████████████████████████████████████████████████████████████████████▏ | 441/681 [33:19<12:23, 3.10s/it] 65%|█████████████████████████████████████████████████████████████████████████▎ | 442/681 [33:22<12:12, 3.07s/it] {'loss': 0.8427, 'grad_norm': 11.554139137268066, 'learning_rate': 1.669113001300851e-07, 'rewards/chosen': -1.3306344747543335, 'rewards/rejected': -2.1431615352630615, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.8125271797180176, 'logps/chosen': -216.06195068359375, 'logps/rejected': -295.6746826171875, 'logps/ref_chosen': -84.77755737304688, 'logps/ref_rejected': -83.82415008544922, 'logits/chosen': -7.492166996002197, 'logits/rejected': -6.9776811599731445, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.010194412432610989, 'kl/avg_steps': 0.6875, 'epoch': 0.65}
65%|█████████████████████████████████████████████████████████████████████████▎ | 442/681 [33:22<12:12, 3.07s/it] 65%|█████████████████████████████████████████████████████████████████████████▌ | 443/681 [33:26<12:27, 3.14s/it] {'loss': 0.9816, 'grad_norm': 11.208428382873535, 'learning_rate': 1.6570202148426815e-07, 'rewards/chosen': -1.2582169771194458, 'rewards/rejected': -1.8541338443756104, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.5959169864654541, 'logps/chosen': -227.578125, 'logps/rejected': -277.5597229003906, 'logps/ref_chosen': -102.64927673339844, 'logps/ref_rejected': -93.03807067871094, 'logits/chosen': -7.71527099609375, 'logits/rejected': -7.359766960144043, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.010124804452061653, 'kl/avg_steps': 0.625, 'epoch': 0.65}
65%|█████████████████████████████████████████████████████████████████████████▌ | 443/681 [33:26<12:27, 3.14s/it] 65%|█████████████████████████████████████████████████████████████████████████▋ | 444/681 [33:29<12:14, 3.10s/it] {'loss': 1.0177, 'grad_norm': 12.04550552368164, 'learning_rate': 1.6449496416858282e-07, 'rewards/chosen': -1.34379243850708, 'rewards/rejected': -1.9272189140319824, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.5834265947341919, 'logps/chosen': -221.87020874023438, 'logps/rejected': -295.9997253417969, 'logps/ref_chosen': -87.91971588134766, 'logps/ref_rejected': -103.32345581054688, 'logits/chosen': -7.471601963043213, 'logits/rejected': -7.507413387298584, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.010061916895210743, 'kl/avg_steps': 0.4375, 'epoch': 0.65}
65%|█████████████████████████████████████████████████████████████████████████▋ | 444/681 [33:29<12:14, 3.10s/it] 65%|█████████████████████████████████████████████████████████████████████████▊ | 445/681 [33:32<12:10, 3.10s/it] {'loss': 0.925, 'grad_norm': 10.32401180267334, 'learning_rate': 1.6329015999011182e-07, 'rewards/chosen': -1.1529053449630737, 'rewards/rejected': -1.8247992992401123, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6718940138816833, 'logps/chosen': -216.97213745117188, 'logps/rejected': -282.439208984375, 'logps/ref_chosen': -101.40087127685547, 'logps/ref_rejected': -99.03790283203125, 'logits/chosen': -7.7422380447387695, 'logits/rejected': -7.081811904907227, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.010018087923526764, 'kl/avg_steps': 0.59375, 'epoch': 0.65}
65%|█████████████████████████████████████████████████████████████████████████▊ | 445/681 [33:32<12:10, 3.10s/it] 65%|██████████████████████████████████████████████████████████████████████████ | 446/681 [33:35<12:21, 3.16s/it] {'loss': 0.9615, 'grad_norm': 12.130196571350098, 'learning_rate': 1.6208764069656578e-07, 'rewards/chosen': -1.1098341941833496, 'rewards/rejected': -1.7301799058914185, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.6203456521034241, 'logps/chosen': -199.480224609375, 'logps/rejected': -281.7625732421875, 'logps/ref_chosen': -87.42234802246094, 'logps/ref_rejected': -106.70075988769531, 'logits/chosen': -7.809151649475098, 'logits/rejected': -7.551025390625, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.009958956390619278, 'kl/avg_steps': 0.5625, 'epoch': 0.65}
65%|██████████████████████████████████████████████████████████████████████████ | 446/681 [33:35<12:21, 3.16s/it] 66%|██████████████████████████████████████████████████████████████████████████▏ | 447/681 [33:38<11:54, 3.06s/it] {'loss': 0.9079, 'grad_norm': 11.679533958435059, 'learning_rate': 1.608874379754465e-07, 'rewards/chosen': -1.1265549659729004, 'rewards/rejected': -1.8664308786392212, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.7398759126663208, 'logps/chosen': -197.861083984375, 'logps/rejected': -294.75396728515625, 'logps/ref_chosen': -83.6152572631836, 'logps/ref_rejected': -104.91239929199219, 'logits/chosen': -7.548259735107422, 'logits/rejected': -7.2763800621032715, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.009903251193463802, 'kl/avg_steps': 0.65625, 'epoch': 0.66}
66%|██████████████████████████████████████████████████████████████████████████▏ | 447/681 [33:38<11:54, 3.06s/it] 66%|██████████████████████████████████████████████████████████████████████████▎ | 448/681 [33:41<11:53, 3.06s/it] {'loss': 0.9455, 'grad_norm': 11.890493392944336, 'learning_rate': 1.5968958345321177e-07, 'rewards/chosen': -1.1929302215576172, 'rewards/rejected': -1.929168939590454, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.7362387180328369, 'logps/chosen': -214.25633239746094, 'logps/rejected': -305.142578125, 'logps/ref_chosen': -92.5757827758789, 'logps/ref_rejected': -107.68977355957031, 'logits/chosen': -7.886592864990234, 'logits/rejected': -7.5247802734375, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.00983868446201086, 'kl/avg_steps': 0.5625, 'epoch': 0.66}
66%|██████████████████████████████████████████████████████████████████████████▎ | 448/681 [33:41<11:53, 3.06s/it] 66%|██████████████████████████████████████████████████████████████████████████▌ | 449/681 [33:44<12:06, 3.13s/it] {'loss': 0.9078, 'grad_norm': 10.460795402526855, 'learning_rate': 1.584941086944423e-07, 'rewards/chosen': -1.1828500032424927, 'rewards/rejected': -1.9290869235992432, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.7462369799613953, 'logps/chosen': -223.71267700195312, 'logps/rejected': -293.65252685546875, 'logps/ref_chosen': -102.39893341064453, 'logps/ref_rejected': -95.14886474609375, 'logits/chosen': -7.6187238693237305, 'logits/rejected': -7.113336086273193, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.009783651679754257, 'kl/avg_steps': 0.53125, 'epoch': 0.66}
66%|██████████████████████████████████████████████████████████████████████████▌ | 449/681 [33:44<12:06, 3.13s/it] 66%|██████████████████████████████████████████████████████████████████████████▋ | 450/681 [33:47<12:07, 3.15s/it] {'loss': 0.9059, 'grad_norm': 11.509246826171875, 'learning_rate': 1.573010452010098e-07, 'rewards/chosen': -1.0168712139129639, 'rewards/rejected': -1.7526460886001587, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.7357749938964844, 'logps/chosen': -191.9837646484375, 'logps/rejected': -290.04888916015625, 'logps/ref_chosen': -86.99285888671875, 'logps/ref_rejected': -108.53203582763672, 'logits/chosen': -8.000127792358398, 'logits/rejected': -7.75508451461792, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.009731950238347054, 'kl/avg_steps': 0.71875, 'epoch': 0.66}
66%|██████████████████████████████████████████████████████████████████████████▋ | 450/681 [33:47<12:07, 3.15s/it] 66%|██████████████████████████████████████████████████████████████████████████▊ | 451/681 [33:50<11:32, 3.01s/it] {'loss': 1.0328, 'grad_norm': 16.418062210083008, 'learning_rate': 1.5611042441124687e-07, 'rewards/chosen': -1.157651424407959, 'rewards/rejected': -1.7588902711868286, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.6012389659881592, 'logps/chosen': -206.88548278808594, 'logps/rejected': -262.9779357910156, 'logps/ref_chosen': -86.81128692626953, 'logps/ref_rejected': -79.8555908203125, 'logits/chosen': -7.632309913635254, 'logits/rejected': -7.299648761749268, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.009662501513957977, 'kl/avg_steps': 0.46875, 'epoch': 0.66}
66%|██████████████████████████████████████████████████████████████████████████▊ | 451/681 [33:50<11:32, 3.01s/it] 66%|███████████████████████████████████████████████████████████████████████████ | 452/681 [33:53<11:30, 3.01s/it] {'loss': 0.9212, 'grad_norm': 10.160775184631348, 'learning_rate': 1.549222776991186e-07, 'rewards/chosen': -1.0104700326919556, 'rewards/rejected': -1.6558433771133423, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.6453733444213867, 'logps/chosen': -185.05380249023438, 'logps/rejected': -277.22369384765625, 'logps/ref_chosen': -79.379638671875, 'logps/ref_rejected': -103.71539306640625, 'logits/chosen': -7.714634895324707, 'logits/rejected': -7.272608757019043, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.00961741991341114, 'kl/avg_steps': 0.6875, 'epoch': 0.66}
66%|███████████████████████████████████████████████████████████████████████████ | 452/681 [33:53<11:30, 3.01s/it] 67%|███████████████████████████████████████████████████████████████████████████▏ | 453/681 [33:56<11:22, 2.99s/it] {'loss': 0.8581, 'grad_norm': 10.804041862487793, 'learning_rate': 1.5373663637339584e-07, 'rewards/chosen': -1.191428303718567, 'rewards/rejected': -1.964081883430481, 'rewards/accuracies': 0.875, 'rewards/margins': 0.7726534605026245, 'logps/chosen': -213.16781616210938, 'logps/rejected': -297.2564697265625, 'logps/ref_chosen': -87.6951904296875, 'logps/ref_rejected': -90.0582275390625, 'logits/chosen': -7.6690826416015625, 'logits/rejected': -7.041682243347168, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.009551751427352428, 'kl/avg_steps': 0.6875, 'epoch': 0.67}
67%|███████████████████████████████████████████████████████████████████████████▏ | 453/681 [33:56<11:22, 2.99s/it] 67%|███████████████████████████████████████████████████████████████████████████▎ | 454/681 [33:59<11:18, 2.99s/it] {'loss': 1.0254, 'grad_norm': 11.586421966552734, 'learning_rate': 1.5255353167683017e-07, 'rewards/chosen': -1.273888349533081, 'rewards/rejected': -1.8924428224563599, 'rewards/accuracies': 0.75, 'rewards/margins': 0.6185543537139893, 'logps/chosen': -224.0977325439453, 'logps/rejected': -293.582275390625, 'logps/ref_chosen': -89.56623840332031, 'logps/ref_rejected': -92.92105102539062, 'logits/chosen': -7.6572160720825195, 'logits/rejected': -7.268322944641113, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.009486531838774681, 'kl/avg_steps': 0.46875, 'epoch': 0.67}
67%|███████████████████████████████████████████████████████████████████████████▎ | 454/681 [33:59<11:18, 2.99s/it] 67%|███████████████████████████████████████████████████████████████████████████▍ | 455/681 [34:02<11:19, 3.01s/it] {'loss': 0.9778, 'grad_norm': 10.893000602722168, 'learning_rate': 1.5137299478533064e-07, 'rewards/chosen': -1.15059232711792, 'rewards/rejected': -1.7739757299423218, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.6233835220336914, 'logps/chosen': -199.83328247070312, 'logps/rejected': -308.0276184082031, 'logps/ref_chosen': -77.6299819946289, 'logps/ref_rejected': -118.97795104980469, 'logits/chosen': -7.797979354858398, 'logits/rejected': -7.566769599914551, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.00944227073341608, 'kl/avg_steps': 0.5, 'epoch': 0.67}
67%|███████████████████████████████████████████████████████████████████████████▍ | 455/681 [34:02<11:19, 3.01s/it] 67%|███████████████████████████████████████████████████████████████████████████▋ | 456/681 [34:05<11:15, 3.00s/it] {'loss': 0.9811, 'grad_norm': 12.390460014343262, 'learning_rate': 1.5019505680714232e-07, 'rewards/chosen': -1.1745624542236328, 'rewards/rejected': -1.7645561695098877, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.5899935960769653, 'logps/chosen': -215.07440185546875, 'logps/rejected': -298.5582580566406, 'logps/ref_chosen': -89.61686706542969, 'logps/ref_rejected': -109.5597152709961, 'logits/chosen': -7.607115745544434, 'logits/rejected': -7.611271858215332, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.009395293891429901, 'kl/avg_steps': 0.5625, 'epoch': 0.67}
67%|███████████████████████████████████████████████████████████████████████████▋ | 456/681 [34:05<11:15, 3.00s/it] 67%|███████████████████████████████████████████████████████████████████████████▊ | 457/681 [34:08<11:32, 3.09s/it] {'loss': 0.842, 'grad_norm': 11.532391548156738, 'learning_rate': 1.4901974878202627e-07, 'rewards/chosen': -0.9881317615509033, 'rewards/rejected': -1.7533208131790161, 'rewards/accuracies': 0.921875, 'rewards/margins': 0.765188992023468, 'logps/chosen': -193.81298828125, 'logps/rejected': -280.0379638671875, 'logps/ref_chosen': -87.32168579101562, 'logps/ref_rejected': -90.76660919189453, 'logits/chosen': -7.829406261444092, 'logits/rejected': -7.343839168548584, 'kl/p_epsilon_steps': 0.90625, 'kl/n_epsilon_steps': 0.09375, 'kl/beta': 0.009342741221189499, 'kl/avg_steps': 0.8125, 'epoch': 0.67}
67%|███████████████████████████████████████████████████████████████████████████▊ | 457/681 [34:08<11:32, 3.09s/it] 67%|███████████████████████████████████████████████████████████████████████████▉ | 458/681 [34:11<11:27, 3.08s/it] {'loss': 1.024, 'grad_norm': 11.981277465820312, 'learning_rate': 1.4784710168044212e-07, 'rewards/chosen': -1.1065789461135864, 'rewards/rejected': -1.687969446182251, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.581390380859375, 'logps/chosen': -213.52423095703125, 'logps/rejected': -286.9295654296875, 'logps/ref_chosen': -93.52044677734375, 'logps/ref_rejected': -103.36898803710938, 'logits/chosen': -7.395938873291016, 'logits/rejected': -7.573344707489014, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.009267443791031837, 'kl/avg_steps': 0.59375, 'epoch': 0.67}
67%|███████████████████████████████████████████████████████████████████████████▉ | 458/681 [34:11<11:27, 3.08s/it] 67%|████████████████████████████████████████████████████████████████████████████▏ | 459/681 [34:14<11:25, 3.09s/it] {'loss': 0.9252, 'grad_norm': 9.090399742126465, 'learning_rate': 1.466771464027316e-07, 'rewards/chosen': -1.05517578125, 'rewards/rejected': -1.7575633525848389, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.7023875713348389, 'logps/chosen': -190.7766571044922, 'logps/rejected': -284.36572265625, 'logps/ref_chosen': -75.68820190429688, 'logps/ref_rejected': -92.17048645019531, 'logits/chosen': -7.5142059326171875, 'logits/rejected': -6.964491844177246, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.009212742559611797, 'kl/avg_steps': 0.65625, 'epoch': 0.67}
67%|████████████████████████████████████████████████████████████████████████████▏ | 459/681 [34:14<11:25, 3.09s/it] 68%|████████████████████████████████████████████████████████████████████████████▎ | 460/681 [34:18<11:29, 3.12s/it] {'loss': 0.9702, 'grad_norm': 12.827125549316406, 'learning_rate': 1.4550991377830423e-07, 'rewards/chosen': -1.0923173427581787, 'rewards/rejected': -1.7237976789474487, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6314802765846252, 'logps/chosen': -201.06686401367188, 'logps/rejected': -300.1006164550781, 'logps/ref_chosen': -81.11788940429688, 'logps/ref_rejected': -110.31238555908203, 'logits/chosen': -7.7618255615234375, 'logits/rejected': -7.418609619140625, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.009152678772807121, 'kl/avg_steps': 0.65625, 'epoch': 0.68}
68%|████████████████████████████████████████████████████████████████████████████▎ | 460/681 [34:18<11:29, 3.12s/it] 68%|████████████████████████████████████████████████████████████████████████████▍ | 461/681 [34:21<11:38, 3.18s/it] {'loss': 1.0504, 'grad_norm': 11.000265121459961, 'learning_rate': 1.4434543456482518e-07, 'rewards/chosen': -1.3213638067245483, 'rewards/rejected': -1.819370985031128, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.4980071187019348, 'logps/chosen': -227.41180419921875, 'logps/rejected': -295.16571044921875, 'logps/ref_chosen': -81.58352661132812, 'logps/ref_rejected': -93.87710571289062, 'logits/chosen': -7.253688335418701, 'logits/rejected': -6.825348854064941, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.009093005210161209, 'kl/avg_steps': 0.46875, 'epoch': 0.68}
68%|████████████████████████████████████████████████████████████████████████████▍ | 461/681 [34:21<11:38, 3.18s/it] 68%|████████████████████████████████████████████████████████████████████████████▋ | 462/681 [34:24<11:21, 3.11s/it] {'loss': 0.9154, 'grad_norm': 10.546554565429688, 'learning_rate': 1.4318373944740484e-07, 'rewards/chosen': -1.00137460231781, 'rewards/rejected': -1.675006628036499, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.6736320853233337, 'logps/chosen': -205.47802734375, 'logps/rejected': -272.15692138671875, 'logps/ref_chosen': -94.19855499267578, 'logps/ref_rejected': -85.63162994384766, 'logits/chosen': -7.895207405090332, 'logits/rejected': -7.166084289550781, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.00905058067291975, 'kl/avg_steps': 0.71875, 'epoch': 0.68}
68%|████████████████████████████████████████████████████████████████████████████▋ | 462/681 [34:24<11:21, 3.11s/it] 68%|████████████████████████████████████████████████████████████████████████████▊ | 463/681 [34:27<11:07, 3.06s/it] {'loss': 0.8797, 'grad_norm': 12.333318710327148, 'learning_rate': 1.4202485903778976e-07, 'rewards/chosen': -0.9655375480651855, 'rewards/rejected': -1.7415142059326172, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7759765386581421, 'logps/chosen': -193.8429718017578, 'logps/rejected': -292.2165222167969, 'logps/ref_chosen': -85.92474365234375, 'logps/ref_rejected': -96.90184020996094, 'logits/chosen': -7.807669639587402, 'logits/rejected': -7.118095397949219, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.008985994383692741, 'kl/avg_steps': 0.65625, 'epoch': 0.68}
68%|████████████████████████████████████████████████████████████████████████████▊ | 463/681 [34:27<11:07, 3.06s/it] 68%|████████████████████████████████████████████████████████████████████████████▉ | 464/681 [34:30<10:55, 3.02s/it] {'loss': 0.9686, 'grad_norm': 14.331416130065918, 'learning_rate': 1.4086882387355658e-07, 'rewards/chosen': -1.1799356937408447, 'rewards/rejected': -1.8003066778182983, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6203708648681641, 'logps/chosen': -212.67556762695312, 'logps/rejected': -310.6097717285156, 'logps/ref_chosen': -79.68920135498047, 'logps/ref_rejected': -107.29232025146484, 'logits/chosen': -7.688302993774414, 'logits/rejected': -7.788723945617676, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.008927407674491405, 'kl/avg_steps': 0.71875, 'epoch': 0.68}
68%|████████████████████████████████████████████████████████████████████████████▉ | 464/681 [34:30<10:55, 3.02s/it] 68%|█████████████████████████████████████████████████████████████████████████████▏ | 465/681 [34:33<11:11, 3.11s/it] {'loss': 0.8775, 'grad_norm': 11.430578231811523, 'learning_rate': 1.3971566441730714e-07, 'rewards/chosen': -1.073095440864563, 'rewards/rejected': -1.8162384033203125, 'rewards/accuracies': 0.875, 'rewards/margins': 0.7431429624557495, 'logps/chosen': -213.5987548828125, 'logps/rejected': -325.18511962890625, 'logps/ref_chosen': -91.8602294921875, 'logps/ref_rejected': -118.71000671386719, 'logits/chosen': -7.3866472244262695, 'logits/rejected': -7.172554016113281, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.008863699622452259, 'kl/avg_steps': 0.71875, 'epoch': 0.68}
68%|█████████████████████████████████████████████████████████████████████████████▏ | 465/681 [34:33<11:11, 3.11s/it] 68%|█████████████████████████████████████████████████████████████████████████████▎ | 466/681 [34:36<11:08, 3.11s/it] {'loss': 0.8759, 'grad_norm': 8.964788436889648, 'learning_rate': 1.3856541105586545e-07, 'rewards/chosen': -0.9667361974716187, 'rewards/rejected': -1.7277427911758423, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.7610065937042236, 'logps/chosen': -195.26803588867188, 'logps/rejected': -294.0559387207031, 'logps/ref_chosen': -84.70140075683594, 'logps/ref_rejected': -96.05084228515625, 'logits/chosen': -7.547239780426025, 'logits/rejected': -7.197183609008789, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.0088004469871521, 'kl/avg_steps': 0.71875, 'epoch': 0.68}
68%|█████████████████████████████████████████████████████████████████████████████▎ | 466/681 [34:36<11:08, 3.11s/it] 69%|█████████████████████████████████████████████████████████████████████████████▍ | 467/681 [34:39<10:59, 3.08s/it] {'loss': 0.9698, 'grad_norm': 11.361374855041504, 'learning_rate': 1.3741809409947729e-07, 'rewards/chosen': -1.1772236824035645, 'rewards/rejected': -1.8375762701034546, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6603525876998901, 'logps/chosen': -244.66354370117188, 'logps/rejected': -320.74517822265625, 'logps/ref_chosen': -109.29832458496094, 'logps/ref_rejected': -108.8436508178711, 'logits/chosen': -8.130990028381348, 'logits/rejected': -7.5615644454956055, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.008737645111978054, 'kl/avg_steps': 0.5625, 'epoch': 0.69}
69%|█████████████████████████████████████████████████████████████████████████████▍ | 467/681 [34:39<10:59, 3.08s/it] 69%|█████████████████████████████████████████████████████████████████████████████▋ | 468/681 [34:42<11:06, 3.13s/it] {'loss': 1.0436, 'grad_norm': 11.03893756866455, 'learning_rate': 1.362737437810114e-07, 'rewards/chosen': -0.9716976881027222, 'rewards/rejected': -1.518856167793274, 'rewards/accuracies': 0.6875, 'rewards/margins': 0.5471584796905518, 'logps/chosen': -210.38668823242188, 'logps/rejected': -282.5191345214844, 'logps/ref_chosen': -98.32164764404297, 'logps/ref_rejected': -106.68048095703125, 'logits/chosen': -7.599516868591309, 'logits/rejected': -7.804562091827393, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.008688771165907383, 'kl/avg_steps': 0.4375, 'epoch': 0.69}
69%|█████████████████████████████████████████████████████████████████████████████▋ | 468/681 [34:42<11:06, 3.13s/it] 69%|█████████████████████████████████████████████████████████████████████████████▊ | 469/681 [34:46<11:11, 3.17s/it] {'loss': 0.9145, 'grad_norm': 12.710182189941406, 'learning_rate': 1.351323902551631e-07, 'rewards/chosen': -1.1398154497146606, 'rewards/rejected': -1.8454160690307617, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.7056005597114563, 'logps/chosen': -229.18951416015625, 'logps/rejected': -324.5375061035156, 'logps/ref_chosen': -96.76420593261719, 'logps/ref_rejected': -109.59500885009766, 'logits/chosen': -7.760175704956055, 'logits/rejected': -7.189078330993652, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.008650923147797585, 'kl/avg_steps': 0.6875, 'epoch': 0.69}
69%|█████████████████████████████████████████████████████████████████████████████▊ | 469/681 [34:46<11:11, 3.17s/it] 69%|█████████████████████████████████████████████████████████████████████████████▉ | 470/681 [34:49<10:46, 3.06s/it] {'loss': 0.9357, 'grad_norm': 11.018048286437988, 'learning_rate': 1.339940635976592e-07, 'rewards/chosen': -1.0692801475524902, 'rewards/rejected': -1.7161200046539307, 'rewards/accuracies': 0.875, 'rewards/margins': 0.6468397378921509, 'logps/chosen': -208.847900390625, 'logps/rejected': -289.9870300292969, 'logps/ref_chosen': -83.49665832519531, 'logps/ref_rejected': -88.48578643798828, 'logits/chosen': -7.906304359436035, 'logits/rejected': -7.4226555824279785, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.008591854013502598, 'kl/avg_steps': 0.78125, 'epoch': 0.69}
69%|█████████████████████████████████████████████████████████████████████████████▉ | 470/681 [34:49<10:46, 3.06s/it] 69%|██████████████████████████████████████████████████████████████████████████████▏ | 471/681 [34:51<10:28, 2.99s/it] {'loss': 0.9497, 'grad_norm': 10.446932792663574, 'learning_rate': 1.3285879380446563e-07, 'rewards/chosen': -1.2270324230194092, 'rewards/rejected': -1.8406031131744385, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.6135708093643188, 'logps/chosen': -233.13998413085938, 'logps/rejected': -307.9644470214844, 'logps/ref_chosen': -88.47430419921875, 'logps/ref_rejected': -90.48171997070312, 'logits/chosen': -7.087899208068848, 'logits/rejected': -6.836267948150635, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.008525250479578972, 'kl/avg_steps': 0.6875, 'epoch': 0.69}
69%|██████████████████████████████████████████████████████████████████████████████▏ | 471/681 [34:51<10:28, 2.99s/it] 69%|██████████████████████████████████████████████████████████████████████████████▎ | 472/681 [34:55<10:58, 3.15s/it] {'loss': 0.8455, 'grad_norm': 9.13286304473877, 'learning_rate': 1.317266107909975e-07, 'rewards/chosen': -1.0882880687713623, 'rewards/rejected': -1.8462860584259033, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.757997989654541, 'logps/chosen': -254.59384155273438, 'logps/rejected': -340.85113525390625, 'logps/ref_chosen': -125.23369598388672, 'logps/ref_rejected': -121.05349731445312, 'logits/chosen': -7.951723098754883, 'logits/rejected': -7.220186233520508, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.008467039093375206, 'kl/avg_steps': 0.71875, 'epoch': 0.69}
69%|██████████████████████████████████████████████████████████████████████████████▎ | 472/681 [34:55<10:58, 3.15s/it] 69%|██████████████████████████████████████████████████████████████████████████████▍ | 473/681 [34:58<11:09, 3.22s/it] {'loss': 1.0855, 'grad_norm': 11.645308494567871, 'learning_rate': 1.3059754439133002e-07, 'rewards/chosen': -1.2393798828125, 'rewards/rejected': -1.711702585220337, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.4723225235939026, 'logps/chosen': -243.3128662109375, 'logps/rejected': -292.8554992675781, 'logps/ref_chosen': -95.61137390136719, 'logps/ref_rejected': -88.15115356445312, 'logits/chosen': -7.586986064910889, 'logits/rejected': -7.300940990447998, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.008406616747379303, 'kl/avg_steps': 0.40625, 'epoch': 0.69}
69%|██████████████████████████████████████████████████████████████████████████████▍ | 473/681 [34:58<11:09, 3.22s/it] 70%|██████████████████████████████████████████████████████████████████████████████▋ | 474/681 [35:01<11:08, 3.23s/it] {'loss': 1.1116, 'grad_norm': 12.829643249511719, 'learning_rate': 1.2947162435741277e-07, 'rewards/chosen': -1.2031913995742798, 'rewards/rejected': -1.6386675834655762, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.43547606468200684, 'logps/chosen': -225.4574737548828, 'logps/rejected': -293.2532958984375, 'logps/ref_chosen': -81.47975158691406, 'logps/ref_rejected': -96.46562957763672, 'logits/chosen': -7.622153282165527, 'logits/rejected': -7.345100402832031, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.008372602984309196, 'kl/avg_steps': 0.4375, 'epoch': 0.7}
70%|██████████████████████████████████████████████████████████████████████████████▋ | 474/681 [35:02<11:08, 3.23s/it] 70%|██████████████████████████████████████████████████████████████████████████████▊ | 475/681 [35:04<10:47, 3.14s/it] {'loss': 0.9232, 'grad_norm': 9.961170196533203, 'learning_rate': 1.2834888035828596e-07, 'rewards/chosen': -0.8945390582084656, 'rewards/rejected': -1.5759094953536987, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6813703775405884, 'logps/chosen': -182.11282348632812, 'logps/rejected': -285.2503662109375, 'logps/ref_chosen': -74.19598388671875, 'logps/ref_rejected': -94.69242095947266, 'logits/chosen': -7.716796875, 'logits/rejected': -7.385658264160156, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.008336132392287254, 'kl/avg_steps': 0.625, 'epoch': 0.7}
70%|██████████████████████████████████████████████████████████████████████████████▊ | 475/681 [35:04<10:47, 3.14s/it] 70%|██████████████████████████████████████████████████████████████████████████████▉ | 476/681 [35:07<10:36, 3.11s/it] {'loss': 0.9888, 'grad_norm': 12.75778579711914, 'learning_rate': 1.2722934197929802e-07, 'rewards/chosen': -1.067098617553711, 'rewards/rejected': -1.656817078590393, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.5897184610366821, 'logps/chosen': -201.364013671875, 'logps/rejected': -281.6759033203125, 'logps/ref_chosen': -71.97109985351562, 'logps/ref_rejected': -80.26224517822266, 'logits/chosen': -7.788543224334717, 'logits/rejected': -7.032910346984863, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.008284355513751507, 'kl/avg_steps': 0.59375, 'epoch': 0.7}
70%|██████████████████████████████████████████████████████████████████████████████▉ | 476/681 [35:07<10:36, 3.11s/it] 70%|███████████████████████████████████████████████████████████████████████████████▏ | 477/681 [35:11<10:38, 3.13s/it] {'loss': 0.9673, 'grad_norm': 10.86681079864502, 'learning_rate': 1.2611303872132631e-07, 'rewards/chosen': -1.0373156070709229, 'rewards/rejected': -1.6613343954086304, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.6240187883377075, 'logps/chosen': -231.5677032470703, 'logps/rejected': -285.14166259765625, 'logps/ref_chosen': -105.00555419921875, 'logps/ref_rejected': -81.87843322753906, 'logits/chosen': -7.685525894165039, 'logits/rejected': -7.131807327270508, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.008235457353293896, 'kl/avg_steps': 0.6875, 'epoch': 0.7}
70%|███████████████████████████████████████████████████████████████████████████████▏ | 477/681 [35:11<10:38, 3.13s/it] 70%|███████████████████████████████████████████████████████████████████████████████▎ | 478/681 [35:14<10:41, 3.16s/it] {'loss': 0.9417, 'grad_norm': 9.899713516235352, 'learning_rate': 1.2500000000000005e-07, 'rewards/chosen': -0.9919052124023438, 'rewards/rejected': -1.6340248584747314, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.6421196460723877, 'logps/chosen': -198.8499755859375, 'logps/rejected': -291.9015808105469, 'logps/ref_chosen': -76.7882080078125, 'logps/ref_rejected': -90.43994140625, 'logits/chosen': -7.471836090087891, 'logits/rejected': -7.25750732421875, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.008179225027561188, 'kl/avg_steps': 0.75, 'epoch': 0.7}
70%|███████████████████████████████████████████████████████████████████████████████▎ | 478/681 [35:14<10:41, 3.16s/it] 70%|███████████████████████████████████████████████████████████████████████████████▍ | 479/681 [35:17<10:28, 3.11s/it] {'loss': 0.9958, 'grad_norm': 12.096019744873047, 'learning_rate': 1.2389025514492456e-07, 'rewards/chosen': -1.0483417510986328, 'rewards/rejected': -1.6027498245239258, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.5544079542160034, 'logps/chosen': -211.1195526123047, 'logps/rejected': -299.9999694824219, 'logps/ref_chosen': -81.3623046875, 'logps/ref_rejected': -101.09114074707031, 'logits/chosen': -7.843846321105957, 'logits/rejected': -7.429350852966309, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.008118337951600552, 'kl/avg_steps': 0.625, 'epoch': 0.7}
70%|███████████████████████████████████████████████████████████████████████████████▍ | 479/681 [35:17<10:28, 3.11s/it] 70%|███████████████████████████████████████████████████████████████████████████████▋ | 480/681 [35:20<10:24, 3.11s/it] {'loss': 1.0276, 'grad_norm': 12.593810081481934, 'learning_rate': 1.227838333989088e-07, 'rewards/chosen': -1.1555697917938232, 'rewards/rejected': -1.6818903684616089, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.5263204574584961, 'logps/chosen': -240.56443786621094, 'logps/rejected': -296.2643127441406, 'logps/ref_chosen': -96.7739028930664, 'logps/ref_rejected': -86.40473937988281, 'logits/chosen': -8.081748962402344, 'logits/rejected': -7.16196346282959, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.008067913353443146, 'kl/avg_steps': 0.5, 'epoch': 0.7}
70%|███████████████████████████████████████████████████████████████████████████████▋ | 480/681 [35:20<10:24, 3.11s/it] 71%|███████████████████████████████████████████████████████████████████████████████▊ | 481/681 [35:23<10:13, 3.07s/it] {'loss': 0.99, 'grad_norm': 11.46150016784668, 'learning_rate': 1.2168076391719489e-07, 'rewards/chosen': -1.1354894638061523, 'rewards/rejected': -1.721273422241211, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.5857840180397034, 'logps/chosen': -233.6854705810547, 'logps/rejected': -314.57574462890625, 'logps/ref_chosen': -91.670166015625, 'logps/ref_rejected': -98.69490051269531, 'logits/chosen': -7.622033596038818, 'logits/rejected': -6.947890281677246, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.008027774281799793, 'kl/avg_steps': 0.59375, 'epoch': 0.71}
71%|███████████████████████████████████████████████████████████████████████████████▊ | 481/681 [35:23<10:13, 3.07s/it] 71%|███████████████████████████████████████████████████████████████████████████████▉ | 482/681 [35:26<10:15, 3.10s/it] {'loss': 0.9871, 'grad_norm': 11.263471603393555, 'learning_rate': 1.2058107576668938e-07, 'rewards/chosen': -1.0913074016571045, 'rewards/rejected': -1.6596999168395996, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.5683927536010742, 'logps/chosen': -235.95123291015625, 'logps/rejected': -304.33599853515625, 'logps/ref_chosen': -98.52011108398438, 'logps/ref_rejected': -94.8294448852539, 'logits/chosen': -7.860917091369629, 'logits/rejected': -7.225180625915527, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.007980390451848507, 'kl/avg_steps': 0.59375, 'epoch': 0.71}
71%|███████████████████████████████████████████████████████████████████████████████▉ | 482/681 [35:26<10:15, 3.10s/it] 71%|████████████████████████████████████████████████████████████████████████████████▏ | 483/681 [35:29<10:07, 3.07s/it] {'loss': 0.9526, 'grad_norm': 9.014310836791992, 'learning_rate': 1.194847979251979e-07, 'rewards/chosen': -1.1138007640838623, 'rewards/rejected': -1.7216558456420898, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.6078551411628723, 'logps/chosen': -248.36256408691406, 'logps/rejected': -319.85467529296875, 'logps/ref_chosen': -107.11860656738281, 'logps/ref_rejected': -101.11499786376953, 'logits/chosen': -8.008612632751465, 'logits/rejected': -7.4816975593566895, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.00793328694999218, 'kl/avg_steps': 0.75, 'epoch': 0.71}
71%|████████████████████████████████████████████████████████████████████████████████▏ | 483/681 [35:29<10:07, 3.07s/it] 71%|████████████████████████████████████████████████████████████████████████████████▎ | 484/681 [35:32<09:46, 2.98s/it] {'loss': 0.9462, 'grad_norm': 8.998906135559082, 'learning_rate': 1.1839195928066101e-07, 'rewards/chosen': -1.049012303352356, 'rewards/rejected': -1.6971848011016846, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.6481724381446838, 'logps/chosen': -220.87344360351562, 'logps/rejected': -307.90826416015625, 'logps/ref_chosen': -86.97991943359375, 'logps/ref_rejected': -90.72367095947266, 'logits/chosen': -7.730357646942139, 'logits/rejected': -6.921565055847168, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.007874229922890663, 'kl/avg_steps': 0.6875, 'epoch': 0.71}
71%|████████████████████████████████████████████████████████████████████████████████▎ | 484/681 [35:32<09:46, 2.98s/it] 71%|████████████████████████████████████████████████████████████████████████████████▍ | 485/681 [35:35<09:51, 3.02s/it] {'loss': 0.8722, 'grad_norm': 9.495361328125, 'learning_rate': 1.1730258863039347e-07, 'rewards/chosen': -0.9427545070648193, 'rewards/rejected': -1.6760468482971191, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.733292281627655, 'logps/chosen': -215.27700805664062, 'logps/rejected': -324.5331726074219, 'logps/ref_chosen': -94.05874633789062, 'logps/ref_rejected': -108.56297302246094, 'logits/chosen': -7.978307247161865, 'logits/rejected': -7.098984241485596, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.007820464670658112, 'kl/avg_steps': 0.6875, 'epoch': 0.71}
71%|████████████████████████████████████████████████████████████████████████████████▍ | 485/681 [35:35<09:51, 3.02s/it] 71%|████████████████████████████████████████████████████████████████████████████████▋ | 486/681 [35:37<09:17, 2.86s/it] {'loss': 0.9519, 'grad_norm': 11.05057144165039, 'learning_rate': 1.1621671468032493e-07, 'rewards/chosen': -1.0858381986618042, 'rewards/rejected': -1.7294615507125854, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.643623411655426, 'logps/chosen': -233.9942626953125, 'logps/rejected': -322.165771484375, 'logps/ref_chosen': -93.74588012695312, 'logps/ref_rejected': -98.07064819335938, 'logits/chosen': -7.851730823516846, 'logits/rejected': -7.302485466003418, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.007767065893858671, 'kl/avg_steps': 0.53125, 'epoch': 0.71}
71%|████████████████████████████████████████████████████████████████████████████████▋ | 486/681 [35:38<09:17, 2.86s/it] 72%|████████████████████████████████████████████████████████████████████████████████▊ | 487/681 [35:41<09:38, 2.98s/it] {'loss': 0.9663, 'grad_norm': 10.726838111877441, 'learning_rate': 1.1513436604424378e-07, 'rewards/chosen': -1.047347068786621, 'rewards/rejected': -1.6354316473007202, 'rewards/accuracies': 0.875, 'rewards/margins': 0.5880845785140991, 'logps/chosen': -224.526123046875, 'logps/rejected': -311.92681884765625, 'logps/ref_chosen': -88.0335693359375, 'logps/ref_rejected': -98.47209930419922, 'logits/chosen': -7.636082649230957, 'logits/rejected': -7.332340240478516, 'kl/p_epsilon_steps': 0.890625, 'kl/n_epsilon_steps': 0.109375, 'kl/beta': 0.0077260215766727924, 'kl/avg_steps': 0.78125, 'epoch': 0.72}
72%|████████████████████████████████████████████████████████████████████████████████▊ | 487/681 [35:41<09:38, 2.98s/it] 72%|████████████████████████████████████████████████████████████████████████████████▉ | 488/681 [35:44<09:57, 3.10s/it] {'loss': 0.935, 'grad_norm': 11.892740249633789, 'learning_rate': 1.1405557124304335e-07, 'rewards/chosen': -0.9786466360092163, 'rewards/rejected': -1.5881898403167725, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.6095430850982666, 'logps/chosen': -213.1976776123047, 'logps/rejected': -299.0603942871094, 'logps/ref_chosen': -84.78964233398438, 'logps/ref_rejected': -90.2734603881836, 'logits/chosen': -7.434564590454102, 'logits/rejected': -6.9452362060546875, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.007666129618883133, 'kl/avg_steps': 0.65625, 'epoch': 0.72}
72%|████████████████████████████████████████████████████████████████████████████████▉ | 488/681 [35:44<09:57, 3.10s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▏ | 489/681 [35:47<09:45, 3.05s/it] {'loss': 0.982, 'grad_norm': 9.741364479064941, 'learning_rate': 1.1298035870396985e-07, 'rewards/chosen': -1.0582023859024048, 'rewards/rejected': -1.6288890838623047, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.5706866979598999, 'logps/chosen': -229.92735290527344, 'logps/rejected': -301.63934326171875, 'logps/ref_chosen': -90.46929931640625, 'logps/ref_rejected': -86.39761352539062, 'logits/chosen': -7.642593860626221, 'logits/rejected': -7.163801670074463, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.007616148795932531, 'kl/avg_steps': 0.53125, 'epoch': 0.72}
72%|█████████████████████████████████████████████████████████████████████████████████▏ | 489/681 [35:47<09:45, 3.05s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▎ | 490/681 [35:50<09:54, 3.11s/it] {'loss': 1.094, 'grad_norm': 10.637678146362305, 'learning_rate': 1.1190875675987355e-07, 'rewards/chosen': -1.1003369092941284, 'rewards/rejected': -1.590024471282959, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.48968762159347534, 'logps/chosen': -230.93482971191406, 'logps/rejected': -327.2318115234375, 'logps/ref_chosen': -85.32012939453125, 'logps/ref_rejected': -115.99385070800781, 'logits/chosen': -7.2537336349487305, 'logits/rejected': -7.196831226348877, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.0075759016908705235, 'kl/avg_steps': 0.453125, 'epoch': 0.72}
72%|█████████████████████████████████████████████████████████████████████████████████▎ | 490/681 [35:50<09:54, 3.11s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▍ | 491/681 [35:53<09:40, 3.05s/it] {'loss': 1.0378, 'grad_norm': 9.847221374511719, 'learning_rate': 1.1084079364846241e-07, 'rewards/chosen': -1.0794801712036133, 'rewards/rejected': -1.5843286514282227, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.5048484802246094, 'logps/chosen': -229.64122009277344, 'logps/rejected': -291.98773193359375, 'logps/ref_chosen': -86.14351654052734, 'logps/ref_rejected': -80.67945861816406, 'logits/chosen': -7.813044548034668, 'logits/rejected': -7.093764305114746, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.00754172820597887, 'kl/avg_steps': 0.5, 'epoch': 0.72}
72%|█████████████████████████████████████████████████████████████████████████████████▍ | 491/681 [35:53<09:40, 3.05s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▋ | 492/681 [35:56<09:43, 3.09s/it] {'loss': 1.151, 'grad_norm': 13.724920272827148, 'learning_rate': 1.097764975115576e-07, 'rewards/chosen': -0.9921280145645142, 'rewards/rejected': -1.3964853286743164, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.40435731410980225, 'logps/chosen': -213.5839080810547, 'logps/rejected': -268.00067138671875, 'logps/ref_chosen': -81.10757446289062, 'logps/ref_rejected': -80.75199890136719, 'logits/chosen': -7.806938171386719, 'logits/rejected': -7.273170471191406, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.007504207547754049, 'kl/avg_steps': 0.46875, 'epoch': 0.72}
72%|█████████████████████████████████████████████████████████████████████████████████▋ | 492/681 [35:56<09:43, 3.09s/it] 72%|█████████████████████████████████████████████████████████████████████████████████▊ | 493/681 [36:00<09:47, 3.13s/it] {'loss': 1.0192, 'grad_norm': 11.94544506072998, 'learning_rate': 1.0871589639435203e-07, 'rewards/chosen': -0.9897406697273254, 'rewards/rejected': -1.4871938228607178, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.49745315313339233, 'logps/chosen': -245.13128662109375, 'logps/rejected': -293.9502868652344, 'logps/ref_chosen': -112.20733642578125, 'logps/ref_rejected': -93.60719299316406, 'logits/chosen': -7.833661079406738, 'logits/rejected': -6.987953186035156, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.007469195406883955, 'kl/avg_steps': 0.5625, 'epoch': 0.72}
72%|█████████████████████████████████████████████████████████████████████████████████▊ | 493/681 [36:00<09:47, 3.13s/it] 73%|█████████████████████████████████████████████████████████████████████████████████▉ | 494/681 [36:02<09:19, 2.99s/it] {'loss': 0.9054, 'grad_norm': 10.316105842590332, 'learning_rate': 1.0765901824467166e-07, 'rewards/chosen': -0.940959632396698, 'rewards/rejected': -1.6217553615570068, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.6807957291603088, 'logps/chosen': -200.50823974609375, 'logps/rejected': -312.24517822265625, 'logps/ref_chosen': -73.11489868164062, 'logps/ref_rejected': -92.16300201416016, 'logits/chosen': -7.544450759887695, 'logits/rejected': -7.313390731811523, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.007427416276186705, 'kl/avg_steps': 0.71875, 'epoch': 0.73}
73%|█████████████████████████████████████████████████████████████████████████████████▉ | 494/681 [36:02<09:19, 2.99s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▏ | 495/681 [36:05<09:28, 3.06s/it] {'loss': 1.0093, 'grad_norm': 10.488041877746582, 'learning_rate': 1.0660589091223854e-07, 'rewards/chosen': -0.950130820274353, 'rewards/rejected': -1.5005515813827515, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.5504207611083984, 'logps/chosen': -228.8547821044922, 'logps/rejected': -302.8748779296875, 'logps/ref_chosen': -99.52032470703125, 'logps/ref_rejected': -97.93089294433594, 'logits/chosen': -7.922597408294678, 'logits/rejected': -7.498479843139648, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.0073744128458201885, 'kl/avg_steps': 0.625, 'epoch': 0.73}
73%|██████████████████████████████████████████████████████████████████████████████████▏ | 495/681 [36:06<09:28, 3.06s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▎ | 496/681 [36:09<09:46, 3.17s/it] {'loss': 1.0384, 'grad_norm': 10.726445198059082, 'learning_rate': 1.0555654214793722e-07, 'rewards/chosen': -1.039567232131958, 'rewards/rejected': -1.520376443862915, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.48080918192863464, 'logps/chosen': -250.06234741210938, 'logps/rejected': -301.3846740722656, 'logps/ref_chosen': -107.85675048828125, 'logps/ref_rejected': -92.77056121826172, 'logits/chosen': -7.784862995147705, 'logits/rejected': -7.454352378845215, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.007328609004616737, 'kl/avg_steps': 0.4375, 'epoch': 0.73}
73%|██████████████████████████████████████████████████████████████████████████████████▎ | 496/681 [36:09<09:46, 3.17s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▍ | 497/681 [36:12<09:39, 3.15s/it] {'loss': 1.1655, 'grad_norm': 10.815900802612305, 'learning_rate': 1.0451099960308374e-07, 'rewards/chosen': -1.1472631692886353, 'rewards/rejected': -1.483197808265686, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.33593475818634033, 'logps/chosen': -249.58370971679688, 'logps/rejected': -286.0672607421875, 'logps/ref_chosen': -92.08322143554688, 'logps/ref_rejected': -81.79503631591797, 'logits/chosen': -7.584544658660889, 'logits/rejected': -7.411238670349121, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.34375, 'kl/beta': 0.007296686060726643, 'kl/avg_steps': 0.3125, 'epoch': 0.73}
73%|██████████████████████████████████████████████████████████████████████████████████▍ | 497/681 [36:12<09:39, 3.15s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▋ | 498/681 [36:15<09:33, 3.13s/it] {'loss': 0.9923, 'grad_norm': 10.606035232543945, 'learning_rate': 1.0346929082869641e-07, 'rewards/chosen': -0.9259578585624695, 'rewards/rejected': -1.4618239402770996, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.5358661413192749, 'logps/chosen': -226.19503784179688, 'logps/rejected': -293.2069396972656, 'logps/ref_chosen': -98.19436645507812, 'logps/ref_rejected': -90.68746185302734, 'logits/chosen': -7.672981262207031, 'logits/rejected': -7.160597801208496, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.007273954804986715, 'kl/avg_steps': 0.6875, 'epoch': 0.73}
73%|██████████████████████████████████████████████████████████████████████████████████▋ | 498/681 [36:15<09:33, 3.13s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▊ | 499/681 [36:18<09:23, 3.10s/it] {'loss': 1.0261, 'grad_norm': 13.12460708618164, 'learning_rate': 1.0243144327477013e-07, 'rewards/chosen': -0.9939165711402893, 'rewards/rejected': -1.5242608785629272, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.5303443074226379, 'logps/chosen': -219.22235107421875, 'logps/rejected': -319.46099853515625, 'logps/ref_chosen': -81.0399169921875, 'logps/ref_rejected': -106.92170715332031, 'logits/chosen': -7.342049598693848, 'logits/rejected': -7.139578342437744, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.007224287837743759, 'kl/avg_steps': 0.625, 'epoch': 0.73}
73%|██████████████████████████████████████████████████████████████████████████████████▊ | 499/681 [36:18<09:23, 3.10s/it] 73%|██████████████████████████████████████████████████████████████████████████████████▉ | 500/681 [36:21<09:15, 3.07s/it] {'loss': 1.0563, 'grad_norm': 11.147185325622559, 'learning_rate': 1.0139748428955333e-07, 'rewards/chosen': -1.0610756874084473, 'rewards/rejected': -1.5483863353729248, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.48731058835983276, 'logps/chosen': -237.38864135742188, 'logps/rejected': -317.2994079589844, 'logps/ref_chosen': -89.248046875, 'logps/ref_rejected': -100.41021728515625, 'logits/chosen': -7.811801433563232, 'logits/rejected': -7.598600387573242, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.007179416250437498, 'kl/avg_steps': 0.4375, 'epoch': 0.73}
73%|██████████████████████████████████████████████████████████████████████████████████▉ | 500/681 [36:21<09:15, 3.07s/it][INFO|trainer.py:4307] 2026-04-24 04:52:28,739 >>
***** Running Evaluation *****
[INFO|trainer.py:4309] 2026-04-24 04:52:28,739 >> Num examples = 2339
[INFO|trainer.py:4312] 2026-04-24 04:52:28,739 >> Batch size = 8
0%| | 0/73 [00:00<?, ?it/s]
3%|███▏ | 2/73 [00:00<00:23, 3.04it/s]
4%|████▊ | 3/73 [00:01<00:32, 2.14it/s]
5%|██████▎ | 4/73 [00:01<00:36, 1.88it/s]
7%|███████▉ | 5/73 [00:02<00:38, 1.76it/s]
8%|█████████▌ | 6/73 [00:03<00:40, 1.65it/s]
10%|███████████ | 7/73 [00:03<00:38, 1.71it/s]
11%|████████████▋ | 8/73 [00:04<00:40, 1.59it/s]
12%|██████████████▎ | 9/73 [00:05<00:41, 1.54it/s]
14%|███████████████▊ | 10/73 [00:05<00:41, 1.51it/s]
15%|█████████████████▎ | 11/73 [00:06<00:40, 1.52it/s]
16%|██████████████████▉ | 12/73 [00:07<00:40, 1.49it/s]
18%|████████████████████▍ | 13/73 [00:07<00:39, 1.52it/s]
19%|██████████████████████ | 14/73 [00:08<00:39, 1.50it/s]
21%|███████████████████████▋ | 15/73 [00:09<00:38, 1.51it/s]
22%|█████████████████████████▏ | 16/73 [00:09<00:39, 1.46it/s]
23%|██████████████████████████▊ | 17/73 [00:10<00:38, 1.44it/s]
25%|████████████████████████████▎ | 18/73 [00:11<00:38, 1.43it/s]
26%|█████████████████████████████▉ | 19/73 [00:12<00:38, 1.41it/s]
27%|███████████████████████████████▌ | 20/73 [00:12<00:37, 1.40it/s]
29%|█████████████████████████████████ | 21/73 [00:13<00:37, 1.40it/s]
30%|██████████████████████████████████▋ | 22/73 [00:14<00:36, 1.39it/s]
32%|████████████████████████████████████▏ | 23/73 [00:14<00:35, 1.42it/s]
33%|█████████████████████████████████████▊ | 24/73 [00:15<00:34, 1.43it/s]
34%|███████████████████████████████████████▍ | 25/73 [00:16<00:33, 1.44it/s]
36%|████████████████████████████████████████▉ | 26/73 [00:17<00:32, 1.45it/s]
37%|██████████████████████████████████████████▌ | 27/73 [00:17<00:28, 1.61it/s]
38%|████████████████████████████████████████████ | 28/73 [00:18<00:28, 1.56it/s]
40%|█████████████████████████████████████████████▋ | 29/73 [00:18<00:28, 1.57it/s]
41%|███████████████████████████████████████████████▎ | 30/73 [00:19<00:26, 1.59it/s]
42%|████████████████████████████████████████████████▊ | 31/73 [00:20<00:27, 1.53it/s]
44%|██████████████████████████████████████████████████▍ | 32/73 [00:20<00:26, 1.56it/s]
45%|███████████████████████████████████████████████████▉ | 33/73 [00:21<00:25, 1.58it/s]
47%|█████████████████████████████████████████████████████▌ | 34/73 [00:22<00:25, 1.54it/s]
48%|███████████████████████████████████████████████████████▏ | 35/73 [00:22<00:25, 1.48it/s]
49%|████████████████████████████████████████████████████████▋ | 36/73 [00:23<00:24, 1.48it/s]
51%|██████████████████████████████████████████████████████████▎ | 37/73 [00:24<00:24, 1.48it/s]
52%|███████████████████████████████████████████████████████████▊ | 38/73 [00:24<00:22, 1.56it/s]
53%|█████████████████████████████████████████████████████████████▍ | 39/73 [00:25<00:22, 1.50it/s]
55%|███████████████████████████████████████████████████████████████ | 40/73 [00:26<00:21, 1.50it/s]
56%|████████████████████████████████████████████████████████████████▌ | 41/73 [00:26<00:20, 1.56it/s]
58%|██████████████████████████████████████████████████████████████████▏ | 42/73 [00:27<00:20, 1.52it/s]
59%|███████████████████████████████████████████████████████████████████▋ | 43/73 [00:27<00:19, 1.54it/s]
60%|█████████████████████████████████████████████████████████████████████▎ | 44/73 [00:28<00:19, 1.52it/s]
62%|██████████████████████████████████████████████████████████████████████▉ | 45/73 [00:29<00:19, 1.46it/s]
63%|████████████████████████████████████████████████████████████████████████▍ | 46/73 [00:29<00:17, 1.53it/s]
64%|██████████████████████████████████████████████████████████████████████████ | 47/73 [00:30<00:17, 1.50it/s]
66%|███████████████████████████████████████████████████████████████████████████▌ | 48/73 [00:31<00:16, 1.49it/s]
67%|█████████████████████████████████████████████████████████████████████████████▏ | 49/73 [00:32<00:16, 1.47it/s]
68%|██████████████████████████████████████████████████████████████████████████████▊ | 50/73 [00:32<00:15, 1.48it/s]
70%|████████████████████████████████████████████████████████████████████████████████▎ | 51/73 [00:33<00:15, 1.46it/s]
71%|█████████████████████████████████████████████████████████████████████████████████▉ | 52/73 [00:34<00:14, 1.44it/s]
73%|███████████████████████████████████████████████████████████████████████████████████▍ | 53/73 [00:34<00:14, 1.41it/s]
74%|█████████████████████████████████████████████████████████████████████████████████████ | 54/73 [00:35<00:12, 1.50it/s]
75%|██████████████████████████████████████████████████████████████████████████████████████▋ | 55/73 [00:36<00:12, 1.49it/s]
77%|████████████████████████████████████████████████████████████████████████████████████████▏ | 56/73 [00:36<00:11, 1.53it/s]
78%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 57/73 [00:37<00:10, 1.47it/s]
79%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 58/73 [00:38<00:09, 1.52it/s]
81%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 59/73 [00:38<00:09, 1.52it/s]
82%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 60/73 [00:39<00:08, 1.50it/s]
84%|████████████████████████████████████████████████████████████████████████████████████████████████ | 61/73 [00:40<00:08, 1.48it/s]
85%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 62/73 [00:40<00:07, 1.50it/s]
86%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 63/73 [00:41<00:06, 1.58it/s]
88%|████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 64/73 [00:41<00:05, 1.60it/s]
89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 65/73 [00:42<00:05, 1.58it/s]
90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 66/73 [00:43<00:04, 1.52it/s]
92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 67/73 [00:43<00:03, 1.54it/s]
93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 68/73 [00:44<00:03, 1.52it/s]
95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 69/73 [00:45<00:02, 1.48it/s]
96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 70/73 [00:46<00:02, 1.48it/s]
97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 71/73 [00:46<00:01, 1.48it/s]
99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 72/73 [00:47<00:00, 1.48it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
{'eval_loss': 0.5370330810546875, 'eval_runtime': 48.5908, 'eval_samples_per_second': 48.137, 'eval_steps_per_second': 1.523, 'eval_rewards/chosen': -1.0134860277175903, 'eval_rewards/rejected': -1.4611784219741821, 'eval_rewards/accuracies': 0.7666952013969421, 'eval_rewards/margins': 0.447692334651947, 'eval_logps/chosen': -242.73046875, 'eval_logps/rejected': -299.77655029296875, 'eval_logps/ref_chosen': -100.49356842041016, 'eval_logps/ref_rejected': -94.06775665283203, 'eval_logits/chosen': -7.587223529815674, 'eval_logits/rejected': -7.184981822967529, 'eval_kl/p_epsilon_steps': 0.7491438388824463, 'eval_kl/n_epsilon_steps': 0.25042808055877686, 'epoch': 0.73}
73%|██████████████████████████████████████████████████████████████████████████████████▉ | 500/681 [37:10<09:15, 3.07s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
 74%|███████████████████████████████████████████████████████████████████████████████████▏ | 501/681 [37:13<52:52, 17.62s/it] {'loss': 1.0588, 'grad_norm': 11.189581871032715, 'learning_rate': 1.0036744111882672e-07, 'rewards/chosen': -0.9249030947685242, 'rewards/rejected': -1.4437575340270996, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.5188544392585754, 'logps/chosen': -220.14817810058594, 'logps/rejected': -288.4246520996094, 'logps/ref_chosen': -90.596923828125, 'logps/ref_rejected': -85.2972640991211, 'logits/chosen': -7.5872368812561035, 'logits/rejected': -7.235992431640625, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.007148143369704485, 'kl/avg_steps': 0.46875, 'epoch': 0.74}
74%|███████████████████████████████████████████████████████████████████████████████████▏ | 501/681 [37:13<52:52, 17.62s/it] 74%|███████████████████████████████████████████████████████████████████████████████████▎ | 502/681 [37:16<39:36, 13.28s/it] {'loss': 1.0789, 'grad_norm': 10.501534461975098, 'learning_rate': 9.934134090518592e-08, 'rewards/chosen': -0.9396613836288452, 'rewards/rejected': -1.4016259908676147, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.46196457743644714, 'logps/chosen': -235.90652465820312, 'logps/rejected': -287.358642578125, 'logps/ref_chosen': -103.47073364257812, 'logps/ref_rejected': -89.04415893554688, 'logits/chosen': -8.039237976074219, 'logits/rejected': -6.9175286293029785, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.007114792708307505, 'kl/avg_steps': 0.5, 'epoch': 0.74}
74%|███████████████████████████████████████████████████████████████████████████████████▎ | 502/681 [37:16<39:36, 13.28s/it] 74%|███████████████████████████████████████████████████████████████████████████████████▍ | 503/681 [37:19<30:33, 10.30s/it] {'loss': 1.0182, 'grad_norm': 9.694924354553223, 'learning_rate': 9.831921068732571e-08, 'rewards/chosen': -0.9058637619018555, 'rewards/rejected': -1.4358644485473633, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.5300005674362183, 'logps/chosen': -222.0538787841797, 'logps/rejected': -291.72247314453125, 'logps/ref_chosen': -93.72999572753906, 'logps/ref_rejected': -87.60896301269531, 'logits/chosen': -7.841567039489746, 'logits/rejected': -7.385000228881836, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.007079395931214094, 'kl/avg_steps': 0.4375, 'epoch': 0.74}
74%|███████████████████████████████████████████████████████████████████████████████████▍ | 503/681 [37:19<30:33, 10.30s/it] 74%|███████████████████████████████████████████████████████████████████████████████████▋ | 504/681 [37:22<24:09, 8.19s/it] {'loss': 0.9324, 'grad_norm': 12.8939208984375, 'learning_rate': 9.730107739932805e-08, 'rewards/chosen': -0.9024004340171814, 'rewards/rejected': -1.5322885513305664, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.629888117313385, 'logps/chosen': -221.08261108398438, 'logps/rejected': -328.2010498046875, 'logps/ref_chosen': -92.2918701171875, 'logps/ref_rejected': -109.03765106201172, 'logits/chosen': -8.119867324829102, 'logits/rejected': -7.42840576171875, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.007048558443784714, 'kl/avg_steps': 0.75, 'epoch': 0.74}
74%|███████████████████████████████████████████████████████████████████████████████████▋ | 504/681 [37:22<24:09, 8.19s/it] 74%|███████████████████████████████████████████████████████████████████████████████████▊ | 505/681 [37:26<19:32, 6.66s/it] {'loss': 1.0176, 'grad_norm': 9.646526336669922, 'learning_rate': 9.628696786995188e-08, 'rewards/chosen': -1.0034873485565186, 'rewards/rejected': -1.5037386417388916, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.5002511143684387, 'logps/chosen': -250.1861572265625, 'logps/rejected': -310.4810791015625, 'logps/ref_chosen': -106.27693176269531, 'logps/ref_rejected': -94.20611572265625, 'logits/chosen': -7.906380653381348, 'logits/rejected': -7.302044868469238, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.006996087729930878, 'kl/avg_steps': 0.53125, 'epoch': 0.74}
74%|███████████████████████████████████████████████████████████████████████████████████▊ | 505/681 [37:26<19:32, 6.66s/it] 74%|███████████████████████████████████████████████████████████████████████████████████▉ | 506/681 [37:28<16:07, 5.53s/it] {'loss': 0.9974, 'grad_norm': 9.696473121643066, 'learning_rate': 9.527690882192635e-08, 'rewards/chosen': -0.9702832698822021, 'rewards/rejected': -1.4915196895599365, 'rewards/accuracies': 0.875, 'rewards/margins': 0.5212363600730896, 'logps/chosen': -221.6510009765625, 'logps/rejected': -301.66949462890625, 'logps/ref_chosen': -81.44086456298828, 'logps/ref_rejected': -85.66439819335938, 'logits/chosen': -7.563765525817871, 'logits/rejected': -7.289196014404297, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.006959117483347654, 'kl/avg_steps': 0.65625, 'epoch': 0.74}
74%|███████████████████████████████████████████████████████████████████████████████████▉ | 506/681 [37:28<16:07, 5.53s/it] 74%|████████████████████████████████████████████████████████████████████████████████████▏ | 507/681 [37:31<13:46, 4.75s/it] {'loss': 1.0131, 'grad_norm': 8.90442943572998, 'learning_rate': 9.427092687124691e-08, 'rewards/chosen': -1.0063550472259521, 'rewards/rejected': -1.550492763519287, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.5441376566886902, 'logps/chosen': -245.88427734375, 'logps/rejected': -328.4168701171875, 'logps/ref_chosen': -99.42694854736328, 'logps/ref_rejected': -102.25296020507812, 'logits/chosen': -7.426417827606201, 'logits/rejected': -7.099169731140137, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.006913745775818825, 'kl/avg_steps': 0.71875, 'epoch': 0.74}
74%|████████████████████████████████████████████████████████████████████████████████████▏ | 507/681 [37:31<13:46, 4.75s/it] 75%|████████████████████████████████████████████████████████████████████████████████████▎ | 508/681 [37:35<12:27, 4.32s/it] {'loss': 1.0875, 'grad_norm': 10.534308433532715, 'learning_rate': 9.326904852647344e-08, 'rewards/chosen': -0.9846047759056091, 'rewards/rejected': -1.4372204542160034, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.4526156783103943, 'logps/chosen': -241.60638427734375, 'logps/rejected': -314.3401794433594, 'logps/ref_chosen': -97.9156494140625, 'logps/ref_rejected': -103.79782104492188, 'logits/chosen': -7.581225395202637, 'logits/rejected': -7.184242248535156, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.006864408031105995, 'kl/avg_steps': 0.46875, 'epoch': 0.75}
75%|████████████████████████████████████████████████████████████████████████████████████▎ | 508/681 [37:35<12:27, 4.32s/it] 75%|████████████████████████████████████████████████████████████████████████████████████▍ | 509/681 [37:38<11:21, 3.96s/it] {'loss': 1.0779, 'grad_norm': 12.450897216796875, 'learning_rate': 9.227130018803195e-08, 'rewards/chosen': -0.9274278879165649, 'rewards/rejected': -1.3628320693969727, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.43540412187576294, 'logps/chosen': -237.10333251953125, 'logps/rejected': -287.923095703125, 'logps/ref_chosen': -100.72984313964844, 'logps/ref_rejected': -86.943359375, 'logits/chosen': -7.861898422241211, 'logits/rejected': -7.100181579589844, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.006832381244748831, 'kl/avg_steps': 0.625, 'epoch': 0.75}
75%|████████████████████████████████████████████████████████████████████████████████████▍ | 509/681 [37:38<11:21, 3.96s/it] 75%|████████████████████████████████████████████████████████████████████████████████████▋ | 510/681 [37:41<10:38, 3.73s/it] {'loss': 1.0245, 'grad_norm': 10.103314399719238, 'learning_rate': 9.127770814751932e-08, 'rewards/chosen': -1.0782501697540283, 'rewards/rejected': -1.5844013690948486, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.5061511993408203, 'logps/chosen': -248.5462646484375, 'logps/rejected': -342.783447265625, 'logps/ref_chosen': -89.16938781738281, 'logps/ref_rejected': -107.91940307617188, 'logits/chosen': -7.755180358886719, 'logits/rejected': -7.310698986053467, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.006789944134652615, 'kl/avg_steps': 0.578125, 'epoch': 0.75}
75%|████████████████████████████████████████████████████████████████████████████████████▋ | 510/681 [37:41<10:38, 3.73s/it] 75%|████████████████████████████████████████████████████████████████████████████████████▊ | 511/681 [37:44<10:17, 3.63s/it] {'loss': 0.998, 'grad_norm': 9.313636779785156, 'learning_rate': 9.028829858700973e-08, 'rewards/chosen': -0.9661585092544556, 'rewards/rejected': -1.5053207874298096, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.539162278175354, 'logps/chosen': -233.65809631347656, 'logps/rejected': -322.42425537109375, 'logps/ref_chosen': -89.74979400634766, 'logps/ref_rejected': -97.73127746582031, 'logits/chosen': -7.98992919921875, 'logits/rejected': -7.07249641418457, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.006750915199518204, 'kl/avg_steps': 0.6875, 'epoch': 0.75}
75%|████████████████████████████████████████████████████████████████████████████████████▊ | 511/681 [37:44<10:17, 3.63s/it] 75%|████████████████████████████████████████████████████████████████████████████████████▉ | 512/681 [37:47<09:25, 3.35s/it] {'loss': 1.032, 'grad_norm': 11.483407974243164, 'learning_rate': 8.930309757836516e-08, 'rewards/chosen': -0.9088408350944519, 'rewards/rejected': -1.4173755645751953, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.5085346698760986, 'logps/chosen': -221.8423614501953, 'logps/rejected': -300.55499267578125, 'logps/ref_chosen': -85.73515319824219, 'logps/ref_rejected': -87.6248779296875, 'logits/chosen': -7.597411155700684, 'logits/rejected': -7.131270408630371, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.006704819854348898, 'kl/avg_steps': 0.625, 'epoch': 0.75}
75%|████████████████████████████████████████████████████████████████████████████████████▉ | 512/681 [37:47<09:25, 3.35s/it] 75%|█████████████████████████████████████████████████████████████████████████████████████ | 513/681 [37:50<09:04, 3.24s/it] {'loss': 1.0548, 'grad_norm': 10.936783790588379, 'learning_rate': 8.832213108254863e-08, 'rewards/chosen': -0.956098198890686, 'rewards/rejected': -1.4060094356536865, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.44991129636764526, 'logps/chosen': -240.4691162109375, 'logps/rejected': -295.57098388671875, 'logps/ref_chosen': -96.19051361083984, 'logps/ref_rejected': -82.91773986816406, 'logits/chosen': -7.599587917327881, 'logits/rejected': -7.238034725189209, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.006663174834102392, 'kl/avg_steps': 0.6875, 'epoch': 0.75}
75%|█████████████████████████████████████████████████████████████████████████████████████ | 513/681 [37:50<09:04, 3.24s/it] 75%|█████████████████████████████████████████████████████████████████████████████████████▎ | 514/681 [37:53<08:59, 3.23s/it] {'loss': 1.0464, 'grad_norm': 10.800374031066895, 'learning_rate': 8.734542494893954e-08, 'rewards/chosen': -0.9116629958152771, 'rewards/rejected': -1.37803316116333, 'rewards/accuracies': 0.75, 'rewards/margins': 0.46637022495269775, 'logps/chosen': -235.91629028320312, 'logps/rejected': -294.8122253417969, 'logps/ref_chosen': -97.82363891601562, 'logps/ref_rejected': -85.36982727050781, 'logits/chosen': -7.524253845214844, 'logits/rejected': -7.1176605224609375, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.006617678329348564, 'kl/avg_steps': 0.46875, 'epoch': 0.75}
75%|█████████████████████████████████████████████████████████████████████████████████████▎ | 514/681 [37:53<08:59, 3.23s/it] 76%|█████████████████████████████████████████████████████████████████████████████████████▍ | 515/681 [37:56<08:49, 3.19s/it] {'loss': 1.091, 'grad_norm': 13.166977882385254, 'learning_rate': 8.637300491465272e-08, 'rewards/chosen': -0.9028570652008057, 'rewards/rejected': -1.3151576519012451, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.41230061650276184, 'logps/chosen': -219.55661010742188, 'logps/rejected': -294.6715087890625, 'logps/ref_chosen': -81.93355560302734, 'logps/ref_rejected': -93.5765609741211, 'logits/chosen': -7.724981307983398, 'logits/rejected': -7.439934730529785, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0065868026576936245, 'kl/avg_steps': 0.59375, 'epoch': 0.76}
76%|█████████████████████████████████████████████████████████████████████████████████████▍ | 515/681 [37:56<08:49, 3.19s/it] 76%|█████████████████████████████████████████████████████████████████████████████████████▌ | 516/681 [37:59<08:26, 3.07s/it] {'loss': 1.1026, 'grad_norm': 9.387166023254395, 'learning_rate': 8.540489660386064e-08, 'rewards/chosen': -0.9115662574768066, 'rewards/rejected': -1.3503837585449219, 'rewards/accuracies': 0.75, 'rewards/margins': 0.43881756067276, 'logps/chosen': -236.6641845703125, 'logps/rejected': -326.1341552734375, 'logps/ref_chosen': -97.09166717529297, 'logps/ref_rejected': -118.55667114257812, 'logits/chosen': -7.62106990814209, 'logits/rejected': -7.103853225708008, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.006547924131155014, 'kl/avg_steps': 0.5, 'epoch': 0.76}
76%|█████████████████████████████████████████████████████████████████████████████████████▌ | 516/681 [37:59<08:26, 3.07s/it] 76%|█████████████████████████████████████████████████████████████████████████████████████▊ | 517/681 [38:02<08:19, 3.04s/it] {'loss': 1.0038, 'grad_norm': 9.433574676513672, 'learning_rate': 8.444112552711752e-08, 'rewards/chosen': -0.9653780460357666, 'rewards/rejected': -1.500288963317871, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.534911036491394, 'logps/chosen': -241.97000122070312, 'logps/rejected': -327.4776611328125, 'logps/ref_chosen': -93.18656921386719, 'logps/ref_rejected': -95.5927963256836, 'logits/chosen': -7.664484024047852, 'logits/rejected': -7.389780521392822, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.006515347398817539, 'kl/avg_steps': 0.5625, 'epoch': 0.76}
76%|█████████████████████████████████████████████████████████████████████████████████████▊ | 517/681 [38:02<08:19, 3.04s/it] 76%|█████████████████████████████████████████████████████████████████████████████████████▉ | 518/681 [38:06<08:36, 3.17s/it] {'loss': 1.1116, 'grad_norm': 11.14731502532959, 'learning_rate': 8.348171708068747e-08, 'rewards/chosen': -0.9949862360954285, 'rewards/rejected': -1.3899054527282715, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.394919216632843, 'logps/chosen': -241.240478515625, 'logps/rejected': -307.2711486816406, 'logps/ref_chosen': -87.0213623046875, 'logps/ref_rejected': -91.24455261230469, 'logits/chosen': -7.792545318603516, 'logits/rejected': -7.39441442489624, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.006478903815150261, 'kl/avg_steps': 0.5625, 'epoch': 0.76}
76%|█████████████████████████████████████████████████████████████████████████████████████▉ | 518/681 [38:06<08:36, 3.17s/it] 76%|██████████████████████████████████████████████████████████████████████████████████████ | 519/681 [38:09<08:40, 3.21s/it] {'loss': 1.0488, 'grad_norm': 9.395498275756836, 'learning_rate': 8.25266965458755e-08, 'rewards/chosen': -0.9199578762054443, 'rewards/rejected': -1.3597040176391602, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.43974611163139343, 'logps/chosen': -244.36495971679688, 'logps/rejected': -321.7382507324219, 'logps/ref_chosen': -100.68755340576172, 'logps/ref_rejected': -109.03042602539062, 'logits/chosen': -7.722479343414307, 'logits/rejected': -7.366518974304199, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.006442663725465536, 'kl/avg_steps': 0.65625, 'epoch': 0.76}
76%|██████████████████████████████████████████████████████████████████████████████████████ | 519/681 [38:09<08:40, 3.21s/it] 76%|██████████████████████████████████████████████████████████████████████████████████████▎ | 520/681 [38:12<08:33, 3.19s/it] {'loss': 0.9971, 'grad_norm': 9.159811973571777, 'learning_rate': 8.15760890883607e-08, 'rewards/chosen': -0.834189772605896, 'rewards/rejected': -1.341964602470398, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.507774829864502, 'logps/chosen': -229.7172393798828, 'logps/rejected': -318.0968322753906, 'logps/ref_chosen': -98.77762603759766, 'logps/ref_rejected': -106.9457015991211, 'logits/chosen': -7.769089221954346, 'logits/rejected': -7.011660575866699, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.006400659214705229, 'kl/avg_steps': 0.625, 'epoch': 0.76}
76%|██████████████████████████████████████████████████████████████████████████████████████▎ | 520/681 [38:12<08:33, 3.19s/it] 77%|██████████████████████████████████████████████████████████████████████████████████████▍ | 521/681 [38:15<08:28, 3.18s/it] {'loss': 1.0088, 'grad_norm': 9.642518043518066, 'learning_rate': 8.062991975753378e-08, 'rewards/chosen': -0.8716791868209839, 'rewards/rejected': -1.3673629760742188, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.4956836700439453, 'logps/chosen': -224.09951782226562, 'logps/rejected': -306.9713134765625, 'logps/ref_chosen': -86.35108947753906, 'logps/ref_rejected': -90.39317321777344, 'logits/chosen': -7.651534080505371, 'logits/rejected': -7.242176055908203, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.006360903847962618, 'kl/avg_steps': 0.65625, 'epoch': 0.77}
77%|██████████████████████████████████████████████████████████████████████████████████████▍ | 521/681 [38:15<08:28, 3.18s/it] 77%|██████████████████████████████████████████████████████████████████████████████████████▌ | 522/681 [38:18<08:19, 3.14s/it] {'loss': 1.0429, 'grad_norm': 9.185672760009766, 'learning_rate': 7.968821348583643e-08, 'rewards/chosen': -0.8779767751693726, 'rewards/rejected': -1.3531558513641357, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.47517913579940796, 'logps/chosen': -215.34011840820312, 'logps/rejected': -289.88787841796875, 'logps/ref_chosen': -76.008056640625, 'logps/ref_rejected': -74.44125366210938, 'logits/chosen': -7.3879499435424805, 'logits/rejected': -6.970926284790039, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.006319432519376278, 'kl/avg_steps': 0.46875, 'epoch': 0.77}
77%|██████████████████████████████████████████████████████████████████████████████████████▌ | 522/681 [38:18<08:19, 3.14s/it] 77%|██████████████████████████████████████████████████████████████████████████████████████▊ | 523/681 [38:21<08:05, 3.07s/it] {'loss': 1.1708, 'grad_norm': 11.464506149291992, 'learning_rate': 7.875099508810484e-08, 'rewards/chosen': -0.9972683787345886, 'rewards/rejected': -1.3228791952133179, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.3256107568740845, 'logps/chosen': -255.38108825683594, 'logps/rejected': -302.5433349609375, 'logps/ref_chosen': -96.49627685546875, 'logps/ref_rejected': -91.0152816772461, 'logits/chosen': -7.647948741912842, 'logits/rejected': -6.6883955001831055, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.006289948243647814, 'kl/avg_steps': 0.4375, 'epoch': 0.77}
77%|██████████████████████████████████████████████████████████████████████████████████████▊ | 523/681 [38:21<08:05, 3.07s/it] 77%|██████████████████████████████████████████████████████████████████████████████████████▉ | 524/681 [38:24<08:10, 3.12s/it] {'loss': 1.0526, 'grad_norm': 10.063695907592773, 'learning_rate': 7.781828926091535e-08, 'rewards/chosen': -0.908073902130127, 'rewards/rejected': -1.3604583740234375, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.45238441228866577, 'logps/chosen': -248.78433227539062, 'logps/rejected': -306.3849792480469, 'logps/ref_chosen': -103.40550231933594, 'logps/ref_rejected': -87.91015625, 'logits/chosen': -8.02735710144043, 'logits/rejected': -7.3874831199646, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.006262549664825201, 'kl/avg_steps': 0.46875, 'epoch': 0.77}
77%|██████████████████████████████████████████████████████████████████████████████████████▉ | 524/681 [38:24<08:10, 3.12s/it] 77%|███████████████████████████████████████████████████████████████████████████████████████ | 525/681 [38:27<07:54, 3.04s/it] {'loss': 1.0194, 'grad_norm': 8.88698673248291, 'learning_rate': 7.689012058193384e-08, 'rewards/chosen': -0.8309329748153687, 'rewards/rejected': -1.344010829925537, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.513077974319458, 'logps/chosen': -218.09078979492188, 'logps/rejected': -323.40283203125, 'logps/ref_chosen': -84.20648193359375, 'logps/ref_rejected': -106.26158905029297, 'logits/chosen': -7.746663570404053, 'logits/rejected': -7.510030746459961, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.006233330816030502, 'kl/avg_steps': 0.5625, 'epoch': 0.77}
77%|███████████████████████████████████████████████████████████████████████████████████████ | 525/681 [38:27<07:54, 3.04s/it] 77%|███████████████████████████████████████████████████████████████████████████████████████▎ | 526/681 [38:31<08:02, 3.11s/it] {'loss': 1.0393, 'grad_norm': 8.236015319824219, 'learning_rate': 7.596651350926836e-08, 'rewards/chosen': -0.8863099813461304, 'rewards/rejected': -1.3675878047943115, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.4812777638435364, 'logps/chosen': -246.24072265625, 'logps/rejected': -315.64605712890625, 'logps/ref_chosen': -102.85411071777344, 'logps/ref_rejected': -93.65145874023438, 'logits/chosen': -7.848876476287842, 'logits/rejected': -7.135056495666504, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.006198464427143335, 'kl/avg_steps': 0.5, 'epoch': 0.77}
77%|███████████████████████████████████████████████████████████████████████████████████████▎ | 526/681 [38:31<08:02, 3.11s/it] 77%|███████████████████████████████████████████████████████████████████████████████████████▍ | 527/681 [38:33<07:51, 3.06s/it] {'loss': 1.0093, 'grad_norm': 8.652771949768066, 'learning_rate': 7.504749238082414e-08, 'rewards/chosen': -0.8550451993942261, 'rewards/rejected': -1.3389368057250977, 'rewards/accuracies': 0.875, 'rewards/margins': 0.4838915765285492, 'logps/chosen': -236.21170043945312, 'logps/rejected': -304.16583251953125, 'logps/ref_chosen': -96.76543426513672, 'logps/ref_rejected': -85.39518737792969, 'logits/chosen': -8.085288047790527, 'logits/rejected': -7.213015556335449, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.006167626474052668, 'kl/avg_steps': 0.6875, 'epoch': 0.77}
77%|███████████████████████████████████████████████████████████████████████████████████████▍ | 527/681 [38:34<07:51, 3.06s/it] 78%|███████████████████████████████████████████████████████████████████████████████████████▌ | 528/681 [38:37<07:55, 3.11s/it] {'loss': 1.0225, 'grad_norm': 8.880172729492188, 'learning_rate': 7.413308141366254e-08, 'rewards/chosen': -0.8029056191444397, 'rewards/rejected': -1.2914564609527588, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.48855090141296387, 'logps/chosen': -225.98036193847656, 'logps/rejected': -312.89617919921875, 'logps/ref_chosen': -94.1649169921875, 'logps/ref_rejected': -100.372314453125, 'logits/chosen': -8.002275466918945, 'logits/rejected': -7.597675800323486, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.006125513464212418, 'kl/avg_steps': 0.6875, 'epoch': 0.78}
78%|███████████████████████████████████████████████████████████████████████████████████████▌ | 528/681 [38:37<07:55, 3.11s/it] 78%|███████████████████████████████████████████████████████████████████████████████████████▊ | 529/681 [38:40<07:56, 3.14s/it] {'loss': 1.0863, 'grad_norm': 10.054601669311523, 'learning_rate': 7.322330470336313e-08, 'rewards/chosen': -0.9308747053146362, 'rewards/rejected': -1.3569061756134033, 'rewards/accuracies': 0.75, 'rewards/margins': 0.42603152990341187, 'logps/chosen': -234.24697875976562, 'logps/rejected': -321.6305847167969, 'logps/ref_chosen': -80.77679443359375, 'logps/ref_rejected': -97.191162109375, 'logits/chosen': -7.928413391113281, 'logits/rejected': -7.295429229736328, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.006083688233047724, 'kl/avg_steps': 0.5, 'epoch': 0.78}
78%|███████████████████████████████████████████████████████████████████████████████████████▊ | 529/681 [38:40<07:56, 3.14s/it] 78%|███████████████████████████████████████████████████████████████████████████████████████▉ | 530/681 [38:43<07:51, 3.12s/it] {'loss': 1.0424, 'grad_norm': 8.395234107971191, 'learning_rate': 7.231818622338822e-08, 'rewards/chosen': -0.852878212928772, 'rewards/rejected': -1.2901506423950195, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.43727239966392517, 'logps/chosen': -221.32212829589844, 'logps/rejected': -306.6104736328125, 'logps/ref_chosen': -79.7757339477539, 'logps/ref_rejected': -92.05390167236328, 'logits/chosen': -7.8860321044921875, 'logits/rejected': -7.2712297439575195, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.006053421180695295, 'kl/avg_steps': 0.65625, 'epoch': 0.78}
78%|███████████████████████████████████████████████████████████████████████████████████████▉ | 530/681 [38:43<07:51, 3.12s/it] 78%|████████████████████████████████████████████████████████████████████████████████████████ | 531/681 [38:46<07:29, 3.00s/it] {'loss': 0.9847, 'grad_norm': 8.446939468383789, 'learning_rate': 7.141774982445147e-08, 'rewards/chosen': -0.8268610239028931, 'rewards/rejected': -1.3529822826385498, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.5261213183403015, 'logps/chosen': -217.52377319335938, 'logps/rejected': -304.80413818359375, 'logps/ref_chosen': -79.31919860839844, 'logps/ref_rejected': -78.10832214355469, 'logits/chosen': -7.656224250793457, 'logits/rejected': -6.923822402954102, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.006013954523950815, 'kl/avg_steps': 0.6875, 'epoch': 0.78}
78%|████████████████████████████████████████████████████████████████████████████████████████ | 531/681 [38:46<07:29, 3.00s/it] 78%|████████████████████████████████████████████████████████████████████████████████████████▎ | 532/681 [38:49<07:31, 3.03s/it] {'loss': 0.9473, 'grad_norm': 9.200221061706543, 'learning_rate': 7.052201923388953e-08, 'rewards/chosen': -0.859551191329956, 'rewards/rejected': -1.4445768594741821, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.5850256681442261, 'logps/chosen': -246.2357177734375, 'logps/rejected': -336.9823913574219, 'logps/ref_chosen': -101.63691711425781, 'logps/ref_rejected': -93.34539031982422, 'logits/chosen': -7.909399032592773, 'logits/rejected': -7.158049583435059, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.005972891114652157, 'kl/avg_steps': 0.625, 'epoch': 0.78}
78%|████████████████████████████████████████████████████████████████████████████████████████▎ | 532/681 [38:49<07:31, 3.03s/it] 78%|████████████████████████████████████████████████████████████████████████████████████████▍ | 533/681 [38:52<07:36, 3.08s/it] {'loss': 1.1378, 'grad_norm': 9.506068229675293, 'learning_rate': 6.963101805503646e-08, 'rewards/chosen': -0.8360397219657898, 'rewards/rejected': -1.177465558052063, 'rewards/accuracies': 0.75, 'rewards/margins': 0.3414258658885956, 'logps/chosen': -232.10377502441406, 'logps/rejected': -282.5700378417969, 'logps/ref_chosen': -90.74664306640625, 'logps/ref_rejected': -82.88848876953125, 'logits/chosen': -7.598127365112305, 'logits/rejected': -6.821909427642822, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.0059357923455536366, 'kl/avg_steps': 0.4375, 'epoch': 0.78}
78%|████████████████████████████████████████████████████████████████████████████████████████▍ | 533/681 [38:52<07:36, 3.08s/it] 78%|████████████████████████████████████████████████████████████████████████████████████████▌ | 534/681 [38:55<07:37, 3.11s/it] {'loss': 1.0025, 'grad_norm': 8.0197114944458, 'learning_rate': 6.874476976660184e-08, 'rewards/chosen': -0.7507482767105103, 'rewards/rejected': -1.2383147478103638, 'rewards/accuracies': 0.875, 'rewards/margins': 0.4875665009021759, 'logps/chosen': -214.4866943359375, 'logps/rejected': -296.1422424316406, 'logps/ref_chosen': -86.63084411621094, 'logps/ref_rejected': -84.87196350097656, 'logits/chosen': -7.659628391265869, 'logits/rejected': -7.227793216705322, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.005909936036914587, 'kl/avg_steps': 0.71875, 'epoch': 0.78}
78%|████████████████████████████████████████████████████████████████████████████████████████▌ | 534/681 [38:55<07:37, 3.11s/it] 79%|████████████████████████████████████████████████████████████████████████████████████████▊ | 535/681 [38:58<07:33, 3.11s/it] {'loss': 1.0009, 'grad_norm': 7.752889633178711, 'learning_rate': 6.786329772205246e-08, 'rewards/chosen': -0.7379162907600403, 'rewards/rejected': -1.2516000270843506, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.5136837363243103, 'logps/chosen': -216.57179260253906, 'logps/rejected': -315.5731201171875, 'logps/ref_chosen': -90.34539031982422, 'logps/ref_rejected': -100.7779769897461, 'logits/chosen': -7.62947940826416, 'logits/rejected': -7.0962138175964355, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.005867761559784412, 'kl/avg_steps': 0.59375, 'epoch': 0.79}
79%|████████████████████████████████████████████████████████████████████████████████████████▊ | 535/681 [38:58<07:33, 3.11s/it] 79%|████████████████████████████████████████████████████████████████████████████████████████▉ | 536/681 [39:01<07:28, 3.10s/it] {'loss': 1.0251, 'grad_norm': 8.29379940032959, 'learning_rate': 6.698662514899638e-08, 'rewards/chosen': -0.7697625756263733, 'rewards/rejected': -1.245697021484375, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.47593456506729126, 'logps/chosen': -211.44839477539062, 'logps/rejected': -308.13739013671875, 'logps/ref_chosen': -78.95956420898438, 'logps/ref_rejected': -93.08779907226562, 'logits/chosen': -7.761280059814453, 'logits/rejected': -7.13836669921875, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.005833127535879612, 'kl/avg_steps': 0.625, 'epoch': 0.79}
79%|████████████████████████████████████████████████████████████████████████████████████████▉ | 536/681 [39:01<07:28, 3.10s/it] 79%|█████████████████████████████████████████████████████████████████████████████████████████ | 537/681 [39:04<07:24, 3.09s/it] {'loss': 0.9757, 'grad_norm': 7.392307758331299, 'learning_rate': 6.611477514857114e-08, 'rewards/chosen': -0.654714047908783, 'rewards/rejected': -1.1971983909606934, 'rewards/accuracies': 0.875, 'rewards/margins': 0.5424842834472656, 'logps/chosen': -200.18478393554688, 'logps/rejected': -286.97747802734375, 'logps/ref_chosen': -86.70939636230469, 'logps/ref_rejected': -78.90767669677734, 'logits/chosen': -7.517266273498535, 'logits/rejected': -6.534526824951172, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.005796896759420633, 'kl/avg_steps': 0.625, 'epoch': 0.79}
79%|█████████████████████████████████████████████████████████████████████████████████████████ | 537/681 [39:04<07:24, 3.09s/it] 79%|█████████████████████████████████████████████████████████████████████████████████████████▎ | 538/681 [39:08<07:23, 3.10s/it] {'loss': 1.0446, 'grad_norm': 8.975393295288086, 'learning_rate': 6.524777069483525e-08, 'rewards/chosen': -0.8716259002685547, 'rewards/rejected': -1.3055297136306763, 'rewards/accuracies': 0.875, 'rewards/margins': 0.43390387296676636, 'logps/chosen': -250.26470947265625, 'logps/rejected': -319.07464599609375, 'logps/ref_chosen': -98.44214630126953, 'logps/ref_rejected': -91.09465026855469, 'logits/chosen': -7.621641159057617, 'logits/rejected': -7.222417831420898, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.005760891363024712, 'kl/avg_steps': 0.53125, 'epoch': 0.79}
79%|█████████████████████████████████████████████████████████████████████████████████████████▎ | 538/681 [39:08<07:23, 3.10s/it] 79%|█████████████████████████████████████████████████████████████████████████████████████████▍ | 539/681 [39:11<07:20, 3.10s/it] {'loss': 1.0608, 'grad_norm': 7.8592424392700195, 'learning_rate': 6.438563463416221e-08, 'rewards/chosen': -0.7778847217559814, 'rewards/rejected': -1.2211490869522095, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.44326433539390564, 'logps/chosen': -227.2099609375, 'logps/rejected': -312.05865478515625, 'logps/ref_chosen': -90.93212127685547, 'logps/ref_rejected': -97.44546508789062, 'logits/chosen': -7.682827949523926, 'logits/rejected': -7.03498649597168, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.005730448290705681, 'kl/avg_steps': 0.59375, 'epoch': 0.79}
79%|█████████████████████████████████████████████████████████████████████████████████████████▍ | 539/681 [39:11<07:20, 3.10s/it] 79%|█████████████████████████████████████████████████████████████████████████████████████████▌ | 540/681 [39:14<07:17, 3.10s/it] {'loss': 1.0036, 'grad_norm': 9.220181465148926, 'learning_rate': 6.352838968463919e-08, 'rewards/chosen': -0.789432942867279, 'rewards/rejected': -1.3127686977386475, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.5233358144760132, 'logps/chosen': -247.24325561523438, 'logps/rejected': -353.7125549316406, 'logps/ref_chosen': -107.8698959350586, 'logps/ref_rejected': -121.39349365234375, 'logits/chosen': -8.04260540008545, 'logits/rejected': -7.764565467834473, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.005696624517440796, 'kl/avg_steps': 0.71875, 'epoch': 0.79}
79%|█████████████████████████████████████████████████████████████████████████████████████████▌ | 540/681 [39:14<07:17, 3.10s/it] 79%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 541/681 [39:17<07:13, 3.09s/it] {'loss': 1.114, 'grad_norm': 8.543745040893555, 'learning_rate': 6.267605843546767e-08, 'rewards/chosen': -0.8885458111763, 'rewards/rejected': -1.2408936023712158, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.3523477613925934, 'logps/chosen': -264.3243713378906, 'logps/rejected': -329.9075927734375, 'logps/ref_chosen': -106.56748962402344, 'logps/ref_rejected': -109.06633758544922, 'logits/chosen': -7.653639793395996, 'logits/rejected': -7.352320671081543, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.00565597228705883, 'kl/avg_steps': 0.53125, 'epoch': 0.79}
79%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 541/681 [39:17<07:13, 3.09s/it] 80%|█████████████████████████████████████████████████████████████████████████████████████████▉ | 542/681 [39:20<07:17, 3.15s/it] {'loss': 0.9943, 'grad_norm': 8.278136253356934, 'learning_rate': 6.182866334636888e-08, 'rewards/chosen': -0.7365444302558899, 'rewards/rejected': -1.248220443725586, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.5116760730743408, 'logps/chosen': -218.92977905273438, 'logps/rejected': -326.39044189453125, 'logps/ref_chosen': -87.36929321289062, 'logps/ref_rejected': -102.82034301757812, 'logits/chosen': -7.408226013183594, 'logits/rejected': -7.1460490226745605, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.005626083817332983, 'kl/avg_steps': 0.65625, 'epoch': 0.8}
80%|█████████████████████████████████████████████████████████████████████████████████████████▉ | 542/681 [39:20<07:17, 3.15s/it] 80%|██████████████████████████████████████████████████████████████████████████████████████████ | 543/681 [39:23<07:08, 3.10s/it] {'loss': 1.2137, 'grad_norm': 9.519865989685059, 'learning_rate': 6.098622674699147e-08, 'rewards/chosen': -0.8755909204483032, 'rewards/rejected': -1.1443732976913452, 'rewards/accuracies': 0.6875, 'rewards/margins': 0.2687823474407196, 'logps/chosen': -244.11569213867188, 'logps/rejected': -317.44921875, 'logps/ref_chosen': -87.152587890625, 'logps/ref_rejected': -111.53172302246094, 'logits/chosen': -7.630051612854004, 'logits/rejected': -7.440648078918457, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.005589403212070465, 'kl/avg_steps': 0.375, 'epoch': 0.8}
80%|██████████████████████████████████████████████████████████████████████████████████████████ | 543/681 [39:23<07:08, 3.10s/it] 80%|██████████████████████████████████████████████████████████████████████████████████████████▎ | 544/681 [39:26<06:57, 3.05s/it] {'loss': 1.1329, 'grad_norm': 10.596867561340332, 'learning_rate': 6.01487708363232e-08, 'rewards/chosen': -0.8795119524002075, 'rewards/rejected': -1.212106466293335, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.33259445428848267, 'logps/chosen': -244.70623779296875, 'logps/rejected': -325.4195861816406, 'logps/ref_chosen': -86.14691162109375, 'logps/ref_rejected': -106.3939208984375, 'logits/chosen': -7.832944869995117, 'logits/rejected': -7.403214931488037, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.005568521562963724, 'kl/avg_steps': 0.46875, 'epoch': 0.8}
80%|██████████████████████████████████████████████████████████████████████████████████████████▎ | 544/681 [39:26<06:57, 3.05s/it] 80%|██████████████████████████████████████████████████████████████████████████████████████████▍ | 545/681 [39:29<06:51, 3.03s/it] {'loss': 1.0893, 'grad_norm': 8.951570510864258, 'learning_rate': 5.9316317682106294e-08, 'rewards/chosen': -0.8078324794769287, 'rewards/rejected': -1.2127079963684082, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.4048755168914795, 'logps/chosen': -242.76431274414062, 'logps/rejected': -321.11676025390625, 'logps/ref_chosen': -96.42424011230469, 'logps/ref_rejected': -100.73405456542969, 'logits/chosen': -7.671846389770508, 'logits/rejected': -7.305430889129639, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.005542540457099676, 'kl/avg_steps': 0.59375, 'epoch': 0.8}
80%|██████████████████████████████████████████████████████████████████████████████████████████▍ | 545/681 [39:29<06:51, 3.03s/it] 80%|██████████████████████████████████████████████████████████████████████████████████████████▌ | 546/681 [39:32<06:48, 3.03s/it] {'loss': 1.0982, 'grad_norm': 8.22799301147461, 'learning_rate': 5.848888922025552e-08, 'rewards/chosen': -0.8111412525177002, 'rewards/rejected': -1.195991039276123, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.3848496675491333, 'logps/chosen': -233.88308715820312, 'logps/rejected': -306.00103759765625, 'logps/ref_chosen': -86.02112579345703, 'logps/ref_rejected': -87.37263488769531, 'logits/chosen': -7.779958724975586, 'logits/rejected': -7.305843353271484, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.005509825889021158, 'kl/avg_steps': 0.5625, 'epoch': 0.8}
80%|██████████████████████████████████████████████████████████████████████████████████████████▌ | 546/681 [39:32<06:48, 3.03s/it] 80%|██████████████████████████████████████████████████████████████████████████████████████████▊ | 547/681 [39:35<07:00, 3.14s/it] {'loss': 1.0372, 'grad_norm': 8.054706573486328, 'learning_rate': 5.7666507254280265e-08, 'rewards/chosen': -0.7865848541259766, 'rewards/rejected': -1.2217838764190674, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.43519890308380127, 'logps/chosen': -244.9899444580078, 'logps/rejected': -322.5784912109375, 'logps/ref_chosen': -100.65100860595703, 'logps/ref_rejected': -97.9117431640625, 'logits/chosen': -7.915996074676514, 'logits/rejected': -7.356800556182861, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.005479006562381983, 'kl/avg_steps': 0.65625, 'epoch': 0.8}
80%|██████████████████████████████████████████████████████████████████████████████████████████▊ | 547/681 [39:35<07:00, 3.14s/it] 80%|██████████████████████████████████████████████████████████████████████████████████████████▉ | 548/681 [39:38<06:49, 3.08s/it] {'loss': 1.1549, 'grad_norm': 9.502042770385742, 'learning_rate': 5.684919345471029e-08, 'rewards/chosen': -0.7902284860610962, 'rewards/rejected': -1.1218115091323853, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.3315829634666443, 'logps/chosen': -241.53533935546875, 'logps/rejected': -305.8096618652344, 'logps/ref_chosen': -96.03010559082031, 'logps/ref_rejected': -98.41322326660156, 'logits/chosen': -7.732963562011719, 'logits/rejected': -7.511667251586914, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.005443285219371319, 'kl/avg_steps': 0.5, 'epoch': 0.8}
80%|██████████████████████████████████████████████████████████████████████████████████████████▉ | 548/681 [39:38<06:49, 3.08s/it] 81%|███████████████████████████████████████████████████████████████████████████████████████████ | 549/681 [39:42<06:52, 3.12s/it] {'loss': 1.0911, 'grad_norm': 9.200112342834473, 'learning_rate': 5.603696935852426e-08, 'rewards/chosen': -0.7132003307342529, 'rewards/rejected': -1.111588716506958, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.39838820695877075, 'logps/chosen': -215.589111328125, 'logps/rejected': -288.0585632324219, 'logps/ref_chosen': -83.45096588134766, 'logps/ref_rejected': -81.38023376464844, 'logits/chosen': -7.810609340667725, 'logits/rejected': -7.074420928955078, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.00541620422154665, 'kl/avg_steps': 0.625, 'epoch': 0.81}
81%|███████████████████████████████████████████████████████████████████████████████████████████ | 549/681 [39:42<06:52, 3.12s/it] 81%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 550/681 [39:45<06:53, 3.15s/it] {'loss': 1.0957, 'grad_norm': 8.178189277648926, 'learning_rate': 5.5229856368582376e-08, 'rewards/chosen': -0.8037484884262085, 'rewards/rejected': -1.171849012374878, 'rewards/accuracies': 0.875, 'rewards/margins': 0.36810043454170227, 'logps/chosen': -239.71810913085938, 'logps/rejected': -320.63720703125, 'logps/ref_chosen': -89.59434509277344, 'logps/ref_rejected': -101.31549835205078, 'logits/chosen': -7.702338218688965, 'logits/rejected': -7.392082214355469, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.00538256298750639, 'kl/avg_steps': 0.65625, 'epoch': 0.81}
81%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 550/681 [39:45<06:53, 3.15s/it] 81%|███████████████████████████████████████████████████████████████████████████████████████████▍ | 551/681 [39:48<06:39, 3.07s/it] {'loss': 1.0428, 'grad_norm': 8.914412498474121, 'learning_rate': 5.4427875753062734e-08, 'rewards/chosen': -0.6989625692367554, 'rewards/rejected': -1.144097924232483, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.4451354146003723, 'logps/chosen': -226.66494750976562, 'logps/rejected': -333.44720458984375, 'logps/ref_chosen': -95.08216094970703, 'logps/ref_rejected': -117.7192611694336, 'logits/chosen': -7.659028053283691, 'logits/rejected': -7.469613075256348, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.005347470287233591, 'kl/avg_steps': 0.71875, 'epoch': 0.81}
81%|███████████████████████████████████████████████████████████████████████████████████████████▍ | 551/681 [39:48<06:39, 3.07s/it] 81%|███████████████████████████████████████████████████████████████████████████████████████████▌ | 552/681 [39:50<06:22, 2.96s/it] {'loss': 0.9501, 'grad_norm': 7.542394638061523, 'learning_rate': 5.363104864490034e-08, 'rewards/chosen': -0.6762351989746094, 'rewards/rejected': -1.2585299015045166, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.582294762134552, 'logps/chosen': -229.49818420410156, 'logps/rejected': -349.8272705078125, 'logps/ref_chosen': -101.40918731689453, 'logps/ref_rejected': -110.83843994140625, 'logits/chosen': -8.226886749267578, 'logits/rejected': -7.813612937927246, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.005309309810400009, 'kl/avg_steps': 0.6875, 'epoch': 0.81}
81%|███████████████████████████████████████████████████████████████████████████████████████████▌ | 552/681 [39:50<06:22, 2.96s/it] 81%|███████████████████████████████████████████████████████████████████████████████████████████▊ | 553/681 [39:53<06:23, 2.99s/it] {'loss': 1.1097, 'grad_norm': 8.038527488708496, 'learning_rate': 5.2839396041230415e-08, 'rewards/chosen': -0.7355344295501709, 'rewards/rejected': -1.0990670919418335, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.3635327219963074, 'logps/chosen': -237.8641815185547, 'logps/rejected': -313.3408203125, 'logps/ref_chosen': -97.96388244628906, 'logps/ref_rejected': -103.57719421386719, 'logits/chosen': -7.498537540435791, 'logits/rejected': -6.907330513000488, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.005273057147860527, 'kl/avg_steps': 0.5, 'epoch': 0.81}
81%|███████████████████████████████████████████████████████████████████████████████████████████▊ | 553/681 [39:54<06:23, 2.99s/it] 81%|███████████████████████████████████████████████████████████████████████████████████████████▉ | 554/681 [39:56<06:21, 3.01s/it] {'loss': 1.1078, 'grad_norm': 7.692688941955566, 'learning_rate': 5.205293880283551e-08, 'rewards/chosen': -0.7710244655609131, 'rewards/rejected': -1.1643975973129272, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.39337316155433655, 'logps/chosen': -251.03372192382812, 'logps/rejected': -316.3883056640625, 'logps/ref_chosen': -103.68680572509766, 'logps/ref_rejected': -92.99210357666016, 'logits/chosen': -7.834980487823486, 'logits/rejected': -7.495749473571777, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.005246823187917471, 'kl/avg_steps': 0.40625, 'epoch': 0.81}
81%|███████████████████████████████████████████████████████████████████████████████████████████▉ | 554/681 [39:57<06:21, 3.01s/it] 81%|████████████████████████████████████████████████████████████████████████████████████████████ | 555/681 [39:59<06:13, 2.96s/it] {'loss': 1.1068, 'grad_norm': 8.527905464172363, 'learning_rate': 5.127169765359515e-08, 'rewards/chosen': -0.8064323663711548, 'rewards/rejected': -1.1860185861587524, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.37958627939224243, 'logps/chosen': -241.05128479003906, 'logps/rejected': -343.023193359375, 'logps/ref_chosen': -86.20362854003906, 'logps/ref_rejected': -114.54719543457031, 'logits/chosen': -7.357694149017334, 'logits/rejected': -7.210930824279785, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.005225594155490398, 'kl/avg_steps': 0.53125, 'epoch': 0.81}
81%|████████████████████████████████████████████████████████████████████████████████████████████ | 555/681 [39:59<06:13, 2.96s/it] 82%|████████████████████████████████████████████████████████████████████████████████████████████▎ | 556/681 [40:03<06:21, 3.06s/it] {'loss': 1.0752, 'grad_norm': 7.89186954498291, 'learning_rate': 5.049569317994012e-08, 'rewards/chosen': -0.7129172086715698, 'rewards/rejected': -1.1216354370117188, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.40871816873550415, 'logps/chosen': -226.0050811767578, 'logps/rejected': -323.8564147949219, 'logps/ref_chosen': -88.40348815917969, 'logps/ref_rejected': -106.63912200927734, 'logits/chosen': -7.9103193283081055, 'logits/rejected': -6.936588287353516, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.005197979975491762, 'kl/avg_steps': 0.53125, 'epoch': 0.82}
82%|████████████████████████████████████████████████████████████████████████████████████████████▎ | 556/681 [40:03<06:21, 3.06s/it] 82%|████████████████████████████████████████████████████████████████████████████████████████████▍ | 557/681 [40:06<06:34, 3.18s/it] {'loss': 1.0491, 'grad_norm': 6.813101291656494, 'learning_rate': 4.9724945830310144e-08, 'rewards/chosen': -0.7568396925926208, 'rewards/rejected': -1.2077887058258057, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.45094895362854004, 'logps/chosen': -245.75643920898438, 'logps/rejected': -350.8326110839844, 'logps/ref_chosen': -98.83523559570312, 'logps/ref_rejected': -115.64888763427734, 'logits/chosen': -7.78956413269043, 'logits/rejected': -7.387689590454102, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.005170511547476053, 'kl/avg_steps': 0.5625, 'epoch': 0.82}
82%|████████████████████████████████████████████████████████████████████████████████████████████▍ | 557/681 [40:06<06:34, 3.18s/it] 82%|████████████████████████████████████████████████████████████████████████████████████████████▌ | 558/681 [40:09<06:26, 3.14s/it] {'loss': 1.0219, 'grad_norm': 8.840348243713379, 'learning_rate': 4.8959475914614554e-08, 'rewards/chosen': -0.7578614950180054, 'rewards/rejected': -1.2178561687469482, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.45999467372894287, 'logps/chosen': -254.44580078125, 'logps/rejected': -347.26983642578125, 'logps/ref_chosen': -106.01183319091797, 'logps/ref_rejected': -108.39131164550781, 'logits/chosen': -8.046621322631836, 'logits/rejected': -7.308237075805664, 'kl/p_epsilon_steps': 0.90625, 'kl/n_epsilon_steps': 0.09375, 'kl/beta': 0.005141590256243944, 'kl/avg_steps': 0.8125, 'epoch': 0.82}
82%|████████████████████████████████████████████████████████████████████████████████████████████▌ | 558/681 [40:09<06:26, 3.14s/it] 82%|████████████████████████████████████████████████████████████████████████████████████████████▊ | 559/681 [40:12<06:18, 3.10s/it] {'loss': 1.0914, 'grad_norm': 9.36201000213623, 'learning_rate': 4.8199303603697614e-08, 'rewards/chosen': -0.79989093542099, 'rewards/rejected': -1.1845612525939941, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.38467034697532654, 'logps/chosen': -257.96856689453125, 'logps/rejected': -334.3748474121094, 'logps/ref_chosen': -100.42474365234375, 'logps/ref_rejected': -100.41270446777344, 'logits/chosen': -7.8494062423706055, 'logits/rejected': -7.495527267456055, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.005100151523947716, 'kl/avg_steps': 0.59375, 'epoch': 0.82}
82%|████████████████████████████████████████████████████████████████████████████████████████████▊ | 559/681 [40:12<06:18, 3.10s/it] 82%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 560/681 [40:15<06:20, 3.15s/it] {'loss': 1.1114, 'grad_norm': 10.975285530090332, 'learning_rate': 4.7444448928806615e-08, 'rewards/chosen': -0.781200647354126, 'rewards/rejected': -1.1401338577270508, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.35893329977989197, 'logps/chosen': -242.33444213867188, 'logps/rejected': -313.7993469238281, 'logps/ref_chosen': -87.63551330566406, 'logps/ref_rejected': -87.37985229492188, 'logits/chosen': -7.894330978393555, 'logits/rejected': -7.052946090698242, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.005070047918707132, 'kl/avg_steps': 0.53125, 'epoch': 0.82}
82%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 560/681 [40:15<06:20, 3.15s/it] 82%|█████████████████████████████████████████████████████████████████████████████████████████████ | 561/681 [40:18<06:09, 3.08s/it] {'loss': 1.104, 'grad_norm': 7.341772556304932, 'learning_rate': 4.669493178106432e-08, 'rewards/chosen': -0.8038696050643921, 'rewards/rejected': -1.1544455289840698, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.35057592391967773, 'logps/chosen': -235.06967163085938, 'logps/rejected': -336.55767822265625, 'logps/ref_chosen': -74.88140106201172, 'logps/ref_rejected': -106.04796600341797, 'logits/chosen': -7.702178001403809, 'logits/rejected': -7.204655647277832, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.005043255630880594, 'kl/avg_steps': 0.65625, 'epoch': 0.82}
82%|█████████████████████████████████████████████████████████████████████████████████████████████ | 561/681 [40:18<06:09, 3.08s/it] 83%|█████████████████████████████████████████████████████████████████████████████████████████████▎ | 562/681 [40:22<06:09, 3.11s/it] {'loss': 1.1451, 'grad_norm': 8.39021110534668, 'learning_rate': 4.5950771910944596e-08, 'rewards/chosen': -0.7275943756103516, 'rewards/rejected': -1.056580901145935, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.32898658514022827, 'logps/chosen': -246.33297729492188, 'logps/rejected': -314.5208740234375, 'logps/ref_chosen': -100.66322326660156, 'logps/ref_rejected': -102.24087524414062, 'logits/chosen': -7.776498794555664, 'logits/rejected': -7.434757232666016, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.005010375287383795, 'kl/avg_steps': 0.5, 'epoch': 0.83}
83%|█████████████████████████████████████████████████████████████████████████████████████████████▎ | 562/681 [40:22<06:09, 3.11s/it] 83%|█████████████████████████████████████████████████████████████████████████████████████████████▍ | 563/681 [40:24<05:59, 3.05s/it] {'loss': 1.151, 'grad_norm': 8.17624568939209, 'learning_rate': 4.521198892775202e-08, 'rewards/chosen': -0.7714153528213501, 'rewards/rejected': -1.0743141174316406, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.30289867520332336, 'logps/chosen': -251.52645874023438, 'logps/rejected': -318.0636901855469, 'logps/ref_chosen': -96.25921630859375, 'logps/ref_rejected': -101.20503997802734, 'logits/chosen': -7.574672698974609, 'logits/rejected': -7.067543983459473, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.004985447973012924, 'kl/avg_steps': 0.5, 'epoch': 0.83}
83%|█████████████████████████████████████████████████████████████████████████████████████████████▍ | 563/681 [40:24<05:59, 3.05s/it] 83%|█████████████████████████████████████████████████████████████████████████████████████████████▌ | 564/681 [40:27<05:56, 3.05s/it] {'loss': 1.1213, 'grad_norm': 8.074674606323242, 'learning_rate': 4.447860229910544e-08, 'rewards/chosen': -0.7416489124298096, 'rewards/rejected': -1.0670380592346191, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.32538917660713196, 'logps/chosen': -263.2978210449219, 'logps/rejected': -316.2669982910156, 'logps/ref_chosen': -112.96040344238281, 'logps/ref_rejected': -99.52803802490234, 'logits/chosen': -7.528448104858398, 'logits/rejected': -7.25858736038208, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0049606445245444775, 'kl/avg_steps': 0.578125, 'epoch': 0.83}
83%|█████████████████████████████████████████████████████████████████████████████████████████████▌ | 564/681 [40:28<05:56, 3.05s/it] 83%|█████████████████████████████████████████████████████████████████████████████████████████████▊ | 565/681 [40:31<06:05, 3.15s/it] {'loss': 1.0586, 'grad_norm': 7.512981414794922, 'learning_rate': 4.375063135042445e-08, 'rewards/chosen': -0.7062567472457886, 'rewards/rejected': -1.1262362003326416, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.41997939348220825, 'logps/chosen': -239.09127807617188, 'logps/rejected': -322.09124755859375, 'logps/ref_chosen': -95.36558532714844, 'logps/ref_rejected': -92.21737670898438, 'logits/chosen': -7.909860610961914, 'logits/rejected': -7.201497554779053, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.004932130686938763, 'kl/avg_steps': 0.53125, 'epoch': 0.83}
83%|█████████████████████████████████████████████████████████████████████████████████████████████▊ | 565/681 [40:31<06:05, 3.15s/it] 83%|█████████████████████████████████████████████████████████████████████████████████████████████▉ | 566/681 [40:34<06:02, 3.15s/it] {'loss': 1.0825, 'grad_norm': 6.797919750213623, 'learning_rate': 4.3028095264420525e-08, 'rewards/chosen': -0.7296649217605591, 'rewards/rejected': -1.1196088790893555, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.3899438977241516, 'logps/chosen': -248.45718383789062, 'logps/rejected': -340.752197265625, 'logps/ref_chosen': -98.97320556640625, 'logps/ref_rejected': -110.82090759277344, 'logits/chosen': -7.698864936828613, 'logits/rejected': -7.01510763168335, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.004906067159026861, 'kl/avg_steps': 0.625, 'epoch': 0.83}
83%|█████████████████████████████████████████████████████████████████████████████████████████████▉ | 566/681 [40:34<06:02, 3.15s/it] 83%|██████████████████████████████████████████████████████████████████████████████████████████████ | 567/681 [40:37<05:45, 3.03s/it] {'loss': 1.1228, 'grad_norm': 8.844162940979004, 'learning_rate': 4.231101308059165e-08, 'rewards/chosen': -0.7517495155334473, 'rewards/rejected': -1.084793210029602, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.3330436944961548, 'logps/chosen': -247.43397521972656, 'logps/rejected': -315.1552734375, 'logps/ref_chosen': -92.7035903930664, 'logps/ref_rejected': -91.22431945800781, 'logits/chosen': -7.589968681335449, 'logits/rejected': -6.949653625488281, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.00487559475004673, 'kl/avg_steps': 0.578125, 'epoch': 0.83}
83%|██████████████████████████████████████████████████████████████████████████████████████████████ | 567/681 [40:37<05:45, 3.03s/it] 83%|██████████████████████████████████████████████████████████████████████████████████████████████▏ | 568/681 [40:40<05:34, 2.96s/it] {'loss': 1.0596, 'grad_norm': 7.321277618408203, 'learning_rate': 4.1599403694720145e-08, 'rewards/chosen': -0.7039542198181152, 'rewards/rejected': -1.1143994331359863, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.4104452133178711, 'logps/chosen': -219.06063842773438, 'logps/rejected': -326.4876708984375, 'logps/ref_chosen': -73.13226318359375, 'logps/ref_rejected': -94.95265197753906, 'logits/chosen': -7.3952226638793945, 'logits/rejected': -7.033419609069824, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.004847569856792688, 'kl/avg_steps': 0.59375, 'epoch': 0.83}
83%|██████████████████████████████████████████████████████████████████████████████████████████████▏ | 568/681 [40:40<05:34, 2.96s/it] 84%|██████████████████████████████████████████████████████████████████████████████████████████████▍ | 569/681 [40:43<05:35, 2.99s/it] {'loss': 1.1345, 'grad_norm': 10.243264198303223, 'learning_rate': 4.089328585837512e-08, 'rewards/chosen': -0.7422770261764526, 'rewards/rejected': -1.0799843072891235, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.3377072811126709, 'logps/chosen': -243.99252319335938, 'logps/rejected': -312.2991943359375, 'logps/ref_chosen': -89.21708679199219, 'logps/ref_rejected': -86.41105651855469, 'logits/chosen': -7.595402240753174, 'logits/rejected': -7.113461494445801, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.004818957298994064, 'kl/avg_steps': 0.625, 'epoch': 0.84}
84%|██████████████████████████████████████████████████████████████████████████████████████████████▍ | 569/681 [40:43<05:35, 2.99s/it] 84%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 570/681 [40:46<05:38, 3.05s/it] {'loss': 1.029, 'grad_norm': 7.365958213806152, 'learning_rate': 4.019267817841834e-08, 'rewards/chosen': -0.6582140922546387, 'rewards/rejected': -1.1009138822555542, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.4426998496055603, 'logps/chosen': -233.00393676757812, 'logps/rejected': -320.5428466796875, 'logps/ref_chosen': -94.76277160644531, 'logps/ref_rejected': -88.84693145751953, 'logits/chosen': -7.986109733581543, 'logits/rejected': -7.3613433837890625, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.004789025988429785, 'kl/avg_steps': 0.6875, 'epoch': 0.84}
84%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 570/681 [40:46<05:38, 3.05s/it] 84%|██████████████████████████████████████████████████████████████████████████████████████████████▋ | 571/681 [40:49<05:34, 3.04s/it] {'loss': 1.0783, 'grad_norm': 9.40793514251709, 'learning_rate': 3.9497599116513705e-08, 'rewards/chosen': -0.6683664917945862, 'rewards/rejected': -1.0698926448822021, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.40152615308761597, 'logps/chosen': -221.9232177734375, 'logps/rejected': -323.5943603515625, 'logps/ref_chosen': -80.8212890625, 'logps/ref_rejected': -97.01528930664062, 'logits/chosen': -7.549665451049805, 'logits/rejected': -7.4702606201171875, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.004756326321512461, 'kl/avg_steps': 0.59375, 'epoch': 0.84}
84%|██████████████████████████████████████████████████████████████████████████████████████████████▋ | 571/681 [40:49<05:34, 3.04s/it] 84%|██████████████████████████████████████████████████████████████████████████████████████████████▉ | 572/681 [40:52<05:35, 3.08s/it] {'loss': 1.0794, 'grad_norm': 7.180628776550293, 'learning_rate': 3.880806698864086e-08, 'rewards/chosen': -0.6801784038543701, 'rewards/rejected': -1.0741183757781982, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.39393994212150574, 'logps/chosen': -221.84637451171875, 'logps/rejected': -319.32598876953125, 'logps/ref_chosen': -77.37992858886719, 'logps/ref_rejected': -90.56597900390625, 'logits/chosen': -7.4798078536987305, 'logits/rejected': -7.4275007247924805, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.004728252068161964, 'kl/avg_steps': 0.5625, 'epoch': 0.84}
84%|██████████████████████████████████████████████████████████████████████████████████████████████▉ | 572/681 [40:52<05:35, 3.08s/it] 84%|███████████████████████████████████████████████████████████████████████████████████████████████ | 573/681 [40:55<05:18, 2.95s/it] {'loss': 1.1016, 'grad_norm': 7.156675338745117, 'learning_rate': 3.812409996461275e-08, 'rewards/chosen': -0.6590225100517273, 'rewards/rejected': -1.023913860321045, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.3648913502693176, 'logps/chosen': -224.49549865722656, 'logps/rejected': -310.0919494628906, 'logps/ref_chosen': -83.7060775756836, 'logps/ref_rejected': -90.69746398925781, 'logits/chosen': -7.6273932456970215, 'logits/rejected': -7.473799228668213, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.004701804369688034, 'kl/avg_steps': 0.65625, 'epoch': 0.84}
84%|███████████████████████████████████████████████████████████████████████████████████████████████ | 573/681 [40:55<05:18, 2.95s/it] 84%|███████████████████████████████████████████████████████████████████████████████████████████████▏ | 574/681 [40:58<05:24, 3.03s/it] {'loss': 1.0924, 'grad_norm': 8.27857494354248, 'learning_rate': 3.74457160675965e-08, 'rewards/chosen': -0.6954778432846069, 'rewards/rejected': -1.074756145477295, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.37927812337875366, 'logps/chosen': -234.46607971191406, 'logps/rejected': -330.6099548339844, 'logps/ref_chosen': -85.12948608398438, 'logps/ref_rejected': -99.05403137207031, 'logits/chosen': -7.626450538635254, 'logits/rejected': -7.274733066558838, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.004671149887144566, 'kl/avg_steps': 0.5625, 'epoch': 0.84}
84%|███████████████████████████████████████████████████████████████████████████████████████████████▏ | 574/681 [40:58<05:24, 3.03s/it] 84%|███████████████████████████████████████████████████████████████████████████████████████████████▍ | 575/681 [41:01<05:24, 3.06s/it] {'loss': 1.0999, 'grad_norm': 9.878924369812012, 'learning_rate': 3.677293317363864e-08, 'rewards/chosen': -0.7044973969459534, 'rewards/rejected': -1.0793570280075073, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.37485966086387634, 'logps/chosen': -258.09442138671875, 'logps/rejected': -335.0401306152344, 'logps/ref_chosen': -105.77084350585938, 'logps/ref_rejected': -100.94944763183594, 'logits/chosen': -7.789747714996338, 'logits/rejected': -7.52932071685791, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.004645021632313728, 'kl/avg_steps': 0.59375, 'epoch': 0.84}
84%|███████████████████████████████████████████████████████████████████████████████████████████████▍ | 575/681 [41:01<05:24, 3.06s/it] 85%|███████████████████████████████████████████████████████████████████████████████████████████████▌ | 576/681 [41:04<05:13, 2.98s/it] {'loss': 1.1064, 'grad_norm': 7.550034046173096, 'learning_rate': 3.6105769011194224e-08, 'rewards/chosen': -0.7074535489082336, 'rewards/rejected': -1.0718638896942139, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.36441028118133545, 'logps/chosen': -234.95172119140625, 'logps/rejected': -339.11627197265625, 'logps/ref_chosen': -81.20452880859375, 'logps/ref_rejected': -105.40940856933594, 'logits/chosen': -7.752475738525391, 'logits/rejected': -7.319489479064941, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.004617604892700911, 'kl/avg_steps': 0.53125, 'epoch': 0.85}
85%|███████████████████████████████████████████████████████████████████████████████████████████████▌ | 576/681 [41:04<05:13, 2.98s/it] 85%|███████████████████████████████████████████████████████████████████████████████████████████████▋ | 577/681 [41:07<05:06, 2.94s/it] {'loss': 1.0486, 'grad_norm': 6.790652275085449, 'learning_rate': 3.5444241160659304e-08, 'rewards/chosen': -0.6245477199554443, 'rewards/rejected': -1.0607309341430664, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.43618321418762207, 'logps/chosen': -238.08453369140625, 'logps/rejected': -322.09918212890625, 'logps/ref_chosen': -101.46419525146484, 'logps/ref_rejected': -89.32637023925781, 'logits/chosen': -7.78272819519043, 'logits/rejected': -7.093866348266602, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.00459320330992341, 'kl/avg_steps': 0.65625, 'epoch': 0.85}
85%|███████████████████████████████████████████████████████████████████████████████████████████████▋ | 577/681 [41:07<05:06, 2.94s/it] 85%|███████████████████████████████████████████████████████████████████████████████████████████████▉ | 578/681 [41:10<05:11, 3.02s/it] {'loss': 1.1523, 'grad_norm': 10.109602928161621, 'learning_rate': 3.478836705390808e-08, 'rewards/chosen': -0.7381951808929443, 'rewards/rejected': -1.0396572351455688, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.3014621138572693, 'logps/chosen': -239.65093994140625, 'logps/rejected': -317.5960693359375, 'logps/ref_chosen': -77.29241943359375, 'logps/ref_rejected': -88.24701690673828, 'logits/chosen': -7.468972682952881, 'logits/rejected': -7.147921562194824, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.004563257098197937, 'kl/avg_steps': 0.53125, 'epoch': 0.85}
85%|███████████████████████████████████████████████████████████████████████████████████████████████▉ | 578/681 [41:10<05:11, 3.02s/it] 85%|████████████████████████████████████████████████████████████████████████████████████████████████ | 579/681 [41:13<05:14, 3.08s/it] {'loss': 1.13, 'grad_norm': 7.376382350921631, 'learning_rate': 3.41381639738331e-08, 'rewards/chosen': -0.6563674211502075, 'rewards/rejected': -0.9787693023681641, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.3224019408226013, 'logps/chosen': -232.49017333984375, 'logps/rejected': -319.19024658203125, 'logps/ref_chosen': -87.34305572509766, 'logps/ref_rejected': -102.12100982666016, 'logits/chosen': -7.745573997497559, 'logits/rejected': -6.99274206161499, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.004539142828434706, 'kl/avg_steps': 0.53125, 'epoch': 0.85}
85%|████████████████████████████████████████████████████████████████████████████████████████████████ | 579/681 [41:13<05:14, 3.08s/it] 85%|████████████████████████████████████████████████████████████████████████████████████████████████▏ | 580/681 [41:16<05:15, 3.13s/it] {'loss': 1.0857, 'grad_norm': 7.219406604766846, 'learning_rate': 3.349364905389032e-08, 'rewards/chosen': -0.6354615092277527, 'rewards/rejected': -1.0177514553070068, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.3822898864746094, 'logps/chosen': -229.9556121826172, 'logps/rejected': -315.407958984375, 'logps/ref_chosen': -88.92879486083984, 'logps/ref_rejected': -88.69661712646484, 'logits/chosen': -7.58662223815918, 'logits/rejected': -7.382083892822266, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.004515156149864197, 'kl/avg_steps': 0.5, 'epoch': 0.85}
85%|████████████████████████████████████████████████████████████████████████████████████████████████▏ | 580/681 [41:16<05:15, 3.13s/it] 85%|████████████████████████████████████████████████████████████████████████████████████████████████▍ | 581/681 [41:19<05:06, 3.06s/it] {'loss': 1.1356, 'grad_norm': 8.329227447509766, 'learning_rate': 3.285483927764726e-08, 'rewards/chosen': -0.671623945236206, 'rewards/rejected': -1.0037708282470703, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.33214688301086426, 'logps/chosen': -242.04396057128906, 'logps/rejected': -324.2420654296875, 'logps/ref_chosen': -92.22323608398438, 'logps/ref_rejected': -99.38943481445312, 'logits/chosen': -7.649691581726074, 'logits/rejected': -7.227565765380859, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.0044926926493644714, 'kl/avg_steps': 0.4375, 'epoch': 0.85}
85%|████████████████████████████████████████████████████████████████████████████████████████████████▍ | 581/681 [41:19<05:06, 3.06s/it] 85%|████████████████████████████████████████████████████████████████████████████████████████████████▌ | 582/681 [41:22<05:03, 3.07s/it] {'loss': 1.1379, 'grad_norm': 6.972781181335449, 'learning_rate': 3.222175147833556e-08, 'rewards/chosen': -0.5807515382766724, 'rewards/rejected': -0.8996415138244629, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.3188899755477905, 'logps/chosen': -217.5670928955078, 'logps/rejected': -317.5841064453125, 'logps/ref_chosen': -87.33561706542969, 'logps/ref_rejected': -115.1544189453125, 'logits/chosen': -7.825651168823242, 'logits/rejected': -7.724842548370361, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.004473122768104076, 'kl/avg_steps': 0.5625, 'epoch': 0.85}
85%|████████████████████████████████████████████████████████████████████████████████████████████████▌ | 582/681 [41:22<05:03, 3.07s/it] 86%|████████████████████████████████████████████████████████████████████████████████████████████████▋ | 583/681 [41:25<05:04, 3.11s/it] {'loss': 1.12, 'grad_norm': 7.694782733917236, 'learning_rate': 3.159440233840763e-08, 'rewards/chosen': -0.695499837398529, 'rewards/rejected': -1.0328271389007568, 'rewards/accuracies': 0.75, 'rewards/margins': 0.337327241897583, 'logps/chosen': -240.53501892089844, 'logps/rejected': -328.16815185546875, 'logps/ref_chosen': -83.71707153320312, 'logps/ref_rejected': -94.58465576171875, 'logits/chosen': -7.3343915939331055, 'logits/rejected': -7.080375671386719, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.004448102321475744, 'kl/avg_steps': 0.4375, 'epoch': 0.86}
86%|████████████████████████████████████████████████████████████████████████████████████████████████▋ | 583/681 [41:26<05:04, 3.11s/it] 86%|████████████████████████████████████████████████████████████████████████████████████████████████▉ | 584/681 [41:28<04:54, 3.04s/it] {'loss': 1.104, 'grad_norm': 7.240791320800781, 'learning_rate': 3.0972808389096635e-08, 'rewards/chosen': -0.6124042272567749, 'rewards/rejected': -0.9732060432434082, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.3608018755912781, 'logps/chosen': -232.9828338623047, 'logps/rejected': -325.73150634765625, 'logps/ref_chosen': -94.25955963134766, 'logps/ref_rejected': -104.48269653320312, 'logits/chosen': -7.599449157714844, 'logits/rejected': -7.185504913330078, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0044287266209721565, 'kl/avg_steps': 0.53125, 'epoch': 0.86}
86%|████████████████████████████████████████████████████████████████████████████████████████████████▉ | 584/681 [41:28<04:54, 3.04s/it] 86%|█████████████████████████████████████████████████████████████████████████████████████████████████ | 585/681 [41:31<04:48, 3.01s/it] {'loss': 1.0747, 'grad_norm': 6.849031925201416, 'learning_rate': 3.035698600998121e-08, 'rewards/chosen': -0.6510626077651978, 'rewards/rejected': -1.0604023933410645, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.4093397259712219, 'logps/chosen': -234.18655395507812, 'logps/rejected': -332.9254150390625, 'logps/ref_chosen': -85.63675689697266, 'logps/ref_rejected': -90.28547668457031, 'logits/chosen': -7.737574577331543, 'logits/rejected': -7.5866241455078125, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.004405323415994644, 'kl/avg_steps': 0.6875, 'epoch': 0.86}
86%|█████████████████████████████████████████████████████████████████████████████████████████████████ | 585/681 [41:31<04:48, 3.01s/it] 86%|█████████████████████████████████████████████████████████████████████████████████████████████████▏ | 586/681 [41:34<04:42, 2.97s/it] {'loss': 1.1595, 'grad_norm': 7.319284439086914, 'learning_rate': 2.974695142855388e-08, 'rewards/chosen': -0.7111270427703857, 'rewards/rejected': -0.9888947606086731, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.2777676582336426, 'logps/chosen': -246.67648315429688, 'logps/rejected': -325.6482238769531, 'logps/ref_chosen': -83.68115234375, 'logps/ref_rejected': -98.35916137695312, 'logits/chosen': -7.4545087814331055, 'logits/rejected': -6.863378524780273, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.0043752435594797134, 'kl/avg_steps': 0.5, 'epoch': 0.86}
86%|█████████████████████████████████████████████████████████████████████████████████████████████████▏ | 586/681 [41:34<04:42, 2.97s/it] 86%|█████████████████████████████████████████████████████████████████████████████████████████████████▍ | 587/681 [41:38<04:49, 3.08s/it] {'loss': 1.1418, 'grad_norm': 6.929393291473389, 'learning_rate': 2.9142720719793122e-08, 'rewards/chosen': -0.5736163258552551, 'rewards/rejected': -0.8761405944824219, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.30252429842948914, 'logps/chosen': -210.05108642578125, 'logps/rejected': -292.1748352050781, 'logps/ref_chosen': -77.5775146484375, 'logps/ref_rejected': -89.3203353881836, 'logits/chosen': -7.802062511444092, 'logits/rejected': -6.934123992919922, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.004353476222604513, 'kl/avg_steps': 0.625, 'epoch': 0.86}
86%|█████████████████████████████████████████████████████████████████████████████████████████████████▍ | 587/681 [41:38<04:49, 3.08s/it] 86%|█████████████████████████████████████████████████████████████████████████████████████████████████▌ | 588/681 [41:40<04:39, 3.00s/it] {'loss': 1.168, 'grad_norm': 7.931462287902832, 'learning_rate': 2.8544309805740018e-08, 'rewards/chosen': -0.6190832257270813, 'rewards/rejected': -0.9055891036987305, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.28650587797164917, 'logps/chosen': -227.4091796875, 'logps/rejected': -323.88861083984375, 'logps/ref_chosen': -83.80426025390625, 'logps/ref_rejected': -113.11943054199219, 'logits/chosen': -7.959348678588867, 'logits/rejected': -7.665192604064941, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.00432643573731184, 'kl/avg_steps': 0.5625, 'epoch': 0.86}
86%|█████████████████████████████████████████████████████████████████████████████████████████████████▌ | 588/681 [41:40<04:39, 3.00s/it] 86%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 589/681 [41:43<04:34, 2.98s/it] {'loss': 1.0798, 'grad_norm': 6.9884934425354, 'learning_rate': 2.7951734455078786e-08, 'rewards/chosen': -0.65859454870224, 'rewards/rejected': -1.0269978046417236, 'rewards/accuracies': 0.875, 'rewards/margins': 0.3684034049510956, 'logps/chosen': -250.2454376220703, 'logps/rejected': -356.3966064453125, 'logps/ref_chosen': -96.46501922607422, 'logps/ref_rejected': -116.03719329833984, 'logits/chosen': -7.721344947814941, 'logits/rejected': -6.941378593444824, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.004302235785871744, 'kl/avg_steps': 0.625, 'epoch': 0.86}
86%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 589/681 [41:43<04:34, 2.98s/it] 87%|█████████████████████████████████████████████████████████████████████████████████████████████████▉ | 590/681 [41:46<04:26, 2.93s/it] {'loss': 1.0942, 'grad_norm': 8.866425514221191, 'learning_rate': 2.736501028272095e-08, 'rewards/chosen': -0.6050465106964111, 'rewards/rejected': -0.957945704460144, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.3528992533683777, 'logps/chosen': -230.829345703125, 'logps/rejected': -336.062744140625, 'logps/ref_chosen': -88.4278564453125, 'logps/ref_rejected': -110.19876861572266, 'logits/chosen': -7.883823394775391, 'logits/rejected': -7.474784851074219, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.0042755138128995895, 'kl/avg_steps': 0.75, 'epoch': 0.87}
87%|█████████████████████████████████████████████████████████████████████████████████████████████████▉ | 590/681 [41:46<04:26, 2.93s/it] 87%|██████████████████████████████████████████████████████████████████████████████████████████████████ | 591/681 [41:49<04:11, 2.79s/it] {'loss': 1.0725, 'grad_norm': 5.826635360717773, 'learning_rate': 2.678415274939408e-08, 'rewards/chosen': -0.577212929725647, 'rewards/rejected': -0.9592036604881287, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.3819907307624817, 'logps/chosen': -224.34735107421875, 'logps/rejected': -317.07794189453125, 'logps/ref_chosen': -87.84968566894531, 'logps/ref_rejected': -89.57919311523438, 'logits/chosen': -7.728172302246094, 'logits/rejected': -6.94543981552124, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.004243686329573393, 'kl/avg_steps': 0.59375, 'epoch': 0.87}
87%|██████████████████████████████████████████████████████████████████████████████████████████████████ | 591/681 [41:49<04:11, 2.79s/it] 87%|██████████████████████████████████████████████████████████████████████████████████████████████████▏ | 592/681 [41:51<04:10, 2.81s/it] {'loss': 1.1533, 'grad_norm': 7.418654441833496, 'learning_rate': 2.6209177161234442e-08, 'rewards/chosen': -0.6733924150466919, 'rewards/rejected': -0.9702633619308472, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.2968708872795105, 'logps/chosen': -235.08880615234375, 'logps/rejected': -313.0411376953125, 'logps/ref_chosen': -74.97573852539062, 'logps/ref_rejected': -81.55262756347656, 'logits/chosen': -7.696051597595215, 'logits/rejected': -7.094331741333008, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.004218637943267822, 'kl/avg_steps': 0.53125, 'epoch': 0.87}
87%|██████████████████████████████████████████████████████████████████████████████████████████████████▏ | 592/681 [41:51<04:10, 2.81s/it] 87%|██████████████████████████████████████████████████████████████████████████████████████████████████▍ | 593/681 [41:54<04:14, 2.89s/it] {'loss': 1.1869, 'grad_norm': 10.643902778625488, 'learning_rate': 2.564009866938349e-08, 'rewards/chosen': -0.6236478090286255, 'rewards/rejected': -0.8773769736289978, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.2537292242050171, 'logps/chosen': -221.35968017578125, 'logps/rejected': -280.5572509765625, 'logps/ref_chosen': -72.23902893066406, 'logps/ref_rejected': -70.09652709960938, 'logits/chosen': -7.23436164855957, 'logits/rejected': -6.888458251953125, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.0041963448747992516, 'kl/avg_steps': 0.5, 'epoch': 0.87}
87%|██████████████████████████████████████████████████████████████████████████████████████████████████▍ | 593/681 [41:55<04:14, 2.89s/it] 87%|██████████████████████████████████████████████████████████████████████████████████████████████████▌ | 594/681 [41:58<04:16, 2.95s/it] {'loss': 1.0774, 'grad_norm': 6.839757442474365, 'learning_rate': 2.5076932269588708e-08, 'rewards/chosen': -0.5646122694015503, 'rewards/rejected': -0.937238335609436, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.37262603640556335, 'logps/chosen': -226.31228637695312, 'logps/rejected': -318.3495788574219, 'logps/ref_chosen': -90.43771362304688, 'logps/ref_rejected': -92.267578125, 'logits/chosen': -7.658824920654297, 'logits/rejected': -7.013692378997803, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.004175467416644096, 'kl/avg_steps': 0.625, 'epoch': 0.87}
87%|██████████████████████████████████████████████████████████████████████████████████████████████████▌ | 594/681 [41:58<04:16, 2.95s/it] 87%|██████████████████████████████████████████████████████████████████████████████████████████████████▋ | 595/681 [42:01<04:24, 3.07s/it] {'loss': 1.1373, 'grad_norm': 7.265661239624023, 'learning_rate': 2.451969280180849e-08, 'rewards/chosen': -0.5944833159446716, 'rewards/rejected': -0.9060162901878357, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.31153303384780884, 'logps/chosen': -224.98826599121094, 'logps/rejected': -306.0340576171875, 'logps/ref_chosen': -81.51480865478516, 'logps/ref_rejected': -86.52047729492188, 'logits/chosen': -7.426346778869629, 'logits/rejected': -6.822681427001953, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.004149532876908779, 'kl/avg_steps': 0.40625, 'epoch': 0.87}
87%|██████████████████████████████████████████████████████████████████████████████████████████████████▋ | 595/681 [42:01<04:24, 3.07s/it] 88%|██████████████████████████████████████████████████████████████████████████████████████████████████▉ | 596/681 [42:04<04:27, 3.14s/it] {'loss': 1.1748, 'grad_norm': 7.548264503479004, 'learning_rate': 2.396839494982103e-08, 'rewards/chosen': -0.6046238541603088, 'rewards/rejected': -0.8767467737197876, 'rewards/accuracies': 0.75, 'rewards/margins': 0.27212288975715637, 'logps/chosen': -234.91661071777344, 'logps/rejected': -299.5952453613281, 'logps/ref_chosen': -88.20805358886719, 'logps/ref_rejected': -86.09375, 'logits/chosen': -7.338626384735107, 'logits/rejected': -6.9880876541137695, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.004132743459194899, 'kl/avg_steps': 0.53125, 'epoch': 0.88}
88%|██████████████████████████████████████████████████████████████████████████████████████████████████▉ | 596/681 [42:04<04:27, 3.14s/it] 88%|███████████████████████████████████████████████████████████████████████████████████████████████████ | 597/681 [42:07<04:24, 3.15s/it] {'loss': 1.0937, 'grad_norm': 6.239217758178711, 'learning_rate': 2.3423053240837514e-08, 'rewards/chosen': -0.5899473428726196, 'rewards/rejected': -0.9528372287750244, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.36288982629776, 'logps/chosen': -227.93858337402344, 'logps/rejected': -340.60699462890625, 'logps/ref_chosen': -83.58998107910156, 'logps/ref_rejected': -106.94439697265625, 'logits/chosen': -7.8197503089904785, 'logits/rejected': -7.281065940856934, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.004110904410481453, 'kl/avg_steps': 0.71875, 'epoch': 0.88}
88%|███████████████████████████████████████████████████████████████████████████████████████████████████ | 597/681 [42:07<04:24, 3.15s/it] 88%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 598/681 [42:10<04:11, 3.03s/it] {'loss': 1.1493, 'grad_norm': 6.588911533355713, 'learning_rate': 2.2883682045119062e-08, 'rewards/chosen': -0.557396411895752, 'rewards/rejected': -0.8516007661819458, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.2942042946815491, 'logps/chosen': -227.22210693359375, 'logps/rejected': -304.92047119140625, 'logps/ref_chosen': -90.18262481689453, 'logps/ref_rejected': -94.7985610961914, 'logits/chosen': -7.641655921936035, 'logits/rejected': -7.323144912719727, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.004081568215042353, 'kl/avg_steps': 0.5625, 'epoch': 0.88}
88%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 598/681 [42:10<04:11, 3.03s/it] 88%|███████████████████████████████████████████████████████████████████████████████████████████████████▍ | 599/681 [42:13<04:13, 3.10s/it] {'loss': 1.1718, 'grad_norm': 6.382659435272217, 'learning_rate': 2.2350295575598367e-08, 'rewards/chosen': -0.570677638053894, 'rewards/rejected': -0.838860034942627, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.2681824266910553, 'logps/chosen': -218.54464721679688, 'logps/rejected': -296.90069580078125, 'logps/ref_chosen': -77.84227752685547, 'logps/ref_rejected': -89.1976318359375, 'logits/chosen': -7.610280990600586, 'logits/rejected': -7.40827751159668, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.004058737773448229, 'kl/avg_steps': 0.375, 'epoch': 0.88}
88%|███████████████████████████████████████████████████████████████████████████████████████████████████▍ | 599/681 [42:13<04:13, 3.10s/it] 88%|███████████████████████████████████████████████████████████████████████████████████████████████████▌ | 600/681 [42:16<04:06, 3.04s/it] {'loss': 1.1296, 'grad_norm': 5.930601119995117, 'learning_rate': 2.1822907887504932e-08, 'rewards/chosen': -0.6067217588424683, 'rewards/rejected': -0.9208986759185791, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.31417691707611084, 'logps/chosen': -242.80270385742188, 'logps/rejected': -322.2016906738281, 'logps/ref_chosen': -92.18781280517578, 'logps/ref_rejected': -92.95429992675781, 'logits/chosen': -7.736866474151611, 'logits/rejected': -7.516294956207275, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.004043574444949627, 'kl/avg_steps': 0.5625, 'epoch': 0.88}
88%|███████████████████████████████████████████████████████████████████████████████████████████████████▌ | 600/681 [42:16<04:06, 3.04s/it][INFO|trainer.py:4307] 2026-04-24 04:58:23,933 >>
***** Running Evaluation *****
[INFO|trainer.py:4309] 2026-04-24 04:58:23,933 >> Num examples = 2339
[INFO|trainer.py:4312] 2026-04-24 04:58:23,933 >> Batch size = 8
0%| | 0/73 [00:00<?, ?it/s]
3%|███▏ | 2/73 [00:00<00:23, 3.04it/s]
4%|████▊ | 3/73 [00:01<00:32, 2.14it/s]
5%|██████▎ | 4/73 [00:01<00:36, 1.88it/s]
7%|███████▉ | 5/73 [00:02<00:38, 1.75it/s]
8%|█████████▌ | 6/73 [00:03<00:40, 1.65it/s]
10%|███████████ | 7/73 [00:03<00:38, 1.70it/s]
11%|████████████▋ | 8/73 [00:04<00:40, 1.59it/s]
12%|██████████████▎ | 9/73 [00:05<00:41, 1.54it/s]
14%|███████████████▊ | 10/73 [00:05<00:41, 1.51it/s]
15%|█████████████████▎ | 11/73 [00:06<00:40, 1.52it/s]
16%|██████████████████▉ | 12/73 [00:07<00:40, 1.49it/s]
18%|████████████████████▍ | 13/73 [00:07<00:39, 1.52it/s]
19%|██████████████████████ | 14/73 [00:08<00:39, 1.50it/s]
21%|███████████████████████▋ | 15/73 [00:09<00:38, 1.51it/s]
22%|█████████████████████████▏ | 16/73 [00:09<00:39, 1.46it/s]
23%|██████████████████████████▊ | 17/73 [00:10<00:38, 1.45it/s]
25%|████████████████████████████▎ | 18/73 [00:11<00:38, 1.43it/s]
26%|█████████████████████████████▉ | 19/73 [00:12<00:38, 1.41it/s]
27%|███████████████████████████████▌ | 20/73 [00:12<00:37, 1.40it/s]
29%|█████████████████████████████████ | 21/73 [00:13<00:37, 1.40it/s]
30%|██████████████████████████████████▋ | 22/73 [00:14<00:36, 1.39it/s]
32%|████████████████████████████████████▏ | 23/73 [00:14<00:35, 1.42it/s]
33%|█████████████████████████████████████▊ | 24/73 [00:15<00:34, 1.43it/s]
34%|███████████████████████████████████████▍ | 25/73 [00:16<00:33, 1.44it/s]
36%|████████████████████████████████████████▉ | 26/73 [00:17<00:32, 1.45it/s]
37%|██████████████████████████████████████████▌ | 27/73 [00:17<00:28, 1.60it/s]
38%|████████████████████████████████████████████ | 28/73 [00:18<00:28, 1.56it/s]
40%|█████████████████████████████████████████████▋ | 29/73 [00:18<00:28, 1.57it/s]
41%|███████████████████████████████████████████████▎ | 30/73 [00:19<00:26, 1.60it/s]
42%|████████████████████████████████████████████████▊ | 31/73 [00:20<00:27, 1.52it/s]
44%|██████████████████████████████████████████████████▍ | 32/73 [00:20<00:26, 1.55it/s]
45%|███████████████████████████████████████████████████▉ | 33/73 [00:21<00:25, 1.58it/s]
47%|█████████████████████████████████████████████████████▌ | 34/73 [00:22<00:25, 1.54it/s]
48%|███████████████████████████████████████████████████████▏ | 35/73 [00:22<00:25, 1.48it/s]
49%|████████████████████████████████████████████████████████▋ | 36/73 [00:23<00:24, 1.49it/s]
51%|██████████████████████████████████████████████████████████▎ | 37/73 [00:24<00:24, 1.48it/s]
52%|███████████████████████████████████████████████████████████▊ | 38/73 [00:24<00:22, 1.56it/s]
53%|█████████████████████████████████████████████████████████████▍ | 39/73 [00:25<00:22, 1.50it/s]
55%|███████████████████████████████████████████████████████████████ | 40/73 [00:26<00:21, 1.51it/s]
56%|████████████████████████████████████████████████████████████████▌ | 41/73 [00:26<00:20, 1.56it/s]
58%|██████████████████████████████████████████████████████████████████▏ | 42/73 [00:27<00:20, 1.52it/s]
59%|███████████████████████████████████████████████████████████████████▋ | 43/73 [00:27<00:19, 1.55it/s]
60%|█████████████████████████████████████████████████████████████████████▎ | 44/73 [00:28<00:19, 1.52it/s]
62%|██████████████████████████████████████████████████████████████████████▉ | 45/73 [00:29<00:19, 1.47it/s]
63%|████████████████████████████████████████████████████████████████████████▍ | 46/73 [00:29<00:17, 1.53it/s]
64%|██████████████████████████████████████████████████████████████████████████ | 47/73 [00:30<00:17, 1.50it/s]
66%|███████████████████████████████████████████████████████████████████████████▌ | 48/73 [00:31<00:16, 1.49it/s]
67%|█████████████████████████████████████████████████████████████████████████████▏ | 49/73 [00:32<00:16, 1.47it/s]
68%|██████████████████████████████████████████████████████████████████████████████▊ | 50/73 [00:32<00:15, 1.49it/s]
70%|████████████████████████████████████████████████████████████████████████████████▎ | 51/73 [00:33<00:15, 1.47it/s]
71%|█████████████████████████████████████████████████████████████████████████████████▉ | 52/73 [00:34<00:14, 1.44it/s]
73%|███████████████████████████████████████████████████████████████████████████████████▍ | 53/73 [00:34<00:14, 1.42it/s]
74%|█████████████████████████████████████████████████████████████████████████████████████ | 54/73 [00:35<00:12, 1.50it/s]
75%|██████████████████████████████████████████████████████████████████████████████████████▋ | 55/73 [00:36<00:12, 1.49it/s]
77%|████████████████████████████████████████████████████████████████████████████████████████▏ | 56/73 [00:36<00:11, 1.53it/s]
78%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 57/73 [00:37<00:10, 1.47it/s]
79%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 58/73 [00:38<00:09, 1.53it/s]
81%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 59/73 [00:38<00:09, 1.53it/s]
82%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 60/73 [00:39<00:08, 1.50it/s]
84%|████████████████████████████████████████████████████████████████████████████████████████████████ | 61/73 [00:40<00:08, 1.49it/s]
85%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 62/73 [00:40<00:07, 1.50it/s]
86%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 63/73 [00:41<00:06, 1.59it/s]
88%|████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 64/73 [00:41<00:05, 1.61it/s]
89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 65/73 [00:42<00:05, 1.58it/s]
90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 66/73 [00:43<00:04, 1.52it/s]
92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 67/73 [00:43<00:03, 1.55it/s]
93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 68/73 [00:44<00:03, 1.52it/s]
95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 69/73 [00:45<00:02, 1.48it/s]
96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 70/73 [00:46<00:02, 1.47it/s]
97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 71/73 [00:46<00:01, 1.48it/s]
99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 72/73 [00:47<00:00, 1.48it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
{'eval_loss': 0.5845144987106323, 'eval_runtime': 48.5484, 'eval_samples_per_second': 48.179, 'eval_steps_per_second': 1.524, 'eval_rewards/chosen': -0.6029341220855713, 'eval_rewards/rejected': -0.8720343112945557, 'eval_rewards/accuracies': 0.7568492889404297, 'eval_rewards/margins': 0.2691001296043396, 'eval_logps/chosen': -250.90887451171875, 'eval_logps/rejected': -312.31573486328125, 'eval_logps/ref_chosen': -100.49356842041016, 'eval_logps/ref_rejected': -94.06775665283203, 'eval_logits/chosen': -7.617927551269531, 'eval_logits/rejected': -7.201268672943115, 'eval_kl/p_epsilon_steps': 0.7465753555297852, 'eval_kl/n_epsilon_steps': 0.252996563911438, 'epoch': 0.88}
88%|███████████████████████████████████████████████████████████████████████████████████████████████████▌ | 600/681 [43:05<04:06, 3.04s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s]
[INFO|trainer.py:3984] 2026-04-24 04:59:26,765 >> Saving model checkpoint to /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-600
[INFO|configuration_utils.py:419] 2026-04-24 04:59:26,775 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-600/config.json
[INFO|configuration_utils.py:911] 2026-04-24 04:59:26,778 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-600/generation_config.json
[INFO|modeling_utils.py:3580] 2026-04-24 05:00:06,997 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-600/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2510] 2026-04-24 05:00:07,002 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-600/tokenizer_config.json
[INFO|tokenization_utils_base.py:2519] 2026-04-24 05:00:07,005 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-600/special_tokens_map.json
[INFO|trainer.py:4083] 2026-04-24 05:03:05,410 >> Deleting older checkpoint [/scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-200] due to args.save_total_limit
88%|█████████████████████████████████████████████████████████████████████████████████████████████████▉ | 601/681 [47:03<1:57:28, 88.10s/it] {'loss': 1.1196, 'grad_norm': 6.323831081390381, 'learning_rate': 2.1301532877994742e-08, 'rewards/chosen': -0.6223502159118652, 'rewards/rejected': -0.9426168203353882, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.32026663422584534, 'logps/chosen': -244.5936279296875, 'logps/rejected': -336.119873046875, 'logps/ref_chosen': -89.21614074707031, 'logps/ref_rejected': -100.17054748535156, 'logits/chosen': -7.606431007385254, 'logits/rejected': -7.300946235656738, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.004020956344902515, 'kl/avg_steps': 0.59375, 'epoch': 0.88}
88%|█████████████████████████████████████████████████████████████████████████████████████████████████▉ | 601/681 [47:03<1:57:28, 88.10s/it] 88%|██████████████████████████████████████████████████████████████████████████████████████████████████ | 602/681 [47:06<1:22:30, 62.66s/it] {'loss': 1.0711, 'grad_norm': 7.207840919494629, 'learning_rate': 2.0786184285784298e-08, 'rewards/chosen': -0.48817095160484314, 'rewards/rejected': -0.8820826411247253, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.3939117193222046, 'logps/chosen': -202.654296875, 'logps/rejected': -315.4969482421875, 'logps/ref_chosen': -80.05760192871094, 'logps/ref_rejected': -93.197509765625, 'logits/chosen': -7.25759744644165, 'logits/rejected': -7.028449058532715, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.003997222986072302, 'kl/avg_steps': 0.65625, 'epoch': 0.88}
88%|██████████████████████████████████████████████████████████████████████████████████████████████████ | 602/681 [47:06<1:22:30, 62.66s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████ | 603/681 [47:09<58:13, 44.78s/it] {'loss': 1.1139, 'grad_norm': 6.062160968780518, 'learning_rate': 2.0276875690788204e-08, 'rewards/chosen': -0.526463508605957, 'rewards/rejected': -0.8603225946426392, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.33385905623435974, 'logps/chosen': -235.39010620117188, 'logps/rejected': -326.241455078125, 'logps/ref_chosen': -102.30957794189453, 'logps/ref_rejected': -108.06884765625, 'logits/chosen': -7.96243953704834, 'logits/rejected': -7.082343101501465, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.003971162252128124, 'kl/avg_steps': 0.5625, 'epoch': 0.89}
89%|████████████████████████████████████████████████████████████████████████████████████████████████████ | 603/681 [47:09<58:13, 44.78s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 604/681 [47:12<41:26, 32.29s/it] {'loss': 1.1058, 'grad_norm': 6.03358268737793, 'learning_rate': 1.977362051376158e-08, 'rewards/chosen': -0.5420645475387573, 'rewards/rejected': -0.8781407475471497, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.33607620000839233, 'logps/chosen': -216.16348266601562, 'logps/rejected': -323.61181640625, 'logps/ref_chosen': -78.17408752441406, 'logps/ref_rejected': -99.4961166381836, 'logits/chosen': -7.427624702453613, 'logits/rejected': -7.069514274597168, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.0039489492774009705, 'kl/avg_steps': 0.6875, 'epoch': 0.89}
89%|████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 604/681 [47:12<41:26, 32.29s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 605/681 [47:16<29:49, 23.55s/it] {'loss': 1.1539, 'grad_norm': 6.287881374359131, 'learning_rate': 1.9276432015946446e-08, 'rewards/chosen': -0.5785641074180603, 'rewards/rejected': -0.8597608208656311, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.2811967134475708, 'logps/chosen': -242.73826599121094, 'logps/rejected': -327.88031005859375, 'logps/ref_chosen': -94.77333068847656, 'logps/ref_rejected': -107.30490112304688, 'logits/chosen': -7.781911373138428, 'logits/rejected': -7.073343276977539, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.003921985626220703, 'kl/avg_steps': 0.46875, 'epoch': 0.89}
89%|████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 605/681 [47:16<29:49, 23.55s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 606/681 [47:18<21:39, 17.33s/it] {'loss': 1.1412, 'grad_norm': 6.576826572418213, 'learning_rate': 1.8785323298722093e-08, 'rewards/chosen': -0.5615659952163696, 'rewards/rejected': -0.8598177433013916, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.298251748085022, 'logps/chosen': -232.45693969726562, 'logps/rejected': -326.3280944824219, 'logps/ref_chosen': -87.7533950805664, 'logps/ref_rejected': -104.2422103881836, 'logits/chosen': -7.757646083831787, 'logits/rejected': -6.858328342437744, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.003903687233105302, 'kl/avg_steps': 0.65625, 'epoch': 0.89}
89%|████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 606/681 [47:18<21:39, 17.33s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 607/681 [47:21<16:06, 13.07s/it] {'loss': 1.1314, 'grad_norm': 6.907278060913086, 'learning_rate': 1.8300307303259904e-08, 'rewards/chosen': -0.5482698082923889, 'rewards/rejected': -0.8778215646743774, 'rewards/accuracies': 0.75, 'rewards/margins': 0.3295517861843109, 'logps/chosen': -229.9647216796875, 'logps/rejected': -314.5696716308594, 'logps/ref_chosen': -88.32904815673828, 'logps/ref_rejected': -86.76811218261719, 'logits/chosen': -7.828334808349609, 'logits/rejected': -6.711248397827148, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.0038782362826168537, 'kl/avg_steps': 0.5, 'epoch': 0.89}
89%|████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 607/681 [47:22<16:06, 13.07s/it] 89%|████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 608/681 [47:25<12:14, 10.06s/it] {'loss': 1.1084, 'grad_norm': 7.614426612854004, 'learning_rate': 1.7821396810182437e-08, 'rewards/chosen': -0.4974094033241272, 'rewards/rejected': -0.8393654823303223, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.34195607900619507, 'logps/chosen': -215.21920776367188, 'logps/rejected': -319.41949462890625, 'logps/ref_chosen': -85.76937103271484, 'logps/ref_rejected': -100.23281860351562, 'logits/chosen': -7.646144866943359, 'logits/rejected': -7.090588569641113, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.0038589416071772575, 'kl/avg_steps': 0.65625, 'epoch': 0.89}
89%|████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 608/681 [47:25<12:14, 10.06s/it] 89%|█████████████████████████████████████████████████████████████████████████████████████████████████████ | 609/681 [47:27<09:27, 7.88s/it] {'loss': 1.1163, 'grad_norm': 6.746026992797852, 'learning_rate': 1.7348604439226617e-08, 'rewards/chosen': -0.531781792640686, 'rewards/rejected': -0.8633979558944702, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.3316161632537842, 'logps/chosen': -232.28611755371094, 'logps/rejected': -322.77618408203125, 'logps/ref_chosen': -92.96656799316406, 'logps/ref_rejected': -95.91818237304688, 'logits/chosen': -7.802424907684326, 'logits/rejected': -7.451755523681641, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.0038337823934853077, 'kl/avg_steps': 0.625, 'epoch': 0.89}
89%|█████████████████████████████████████████████████████████████████████████████████████████████████████ | 609/681 [47:27<09:27, 7.88s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 610/681 [47:30<07:37, 6.44s/it] {'loss': 1.1011, 'grad_norm': 6.536015510559082, 'learning_rate': 1.6881942648911074e-08, 'rewards/chosen': -0.5073626637458801, 'rewards/rejected': -0.8506356477737427, 'rewards/accuracies': 0.875, 'rewards/margins': 0.3432729244232178, 'logps/chosen': -228.5191650390625, 'logps/rejected': -314.64813232421875, 'logps/ref_chosen': -94.70028686523438, 'logps/ref_rejected': -89.68739318847656, 'logits/chosen': -7.566076278686523, 'logits/rejected': -7.088051795959473, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.0038099701050668955, 'kl/avg_steps': 0.65625, 'epoch': 0.9}
90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 610/681 [47:30<07:37, 6.44s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 611/681 [47:33<06:14, 5.35s/it] {'loss': 1.1105, 'grad_norm': 7.131064414978027, 'learning_rate': 1.6421423736208e-08, 'rewards/chosen': -0.52030348777771, 'rewards/rejected': -0.8540895581245422, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.3337860703468323, 'logps/chosen': -224.7576446533203, 'logps/rejected': -316.98431396484375, 'logps/ref_chosen': -86.78334045410156, 'logps/ref_rejected': -89.84307861328125, 'logits/chosen': -7.605221748352051, 'logits/rejected': -7.5947184562683105, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.003785130102187395, 'kl/avg_steps': 0.5625, 'epoch': 0.9}
90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 611/681 [47:33<06:14, 5.35s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 612/681 [47:36<05:16, 4.58s/it] {'loss': 1.1023, 'grad_norm': 6.337128639221191, 'learning_rate': 1.5967059836219042e-08, 'rewards/chosen': -0.5628792643547058, 'rewards/rejected': -0.9013561010360718, 'rewards/accuracies': 0.90625, 'rewards/margins': 0.33847683668136597, 'logps/chosen': -251.38430786132812, 'logps/rejected': -335.0631103515625, 'logps/ref_chosen': -101.02015686035156, 'logps/ref_rejected': -93.78302764892578, 'logits/chosen': -7.775234699249268, 'logits/rejected': -6.849664688110352, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.003763957880437374, 'kl/avg_steps': 0.6875, 'epoch': 0.9}
90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 612/681 [47:36<05:16, 4.58s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 613/681 [47:39<04:39, 4.12s/it] {'loss': 1.0762, 'grad_norm': 6.324493885040283, 'learning_rate': 1.551886292185553e-08, 'rewards/chosen': -0.46113917231559753, 'rewards/rejected': -0.8275177478790283, 'rewards/accuracies': 0.921875, 'rewards/margins': 0.3663785457611084, 'logps/chosen': -213.39401245117188, 'logps/rejected': -333.4338073730469, 'logps/ref_chosen': -88.9886245727539, 'logps/ref_rejected': -109.99551391601562, 'logits/chosen': -8.078614234924316, 'logits/rejected': -7.592404365539551, 'kl/p_epsilon_steps': 0.953125, 'kl/n_epsilon_steps': 0.046875, 'kl/beta': 0.003738257335498929, 'kl/avg_steps': 0.90625, 'epoch': 0.9}
90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 613/681 [47:39<04:39, 4.12s/it] 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 614/681 [47:42<04:12, 3.77s/it] {'loss': 1.1461, 'grad_norm': 7.8051652908325195, 'learning_rate': 1.507684480352292e-08, 'rewards/chosen': -0.5560883283615112, 'rewards/rejected': -0.8495379686355591, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.29344964027404785, 'logps/chosen': -230.80239868164062, 'logps/rejected': -340.70416259765625, 'logps/ref_chosen': -80.20005798339844, 'logps/ref_rejected': -109.86239624023438, 'logits/chosen': -7.316326141357422, 'logits/rejected': -7.080374240875244, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0037046836223453283, 'kl/avg_steps': 0.5625, 'epoch': 0.9}
90%|█████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 614/681 [47:42<04:12, 3.77s/it] 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████ | 615/681 [47:45<03:56, 3.58s/it] {'loss': 1.1669, 'grad_norm': 6.5305585861206055, 'learning_rate': 1.4641017128809801e-08, 'rewards/chosen': -0.525856077671051, 'rewards/rejected': -0.7832653522491455, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.2574092149734497, 'logps/chosen': -243.8656463623047, 'logps/rejected': -315.3383483886719, 'logps/ref_chosen': -100.43526458740234, 'logps/ref_rejected': -101.1800537109375, 'logits/chosen': -7.641972541809082, 'logits/rejected': -6.816801071166992, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0036839614622294903, 'kl/avg_steps': 0.59375, 'epoch': 0.9}
90%|██████████████████████████████████████████████████████████████████████████████████████████████████████ | 615/681 [47:45<03:56, 3.58s/it] 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 616/681 [47:48<03:45, 3.47s/it] {'loss': 1.1124, 'grad_norm': 5.881459712982178, 'learning_rate': 1.4211391382180637e-08, 'rewards/chosen': -0.5267339944839478, 'rewards/rejected': -0.8600709438323975, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.3333369195461273, 'logps/chosen': -236.98391723632812, 'logps/rejected': -318.6792297363281, 'logps/ref_chosen': -92.49292755126953, 'logps/ref_rejected': -82.06065368652344, 'logits/chosen': -7.507961273193359, 'logits/rejected': -7.136435508728027, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.003662216942757368, 'kl/avg_steps': 0.625, 'epoch': 0.9}
90%|██████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 616/681 [47:48<03:45, 3.47s/it] 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 617/681 [47:52<03:38, 3.42s/it] {'loss': 1.1664, 'grad_norm': 6.169250011444092, 'learning_rate': 1.378797888467345e-08, 'rewards/chosen': -0.5471093654632568, 'rewards/rejected': -0.8117498755455017, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.26464053988456726, 'logps/chosen': -241.96456909179688, 'logps/rejected': -294.9502868652344, 'logps/ref_chosen': -91.09699249267578, 'logps/ref_rejected': -70.41004943847656, 'logits/chosen': -7.898214817047119, 'logits/rejected': -7.082752227783203, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0036394703201949596, 'kl/avg_steps': 0.59375, 'epoch': 0.91}
91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 617/681 [47:52<03:38, 3.42s/it] 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 618/681 [47:55<03:33, 3.39s/it] {'loss': 1.1462, 'grad_norm': 7.049670696258545, 'learning_rate': 1.3370790793601371e-08, 'rewards/chosen': -0.5415487289428711, 'rewards/rejected': -0.8314208388328552, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.2898721694946289, 'logps/chosen': -252.33189392089844, 'logps/rejected': -331.26397705078125, 'logps/ref_chosen': -102.02059936523438, 'logps/ref_rejected': -99.80119323730469, 'logits/chosen': -7.613478660583496, 'logits/rejected': -7.167911052703857, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.003617988433688879, 'kl/avg_steps': 0.5625, 'epoch': 0.91}
91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 618/681 [47:55<03:33, 3.39s/it] 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 619/681 [47:58<03:24, 3.30s/it] {'loss': 1.1708, 'grad_norm': 6.142784118652344, 'learning_rate': 1.2959838102258535e-08, 'rewards/chosen': -0.5201643705368042, 'rewards/rejected': -0.7863855361938477, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.26622116565704346, 'logps/chosen': -234.5894775390625, 'logps/rejected': -319.714111328125, 'logps/ref_chosen': -89.74136352539062, 'logps/ref_rejected': -99.90138244628906, 'logits/chosen': -7.600045204162598, 'logits/rejected': -7.366483688354492, 'kl/p_epsilon_steps': 0.6875, 'kl/n_epsilon_steps': 0.3125, 'kl/beta': 0.003597751259803772, 'kl/avg_steps': 0.375, 'epoch': 0.91}
91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 619/681 [47:58<03:24, 3.30s/it] 91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 620/681 [48:01<03:15, 3.20s/it] {'loss': 1.1526, 'grad_norm': 5.838797092437744, 'learning_rate': 1.2555131639630567e-08, 'rewards/chosen': -0.5036029815673828, 'rewards/rejected': -0.7929166555404663, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.28931373357772827, 'logps/chosen': -225.7647247314453, 'logps/rejected': -307.8377380371094, 'logps/ref_chosen': -85.12431335449219, 'logps/ref_rejected': -85.41253662109375, 'logits/chosen': -8.057722091674805, 'logits/rejected': -7.399764060974121, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.003584309946745634, 'kl/avg_steps': 0.40625, 'epoch': 0.91}
91%|██████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 620/681 [48:01<03:15, 3.20s/it] 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████ | 621/681 [48:04<03:10, 3.17s/it] {'loss': 1.1604, 'grad_norm': 5.572709083557129, 'learning_rate': 1.2156682070109086e-08, 'rewards/chosen': -0.49918726086616516, 'rewards/rejected': -0.7818341851234436, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.28264692425727844, 'logps/chosen': -229.43209838867188, 'logps/rejected': -315.90655517578125, 'logps/ref_chosen': -89.24842071533203, 'logps/ref_rejected': -95.46463775634766, 'logits/chosen': -7.758251190185547, 'logits/rejected': -7.50982666015625, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.0035698076244443655, 'kl/avg_steps': 0.46875, 'epoch': 0.91}
91%|███████████████████████████████████████████████████████████████████████████████████████████████████████ | 621/681 [48:04<03:10, 3.17s/it] 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 622/681 [48:07<03:04, 3.13s/it] {'loss': 1.1355, 'grad_norm': 6.386410713195801, 'learning_rate': 1.1764499893210878e-08, 'rewards/chosen': -0.5042383670806885, 'rewards/rejected': -0.8008482456207275, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.29660987854003906, 'logps/chosen': -242.18194580078125, 'logps/rejected': -317.67083740234375, 'logps/ref_chosen': -99.79413604736328, 'logps/ref_rejected': -90.82821655273438, 'logits/chosen': -7.883831977844238, 'logits/rejected': -6.893963813781738, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0035531523171812296, 'kl/avg_steps': 0.5625, 'epoch': 0.91}
91%|███████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 622/681 [48:07<03:04, 3.13s/it] 91%|███████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 623/681 [48:10<02:51, 2.96s/it] {'loss': 1.2219, 'grad_norm': 7.438137531280518, 'learning_rate': 1.1378595443300998e-08, 'rewards/chosen': -0.5164909958839417, 'rewards/rejected': -0.7255691289901733, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.20907816290855408, 'logps/chosen': -236.72573852539062, 'logps/rejected': -297.7119445800781, 'logps/ref_chosen': -90.45555114746094, 'logps/ref_rejected': -91.32276916503906, 'logits/chosen': -7.771790504455566, 'logits/rejected': -7.093353748321533, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.0035332776606082916, 'kl/avg_steps': 0.40625, 'epoch': 0.91}
91%|███████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 623/681 [48:10<02:51, 2.96s/it] 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 624/681 [48:13<02:50, 2.99s/it] {'loss': 1.1213, 'grad_norm': 6.896797180175781, 'learning_rate': 1.0998978889320582e-08, 'rewards/chosen': -0.5218902826309204, 'rewards/rejected': -0.8390946388244629, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.3172043263912201, 'logps/chosen': -258.7445373535156, 'logps/rejected': -344.84759521484375, 'logps/ref_chosen': -109.87522888183594, 'logps/ref_rejected': -104.77320861816406, 'logits/chosen': -8.146064758300781, 'logits/rejected': -7.289480209350586, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0035189816262573004, 'kl/avg_steps': 0.5625, 'epoch': 0.92}
92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 624/681 [48:13<02:50, 2.99s/it] 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 625/681 [48:16<02:52, 3.08s/it] {'loss': 1.1839, 'grad_norm': 5.683692455291748, 'learning_rate': 1.0625660234518913e-08, 'rewards/chosen': -0.5464307069778442, 'rewards/rejected': -0.786613941192627, 'rewards/accuracies': 0.75, 'rewards/margins': 0.24018320441246033, 'logps/chosen': -243.88046264648438, 'logps/rejected': -318.0606994628906, 'logps/ref_chosen': -87.16815948486328, 'logps/ref_rejected': -91.86148071289062, 'logits/chosen': -7.770914554595947, 'logits/rejected': -7.289626121520996, 'kl/p_epsilon_steps': 0.75, 'kl/n_epsilon_steps': 0.25, 'kl/beta': 0.0034992981236428022, 'kl/avg_steps': 0.5, 'epoch': 0.92}
92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 625/681 [48:16<02:52, 3.08s/it] 92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 626/681 [48:19<02:54, 3.17s/it] {'loss': 1.2005, 'grad_norm': 7.084156513214111, 'learning_rate': 1.0258649316189721e-08, 'rewards/chosen': -0.5199881792068481, 'rewards/rejected': -0.7504914402961731, 'rewards/accuracies': 0.75, 'rewards/margins': 0.23050320148468018, 'logps/chosen': -254.01075744628906, 'logps/rejected': -321.1068115234375, 'logps/ref_chosen': -104.22421264648438, 'logps/ref_rejected': -104.1774673461914, 'logits/chosen': -7.743939399719238, 'logits/rejected': -7.482439994812012, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.0034818886779248714, 'kl/avg_steps': 0.40625, 'epoch': 0.92}
92%|███████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 626/681 [48:19<02:54, 3.17s/it] 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████ | 627/681 [48:23<02:49, 3.15s/it] {'loss': 1.1507, 'grad_norm': 5.33841609954834, 'learning_rate': 9.897955805412e-09, 'rewards/chosen': -0.46203625202178955, 'rewards/rejected': -0.7497316002845764, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.2876952886581421, 'logps/chosen': -208.89431762695312, 'logps/rejected': -330.51983642578125, 'logps/ref_chosen': -74.93461608886719, 'logps/ref_rejected': -112.57289123535156, 'logits/chosen': -7.591939926147461, 'logits/rejected': -7.3537187576293945, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.0034678007941693068, 'kl/avg_steps': 0.625, 'epoch': 0.92}
92%|████████████████████████████████████████████████████████████████████████████████████████████████████████ | 627/681 [48:23<02:49, 3.15s/it] 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 628/681 [48:26<02:45, 3.12s/it] {'loss': 1.1251, 'grad_norm': 6.301426410675049, 'learning_rate': 9.543589206795238e-09, 'rewards/chosen': -0.4717589020729065, 'rewards/rejected': -0.7812816500663757, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.30952274799346924, 'logps/chosen': -231.39364624023438, 'logps/rejected': -335.9172668457031, 'logps/ref_chosen': -93.69107818603516, 'logps/ref_rejected': -107.34395599365234, 'logits/chosen': -7.962158679962158, 'logits/rejected': -7.421542644500732, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.0034462616313248873, 'kl/avg_steps': 0.6875, 'epoch': 0.92}
92%|████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 628/681 [48:26<02:45, 3.12s/it] 92%|████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 629/681 [48:29<02:41, 3.10s/it] {'loss': 1.1605, 'grad_norm': 5.8255295753479, 'learning_rate': 9.19555885822887e-09, 'rewards/chosen': -0.5139710903167725, 'rewards/rejected': -0.7815043926239014, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.2675333023071289, 'logps/chosen': -253.8419952392578, 'logps/rejected': -326.94293212890625, 'logps/ref_chosen': -103.23037719726562, 'logps/ref_rejected': -97.16841888427734, 'logits/chosen': -7.680126190185547, 'logits/rejected': -7.379844665527344, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0034227303694933653, 'kl/avg_steps': 0.53125, 'epoch': 0.92}
92%|████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 629/681 [48:29<02:41, 3.10s/it] 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 630/681 [48:32<02:38, 3.10s/it] {'loss': 1.2339, 'grad_norm': 7.221403121948242, 'learning_rate': 8.85387393063622e-09, 'rewards/chosen': -0.4714937210083008, 'rewards/rejected': -0.6566429138183594, 'rewards/accuracies': 0.75, 'rewards/margins': 0.1851492077112198, 'logps/chosen': -232.68309020996094, 'logps/rejected': -283.4253234863281, 'logps/ref_chosen': -93.89755249023438, 'logps/ref_rejected': -89.3743896484375, 'logits/chosen': -7.831258773803711, 'logits/rejected': -7.15199089050293, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.003404643153771758, 'kl/avg_steps': 0.46875, 'epoch': 0.93}
93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 630/681 [48:32<02:38, 3.10s/it] 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 631/681 [48:35<02:32, 3.05s/it] {'loss': 1.1612, 'grad_norm': 5.580920219421387, 'learning_rate': 8.518543427732949e-09, 'rewards/chosen': -0.4492769241333008, 'rewards/rejected': -0.7134263515472412, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.2641494572162628, 'logps/chosen': -220.7677764892578, 'logps/rejected': -300.57403564453125, 'logps/ref_chosen': -87.77082061767578, 'logps/ref_rejected': -88.68241882324219, 'logits/chosen': -7.815197944641113, 'logits/rejected': -7.3377485275268555, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.003388758283108473, 'kl/avg_steps': 0.53125, 'epoch': 0.93}
93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 631/681 [48:35<02:32, 3.05s/it] 93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 632/681 [48:38<02:27, 3.00s/it] {'loss': 1.1516, 'grad_norm': 6.467636585235596, 'learning_rate': 8.189576185789637e-09, 'rewards/chosen': -0.44607216119766235, 'rewards/rejected': -0.7322953343391418, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.2862231731414795, 'logps/chosen': -221.4307861328125, 'logps/rejected': -310.26220703125, 'logps/ref_chosen': -88.62652587890625, 'logps/ref_rejected': -91.45091247558594, 'logits/chosen': -7.605412483215332, 'logits/rejected': -7.2681169509887695, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0033708508126437664, 'kl/avg_steps': 0.53125, 'epoch': 0.93}
93%|████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 632/681 [48:38<02:27, 3.00s/it] 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████ | 633/681 [48:41<02:27, 3.07s/it] {'loss': 1.2589, 'grad_norm': 6.414995193481445, 'learning_rate': 7.866980873399015e-09, 'rewards/chosen': -0.5550001859664917, 'rewards/rejected': -0.7260842323303223, 'rewards/accuracies': 0.671875, 'rewards/margins': 0.17108407616615295, 'logps/chosen': -247.08786010742188, 'logps/rejected': -316.35614013671875, 'logps/ref_chosen': -81.37442016601562, 'logps/ref_rejected': -98.62571716308594, 'logits/chosen': -7.696202278137207, 'logits/rejected': -7.264307498931885, 'kl/p_epsilon_steps': 0.640625, 'kl/n_epsilon_steps': 0.359375, 'kl/beta': 0.00335303763858974, 'kl/avg_steps': 0.28125, 'epoch': 0.93}
93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████ | 633/681 [48:41<02:27, 3.07s/it] 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 634/681 [48:44<02:26, 3.12s/it] {'loss': 1.1935, 'grad_norm': 6.248257637023926, 'learning_rate': 7.550765991247654e-09, 'rewards/chosen': -0.5128281712532043, 'rewards/rejected': -0.7364822030067444, 'rewards/accuracies': 0.75, 'rewards/margins': 0.22365406155586243, 'logps/chosen': -250.07144165039062, 'logps/rejected': -334.539306640625, 'logps/ref_chosen': -96.12284851074219, 'logps/ref_rejected': -112.84780883789062, 'logits/chosen': -7.881155490875244, 'logits/rejected': -7.148515701293945, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0033436338417232037, 'kl/avg_steps': 0.53125, 'epoch': 0.93}
93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 634/681 [48:44<02:26, 3.12s/it] 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 635/681 [48:47<02:20, 3.06s/it] {'loss': 1.2457, 'grad_norm': 6.869739055633545, 'learning_rate': 7.240939871891699e-09, 'rewards/chosen': -0.5283612012863159, 'rewards/rejected': -0.7009052634239197, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.17254400253295898, 'logps/chosen': -258.06280517578125, 'logps/rejected': -301.9739990234375, 'logps/ref_chosen': -98.68411254882812, 'logps/ref_rejected': -89.8991928100586, 'logits/chosen': -7.332803249359131, 'logits/rejected': -6.858902931213379, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.003325964557006955, 'kl/avg_steps': 0.46875, 'epoch': 0.93}
93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 635/681 [48:47<02:20, 3.06s/it] 93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 636/681 [48:50<02:19, 3.10s/it] {'loss': 1.1499, 'grad_norm': 5.351519584655762, 'learning_rate': 6.937510679537628e-09, 'rewards/chosen': -0.4755138158798218, 'rewards/rejected': -0.7524248361587524, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.27691105008125305, 'logps/chosen': -234.77163696289062, 'logps/rejected': -316.6680603027344, 'logps/ref_chosen': -90.41796112060547, 'logps/ref_rejected': -87.70687866210938, 'logits/chosen': -7.650508880615234, 'logits/rejected': -6.78857946395874, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.003310446860268712, 'kl/avg_steps': 0.625, 'epoch': 0.93}
93%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 636/681 [48:50<02:19, 3.10s/it] 94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 637/681 [48:53<02:15, 3.09s/it] {'loss': 1.1395, 'grad_norm': 5.5197577476501465, 'learning_rate': 6.640486409826785e-09, 'rewards/chosen': -0.4761905372142792, 'rewards/rejected': -0.7680450677871704, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.29185453057289124, 'logps/chosen': -227.92446899414062, 'logps/rejected': -339.26043701171875, 'logps/ref_chosen': -82.44971466064453, 'logps/ref_rejected': -104.02860260009766, 'logits/chosen': -7.380791664123535, 'logits/rejected': -7.242560386657715, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0032898851204663515, 'kl/avg_steps': 0.59375, 'epoch': 0.94}
94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 637/681 [48:53<02:15, 3.09s/it] 94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 638/681 [48:56<02:11, 3.06s/it] {'loss': 1.1318, 'grad_norm': 5.237280368804932, 'learning_rate': 6.349874889624962e-09, 'rewards/chosen': -0.43678027391433716, 'rewards/rejected': -0.7282345294952393, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.2914542555809021, 'logps/chosen': -226.06134033203125, 'logps/rejected': -310.4959716796875, 'logps/ref_chosen': -91.92498779296875, 'logps/ref_rejected': -86.28703308105469, 'logits/chosen': -7.648863792419434, 'logits/rejected': -6.924067497253418, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0032704665791243315, 'kl/avg_steps': 0.59375, 'epoch': 0.94}
94%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 638/681 [48:56<02:11, 3.06s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████ | 639/681 [48:59<02:09, 3.09s/it] {'loss': 1.2695, 'grad_norm': 5.765613079071045, 'learning_rate': 6.065683776815933e-09, 'rewards/chosen': -0.5761454105377197, 'rewards/rejected': -0.7252421975135803, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.1490968018770218, 'logps/chosen': -281.9426574707031, 'logps/rejected': -305.64599609375, 'logps/ref_chosen': -104.52755737304688, 'logps/ref_rejected': -81.4803466796875, 'logits/chosen': -7.779665470123291, 'logits/rejected': -6.977321624755859, 'kl/p_epsilon_steps': 0.671875, 'kl/n_epsilon_steps': 0.328125, 'kl/beta': 0.003251162823289633, 'kl/avg_steps': 0.34375, 'epoch': 0.94}
94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████ | 639/681 [48:59<02:09, 3.09s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 640/681 [49:02<02:06, 3.09s/it] {'loss': 1.1573, 'grad_norm': 5.791350364685059, 'learning_rate': 5.7879205600998296e-09, 'rewards/chosen': -0.47469913959503174, 'rewards/rejected': -0.7432272434234619, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.26852816343307495, 'logps/chosen': -245.22933959960938, 'logps/rejected': -343.93792724609375, 'logps/ref_chosen': -97.88526916503906, 'logps/ref_rejected': -112.70501708984375, 'logits/chosen': -7.959763526916504, 'logits/rejected': -7.383049964904785, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.00324002536945045, 'kl/avg_steps': 0.71875, 'epoch': 0.94}
94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 640/681 [49:02<02:06, 3.09s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 641/681 [49:06<02:05, 3.14s/it] {'loss': 1.1614, 'grad_norm': 5.238857269287109, 'learning_rate': 5.516592558795746e-09, 'rewards/chosen': -0.4384271800518036, 'rewards/rejected': -0.7108187079429626, 'rewards/accuracies': 0.71875, 'rewards/margins': 0.27239149808883667, 'logps/chosen': -232.91995239257812, 'logps/rejected': -317.35809326171875, 'logps/ref_chosen': -96.4456787109375, 'logps/ref_rejected': -95.13568878173828, 'logits/chosen': -7.708554267883301, 'logits/rejected': -7.541186332702637, 'kl/p_epsilon_steps': 0.65625, 'kl/n_epsilon_steps': 0.34375, 'kl/beta': 0.0032169038895517588, 'kl/avg_steps': 0.3125, 'epoch': 0.94}
94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 641/681 [49:06<02:05, 3.14s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 642/681 [49:09<02:01, 3.12s/it] {'loss': 1.1825, 'grad_norm': 5.874085903167725, 'learning_rate': 5.251706922648868e-09, 'rewards/chosen': -0.497159868478775, 'rewards/rejected': -0.7433220148086548, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.24616217613220215, 'logps/chosen': -256.21673583984375, 'logps/rejected': -347.91558837890625, 'logps/ref_chosen': -100.75984954833984, 'logps/ref_rejected': -114.70763397216797, 'logits/chosen': -7.545126914978027, 'logits/rejected': -7.102571487426758, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.003206882392987609, 'kl/avg_steps': 0.53125, 'epoch': 0.94}
94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 642/681 [49:09<02:01, 3.12s/it] 94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 643/681 [49:12<01:59, 3.14s/it] {'loss': 1.1822, 'grad_norm': 6.670008659362793, 'learning_rate': 4.993270631642038e-09, 'rewards/chosen': -0.4609188437461853, 'rewards/rejected': -0.7005389928817749, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.2396201640367508, 'logps/chosen': -229.69638061523438, 'logps/rejected': -315.35858154296875, 'logps/ref_chosen': -84.74365997314453, 'logps/ref_rejected': -94.31842041015625, 'logits/chosen': -7.875416278839111, 'logits/rejected': -7.243396759033203, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.003189935814589262, 'kl/avg_steps': 0.5625, 'epoch': 0.94}
94%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 643/681 [49:12<01:59, 3.14s/it] 95%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 644/681 [49:15<01:56, 3.14s/it] {'loss': 1.2111, 'grad_norm': 6.133624076843262, 'learning_rate': 4.741290495811873e-09, 'rewards/chosen': -0.45997491478919983, 'rewards/rejected': -0.6619677543640137, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.20199283957481384, 'logps/chosen': -230.85415649414062, 'logps/rejected': -304.661865234375, 'logps/ref_chosen': -85.32275390625, 'logps/ref_rejected': -94.60861206054688, 'logits/chosen': -7.913561820983887, 'logits/rejected': -7.103449821472168, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0031720928382128477, 'kl/avg_steps': 0.5625, 'epoch': 0.95}
95%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 644/681 [49:15<01:56, 3.14s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 645/681 [49:18<01:52, 3.12s/it] {'loss': 1.2771, 'grad_norm': 5.844942092895508, 'learning_rate': 4.495773155069299e-09, 'rewards/chosen': -0.5023009777069092, 'rewards/rejected': -0.6405566930770874, 'rewards/accuracies': 0.625, 'rewards/margins': 0.13825571537017822, 'logps/chosen': -241.77206420898438, 'logps/rejected': -306.9005432128906, 'logps/ref_chosen': -82.59024047851562, 'logps/ref_rejected': -103.00375366210938, 'logits/chosen': -7.571622371673584, 'logits/rejected': -7.209532737731934, 'kl/p_epsilon_steps': 0.59375, 'kl/n_epsilon_steps': 0.40625, 'kl/beta': 0.0031543495133519173, 'kl/avg_steps': 0.1875, 'epoch': 0.95}
95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 645/681 [49:18<01:52, 3.12s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 646/681 [49:21<01:46, 3.05s/it] {'loss': 1.1663, 'grad_norm': 5.194578647613525, 'learning_rate': 4.256725079024553e-09, 'rewards/chosen': -0.43383896350860596, 'rewards/rejected': -0.6858773231506348, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.2520383596420288, 'logps/chosen': -229.54640197753906, 'logps/rejected': -303.084716796875, 'logps/ref_chosen': -91.1920394897461, 'logps/ref_rejected': -83.77833557128906, 'logits/chosen': -7.935370922088623, 'logits/rejected': -7.05921745300293, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0031484460923820734, 'kl/avg_steps': 0.59375, 'epoch': 0.95}
95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 646/681 [49:21<01:46, 3.05s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 647/681 [49:24<01:46, 3.12s/it] {'loss': 1.2089, 'grad_norm': 5.19912576675415, 'learning_rate': 4.024152566816791e-09, 'rewards/chosen': -0.47964316606521606, 'rewards/rejected': -0.694766104221344, 'rewards/accuracies': 0.703125, 'rewards/margins': 0.21512295305728912, 'logps/chosen': -242.48236083984375, 'logps/rejected': -322.8554382324219, 'logps/ref_chosen': -88.84446716308594, 'logps/ref_rejected': -99.49832916259766, 'logits/chosen': -7.209627151489258, 'logits/rejected': -7.173065185546875, 'kl/p_epsilon_steps': 0.703125, 'kl/n_epsilon_steps': 0.296875, 'kl/beta': 0.00312986271455884, 'kl/avg_steps': 0.40625, 'epoch': 0.95}
95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 647/681 [49:24<01:46, 3.12s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 648/681 [49:27<01:41, 3.07s/it] {'loss': 1.127, 'grad_norm': 5.339207172393799, 'learning_rate': 3.798061746947995e-09, 'rewards/chosen': -0.4329220652580261, 'rewards/rejected': -0.7390461564064026, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.30612409114837646, 'logps/chosen': -227.42025756835938, 'logps/rejected': -343.5663146972656, 'logps/ref_chosen': -87.84810638427734, 'logps/ref_rejected': -104.67005920410156, 'logits/chosen': -7.534468173980713, 'logits/rejected': -7.395105838775635, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.003117199055850506, 'kl/avg_steps': 0.6875, 'epoch': 0.95}
95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 648/681 [49:27<01:41, 3.07s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 649/681 [49:31<01:41, 3.17s/it] {'loss': 1.1676, 'grad_norm': 5.596925258636475, 'learning_rate': 3.5784585771215235e-09, 'rewards/chosen': -0.39361506700515747, 'rewards/rejected': -0.6433808207511902, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.24976572394371033, 'logps/chosen': -217.3938446044922, 'logps/rejected': -297.97711181640625, 'logps/ref_chosen': -89.6925048828125, 'logps/ref_rejected': -88.70658111572266, 'logits/chosen': -7.8632001876831055, 'logits/rejected': -7.287814140319824, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0030959146097302437, 'kl/avg_steps': 0.5625, 'epoch': 0.95}
95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 649/681 [49:31<01:41, 3.17s/it] 95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 650/681 [49:34<01:35, 3.08s/it] {'loss': 1.1869, 'grad_norm': 5.370362281799316, 'learning_rate': 3.3653488440851253e-09, 'rewards/chosen': -0.463270366191864, 'rewards/rejected': -0.7010980844497681, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.23782768845558167, 'logps/chosen': -241.04432678222656, 'logps/rejected': -332.0096435546875, 'logps/ref_chosen': -89.93060302734375, 'logps/ref_rejected': -102.61282348632812, 'logits/chosen': -7.673471450805664, 'logits/rejected': -7.417351722717285, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.0030785975977778435, 'kl/avg_steps': 0.625, 'epoch': 0.95}
95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 650/681 [49:34<01:35, 3.08s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 651/681 [49:37<01:32, 3.07s/it] {'loss': 1.1412, 'grad_norm': 5.453701972961426, 'learning_rate': 3.158738163478475e-09, 'rewards/chosen': -0.3932613134384155, 'rewards/rejected': -0.675163745880127, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.2819024920463562, 'logps/chosen': -208.45730590820312, 'logps/rejected': -328.37841796875, 'logps/ref_chosen': -79.18731689453125, 'logps/ref_rejected': -105.93333435058594, 'logits/chosen': -7.594956398010254, 'logits/rejected': -6.981936454772949, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.0030594756826758385, 'kl/avg_steps': 0.65625, 'epoch': 0.96}
96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 651/681 [49:37<01:32, 3.07s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 652/681 [49:40<01:28, 3.06s/it] {'loss': 1.1713, 'grad_norm': 4.904377460479736, 'learning_rate': 2.9586319796851555e-09, 'rewards/chosen': -0.40263664722442627, 'rewards/rejected': -0.6452191472053528, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.24258247017860413, 'logps/chosen': -234.99612426757812, 'logps/rejected': -330.2041015625, 'logps/ref_chosen': -101.79022979736328, 'logps/ref_rejected': -116.3245849609375, 'logits/chosen': -7.993418216705322, 'logits/rejected': -7.6948065757751465, 'kl/p_epsilon_steps': 0.84375, 'kl/n_epsilon_steps': 0.15625, 'kl/beta': 0.003039528848603368, 'kl/avg_steps': 0.6875, 'epoch': 0.96}
96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 652/681 [49:40<01:28, 3.06s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 653/681 [49:43<01:25, 3.04s/it] {'loss': 1.1885, 'grad_norm': 5.360039234161377, 'learning_rate': 2.7650355656892166e-09, 'rewards/chosen': -0.45035725831985474, 'rewards/rejected': -0.678409218788147, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.22805194556713104, 'logps/chosen': -243.21240234375, 'logps/rejected': -335.51324462890625, 'logps/ref_chosen': -93.35359191894531, 'logps/ref_rejected': -109.12324523925781, 'logits/chosen': -7.770810127258301, 'logits/rejected': -7.401587963104248, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.003018774790689349, 'kl/avg_steps': 0.59375, 'epoch': 0.96}
96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 653/681 [49:43<01:25, 3.04s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 654/681 [49:46<01:22, 3.04s/it] {'loss': 1.2262, 'grad_norm': 7.331425666809082, 'learning_rate': 2.577954022936174e-09, 'rewards/chosen': -0.45708924531936646, 'rewards/rejected': -0.6516642570495605, 'rewards/accuracies': 0.75, 'rewards/margins': 0.1945749968290329, 'logps/chosen': -241.98892211914062, 'logps/rejected': -323.5856628417969, 'logps/ref_chosen': -89.11553955078125, 'logps/ref_rejected': -104.91995239257812, 'logits/chosen': -7.537060260772705, 'logits/rejected': -7.259842872619629, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.003000956494361162, 'kl/avg_steps': 0.53125, 'epoch': 0.96}
96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 654/681 [49:46<01:22, 3.04s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 655/681 [49:49<01:19, 3.05s/it] {'loss': 1.1856, 'grad_norm': 5.144998550415039, 'learning_rate': 2.397392281198729e-09, 'rewards/chosen': -0.4156780540943146, 'rewards/rejected': -0.65021151304245, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.23453345894813538, 'logps/chosen': -220.708984375, 'logps/rejected': -322.0316162109375, 'logps/ref_chosen': -81.03610229492188, 'logps/ref_rejected': -102.80233764648438, 'logits/chosen': -7.6725006103515625, 'logits/rejected': -7.398637771606445, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0029850981663912535, 'kl/avg_steps': 0.53125, 'epoch': 0.96}
96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 655/681 [49:49<01:19, 3.05s/it] 96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 656/681 [49:52<01:19, 3.18s/it] {'loss': 1.1311, 'grad_norm': 5.95790958404541, 'learning_rate': 2.223355098446622e-09, 'rewards/chosen': -0.4093048572540283, 'rewards/rejected': -0.7035213708877563, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.294216513633728, 'logps/chosen': -223.98876953125, 'logps/rejected': -357.14013671875, 'logps/ref_chosen': -85.32534790039062, 'logps/ref_rejected': -118.33866882324219, 'logits/chosen': -7.680576324462891, 'logits/rejected': -7.717093467712402, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.00296932365745306, 'kl/avg_steps': 0.71875, 'epoch': 0.96}
96%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 656/681 [49:52<01:19, 3.18s/it] 96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 657/681 [49:55<01:13, 3.07s/it] {'loss': 1.1867, 'grad_norm': 6.30833625793457, 'learning_rate': 2.055847060721566e-09, 'rewards/chosen': -0.4302343726158142, 'rewards/rejected': -0.666432797908783, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.23619841039180756, 'logps/chosen': -226.81549072265625, 'logps/rejected': -330.3614501953125, 'logps/ref_chosen': -80.19772338867188, 'logps/ref_rejected': -102.581298828125, 'logits/chosen': -7.752594947814941, 'logits/rejected': -7.609277725219727, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.002948133973404765, 'kl/avg_steps': 0.59375, 'epoch': 0.96}
96%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 657/681 [49:55<01:13, 3.07s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 658/681 [49:58<01:08, 2.99s/it] {'loss': 1.2098, 'grad_norm': 5.188457012176514, 'learning_rate': 1.8948725820160662e-09, 'rewards/chosen': -0.44880080223083496, 'rewards/rejected': -0.6514099836349487, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.202609121799469, 'logps/chosen': -248.58145141601562, 'logps/rejected': -325.5663146972656, 'logps/ref_chosen': -94.634521484375, 'logps/ref_rejected': -101.63162231445312, 'logits/chosen': -7.566948890686035, 'logits/rejected': -7.049898147583008, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.002930732909590006, 'kl/avg_steps': 0.65625, 'epoch': 0.97}
97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 658/681 [49:58<01:08, 2.99s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 659/681 [50:01<01:06, 3.01s/it] {'loss': 1.1774, 'grad_norm': 4.891656875610352, 'learning_rate': 1.7404359041573723e-09, 'rewards/chosen': -0.3891860246658325, 'rewards/rejected': -0.6415092945098877, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.25232329964637756, 'logps/chosen': -246.71299743652344, 'logps/rejected': -315.93914794921875, 'logps/ref_chosen': -112.55587005615234, 'logps/ref_rejected': -93.9216079711914, 'logits/chosen': -8.355998039245605, 'logits/rejected': -7.4629597663879395, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0029116251971572638, 'kl/avg_steps': 0.5625, 'epoch': 0.97}
97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 659/681 [50:01<01:06, 3.01s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 660/681 [50:04<01:02, 2.98s/it] {'loss': 1.1675, 'grad_norm': 7.11164665222168, 'learning_rate': 1.592541096695571e-09, 'rewards/chosen': -0.4076111614704132, 'rewards/rejected': -0.6608085632324219, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.25319740176200867, 'logps/chosen': -234.69586181640625, 'logps/rejected': -311.2170715332031, 'logps/ref_chosen': -93.37742614746094, 'logps/ref_rejected': -81.39482116699219, 'logits/chosen': -7.641197681427002, 'logits/rejected': -7.153841972351074, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0028953389264643192, 'kl/avg_steps': 0.59375, 'epoch': 0.97}
97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 660/681 [50:04<01:02, 2.98s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 661/681 [50:06<00:57, 2.86s/it] {'loss': 1.1572, 'grad_norm': 4.977914333343506, 'learning_rate': 1.4511920567963908e-09, 'rewards/chosen': -0.41780516505241394, 'rewards/rejected': -0.6864430904388428, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.26863789558410645, 'logps/chosen': -233.6283416748047, 'logps/rejected': -332.63751220703125, 'logps/ref_chosen': -87.85516357421875, 'logps/ref_rejected': -92.40330505371094, 'logits/chosen': -7.581234931945801, 'logits/rejected': -6.8354692459106445, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.002878249390050769, 'kl/avg_steps': 0.65625, 'epoch': 0.97}
97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 661/681 [50:06<00:57, 2.86s/it] 97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 662/681 [50:10<00:57, 3.02s/it] {'loss': 1.1704, 'grad_norm': 4.641286373138428, 'learning_rate': 1.3163925091384532e-09, 'rewards/chosen': -0.4002068340778351, 'rewards/rejected': -0.6452619433403015, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.24505510926246643, 'logps/chosen': -243.25115966796875, 'logps/rejected': -322.36285400390625, 'logps/ref_chosen': -102.77980041503906, 'logps/ref_rejected': -95.22531127929688, 'logits/chosen': -7.912067413330078, 'logits/rejected': -7.321425437927246, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.002859483938664198, 'kl/avg_steps': 0.59375, 'epoch': 0.97}
97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 662/681 [50:10<00:57, 3.02s/it] 97%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 663/681 [50:13<00:56, 3.14s/it] {'loss': 1.1945, 'grad_norm': 5.062544822692871, 'learning_rate': 1.1881460058152382e-09, 'rewards/chosen': -0.39547061920166016, 'rewards/rejected': -0.6204517483711243, 'rewards/accuracies': 0.75, 'rewards/margins': 0.2249811291694641, 'logps/chosen': -235.74203491210938, 'logps/rejected': -340.11309814453125, 'logps/ref_chosen': -96.34658813476562, 'logps/ref_rejected': -120.52645111083984, 'logits/chosen': -7.826132774353027, 'logits/rejected': -7.58699369430542, 'kl/p_epsilon_steps': 0.734375, 'kl/n_epsilon_steps': 0.265625, 'kl/beta': 0.002842606045305729, 'kl/avg_steps': 0.46875, 'epoch': 0.97}
97%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 663/681 [50:13<00:56, 3.14s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 664/681 [50:17<00:54, 3.20s/it] {'loss': 1.2061, 'grad_norm': 5.462507724761963, 'learning_rate': 1.066455926241383e-09, 'rewards/chosen': -0.43186432123184204, 'rewards/rejected': -0.6376165151596069, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.20575222373008728, 'logps/chosen': -245.00120544433594, 'logps/rejected': -338.63543701171875, 'logps/ref_chosen': -91.84242248535156, 'logps/ref_rejected': -111.83668518066406, 'logits/chosen': -7.908196449279785, 'logits/rejected': -7.449959754943848, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.002829343546181917, 'kl/avg_steps': 0.53125, 'epoch': 0.98}
98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 664/681 [50:17<00:54, 3.20s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 665/681 [50:19<00:49, 3.07s/it] {'loss': 1.1848, 'grad_norm': 4.550708293914795, 'learning_rate': 9.513254770636137e-10, 'rewards/chosen': -0.36321356892585754, 'rewards/rejected': -0.5911097526550293, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.22789615392684937, 'logps/chosen': -217.97030639648438, 'logps/rejected': -303.5909423828125, 'logps/ref_chosen': -88.18618774414062, 'logps/ref_rejected': -91.9120101928711, 'logits/chosen': -7.470752716064453, 'logits/rejected': -7.1154465675354, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.0028143920935690403, 'kl/avg_steps': 0.71875, 'epoch': 0.98}
98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 665/681 [50:19<00:49, 3.07s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 666/681 [50:23<00:46, 3.11s/it] {'loss': 1.2025, 'grad_norm': 6.450153350830078, 'learning_rate': 8.427576920763956e-10, 'rewards/chosen': -0.41387584805488586, 'rewards/rejected': -0.628452479839325, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.2145766168832779, 'logps/chosen': -249.63475036621094, 'logps/rejected': -327.71697998046875, 'logps/ref_chosen': -100.97460174560547, 'logps/ref_rejected': -101.24992370605469, 'logits/chosen': -7.702958583831787, 'logits/rejected': -7.515384674072266, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.002794307889416814, 'kl/avg_steps': 0.53125, 'epoch': 0.98}
98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 666/681 [50:23<00:46, 3.11s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 667/681 [50:26<00:43, 3.13s/it] {'loss': 1.1737, 'grad_norm': 4.641995429992676, 'learning_rate': 7.407554321417764e-10, 'rewards/chosen': -0.4399784505367279, 'rewards/rejected': -0.6796703934669495, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.23969195783138275, 'logps/chosen': -256.4490661621094, 'logps/rejected': -339.6527404785156, 'logps/ref_chosen': -97.5711669921875, 'logps/ref_rejected': -93.58476257324219, 'logits/chosen': -7.869577884674072, 'logits/rejected': -7.049417495727539, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.002779541537165642, 'kl/avg_steps': 0.59375, 'epoch': 0.98}
98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 667/681 [50:26<00:43, 3.13s/it] 98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 668/681 [50:29<00:40, 3.14s/it] {'loss': 1.1573, 'grad_norm': 6.144704341888428, 'learning_rate': 6.453213851142225e-10, 'rewards/chosen': -0.38403064012527466, 'rewards/rejected': -0.6493211388587952, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.2652904689311981, 'logps/chosen': -242.12380981445312, 'logps/rejected': -345.40899658203125, 'logps/ref_chosen': -102.5750503540039, 'logps/ref_rejected': -108.81768798828125, 'logits/chosen': -8.091211318969727, 'logits/rejected': -7.701225280761719, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.002763135591521859, 'kl/avg_steps': 0.53125, 'epoch': 0.98}
98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 668/681 [50:29<00:40, 3.14s/it] 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 669/681 [50:32<00:37, 3.16s/it] {'loss': 1.1609, 'grad_norm': 5.321188926696777, 'learning_rate': 5.564580657695939e-10, 'rewards/chosen': -0.3517535924911499, 'rewards/rejected': -0.6092487573623657, 'rewards/accuracies': 0.8125, 'rewards/margins': 0.2574951946735382, 'logps/chosen': -217.85406494140625, 'logps/rejected': -305.6316223144531, 'logps/ref_chosen': -89.49478149414062, 'logps/ref_rejected': -82.51950073242188, 'logits/chosen': -7.704500198364258, 'logits/rejected': -6.9445672035217285, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0027485338505357504, 'kl/avg_steps': 0.59375, 'epoch': 0.98}
98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 669/681 [50:32<00:37, 3.16s/it] 98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 670/681 [50:35<00:35, 3.19s/it] {'loss': 1.1435, 'grad_norm': 4.645162582397461, 'learning_rate': 4.741678157389739e-10, 'rewards/chosen': -0.3478962182998657, 'rewards/rejected': -0.6254346370697021, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.2775384485721588, 'logps/chosen': -223.52005004882812, 'logps/rejected': -332.3464050292969, 'logps/ref_chosen': -95.45459747314453, 'logps/ref_rejected': -101.53292846679688, 'logits/chosen': -7.779049873352051, 'logits/rejected': -7.034740447998047, 'kl/p_epsilon_steps': 0.875, 'kl/n_epsilon_steps': 0.125, 'kl/beta': 0.0027323109097778797, 'kl/avg_steps': 0.75, 'epoch': 0.98}
98%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 670/681 [50:35<00:35, 3.19s/it] 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 671/681 [50:38<00:31, 3.17s/it] {'loss': 1.2244, 'grad_norm': 4.930266380310059, 'learning_rate': 3.9845280344705245e-10, 'rewards/chosen': -0.42587053775787354, 'rewards/rejected': -0.6129598617553711, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.18708932399749756, 'logps/chosen': -239.85293579101562, 'logps/rejected': -317.8214111328125, 'logps/ref_chosen': -82.12312316894531, 'logps/ref_rejected': -90.21969604492188, 'logits/chosen': -7.417923927307129, 'logits/rejected': -7.285045623779297, 'kl/p_epsilon_steps': 0.78125, 'kl/n_epsilon_steps': 0.21875, 'kl/beta': 0.0027119710575789213, 'kl/avg_steps': 0.5625, 'epoch': 0.99}
99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 671/681 [50:39<00:31, 3.17s/it] 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 672/681 [50:42<00:28, 3.14s/it] {'loss': 1.2405, 'grad_norm': 5.901381969451904, 'learning_rate': 3.293150240547549e-10, 'rewards/chosen': -0.423230916261673, 'rewards/rejected': -0.5902296304702759, 'rewards/accuracies': 0.734375, 'rewards/margins': 0.1669987291097641, 'logps/chosen': -247.38026428222656, 'logps/rejected': -320.5558166503906, 'logps/ref_chosen': -90.0619125366211, 'logps/ref_rejected': -100.45323181152344, 'logits/chosen': -7.819128513336182, 'logits/rejected': -7.268359184265137, 'kl/p_epsilon_steps': 0.71875, 'kl/n_epsilon_steps': 0.28125, 'kl/beta': 0.0026968014426529408, 'kl/avg_steps': 0.4375, 'epoch': 0.99}
99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 672/681 [50:42<00:28, 3.14s/it] 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 673/681 [50:44<00:24, 3.02s/it] {'loss': 1.2187, 'grad_norm': 5.085114479064941, 'learning_rate': 2.6675629940689504e-10, 'rewards/chosen': -0.3727704882621765, 'rewards/rejected': -0.562617301940918, 'rewards/accuracies': 0.78125, 'rewards/margins': 0.18984678387641907, 'logps/chosen': -218.5710906982422, 'logps/rejected': -302.2262268066406, 'logps/ref_chosen': -79.26315307617188, 'logps/ref_rejected': -91.34925079345703, 'logits/chosen': -7.903676986694336, 'logits/rejected': -7.309889793395996, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.002685054438188672, 'kl/avg_steps': 0.53125, 'epoch': 0.99}
99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 673/681 [50:44<00:24, 3.02s/it] 99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 674/681 [50:47<00:21, 3.06s/it] {'loss': 1.1642, 'grad_norm': 4.563065052032471, 'learning_rate': 2.1077827798404725e-10, 'rewards/chosen': -0.34820300340652466, 'rewards/rejected': -0.600648045539856, 'rewards/accuracies': 0.890625, 'rewards/margins': 0.2524449825286865, 'logps/chosen': -206.57798767089844, 'logps/rejected': -302.875, 'logps/ref_chosen': -75.45831298828125, 'logps/ref_rejected': -76.20362854003906, 'logits/chosen': -7.53197717666626, 'logits/rejected': -7.05259895324707, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.0026708655059337616, 'kl/avg_steps': 0.71875, 'epoch': 0.99}
99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 674/681 [50:47<00:21, 3.06s/it] 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 675/681 [50:50<00:17, 2.98s/it] {'loss': 1.1993, 'grad_norm': 5.335831165313721, 'learning_rate': 1.6138243485910863e-10, 'rewards/chosen': -0.3907131552696228, 'rewards/rejected': -0.6111031174659729, 'rewards/accuracies': 0.765625, 'rewards/margins': 0.2203899621963501, 'logps/chosen': -227.656494140625, 'logps/rejected': -313.17236328125, 'logps/ref_chosen': -79.90953063964844, 'logps/ref_rejected': -81.21824645996094, 'logits/chosen': -7.707300186157227, 'logits/rejected': -7.3889970779418945, 'kl/p_epsilon_steps': 0.765625, 'kl/n_epsilon_steps': 0.234375, 'kl/beta': 0.0026518055237829685, 'kl/avg_steps': 0.53125, 'epoch': 0.99}
99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 675/681 [50:50<00:17, 2.98s/it] 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏| 676/681 [50:53<00:15, 3.04s/it] {'loss': 1.1933, 'grad_norm': 4.61472749710083, 'learning_rate': 1.1857007165852472e-10, 'rewards/chosen': -0.41136687994003296, 'rewards/rejected': -0.632805585861206, 'rewards/accuracies': 0.796875, 'rewards/margins': 0.2214387059211731, 'logps/chosen': -254.79995727539062, 'logps/rejected': -336.66851806640625, 'logps/ref_chosen': -98.17111206054688, 'logps/ref_rejected': -95.024658203125, 'logits/chosen': -8.043817520141602, 'logits/rejected': -7.283336639404297, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.0026377923786640167, 'kl/avg_steps': 0.59375, 'epoch': 0.99}
99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏| 676/681 [50:53<00:15, 3.04s/it] 99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎| 677/681 [50:56<00:11, 2.96s/it] {'loss': 1.186, 'grad_norm': 4.470887660980225, 'learning_rate': 8.23423165278725e-11, 'rewards/chosen': -0.3634149134159088, 'rewards/rejected': -0.5882839560508728, 'rewards/accuracies': 0.84375, 'rewards/margins': 0.22486907243728638, 'logps/chosen': -230.60610961914062, 'logps/rejected': -308.8211669921875, 'logps/ref_chosen': -91.37928009033203, 'logps/ref_rejected': -82.87776947021484, 'logits/chosen': -7.727552890777588, 'logits/rejected': -6.984161376953125, 'kl/p_epsilon_steps': 0.796875, 'kl/n_epsilon_steps': 0.203125, 'kl/beta': 0.002622222760692239, 'kl/avg_steps': 0.59375, 'epoch': 0.99}
99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎| 677/681 [50:56<00:11, 2.96s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌| 678/681 [50:59<00:09, 3.00s/it] {'loss': 1.1519, 'grad_norm': 4.6202616691589355, 'learning_rate': 5.270012410216185e-11, 'rewards/chosen': -0.3453512489795685, 'rewards/rejected': -0.6112858057022095, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.265934556722641, 'logps/chosen': -208.89874267578125, 'logps/rejected': -323.3565979003906, 'logps/ref_chosen': -75.64586639404297, 'logps/ref_rejected': -86.96611022949219, 'logits/chosen': -7.629189491271973, 'logits/rejected': -6.984808921813965, 'kl/p_epsilon_steps': 0.859375, 'kl/n_epsilon_steps': 0.140625, 'kl/beta': 0.002606745343655348, 'kl/avg_steps': 0.71875, 'epoch': 1.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌| 678/681 [50:59<00:09, 3.00s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋| 679/681 [51:02<00:06, 3.01s/it] {'loss': 1.1954, 'grad_norm': 4.485384464263916, 'learning_rate': 2.9644275480772416e-11, 'rewards/chosen': -0.39101141691207886, 'rewards/rejected': -0.6033735871315002, 'rewards/accuracies': 0.875, 'rewards/margins': 0.2123621702194214, 'logps/chosen': -232.62774658203125, 'logps/rejected': -317.65594482421875, 'logps/ref_chosen': -80.77344512939453, 'logps/ref_rejected': -82.87850189208984, 'logits/chosen': -7.64573860168457, 'logits/rejected': -6.9826436042785645, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.0025881431065499783, 'kl/avg_steps': 0.65625, 'epoch': 1.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋| 679/681 [51:02<00:06, 3.01s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 680/681 [51:05<00:03, 3.03s/it] {'loss': 1.1887, 'grad_norm': 4.077805042266846, 'learning_rate': 1.31753782067201e-11, 'rewards/chosen': -0.3943979740142822, 'rewards/rejected': -0.6201244592666626, 'rewards/accuracies': 0.859375, 'rewards/margins': 0.22572645545005798, 'logps/chosen': -261.8695373535156, 'logps/rejected': -359.0973815917969, 'logps/ref_chosen': -107.68292999267578, 'logps/ref_rejected': -116.09486389160156, 'logits/chosen': -7.682188987731934, 'logits/rejected': -7.54879093170166, 'kl/p_epsilon_steps': 0.8125, 'kl/n_epsilon_steps': 0.1875, 'kl/beta': 0.0025712691713124514, 'kl/avg_steps': 0.625, 'epoch': 1.0}
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 680/681 [51:05<00:03, 3.03s/it] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [51:08<00:00, 3.02s/it] {'loss': 1.2117, 'grad_norm': 4.475795269012451, 'learning_rate': 3.2938662507808745e-12, 'rewards/chosen': -0.3517065942287445, 'rewards/rejected': -0.5499787330627441, 'rewards/accuracies': 0.828125, 'rewards/margins': 0.19827213883399963, 'logps/chosen': -231.38394165039062, 'logps/rejected': -311.75775146484375, 'logps/ref_chosen': -93.01106262207031, 'logps/ref_rejected': -94.82217407226562, 'logits/chosen': -7.837874412536621, 'logits/rejected': -7.355884552001953, 'kl/p_epsilon_steps': 0.828125, 'kl/n_epsilon_steps': 0.171875, 'kl/beta': 0.002555298386141658, 'kl/avg_steps': 0.65625, 'epoch': 1.0}
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [51:08<00:00, 3.02s/it][INFO|trainer.py:3984] 2026-04-24 05:07:30,523 >> Saving model checkpoint to /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-681
[INFO|configuration_utils.py:419] 2026-04-24 05:07:30,528 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-681/config.json
[INFO|configuration_utils.py:911] 2026-04-24 05:07:30,533 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-681/generation_config.json
[INFO|modeling_utils.py:3580] 2026-04-24 05:08:10,118 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 6 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-681/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2510] 2026-04-24 05:08:10,123 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-681/tokenizer_config.json
[INFO|tokenization_utils_base.py:2519] 2026-04-24 05:08:10,142 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-681/special_tokens_map.json
[INFO|trainer.py:4083] 2026-04-24 05:11:08,116 >> Deleting older checkpoint [/scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/checkpoint-400] due to args.save_total_limit
[INFO|trainer.py:2681] 2026-04-24 05:11:10,478 >>
Training completed. Do not forget to share your model on huggingface.co/models =)
{'train_runtime': 3308.3388, 'train_samples_per_second': 13.178, 'train_steps_per_second': 0.206, 'train_loss': 1.0212064078200755, 'epoch': 1.0}
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [55:03<00:00, 3.02s/it] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 681/681 [55:03<00:00, 4.85s/it]
***** train metrics *****
epoch = 1.0
total_flos = 0GF
train_loss = 1.0212
train_runtime = 0:55:08.33
train_samples = 43598
train_samples_per_second = 13.178
train_steps_per_second = 0.206
2026-04-24 05:11:10 - INFO - __main__ - *** Training complete ***
2026-04-24 05:11:10 - INFO - __main__ - *** Save model ***
[INFO|configuration_utils.py:419] 2026-04-24 05:11:26,900 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/config.json
[INFO|configuration_utils.py:911] 2026-04-24 05:11:26,907 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/generation_config.json
[INFO|modeling_utils.py:3580] 2026-04-24 05:12:11,170 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 7 checkpoint shards. You can find where each parameters has been saved in the index located at /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/model.safetensors.index.json.
[INFO|tokenization_utils_base.py:2510] 2026-04-24 05:12:11,174 >> tokenizer config file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/tokenizer_config.json
[INFO|tokenization_utils_base.py:2519] 2026-04-24 05:12:11,177 >> Special tokens file saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/special_tokens_map.json
2026-04-24 05:12:11 - INFO - __main__ - Saved HF-compatible model artifacts to /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306
[INFO|modelcard.py:450] 2026-04-24 05:12:11,571 >> Dropping the following result as it does not have all the necessary fields:
{'dataset': {'name': 'Anthropic/hh-rlhf', 'type': 'Anthropic/hh-rlhf'}}
[INFO|configuration_utils.py:419] 2026-04-24 05:12:11,579 >> Configuration saved in /scratch/qu.yang1/outputs/qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306/config.json
2026-04-24 05:12:11 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:4307] 2026-04-24 05:12:11,587 >>
***** Running Evaluation *****
[INFO|trainer.py:4309] 2026-04-24 05:12:11,587 >> Num examples = 2339
[INFO|trainer.py:4312] 2026-04-24 05:12:11,587 >> Batch size = 8
0%| | 0/73 [00:00<?, ?it/s] 3%|███▏ | 2/73 [00:00<00:23, 3.06it/s] 4%|████▊ | 3/73 [00:01<00:32, 2.16it/s] 5%|██████▎ | 4/73 [00:01<00:36, 1.89it/s] 7%|███████▉ | 5/73 [00:02<00:38, 1.77it/s] 8%|█████████▌ | 6/73 [00:03<00:40, 1.67it/s] 10%|███████████ | 7/73 [00:03<00:38, 1.72it/s] 11%|████████████▋ | 8/73 [00:04<00:40, 1.60it/s] 12%|██████████████▎ | 9/73 [00:05<00:41, 1.55it/s] 14%|███████████████▊ | 10/73 [00:05<00:41, 1.53it/s] 15%|█████████████████▎ | 11/73 [00:06<00:40, 1.52it/s] 16%|██████████████████▉ | 12/73 [00:07<00:40, 1.50it/s] 18%|████████████████████▍ | 13/73 [00:07<00:39, 1.53it/s] 19%|██████████████████████ | 14/73 [00:08<00:39, 1.51it/s] 21%|███████████████████████▋ | 15/73 [00:09<00:38, 1.51it/s] 22%|█████████████████████████▏ | 16/73 [00:09<00:39, 1.46it/s] 23%|██████████████████████████▊ | 17/73 [00:10<00:38, 1.45it/s] 25%|████████████████████████████▎ | 18/73 [00:11<00:38, 1.43it/s] 26%|█████████████████████████████▉ | 19/73 [00:12<00:38, 1.42it/s] 27%|███████████████████████████████▌ | 20/73 [00:12<00:37, 1.40it/s] 29%|█████████████████████████████████ | 21/73 [00:13<00:37, 1.40it/s] 30%|██████████████████████████████████▋ | 22/73 [00:14<00:36, 1.40it/s] 32%|████████████████████████████████████▏ | 23/73 [00:14<00:35, 1.42it/s] 33%|█████████████████████████████████████▊ | 24/73 [00:15<00:34, 1.43it/s] 34%|███████████████████████████████████████▍ | 25/73 [00:16<00:33, 1.45it/s] 36%|████████████████████████████████████████▉ | 26/73 [00:16<00:32, 1.45it/s] 37%|██████████████████████████████████████████▌ | 27/73 [00:17<00:28, 1.61it/s] 38%|████████████████████████████████████████████ | 28/73 [00:18<00:28, 1.56it/s] 40%|█████████████████████████████████████████████▋ | 29/73 [00:18<00:27, 1.57it/s] 41%|███████████████████████████████████████████████▎ | 30/73 [00:19<00:27, 1.59it/s] 42%|████████████████████████████████████████████████▊ | 31/73 [00:20<00:27, 1.53it/s] 44%|██████████████████████████████████████████████████▍ | 32/73 [00:20<00:26, 1.56it/s] 45%|███████████████████████████████████████████████████▉ | 33/73 [00:21<00:25, 1.58it/s] 47%|█████████████████████████████████████████████████████▌ | 34/73 [00:21<00:25, 1.55it/s] 48%|███████████████████████████████████████████████████████▏ | 35/73 [00:22<00:25, 1.49it/s] 49%|████████████████████████████████████████████████████████▋ | 36/73 [00:23<00:24, 1.49it/s] 51%|██████████████████████████████████████████████████████████▎ | 37/73 [00:24<00:24, 1.48it/s] 52%|███████████████████████████████████████████████████████████▊ | 38/73 [00:24<00:22, 1.56it/s] 53%|█████████████████████████████████████████████████████████████▍ | 39/73 [00:25<00:22, 1.49it/s] 55%|███████████████████████████████████████████████████████████████ | 40/73 [00:25<00:21, 1.51it/s] 56%|████████████████████████████████████████████████████████████████▌ | 41/73 [00:26<00:20, 1.56it/s] 58%|██████████████████████████████████████████████████████████████████▏ | 42/73 [00:27<00:20, 1.52it/s] 59%|███████████████████████████████████████████████████████████████████▋ | 43/73 [00:27<00:19, 1.55it/s] 60%|█████████████████████████████████████████████████████████████████████▎ | 44/73 [00:28<00:19, 1.53it/s] 62%|██████████████████████████████████████████████████████████████████████▉ | 45/73 [00:29<00:18, 1.47it/s] 63%|████████████████████████████████████████████████████████████████████████▍ | 46/73 [00:29<00:17, 1.54it/s] 64%|██████████████████████████████████████████████████████████████████████████ | 47/73 [00:30<00:17, 1.50it/s] 66%|███████████████████████████████████████████████████████████████████████████▌ | 48/73 [00:31<00:16, 1.50it/s] 67%|█████████████████████████████████████████████████████████████████████████████▏ | 49/73 [00:31<00:16, 1.48it/s] 68%|██████████████████████████████████████████████████████████████████████████████▊ | 50/73 [00:32<00:15, 1.49it/s] 70%|████████████████████████████████████████████████████████████████████████████████▎ | 51/73 [00:33<00:14, 1.47it/s] 71%|█████████████████████████████████████████████████████████████████████████████████▉ | 52/73 [00:34<00:14, 1.44it/s] 73%|███████████████████████████████████████████████████████████████████████████████████▍ | 53/73 [00:34<00:14, 1.41it/s] 74%|█████████████████████████████████████████████████████████████████████████████████████ | 54/73 [00:35<00:12, 1.50it/s] 75%|██████████████████████████████████████████████████████████████████████████████████████▋ | 55/73 [00:36<00:12, 1.50it/s] 77%|████████████████████████████████████████████████████████████████████████████████████████▏ | 56/73 [00:36<00:11, 1.53it/s] 78%|█████████████████████████████████████████████████████████████████████████████████████████▊ | 57/73 [00:37<00:10, 1.48it/s] 79%|███████████████████████████████████████████████████████████████████████████████████████████▎ | 58/73 [00:37<00:09, 1.54it/s] 81%|████████████████████████████████████████████████████████████████████████████████████████████▉ | 59/73 [00:38<00:09, 1.54it/s] 82%|██████████████████████████████████████████████████████████████████████████████████████████████▌ | 60/73 [00:39<00:08, 1.51it/s] 84%|████████████████████████████████████████████████████████████████████████████████████████████████ | 61/73 [00:39<00:08, 1.49it/s] 85%|█████████████████████████████████████████████████████████████████████████████████████████████████▋ | 62/73 [00:40<00:07, 1.51it/s] 86%|███████████████████████████████████████████████████████████████████████████████████████████████████▏ | 63/73 [00:41<00:06, 1.59it/s] 88%|████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 64/73 [00:41<00:05, 1.61it/s] 89%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 65/73 [00:42<00:05, 1.59it/s] 90%|███████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 66/73 [00:43<00:04, 1.53it/s] 92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 67/73 [00:43<00:03, 1.55it/s] 93%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 68/73 [00:44<00:03, 1.53it/s] 95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████▋ | 69/73 [00:45<00:02, 1.49it/s] 96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████▎ | 70/73 [00:45<00:02, 1.48it/s] 97%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 71/73 [00:46<00:01, 1.48it/s] 99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 72/73 [00:47<00:00, 1.48it/s] 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.61it/s] 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73/73 [00:47<00:00, 1.53it/s]
***** eval metrics *****
epoch = 1.0
eval_kl/n_epsilon_steps = 0.2517
eval_kl/p_epsilon_steps = 0.7479
eval_logits/chosen = -7.607
eval_logits/rejected = -7.1863
eval_logps/chosen = -252.7017
eval_logps/ref_chosen = -100.4936
eval_logps/ref_rejected = -94.0678
eval_logps/rejected = -314.7497
eval_loss = 0.6181
eval_rewards/accuracies = 0.7577
eval_rewards/chosen = -0.3852
eval_rewards/margins = 0.1715
eval_rewards/rejected = -0.5567
eval_runtime = 0:00:48.39
eval_samples = 2339
eval_samples_per_second = 48.331
eval_steps_per_second = 1.529
2026-04-24 05:12:59 - INFO - __main__ - *** Training complete! ***
wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.014 MB of 0.014 MB uploaded wandb: | 0.014 MB of 0.014 MB uploaded wandb: / 0.048 MB of 0.310 MB uploaded wandb: - 0.313 MB of 0.313 MB uploaded wandb:
wandb: Run history:
wandb: eval/kl/n_epsilon_steps █▄▂▁▂▂▂
wandb: eval/kl/p_epsilon_steps ▁▅▇█▇▇▇
wandb: eval/logits/chosen █▄▂▁▁▁▁
wandb: eval/logits/rejected █▃▂▁▁▁▁
wandb: eval/logps/chosen █▇▅▂▁▁▁
wandb: eval/logps/ref_chosen ▁▁▁▁▁▁▁
wandb: eval/logps/ref_rejected ▁▁▁▁▁▁▁
wandb: eval/logps/rejected █▇▅▂▁▁▁
wandb: eval/loss █▂▁▁▄▆█
wandb: eval/rewards/accuracies ▁▇██▇▆▆
wandb: eval/rewards/chosen █▅▁▁▄▆█
wandb: eval/rewards/margins ▂▇█▇▅▂▁
wandb: eval/rewards/rejected █▄▁▁▄▇█
wandb: eval/runtime █▆▇▅█▆▁
wandb: eval/samples_per_second ▁▃▂▄▁▃█
wandb: eval/steps_per_second ▁▃▂▅▁▂█
wandb: train/epoch ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb: train/global_step ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb: train/grad_norm ▆█▅▅▅▄▃▄▄▃▃▄▃▃▃▃▃▃▃▃▃▃▃▃▂▂▂▂▂▃▂▂▂▂▂▂▁▁▁▁
wandb: train/kl/avg_steps ▁▂▅▇▇▅▅▅▅▆▆▆▇▆▇▇▇▇▇▇▇▇▇▇▇▇▇█▇█▇▇▇▆▆▇▇██▇
wandb: train/kl/beta ███▇▇▆▆▅▅▅▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: train/kl/n_epsilon_steps █▇▄▃▃▄▄▄▃▃▃▃▂▃▃▃▂▃▁▁▃▂▂▁▃▂▃▁▂▁▂▂▁▃▃▂▃▁▁▂
wandb: train/kl/p_epsilon_steps ▁▂▅▇▇▅▅▅▅▆▆▆▇▆▇▇▇▇▇▇▇▇▇▇▇▇▇█▇█▇▇▇▆▆▇▇██▇
wandb: train/learning_rate ▂▃▅▇██████▇▇▇▇▇▆▆▆▆▅▅▅▄▄▄▄▃▃▃▂▂▂▂▂▁▁▁▁▁▁
wandb: train/logits/chosen ▇██▇▆▆▅▄▄▃▃▃▂▂▂▂▂▂▂▂▁▂▁▁▂▁▂▁▂▁▁▁▁▁▂▂▁▁▂▂
wandb: train/logits/rejected ████▇▆▅▅▄▄▃▃▂▂▂▂▂▂▂▂▂▂▁▂▂▁▁▂▂▁▂▂▁▁▂▂▁▁▁▁
wandb: train/logps/chosen ▇██▇▇▇▇▇▇▇▇▆▆▆▆▆▆▅▅▅▄▄▃▃▂▃▃▁▃▂▂▃▂▂▂▃▁▂▂▁
wandb: train/logps/ref_chosen ▅█▆▆▄▄▅▅▄▄▇▄▅▅▇▅▆▄▇▆▅█▇▄▅▄▆▁▆▅▄▇▄▇▇▆▃▄▆▃
wandb: train/logps/ref_rejected ▇▃█▅▅▂▅█▇▄▄▃▄▆▅▅▄▄█▅▇▄▆▃▅▄▃▁▆▃▃▆▃▄█▅▄▂▁▂
wandb: train/logps/rejected █▇█▇▇▇▇█▇▇▇▆▆▇▆▆▆▅▅▄▄▄▃▃▂▃▃▁▃▂▂▂▁▂▂▂▁▁▁▁
wandb: train/loss ███▇▆▆▄▅▄▃▃▃▂▃▃▂▂▂▂▁▂▂▃▁▂▂▄▂▃▃▄▄▃▅▅▅▅▅▅▆
wandb: train/rewards/accuracies ▁▁▅▆▆▅▇▆▇▇▆▆█▆▆▇▇▇▆▇▇▇▇▇▆▇▆███▇▇▇▆▆▇▆▇▇▇
wandb: train/rewards/chosen █████▇▇▇▇▇▆▄▄▄▃▄▄▁▂▂▂▂▁▃▁▃▃▃▄▄▄▅▅▅▅▆▆▆▆▆
wandb: train/rewards/margins ▁▁▁▂▂▃▄▄▄▅▆▆▇▆▆▇▇▇▇█▇▇▆█▆▆▅▆▅▅▄▄▅▃▃▃▃▃▃▂
wandb: train/rewards/rejected ████▇▇▆▆▆▆▄▄▃▄▃▃▃▁▂▁▂▂▂▂▂▃▃▃▄▄▄▅▅▅▅▆▆▆▆▆
wandb:
wandb: Run summary:
wandb: eval/kl/n_epsilon_steps 0.25171
wandb: eval/kl/p_epsilon_steps 0.74786
wandb: eval/logits/chosen -7.60696
wandb: eval/logits/rejected -7.18629
wandb: eval/logps/chosen -252.70166
wandb: eval/logps/ref_chosen -100.49357
wandb: eval/logps/ref_rejected -94.06776
wandb: eval/logps/rejected -314.74973
wandb: eval/loss 0.61809
wandb: eval/rewards/accuracies 0.75771
wandb: eval/rewards/chosen -0.38518
wandb: eval/rewards/margins 0.1715
wandb: eval/rewards/rejected -0.55669
wandb: eval/runtime 48.3953
wandb: eval/samples_per_second 48.331
wandb: eval/steps_per_second 1.529
wandb: total_flos 0.0
wandb: train/epoch 1.0
wandb: train/global_step 681
wandb: train/grad_norm 4.4758
wandb: train/kl/avg_steps 0.65625
wandb: train/kl/beta 0.00256
wandb: train/kl/n_epsilon_steps 0.17188
wandb: train/kl/p_epsilon_steps 0.82812
wandb: train/learning_rate 0.0
wandb: train/logits/chosen -7.83787
wandb: train/logits/rejected -7.35588
wandb: train/logps/chosen -231.38394
wandb: train/logps/ref_chosen -93.01106
wandb: train/logps/ref_rejected -94.82217
wandb: train/logps/rejected -311.75775
wandb: train/loss 1.2117
wandb: train/rewards/accuracies 0.82812
wandb: train/rewards/chosen -0.35171
wandb: train/rewards/margins 0.19827
wandb: train/rewards/rejected -0.54998
wandb: train_loss 1.02121
wandb: train_runtime 3308.3388
wandb: train_samples_per_second 13.178
wandb: train_steps_per_second 0.206
wandb:
wandb: 🚀 View run qwen3-8b-base-epsilon-dpo-hh-helpful-4xh200-batch-64-20260424-040306 at: https://wandb.ai/feng-cheng-northeastern-university/qwen3_hh_4xh200_beta_0.1/runs/gfncx0q7
wandb: ⭐️ View project at: https://wandb.ai/feng-cheng-northeastern-university/qwen3_hh_4xh200_beta_0.1
wandb: Synced 6 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: /scratch/qu.yang1/wandb/wandb/run-20260424_041603-gfncx0q7/logs
wandb: WARNING The new W&B backend becomes opt-out in version 0.18.0; try it out with `wandb.require("core")`! See https://wandb.me/wandb-core for more information.
Reference in New Issue View Git Blame Copy Permalink