SeongryongJung/Qwen3-8B-Chemistry-RLSD-TR
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
main
Qwen3-8B-Chemistry-RLSD-TR/artifacts/queue.log
ModelHub XC d3719d800b 初始化项目，由ModelHub XC社区提供模型 
Model: SeongryongJung/Qwen3-8B-Chemistry-RLSD-TR
Source: Original Platform
2026-07-21 17:57:12 +08:00

5790 lines
1.9 MiB
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

[2026-07-02 23:56:17] Waiting for PID 999999999.
[2026-07-02 23:56:17] PID 999999999 finished. Starting Qwen3 generalization queue.
[2026-07-02 23:56:17] Model: Qwen/Qwen3-8B
[2026-07-02 23:56:17] Datasets: chemistry physics biology material tooluse
[2026-07-02 23:56:17] Methods: grpo rlsd_tr
[2026-07-02 23:56:17] Starting Qwen/Qwen3-8B chemistry grpo.
Experiment: qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
Dataset: chemistry
Method: grpo
Config: baseline_grpo
Model: Qwen/Qwen3-8B
Train: /mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet
Val: /mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet
Ray tmp: /tmp/ray_q3g_chemistry_grpo_Qwen3_8B
2026-07-02 23:56:19,612 INFO scripts.py:1364 -- Did not find any active Ray processes.
Experiment: qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
Config: baseline_grpo
Task: datasets/sciknoweval/chemistry
Arguments: max_model_len=10240 data.train_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet] data.val_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet] data.train_batch_size=32 data.train_max_samples=3200 data.max_prompt_length=2048 data.max_response_length=8192 data.apply_chat_template_kwargs.enable_thinking=false trainer.group_name=QWEN3-GRPO-generalization trainer.n_gpus_per_node=8 trainer.total_training_steps=100 trainer.val_before_train=False trainer.save_freq=10 trainer.test_freq=10 trainer.default_local_dir=/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8 custom_reward_function.path=/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py +ray_kwargs.ray_init._temp_dir=/tmp/ray_q3g_chemistry_grpo_Qwen3_8B +ray_kwargs.ray_init.include_dashboard=False +ray_kwargs.ray_init.runtime_env.env_vars.USER=root +ray_kwargs.ray_init.runtime_env.env_vars.TASK=datasets/sciknoweval/chemistry +ray_kwargs.ray_init.runtime_env.env_vars.EXPERIMENT=qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8 actor_rollout_ref.model.path=Qwen/Qwen3-8B actor_rollout_ref.actor.optim.lr=1e-6 actor_rollout_ref.actor.optim.lr_warmup_steps=10 actor_rollout_ref.actor.optim.weight_decay=0.01 actor_rollout_ref.actor.grad_clip=1.0 actor_rollout_ref.actor.ppo_mini_batch_size=8 actor_rollout_ref.actor.clip_ratio_high=0.28 actor_rollout_ref.actor.clip_ratio_low=0.2 actor_rollout_ref.actor.ppo_max_token_len_per_gpu=10240 actor_rollout_ref.rollout.n=8 actor_rollout_ref.rollout.temperature=1.0 actor_rollout_ref.rollout.top_p=1.0 actor_rollout_ref.rollout.max_model_len=10240 actor_rollout_ref.rollout.max_num_batched_tokens=10240 actor_rollout_ref.rollout.gpu_memory_utilization=0.8 actor_rollout_ref.rollout.calculate_log_probs=True actor_rollout_ref.rollout.val_kwargs.n=16 actor_rollout_ref.rollout.val_kwargs.temperature=0.6 actor_rollout_ref.rollout.val_kwargs.top_p=0.95 actor_rollout_ref.rollout.val_kwargs.do_sample=True algorithm.norm_adv_by_std_in_grpo=False algorithm.rollout_correction.rollout_is=token algorithm.rollout_correction.rollout_is_threshold=2.0 trainer.max_actor_ckpt_to_keep=100 actor_rollout_ref.actor.policy_loss.loss_mode=vanilla actor_rollout_ref.actor.kl_loss_coef=0.0
ray init kwargs: {'num_cpus': None, '_temp_dir': '/tmp/ray_q3g_chemistry_grpo_Qwen3_8B', 'include_dashboard': False, 'runtime_env': {'env_vars': {'TOKENIZERS_PARALLELISM': 'true', 'NCCL_DEBUG': 'WARN', 'VLLM_LOGGING_LEVEL': 'WARN', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': 'true', 'VLLM_ALLREDUCE_USE_SYMM_MEM': '0', 'CUDA_DEVICE_MAX_CONNECTIONS': '1', 'NCCL_CUMEM_ENABLE': '0', 'USER': 'root', 'TASK': 'datasets/sciknoweval/chemistry', 'EXPERIMENT': 'qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8'}, 'working_dir': None}}
2026-07-02 23:56:29,407 INFO worker.py:2007 -- Started a local Ray instance.
/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(TaskRunner pid=2224264) TaskRunner hostname: mole-workspace-rw, PID: 2224264
(TaskRunner pid=2224264) {'actor_rollout_ref': {'actor': {'_target_': 'verl.workers.config.FSDPActorConfig',
(TaskRunner pid=2224264) 'calculate_entropy': False,
(TaskRunner pid=2224264) 'calculate_sum_pi_squared': False,
(TaskRunner pid=2224264) 'checkpoint': {'_target_': 'verl.trainer.config.CheckpointConfig',
(TaskRunner pid=2224264) 'async_save': False,
(TaskRunner pid=2224264) 'load_contents': ['model',
(TaskRunner pid=2224264) 'optimizer',
(TaskRunner pid=2224264) 'extra'],
(TaskRunner pid=2224264) 'save_contents': ['model',
(TaskRunner pid=2224264) 'optimizer',
(TaskRunner pid=2224264) 'extra']},
(TaskRunner pid=2224264) 'clip_ratio': 0.2,
(TaskRunner pid=2224264) 'clip_ratio_c': 3.0,
(TaskRunner pid=2224264) 'clip_ratio_high': 0.28,
(TaskRunner pid=2224264) 'clip_ratio_low': 0.2,
(TaskRunner pid=2224264) 'data_loader_seed': 42,
(TaskRunner pid=2224264) 'entropy_checkpointing': False,
(TaskRunner pid=2224264) 'entropy_coeff': 0,
(TaskRunner pid=2224264) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2224264) 'freeze_vision_tower': False,
(TaskRunner pid=2224264) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2224264) 'dtype': 'bfloat16',
(TaskRunner pid=2224264) 'entropy_checkpointing': False,
(TaskRunner pid=2224264) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2224264) 'forward_only': False,
(TaskRunner pid=2224264) 'forward_prefetch': False,
(TaskRunner pid=2224264) 'fsdp_size': -1,
(TaskRunner pid=2224264) 'full_determinism': False,
(TaskRunner pid=2224264) 'model_dtype': 'fp32',
(TaskRunner pid=2224264) 'offload_policy': False,
(TaskRunner pid=2224264) 'optimizer_offload': False,
(TaskRunner pid=2224264) 'param_offload': False,
(TaskRunner pid=2224264) 'reshard_after_forward': True,
(TaskRunner pid=2224264) 'seed': 42,
(TaskRunner pid=2224264) 'strategy': 'fsdp',
(TaskRunner pid=2224264) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2224264) 'use_orig_params': False,
(TaskRunner pid=2224264) 'use_torch_compile': True,
(TaskRunner pid=2224264) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2224264) 'grad_clip': 1.0,
(TaskRunner pid=2224264) 'kl_loss_coef': 0.0,
(TaskRunner pid=2224264) 'kl_loss_type': 'low_var_kl',
(TaskRunner pid=2224264) 'loss_agg_mode': 'token-mean',
(TaskRunner pid=2224264) 'loss_scale_factor': None,
(TaskRunner pid=2224264) 'optim': {'_target_': 'verl.workers.config.FSDPOptimizerConfig',
(TaskRunner pid=2224264) 'betas': [0.9, 0.999],
(TaskRunner pid=2224264) 'clip_grad': 1.0,
(TaskRunner pid=2224264) 'lr': 1e-06,
(TaskRunner pid=2224264) 'lr_scheduler_type': 'constant',
(TaskRunner pid=2224264) 'lr_warmup_steps': 10,
(TaskRunner pid=2224264) 'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=2224264) 'min_lr_ratio': 0.0,
(TaskRunner pid=2224264) 'num_cycles': 0.5,
(TaskRunner pid=2224264) 'optimizer': 'AdamW',
(TaskRunner pid=2224264) 'optimizer_impl': 'torch.optim',
(TaskRunner pid=2224264) 'override_optimizer_config': None,
(TaskRunner pid=2224264) 'total_training_steps': -1,
(TaskRunner pid=2224264) 'warmup_style': None,
(TaskRunner pid=2224264) 'weight_decay': 0.01},
(TaskRunner pid=2224264) 'policy_loss': {'_target_': 'verl.workers.config.PolicyLossConfig',
(TaskRunner pid=2224264) 'clip_cov_lb': 1.0,
(TaskRunner pid=2224264) 'clip_cov_ratio': 0.0002,
(TaskRunner pid=2224264) 'clip_cov_ub': 5.0,
(TaskRunner pid=2224264) 'kl_cov_ratio': 0.0002,
(TaskRunner pid=2224264) 'loss_mode': 'vanilla',
(TaskRunner pid=2224264) 'ppo_kl_coef': 0.1},
(TaskRunner pid=2224264) 'ppo_epochs': 1,
(TaskRunner pid=2224264) 'ppo_max_token_len_per_gpu': 10240,
(TaskRunner pid=2224264) 'ppo_micro_batch_size': None,
(TaskRunner pid=2224264) 'ppo_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2224264) 'ppo_mini_batch_size': 8,
(TaskRunner pid=2224264) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2224264) 'all_ranks': False,
(TaskRunner pid=2224264) 'enable': False,
(TaskRunner pid=2224264) 'ranks': [],
(TaskRunner pid=2224264) 'save_path': 'outputs/profile',
(TaskRunner pid=2224264) 'tool': None,
(TaskRunner pid=2224264) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2224264) 'analysis': True,
(TaskRunner pid=2224264) 'contents': [],
(TaskRunner pid=2224264) 'discrete': False,
(TaskRunner pid=2224264) 'level': 'level0'},
(TaskRunner pid=2224264) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2224264) 'discrete': False},
(TaskRunner pid=2224264) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2224264) 'step_end': None,
(TaskRunner pid=2224264) 'step_start': 0},
(TaskRunner pid=2224264) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2224264) 'stack_depth': 32,
(TaskRunner pid=2224264) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2224264) 'rollout_n': 8,
(TaskRunner pid=2224264) 'router_replay': {'_target_': 'verl.workers.config.RouterReplayConfig',
(TaskRunner pid=2224264) 'mode': 'disabled',
(TaskRunner pid=2224264) 'record_file': None,
(TaskRunner pid=2224264) 'replay_file': None},
(TaskRunner pid=2224264) 'self_distillation': {'_target_': 'verl.workers.config.SelfDistillationConfig',
(TaskRunner pid=2224264) 'alpha': 0.5,
(TaskRunner pid=2224264) 'distillation_add_tail': True,
(TaskRunner pid=2224264) 'distillation_topk': 100,
(TaskRunner pid=2224264) 'dont_reprompt_on_self_success': True,
(TaskRunner pid=2224264) 'environment_feedback_only_without_solution': True,
(TaskRunner pid=2224264) 'evolving_teacher': {'_target_': 'verl.workers.config.EvolvingTeacherConfig',
(TaskRunner pid=2224264) 'enable': False,
(TaskRunner pid=2224264) 'loss_weight': 0.0,
(TaskRunner pid=2224264) 'mask': 'all'},
(TaskRunner pid=2224264) 'feedback_template': '\n'
(TaskRunner pid=2224264) 'The '
(TaskRunner pid=2224264) 'following '
(TaskRunner pid=2224264) 'is '
(TaskRunner pid=2224264) 'feedback '
(TaskRunner pid=2224264) 'from '
(TaskRunner pid=2224264) 'your '
(TaskRunner pid=2224264) 'unsuccessful '
(TaskRunner pid=2224264) 'earlier '
(TaskRunner pid=2224264) 'attempt:\n'
(TaskRunner pid=2224264) '\n'
(TaskRunner pid=2224264) '{feedback_raw}',
(TaskRunner pid=2224264) 'full_logit_distillation': True,
(TaskRunner pid=2224264) 'include_environment_feedback': True,
(TaskRunner pid=2224264) 'is_clip': 2,
(TaskRunner pid=2224264) 'max_reprompt_len': 22528,
(TaskRunner pid=2224264) 'remove_thinking_from_demonstration': False,
(TaskRunner pid=2224264) 'reprompt_template': '{prompt}{solution}{feedback}\n'
(TaskRunner pid=2224264) '\n'
(TaskRunner pid=2224264) 'Correctly '
(TaskRunner pid=2224264) 'solve '
(TaskRunner pid=2224264) 'the '
(TaskRunner pid=2224264) 'original '
(TaskRunner pid=2224264) 'question.',
(TaskRunner pid=2224264) 'reprompt_truncation': 'right',
(TaskRunner pid=2224264) 'solution_template': '\n'
(TaskRunner pid=2224264) 'Correct '
(TaskRunner pid=2224264) 'solution:\n'
(TaskRunner pid=2224264) '\n'
(TaskRunner pid=2224264) '{successful_previous_attempt}',
(TaskRunner pid=2224264) 'srpo_dynamic_weighting': False,
(TaskRunner pid=2224264) 'srpo_dynamic_weighting_temperature': 1.0,
(TaskRunner pid=2224264) 'success_reward_threshold': 0.5,
(TaskRunner pid=2224264) 'teacher_regularization': 'ema',
(TaskRunner pid=2224264) 'teacher_update_rate': 0.0,
(TaskRunner pid=2224264) 'token_reweight_decay_steps': None,
(TaskRunner pid=2224264) 'token_reweight_eps_w': 0.2,
(TaskRunner pid=2224264) 'token_reweight_lambda': 0.5},
(TaskRunner pid=2224264) 'shuffle': False,
(TaskRunner pid=2224264) 'strategy': 'fsdp',
(TaskRunner pid=2224264) 'sum_pi_squared_checkpointing': False,
(TaskRunner pid=2224264) 'tau_neg': 1.05,
(TaskRunner pid=2224264) 'tau_pos': 1.0,
(TaskRunner pid=2224264) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2224264) 'use_dynamic_bsz': False,
(TaskRunner pid=2224264) 'use_fused_kernels': False,
(TaskRunner pid=2224264) 'use_kl_loss': False,
(TaskRunner pid=2224264) 'use_prefix_grouper': False,
(TaskRunner pid=2224264) 'use_remove_padding': True,
(TaskRunner pid=2224264) 'use_torch_compile': True},
(TaskRunner pid=2224264) 'hybrid_engine': True,
(TaskRunner pid=2224264) 'model': {'_target_': 'verl.workers.config.HFModelConfig',
(TaskRunner pid=2224264) 'custom_chat_template': None,
(TaskRunner pid=2224264) 'enable_activation_offload': False,
(TaskRunner pid=2224264) 'enable_gradient_checkpointing': True,
(TaskRunner pid=2224264) 'exclude_modules': None,
(TaskRunner pid=2224264) 'external_lib': None,
(TaskRunner pid=2224264) 'fused_kernel_options': {'impl_backend': 'torch'},
(TaskRunner pid=2224264) 'hf_config_path': None,
(TaskRunner pid=2224264) 'lora_adapter_path': None,
(TaskRunner pid=2224264) 'lora_alpha': 16,
(TaskRunner pid=2224264) 'lora_rank': 0,
(TaskRunner pid=2224264) 'override_config': {},
(TaskRunner pid=2224264) 'path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2224264) 'target_modules': 'all-linear',
(TaskRunner pid=2224264) 'tiled_mlp': {'enabled': False,
(TaskRunner pid=2224264) 'num_shards': 4},
(TaskRunner pid=2224264) 'tokenizer_path': None,
(TaskRunner pid=2224264) 'trust_remote_code': True,
(TaskRunner pid=2224264) 'use_fused_kernels': False,
(TaskRunner pid=2224264) 'use_liger': False,
(TaskRunner pid=2224264) 'use_remove_padding': True,
(TaskRunner pid=2224264) 'use_shm': False},
(TaskRunner pid=2224264) 'nccl_timeout': 600,
(TaskRunner pid=2224264) 'ref': {'_target_': 'verl.workers.config.FSDPActorConfig',
(TaskRunner pid=2224264) 'entropy_checkpointing': False,
(TaskRunner pid=2224264) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2224264) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2224264) 'dtype': 'bfloat16',
(TaskRunner pid=2224264) 'entropy_checkpointing': False,
(TaskRunner pid=2224264) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2224264) 'forward_only': True,
(TaskRunner pid=2224264) 'forward_prefetch': False,
(TaskRunner pid=2224264) 'fsdp_size': -1,
(TaskRunner pid=2224264) 'full_determinism': False,
(TaskRunner pid=2224264) 'model_dtype': 'fp32',
(TaskRunner pid=2224264) 'offload_policy': False,
(TaskRunner pid=2224264) 'optimizer_offload': False,
(TaskRunner pid=2224264) 'param_offload': False,
(TaskRunner pid=2224264) 'reshard_after_forward': True,
(TaskRunner pid=2224264) 'seed': 42,
(TaskRunner pid=2224264) 'strategy': 'fsdp',
(TaskRunner pid=2224264) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2224264) 'use_orig_params': False,
(TaskRunner pid=2224264) 'use_torch_compile': True,
(TaskRunner pid=2224264) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2224264) 'log_prob_max_token_len_per_gpu': 10240,
(TaskRunner pid=2224264) 'log_prob_micro_batch_size': None,
(TaskRunner pid=2224264) 'log_prob_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2224264) 'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=2224264) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2224264) 'all_ranks': False,
(TaskRunner pid=2224264) 'enable': False,
(TaskRunner pid=2224264) 'ranks': [],
(TaskRunner pid=2224264) 'save_path': 'outputs/profile',
(TaskRunner pid=2224264) 'tool': None,
(TaskRunner pid=2224264) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2224264) 'analysis': True,
(TaskRunner pid=2224264) 'contents': [],
(TaskRunner pid=2224264) 'discrete': False,
(TaskRunner pid=2224264) 'level': 'level0'},
(TaskRunner pid=2224264) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2224264) 'discrete': False},
(TaskRunner pid=2224264) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2224264) 'step_end': None,
(TaskRunner pid=2224264) 'step_start': 0},
(TaskRunner pid=2224264) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2224264) 'stack_depth': 32,
(TaskRunner pid=2224264) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2224264) 'rollout_n': 8,
(TaskRunner pid=2224264) 'router_replay': {'_target_': 'verl.workers.config.RouterReplayConfig',
(TaskRunner pid=2224264) 'mode': 'disabled',
(TaskRunner pid=2224264) 'record_file': None,
(TaskRunner pid=2224264) 'replay_file': None},
(TaskRunner pid=2224264) 'strategy': 'fsdp',
(TaskRunner pid=2224264) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2224264) 'use_torch_compile': True},
(TaskRunner pid=2224264) 'rollout': {'_target_': 'verl.workers.config.RolloutConfig',
(TaskRunner pid=2224264) 'agent': {'_target_': 'verl.workers.config.AgentLoopConfig',
(TaskRunner pid=2224264) 'agent_loop_config_path': None,
(TaskRunner pid=2224264) 'custom_async_server': {'_target_': 'verl.workers.config.CustomAsyncServerConfig',
(TaskRunner pid=2224264) 'name': None,
(TaskRunner pid=2224264) 'path': None},
(TaskRunner pid=2224264) 'default_agent_loop': 'single_turn_agent',
(TaskRunner pid=2224264) 'num_workers': 8},
(TaskRunner pid=2224264) 'calculate_log_probs': True,
(TaskRunner pid=2224264) 'cudagraph_capture_sizes': None,
(TaskRunner pid=2224264) 'data_parallel_size': 1,
(TaskRunner pid=2224264) 'disable_log_stats': True,
(TaskRunner pid=2224264) 'do_sample': True,
(TaskRunner pid=2224264) 'dtype': 'bfloat16',
(TaskRunner pid=2224264) 'enable_chunked_prefill': True,
(TaskRunner pid=2224264) 'enable_prefix_caching': True,
(TaskRunner pid=2224264) 'enable_rollout_routing_replay': False,
(TaskRunner pid=2224264) 'enforce_eager': False,
(TaskRunner pid=2224264) 'engine_kwargs': {'sglang': {}, 'vllm': {}},
(TaskRunner pid=2224264) 'expert_parallel_size': 1,
(TaskRunner pid=2224264) 'free_cache_engine': True,
(TaskRunner pid=2224264) 'gpu_memory_utilization': 0.8,
(TaskRunner pid=2224264) 'ignore_eos': False,
(TaskRunner pid=2224264) 'layered_summon': False,
(TaskRunner pid=2224264) 'load_format': 'dummy',
(TaskRunner pid=2224264) 'log_prob_max_token_len_per_gpu': 10240,
(TaskRunner pid=2224264) 'log_prob_micro_batch_size': None,
(TaskRunner pid=2224264) 'log_prob_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2224264) 'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=2224264) 'logprobs_mode': 'processed_logprobs',
(TaskRunner pid=2224264) 'max_model_len': 10240,
(TaskRunner pid=2224264) 'max_num_batched_tokens': 10240,
(TaskRunner pid=2224264) 'max_num_seqs': 1024,
(TaskRunner pid=2224264) 'mode': 'async',
(TaskRunner pid=2224264) 'multi_stage_wake_up': False,
(TaskRunner pid=2224264) 'multi_turn': {'_target_': 'verl.workers.config.MultiTurnConfig',
(TaskRunner pid=2224264) 'enable': False,
(TaskRunner pid=2224264) 'format': 'hermes',
(TaskRunner pid=2224264) 'interaction_config_path': None,
(TaskRunner pid=2224264) 'max_assistant_turns': None,
(TaskRunner pid=2224264) 'max_parallel_calls': 1,
(TaskRunner pid=2224264) 'max_tool_response_length': 256,
(TaskRunner pid=2224264) 'max_user_turns': None,
(TaskRunner pid=2224264) 'num_repeat_rollouts': None,
(TaskRunner pid=2224264) 'tokenization_sanity_check_mode': 'strict',
(TaskRunner pid=2224264) 'tool_config_path': None,
(TaskRunner pid=2224264) 'tool_response_truncate_side': 'middle',
(TaskRunner pid=2224264) 'use_inference_chat_template': False},
(TaskRunner pid=2224264) 'n': 8,
(TaskRunner pid=2224264) 'name': 'vllm',
(TaskRunner pid=2224264) 'over_sample_rate': 0,
(TaskRunner pid=2224264) 'pipeline_model_parallel_size': 1,
(TaskRunner pid=2224264) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2224264) 'all_ranks': False,
(TaskRunner pid=2224264) 'enable': False,
(TaskRunner pid=2224264) 'ranks': [],
(TaskRunner pid=2224264) 'save_path': 'outputs/profile',
(TaskRunner pid=2224264) 'tool': None,
(TaskRunner pid=2224264) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2224264) 'analysis': True,
(TaskRunner pid=2224264) 'contents': [],
(TaskRunner pid=2224264) 'discrete': False,
(TaskRunner pid=2224264) 'level': 'level0'},
(TaskRunner pid=2224264) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2224264) 'discrete': False},
(TaskRunner pid=2224264) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2224264) 'step_end': None,
(TaskRunner pid=2224264) 'step_start': 0},
(TaskRunner pid=2224264) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2224264) 'stack_depth': 32,
(TaskRunner pid=2224264) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2224264) 'prometheus': {'_target_': 'verl.workers.config.PrometheusConfig',
(TaskRunner pid=2224264) 'enable': False,
(TaskRunner pid=2224264) 'file': '/tmp/ray/session_latest/metrics/prometheus/prometheus.yml',
(TaskRunner pid=2224264) 'port': 9090,
(TaskRunner pid=2224264) 'served_model_name': 'Qwen/Qwen3-8B'},
(TaskRunner pid=2224264) 'prompt_length': 2048,
(TaskRunner pid=2224264) 'quantization': None,
(TaskRunner pid=2224264) 'quantization_config_file': None,
(TaskRunner pid=2224264) 'response_length': 8192,
(TaskRunner pid=2224264) 'scheduling_policy': 'fcfs',
(TaskRunner pid=2224264) 'skip_dump_dir': '/tmp/rollout_dump',
(TaskRunner pid=2224264) 'skip_rollout': False,
(TaskRunner pid=2224264) 'skip_tokenizer_init': True,
(TaskRunner pid=2224264) 'temperature': 1.0,
(TaskRunner pid=2224264) 'tensor_model_parallel_size': 2,
(TaskRunner pid=2224264) 'top_k': -1,
(TaskRunner pid=2224264) 'top_p': 1.0,
(TaskRunner pid=2224264) 'trace': {'_target_': 'verl.workers.config.TraceConfig',
(TaskRunner pid=2224264) 'backend': None,
(TaskRunner pid=2224264) 'max_samples_per_step_per_worker': None,
(TaskRunner pid=2224264) 'token2text': False},
(TaskRunner pid=2224264) 'update_weights_bucket_megabytes': 512,
(TaskRunner pid=2224264) 'val_kwargs': {'_target_': 'verl.workers.config.SamplingConfig',
(TaskRunner pid=2224264) 'do_sample': True,
(TaskRunner pid=2224264) 'n': 16,
(TaskRunner pid=2224264) 'temperature': 0.6,
(TaskRunner pid=2224264) 'top_k': -1,
(TaskRunner pid=2224264) 'top_p': 0.95}}},
(TaskRunner pid=2224264) 'algorithm': {'_target_': 'verl.trainer.config.AlgoConfig',
(TaskRunner pid=2224264) 'adv_estimator': 'grpo',
(TaskRunner pid=2224264) 'gamma': 1.0,
(TaskRunner pid=2224264) 'kl_ctrl': {'_target_': 'verl.trainer.config.KLControlConfig',
(TaskRunner pid=2224264) 'horizon': 10000,
(TaskRunner pid=2224264) 'kl_coef': 0.001,
(TaskRunner pid=2224264) 'target_kl': 0.1,
(TaskRunner pid=2224264) 'type': 'fixed'},
(TaskRunner pid=2224264) 'kl_penalty': 'kl',
(TaskRunner pid=2224264) 'lam': 1.0,
(TaskRunner pid=2224264) 'norm_adv_by_std_in_grpo': False,
(TaskRunner pid=2224264) 'pf_ppo': {'reweight_method': 'pow', 'weight_pow': 2.0},
(TaskRunner pid=2224264) 'rollout_correction': {'bypass_mode': False,
(TaskRunner pid=2224264) 'loss_type': 'ppo_clip',
(TaskRunner pid=2224264) 'rollout_is': 'token',
(TaskRunner pid=2224264) 'rollout_is_batch_normalize': False,
(TaskRunner pid=2224264) 'rollout_is_threshold': 2.0,
(TaskRunner pid=2224264) 'rollout_rs': None,
(TaskRunner pid=2224264) 'rollout_rs_threshold': None},
(TaskRunner pid=2224264) 'use_kl_in_reward': False,
(TaskRunner pid=2224264) 'use_pf_ppo': False},
(TaskRunner pid=2224264) 'critic': {'_target_': 'verl.workers.config.FSDPCriticConfig',
(TaskRunner pid=2224264) 'checkpoint': {'_target_': 'verl.trainer.config.CheckpointConfig',
(TaskRunner pid=2224264) 'async_save': False,
(TaskRunner pid=2224264) 'load_contents': ['model', 'optimizer', 'extra'],
(TaskRunner pid=2224264) 'save_contents': ['model', 'optimizer', 'extra']},
(TaskRunner pid=2224264) 'cliprange_value': 0.5,
(TaskRunner pid=2224264) 'data_loader_seed': 42,
(TaskRunner pid=2224264) 'enable': None,
(TaskRunner pid=2224264) 'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=2224264) 'forward_micro_batch_size': None,
(TaskRunner pid=2224264) 'forward_micro_batch_size_per_gpu': None,
(TaskRunner pid=2224264) 'grad_clip': 1.0,
(TaskRunner pid=2224264) 'loss_agg_mode': 'token-mean',
(TaskRunner pid=2224264) 'model': {'_target_': 'verl.workers.config.FSDPCriticModelCfg',
(TaskRunner pid=2224264) 'enable_activation_offload': False,
(TaskRunner pid=2224264) 'enable_gradient_checkpointing': True,
(TaskRunner pid=2224264) 'external_lib': None,
(TaskRunner pid=2224264) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2224264) 'dtype': 'bfloat16',
(TaskRunner pid=2224264) 'entropy_checkpointing': False,
(TaskRunner pid=2224264) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2224264) 'forward_only': False,
(TaskRunner pid=2224264) 'forward_prefetch': False,
(TaskRunner pid=2224264) 'fsdp_size': -1,
(TaskRunner pid=2224264) 'full_determinism': False,
(TaskRunner pid=2224264) 'model_dtype': 'fp32',
(TaskRunner pid=2224264) 'offload_policy': False,
(TaskRunner pid=2224264) 'optimizer_offload': False,
(TaskRunner pid=2224264) 'param_offload': False,
(TaskRunner pid=2224264) 'reshard_after_forward': True,
(TaskRunner pid=2224264) 'seed': 42,
(TaskRunner pid=2224264) 'strategy': 'fsdp',
(TaskRunner pid=2224264) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2224264) 'use_orig_params': False,
(TaskRunner pid=2224264) 'use_torch_compile': True,
(TaskRunner pid=2224264) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2224264) 'lora_alpha': 16,
(TaskRunner pid=2224264) 'lora_rank': 0,
(TaskRunner pid=2224264) 'override_config': {},
(TaskRunner pid=2224264) 'path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2224264) 'target_modules': 'all-linear',
(TaskRunner pid=2224264) 'tiled_mlp': {'enabled': False, 'num_shards': 4},
(TaskRunner pid=2224264) 'tokenizer_path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2224264) 'trust_remote_code': True,
(TaskRunner pid=2224264) 'use_remove_padding': False,
(TaskRunner pid=2224264) 'use_shm': False},
(TaskRunner pid=2224264) 'optim': {'_target_': 'verl.workers.config.FSDPOptimizerConfig',
(TaskRunner pid=2224264) 'betas': [0.9, 0.999],
(TaskRunner pid=2224264) 'clip_grad': 1.0,
(TaskRunner pid=2224264) 'lr': 1e-05,
(TaskRunner pid=2224264) 'lr_scheduler_type': 'constant',
(TaskRunner pid=2224264) 'lr_warmup_steps': -1,
(TaskRunner pid=2224264) 'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=2224264) 'min_lr_ratio': 0.0,
(TaskRunner pid=2224264) 'num_cycles': 0.5,
(TaskRunner pid=2224264) 'optimizer': 'AdamW',
(TaskRunner pid=2224264) 'optimizer_impl': 'torch.optim',
(TaskRunner pid=2224264) 'override_optimizer_config': None,
(TaskRunner pid=2224264) 'total_training_steps': -1,
(TaskRunner pid=2224264) 'warmup_style': None,
(TaskRunner pid=2224264) 'weight_decay': 0.01},
(TaskRunner pid=2224264) 'ppo_epochs': 1,
(TaskRunner pid=2224264) 'ppo_max_token_len_per_gpu': 32768,
(TaskRunner pid=2224264) 'ppo_micro_batch_size': None,
(TaskRunner pid=2224264) 'ppo_micro_batch_size_per_gpu': None,
(TaskRunner pid=2224264) 'ppo_mini_batch_size': 8,
(TaskRunner pid=2224264) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2224264) 'all_ranks': False,
(TaskRunner pid=2224264) 'enable': False,
(TaskRunner pid=2224264) 'ranks': [],
(TaskRunner pid=2224264) 'save_path': 'outputs/profile',
(TaskRunner pid=2224264) 'tool': None,
(TaskRunner pid=2224264) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2224264) 'analysis': True,
(TaskRunner pid=2224264) 'contents': [],
(TaskRunner pid=2224264) 'discrete': False,
(TaskRunner pid=2224264) 'level': 'level0'},
(TaskRunner pid=2224264) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2224264) 'discrete': False},
(TaskRunner pid=2224264) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2224264) 'step_end': None,
(TaskRunner pid=2224264) 'step_start': 0},
(TaskRunner pid=2224264) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2224264) 'stack_depth': 32,
(TaskRunner pid=2224264) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2224264) 'rollout_n': 8,
(TaskRunner pid=2224264) 'shuffle': False,
(TaskRunner pid=2224264) 'strategy': 'fsdp',
(TaskRunner pid=2224264) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2224264) 'use_dynamic_bsz': False},
(TaskRunner pid=2224264) 'custom_reward_function': {'name': 'compute_score',
(TaskRunner pid=2224264) 'path': '/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py'},
(TaskRunner pid=2224264) 'data': {'apply_chat_template_kwargs': {'enable_thinking': False},
(TaskRunner pid=2224264) 'custom_cls': {'name': None, 'path': None},
(TaskRunner pid=2224264) 'datagen': {'name': None, 'path': None},
(TaskRunner pid=2224264) 'dataloader_num_workers': 8,
(TaskRunner pid=2224264) 'filter_overlong_prompts': True,
(TaskRunner pid=2224264) 'filter_overlong_prompts_workers': 1,
(TaskRunner pid=2224264) 'image_key': 'images',
(TaskRunner pid=2224264) 'image_patch_size': 14,
(TaskRunner pid=2224264) 'max_prompt_length': 2048,
(TaskRunner pid=2224264) 'max_response_length': 8192,
(TaskRunner pid=2224264) 'prompt_key': 'prompt',
(TaskRunner pid=2224264) 'return_full_prompt': False,
(TaskRunner pid=2224264) 'return_multi_modal_inputs': True,
(TaskRunner pid=2224264) 'return_raw_chat': True,
(TaskRunner pid=2224264) 'return_raw_input_ids': False,
(TaskRunner pid=2224264) 'reward_fn_key': 'data_source',
(TaskRunner pid=2224264) 'sampler': {'class_name': None, 'class_path': None},
(TaskRunner pid=2224264) 'seed': None,
(TaskRunner pid=2224264) 'shuffle': True,
(TaskRunner pid=2224264) 'tokenizer': None,
(TaskRunner pid=2224264) 'tool_config_path': None,
(TaskRunner pid=2224264) 'train_batch_size': 32,
(TaskRunner pid=2224264) 'train_files': ['/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet'],
(TaskRunner pid=2224264) 'train_max_samples': 3200,
(TaskRunner pid=2224264) 'truncation': 'error',
(TaskRunner pid=2224264) 'trust_remote_code': True,
(TaskRunner pid=2224264) 'use_shm': False,
(TaskRunner pid=2224264) 'val_batch_size': None,
(TaskRunner pid=2224264) 'val_files': ['/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet'],
(TaskRunner pid=2224264) 'val_max_samples': -1,
(TaskRunner pid=2224264) 'validation_shuffle': False,
(TaskRunner pid=2224264) 'video_key': 'videos'},
(TaskRunner pid=2224264) 'global_profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2224264) 'global_tool_config': {'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2224264) 'controller_nsight_options': {'cuda-graph-trace': 'graph',
(TaskRunner pid=2224264) 'cuda-memory-usage': 'true',
(TaskRunner pid=2224264) 'trace': 'cuda,nvtx,cublas,ucx'},
(TaskRunner pid=2224264) 'discrete': False,
(TaskRunner pid=2224264) 'worker_nsight_options': {'capture-range': 'cudaProfilerApi',
(TaskRunner pid=2224264) 'capture-range-end': None,
(TaskRunner pid=2224264) 'cuda-graph-trace': 'graph',
(TaskRunner pid=2224264) 'cuda-memory-usage': 'true',
(TaskRunner pid=2224264) 'kill': 'none',
(TaskRunner pid=2224264) 'trace': 'cuda,nvtx,cublas,ucx'}},
(TaskRunner pid=2224264) 'torch_memory': {'context': 'all',
(TaskRunner pid=2224264) 'kw_args': {},
(TaskRunner pid=2224264) 'stack_depth': 32,
(TaskRunner pid=2224264) 'stacks': 'all',
(TaskRunner pid=2224264) 'trace_alloc_max_entries': 100000}},
(TaskRunner pid=2224264) 'profile_continuous_steps': False,
(TaskRunner pid=2224264) 'save_path': 'outputs/profile',
(TaskRunner pid=2224264) 'steps': None,
(TaskRunner pid=2224264) 'tool': None},
(TaskRunner pid=2224264) 'max_model_len': 10240,
(TaskRunner pid=2224264) 'ray_kwargs': {'ray_init': {'_temp_dir': '/tmp/ray_q3g_chemistry_grpo_Qwen3_8B',
(TaskRunner pid=2224264) 'include_dashboard': False,
(TaskRunner pid=2224264) 'num_cpus': None,
(TaskRunner pid=2224264) 'runtime_env': {'env_vars': {'EXPERIMENT': 'qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2224264) 'TASK': 'datasets/sciknoweval/chemistry',
(TaskRunner pid=2224264) 'USER': 'root'}}},
(TaskRunner pid=2224264) 'timeline_json_file': None},
(TaskRunner pid=2224264) 'reward_manager': {'_target_': 'verl.trainer.config.config.RewardManagerConfig',
(TaskRunner pid=2224264) 'module': {'_target_': 'verl.trainer.config.config.ModuleConfig',
(TaskRunner pid=2224264) 'name': 'custom_reward_manager',
(TaskRunner pid=2224264) 'path': None},
(TaskRunner pid=2224264) 'name': 'naive',
(TaskRunner pid=2224264) 'source': 'register'},
(TaskRunner pid=2224264) 'reward_model': {'enable': False,
(TaskRunner pid=2224264) 'enable_resource_pool': False,
(TaskRunner pid=2224264) 'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=2224264) 'launch_reward_fn_async': False,
(TaskRunner pid=2224264) 'max_length': None,
(TaskRunner pid=2224264) 'micro_batch_size': None,
(TaskRunner pid=2224264) 'micro_batch_size_per_gpu': None,
(TaskRunner pid=2224264) 'model': {'external_lib': None,
(TaskRunner pid=2224264) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2224264) 'forward_prefetch': False,
(TaskRunner pid=2224264) 'fsdp_size': -1,
(TaskRunner pid=2224264) 'param_offload': False,
(TaskRunner pid=2224264) 'reshard_after_forward': True,
(TaskRunner pid=2224264) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2224264) 'input_tokenizer': 'Qwen/Qwen3-8B',
(TaskRunner pid=2224264) 'override_config': {},
(TaskRunner pid=2224264) 'path': '~/models/FsfairX-LLaMA3-RM-v0.1',
(TaskRunner pid=2224264) 'trust_remote_code': False,
(TaskRunner pid=2224264) 'use_fused_kernels': False,
(TaskRunner pid=2224264) 'use_remove_padding': False,
(TaskRunner pid=2224264) 'use_shm': False},
(TaskRunner pid=2224264) 'n_gpus_per_node': 8,
(TaskRunner pid=2224264) 'nnodes': 0,
(TaskRunner pid=2224264) 'num_workers': 1,
(TaskRunner pid=2224264) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2224264) 'all_ranks': False,
(TaskRunner pid=2224264) 'enable': False,
(TaskRunner pid=2224264) 'ranks': [],
(TaskRunner pid=2224264) 'save_path': 'outputs/profile',
(TaskRunner pid=2224264) 'tool': None,
(TaskRunner pid=2224264) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2224264) 'analysis': True,
(TaskRunner pid=2224264) 'contents': [],
(TaskRunner pid=2224264) 'discrete': False,
(TaskRunner pid=2224264) 'level': 'level0'},
(TaskRunner pid=2224264) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2224264) 'discrete': False},
(TaskRunner pid=2224264) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2224264) 'step_end': None,
(TaskRunner pid=2224264) 'step_start': 0},
(TaskRunner pid=2224264) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2224264) 'stack_depth': 32,
(TaskRunner pid=2224264) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2224264) 'reward_loop_class_name': None,
(TaskRunner pid=2224264) 'reward_loop_module_path': None,
(TaskRunner pid=2224264) 'reward_loop_source': 'register',
(TaskRunner pid=2224264) 'reward_manager': 'naive',
(TaskRunner pid=2224264) 'rollout': {'_target_': 'verl.workers.config.RolloutConfig',
(TaskRunner pid=2224264) 'cudagraph_capture_sizes': None,
(TaskRunner pid=2224264) 'data_parallel_size': 1,
(TaskRunner pid=2224264) 'disable_log_stats': True,
(TaskRunner pid=2224264) 'dtype': 'bfloat16',
(TaskRunner pid=2224264) 'enable_chunked_prefill': True,
(TaskRunner pid=2224264) 'enable_prefix_caching': True,
(TaskRunner pid=2224264) 'enforce_eager': True,
(TaskRunner pid=2224264) 'engine_kwargs': {},
(TaskRunner pid=2224264) 'expert_parallel_size': 1,
(TaskRunner pid=2224264) 'free_cache_engine': True,
(TaskRunner pid=2224264) 'gpu_memory_utilization': 0.5,
(TaskRunner pid=2224264) 'limit_images': None,
(TaskRunner pid=2224264) 'load_format': 'auto',
(TaskRunner pid=2224264) 'max_model_len': None,
(TaskRunner pid=2224264) 'max_num_batched_tokens': 8192,
(TaskRunner pid=2224264) 'max_num_seqs': 1024,
(TaskRunner pid=2224264) 'name': '???',
(TaskRunner pid=2224264) 'prompt_length': 2048,
(TaskRunner pid=2224264) 'response_length': 2048,
(TaskRunner pid=2224264) 'skip_tokenizer_init': False,
(TaskRunner pid=2224264) 'tensor_model_parallel_size': 2},
(TaskRunner pid=2224264) 'sandbox_fusion': {'max_concurrent': 64,
(TaskRunner pid=2224264) 'memory_limit_mb': 1024,
(TaskRunner pid=2224264) 'url': None},
(TaskRunner pid=2224264) 'strategy': 'fsdp',
(TaskRunner pid=2224264) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2224264) 'use_dynamic_bsz': False,
(TaskRunner pid=2224264) 'use_reward_loop': False},
(TaskRunner pid=2224264) 'trainer': {'balance_batch': True,
(TaskRunner pid=2224264) 'critic_warmup': 0,
(TaskRunner pid=2224264) 'default_hdfs_dir': None,
(TaskRunner pid=2224264) 'default_local_dir': '/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2224264) 'del_local_ckpt_after_load': False,
(TaskRunner pid=2224264) 'device': 'cuda',
(TaskRunner pid=2224264) 'esi_redundant_time': 0,
(TaskRunner pid=2224264) 'experiment_name': 'qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2224264) 'group_name': 'QWEN3-GRPO-generalization',
(TaskRunner pid=2224264) 'log_val_generations': 0,
(TaskRunner pid=2224264) 'logger': ['console', 'wandb'],
(TaskRunner pid=2224264) 'max_actor_ckpt_to_keep': 100,
(TaskRunner pid=2224264) 'max_critic_ckpt_to_keep': None,
(TaskRunner pid=2224264) 'n_gpus_per_node': 8,
(TaskRunner pid=2224264) 'nnodes': 1,
(TaskRunner pid=2224264) 'project_name': 'SDPO-root',
(TaskRunner pid=2224264) 'ray_wait_register_center_timeout': 300,
(TaskRunner pid=2224264) 'resume_from_path': None,
(TaskRunner pid=2224264) 'resume_mode': 'auto',
(TaskRunner pid=2224264) 'rollout_data_dir': None,
(TaskRunner pid=2224264) 'save_freq': 10,
(TaskRunner pid=2224264) 'test_freq': 10,
(TaskRunner pid=2224264) 'total_epochs': 30,
(TaskRunner pid=2224264) 'total_training_steps': 100,
(TaskRunner pid=2224264) 'use_legacy_worker_impl': 'auto',
(TaskRunner pid=2224264) 'val_before_train': False,
(TaskRunner pid=2224264) 'val_only': False,
(TaskRunner pid=2224264) 'validation_data_dir': None},
(TaskRunner pid=2224264) 'transfer_queue': {'enable': False},
(TaskRunner pid=2224264) 'vars': {'ckpt_dir': '/capstor/scratch/cscs/root/ttrl_runs/datasets/sciknoweval/chemistry',
(TaskRunner pid=2224264) 'dir': '/users/root/SDPO',
(TaskRunner pid=2224264) 'log_dir': '/users/root/output',
(TaskRunner pid=2224264) 'task': 'datasets/sciknoweval/chemistry'}}
(TaskRunner pid=2224264) [validate_config] All configuration checks passed successfully!
(TaskRunner pid=2224264) /mnt/mole/SDPO/L2T/verl/trainer/main_ppo.py:326: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=2224264) use_critic=need_critic(config),
(TaskRunner pid=2224264) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=2224264) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(TaskRunner pid=2224264) Using dataset class: RLHFDataset
(TaskRunner pid=2224264) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(TaskRunner pid=2224264) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(TaskRunner pid=2224264) dataset len: 1890
(TaskRunner pid=2224264) Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2224264) WARNING:2026-07-02 23:56:38,560:Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2224264) Filtering prompts longer than 2048 tokens (num_proc=1): 0%| | 0/1890 [00:00<?, ? examples/s]
(TaskRunner pid=2224264) Filtering prompts longer than 2048 tokens (num_proc=1): 53%|█████▎ | 1000/1890 [00:00<00:00, 1041.98 examples/s]
(TaskRunner pid=2224264) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 1890/1890 [00:01<00:00, 1506.61 examples/s]
(TaskRunner pid=2224264) filter dataset len: 1890
(TaskRunner pid=2224264) Using dataset class: RLHFDataset
(TaskRunner pid=2224264) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 1890/1890 [00:01<00:00, 1304.24 examples/s]
(TaskRunner pid=2224264) dataset len: 210
(TaskRunner pid=2224264) Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2224264) WARNING:2026-07-02 23:56:40,469:Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2224264) Filtering prompts longer than 2048 tokens (num_proc=1): 0%| | 0/210 [00:00<?, ? examples/s]
(TaskRunner pid=2224264) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 210/210 [00:00<00:00, 349.33 examples/s]
(TaskRunner pid=2224264) filter dataset len: 210
(TaskRunner pid=2224264) Size of train dataloader: 59, Size of val dataloader: 1
(TaskRunner pid=2224264) Total training steps: 100
(TaskRunner pid=2224264) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 210/210 [00:00<00:00, 298.23 examples/s]
(TaskRunner pid=2224264) /mnt/mole/SDPO/L2T/verl/trainer/ppo/ray_trainer.py:354: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=2224264) self.use_critic = need_critic(self.config)
(TaskRunner pid=2224264) colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
(TaskRunner pid=2224264) bind role actor_rollout method chat_completion to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2224264) bind role actor_rollout method generate to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2224264) bind role actor_rollout method get_zeromq_address to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2224264) bind role actor_rollout method sleep to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2224264) bind role actor_rollout method wake_up to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(WorkerDict pid=2224678) [W702 23:56:48.468298935 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [::ffff:198.19.44.212]:38433 (errno: 97 - Address family not supported by protocol).
(WorkerDict pid=2224673) [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
(WorkerDict pid=2224673) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(WorkerDict pid=2224673) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(WorkerDict pid=2224680) [W702 23:56:48.948596036 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [::ffff:198.19.44.212]:38433 (errno: 97 - Address family not supported by protocol). [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(WorkerDict pid=2224673) Model config after override: Qwen3Config {
(WorkerDict pid=2224673) "architectures": [
(WorkerDict pid=2224673) "Qwen3ForCausalLM"
(WorkerDict pid=2224673) ],
(WorkerDict pid=2224673) "attention_bias": false,
(WorkerDict pid=2224673) "attention_dropout": 0.0,
(WorkerDict pid=2224673) "dtype": "bfloat16",
(WorkerDict pid=2224673) "eos_token_id": 151645,
(WorkerDict pid=2224673) "head_dim": 128,
(WorkerDict pid=2224673) "hidden_act": "silu",
(WorkerDict pid=2224673) "hidden_size": 4096,
(WorkerDict pid=2224673) "initializer_range": 0.02,
(WorkerDict pid=2224673) "intermediate_size": 12288,
(WorkerDict pid=2224673) "layer_types": [
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention",
(WorkerDict pid=2224673) "full_attention"
(WorkerDict pid=2224673) ],
(WorkerDict pid=2224673) "max_position_embeddings": 40960,
(WorkerDict pid=2224673) "max_window_layers": 36,
(WorkerDict pid=2224673) "model_type": "qwen3",
(WorkerDict pid=2224673) "num_attention_heads": 32,
(WorkerDict pid=2224673) "num_hidden_layers": 36,
(WorkerDict pid=2224673) "num_key_value_heads": 8,
(WorkerDict pid=2224673) "pad_token_id": 151643,
(WorkerDict pid=2224673) "rms_norm_eps": 1e-06,
(WorkerDict pid=2224673) "rope_scaling": null,
(WorkerDict pid=2224673) "rope_theta": 1000000,
(WorkerDict pid=2224673) "sliding_window": null,
(WorkerDict pid=2224673) "tie_word_embeddings": false,
(WorkerDict pid=2224673) "transformers_version": "4.57.1",
(WorkerDict pid=2224673) "use_cache": true,
(WorkerDict pid=2224673) "use_sliding_window": false,
(WorkerDict pid=2224673) "vocab_size": 151936
(WorkerDict pid=2224673) }
(WorkerDict pid=2224673)
(WorkerDict pid=2224673) `torch_dtype` is deprecated! Use `dtype` instead!
(WorkerDict pid=2224673) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
(WorkerDict pid=2224673) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
(WorkerDict pid=2224673) Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
(WorkerDict pid=2224673) Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.96s/it]
(WorkerDict pid=2224680) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(WorkerDict pid=2224680) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2224680) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(WorkerDict pid=2224680) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` [repeated 14x across cluster]
(WorkerDict pid=2224680) Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s] [repeated 7x across cluster]
(WorkerDict pid=2224673) Loading checkpoint shards: 60%|██████ | 3/5 [00:08<00:05, 2.94s/it] [repeated 16x across cluster]
(WorkerDict pid=2224673) Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.06s/it] Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.42s/it]
(WorkerDict pid=2224673) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=2224673) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch
(WorkerDict pid=2224680) [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7 [repeated 7x across cluster]
(WorkerDict pid=2224673) Qwen3ForCausalLM contains 8.19B parameters
(WorkerDict pid=2224673) wrap_policy: functools.partial(<function _or_policy at 0x7283ffa8c540>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7283ffa8c400>, transformer_layer_cls={<class 'transformers.models.qwen3.modeling_qwen3.Qwen3DecoderLayer'>})])
(WorkerDict pid=2224673) NCCL version 2.27.5+cuda12.9
(WorkerDict pid=2224674) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(WorkerDict pid=2224674) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(WorkerDict pid=2224680) Loading checkpoint shards: 80%|████████ | 4/5 [00:12<00:02, 2.93s/it] [repeated 15x across cluster]
(WorkerDict pid=2224680) Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.22s/it] Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.62s/it] [repeated 7x across cluster]
(WorkerDict pid=2224673) Total steps: 100, num_warmup_steps: 10
(WorkerDict pid=2224673) Actor use_remove_padding=True
(WorkerDict pid=2224673) Actor use_fused_kernels=False
(WorkerDict pid=2224673) Actor use_prefix_grouper=False
(WorkerDict pid=2224680) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 7x across cluster]
(WorkerDict pid=2224680) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch [repeated 7x across cluster]
(WorkerDict pid=2224674) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(WorkerDict pid=2224674) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(WorkerDict pid=2224674) [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
(WorkerDict pid=2224673) /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=2224673) warnings.warn(
(WorkerDict pid=2224673) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now. [repeated 7x across cluster]
(WorkerDict pid=2224673) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2224673) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(WorkerDict pid=2224673) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2224680) /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . [repeated 7x across cluster]
(WorkerDict pid=2224680) warnings.warn( [repeated 7x across cluster]
(TaskRunner pid=2224264) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=2224264) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(vLLMHttpServer pid=2225569) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(vLLMHttpServer pid=2225569) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(vLLMHttpServer pid=2225570) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(vLLMHttpServer pid=2225570) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(vLLMHttpServer pid=2225569) ['serve',
(vLLMHttpServer pid=2225569) 'Qwen/Qwen3-8B',
(vLLMHttpServer pid=2225569) '--dtype',
(vLLMHttpServer pid=2225569) 'bfloat16',
(vLLMHttpServer pid=2225569) '--load_format',
(vLLMHttpServer pid=2225569) 'dummy',
(vLLMHttpServer pid=2225569) '--trust_remote_code',
(vLLMHttpServer pid=2225569) '--max_model_len',
(vLLMHttpServer pid=2225569) '40960',
(vLLMHttpServer pid=2225569) '--max_num_seqs',
(vLLMHttpServer pid=2225569) '1024',
(vLLMHttpServer pid=2225569) '--enable_chunked_prefill',
(vLLMHttpServer pid=2225569) '--max_num_batched_tokens',
(vLLMHttpServer pid=2225569) '10240',
(vLLMHttpServer pid=2225569) '--enable_prefix_caching',
(vLLMHttpServer pid=2225569) '--enable_sleep_mode',
(vLLMHttpServer pid=2225569) '--logprobs_mode',
(vLLMHttpServer pid=2225569) 'processed_logprobs',
(vLLMHttpServer pid=2225569) '--disable_custom_all_reduce',
(vLLMHttpServer pid=2225569) '--gpu_memory_utilization',
(vLLMHttpServer pid=2225569) '0.8',
(vLLMHttpServer pid=2225569) '--disable_log_stats',
(vLLMHttpServer pid=2225569) '--tensor_parallel_size',
(vLLMHttpServer pid=2225569) '2',
(vLLMHttpServer pid=2225569) '--seed',
(vLLMHttpServer pid=2225569) '0',
(vLLMHttpServer pid=2225569) '--override_generation_config',
(vLLMHttpServer pid=2225569) '{"temperature": 1.0, "top_k": -1, "top_p": 1.0, "repetition_penalty": 1.0, '
(vLLMHttpServer pid=2225569) '"max_new_tokens": 8192}',
(vLLMHttpServer pid=2225569) '--hf_overrides',
(vLLMHttpServer pid=2225569) '{}',
(vLLMHttpServer pid=2225569) '--scheduling_policy',
(vLLMHttpServer pid=2225569) 'fcfs']
(WorkerDict pid=2224680) [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [repeated 23x across cluster]
(vLLMHttpServer pid=2225569) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(vLLMHttpServer pid=2225569) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(vLLMHttpServer pid=2225569) WARNING 07-02 23:57:38 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(WorkerDict pid=2224673) WARNING 07-02 23:57:46 [worker_base.py:301] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'.
(vLLMHttpServer pid=2225571) WARNING 07-02 23:57:39 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(WorkerDict pid=2224675) NCCL version 2.27.5+cuda12.9
(WorkerDict pid=2224677) 2026-07-02 23:58:01,108 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(vLLMHttpServer pid=2225571) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 2x across cluster]
(vLLMHttpServer pid=2225571) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 2x across cluster]
(vLLMHttpServer pid=2225571) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 3x across cluster]
(vLLMHttpServer pid=2225571) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 3x across cluster]
(WorkerDict pid=2224678) 2026-07-02 23:58:01,122 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 2%|▏ | 1/51 [00:00<00:05, 9.84it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 6%|▌ | 3/51 [00:00<00:03, 15.33it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 10%|▉ | 5/51 [00:00<00:02, 16.95it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 14%|█▎ | 7/51 [00:00<00:02, 17.32it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 18%|█▊ | 9/51 [00:00<00:02, 17.07it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 22%|██▏ | 11/51 [00:00<00:02, 17.25it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 25%|██▌ | 13/51 [00:00<00:02, 17.04it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 29%|██▉ | 15/51 [00:00<00:02, 16.86it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 33%|███▎ | 17/51 [00:01<00:02, 16.82it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 37%|███▋ | 19/51 [00:01<00:01, 17.13it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 41%|████ | 21/51 [00:01<00:01, 17.13it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 45%|████▌ | 23/51 [00:01<00:01, 16.62it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 49%|████▉ | 25/51 [00:01<00:01, 16.48it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 53%|█████▎ | 27/51 [00:01<00:01, 16.36it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 57%|█████▋ | 29/51 [00:01<00:01, 15.74it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 61%|██████ | 31/51 [00:01<00:01, 15.80it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 65%|██████▍ | 33/51 [00:02<00:01, 15.50it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 69%|██████▊ | 35/51 [00:02<00:01, 15.58it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 73%|███████▎ | 37/51 [00:02<00:00, 15.29it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 76%|███████▋ | 39/51 [00:02<00:00, 14.89it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 80%|████████ | 41/51 [00:02<00:00, 14.86it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 84%|████████▍ | 43/51 [00:02<00:00, 13.86it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 88%|████████▊ | 45/51 [00:02<00:00, 13.92it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 92%|█████████▏| 47/51 [00:03<00:00, 13.81it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 96%|█████████▌| 49/51 [00:03<00:00, 12.90it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 13.17it/s] Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 15.25it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 0%| | 0/51 [00:00<?, ?it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 4%|▍ | 2/51 [00:00<00:02, 18.50it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 10%|▉ | 5/51 [00:00<00:02, 20.71it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 16%|█▌ | 8/51 [00:00<00:02, 20.25it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 22%|██▏ | 11/51 [00:00<00:01, 21.00it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 27%|██▋ | 14/51 [00:00<00:01, 21.25it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 33%|███▎ | 17/51 [00:00<00:01, 21.37it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 39%|███▉ | 20/51 [00:00<00:01, 21.58it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 45%|████▌ | 23/51 [00:01<00:01, 21.93it/s]
(WorkerDict pid=2224674) 2026-07-02 23:58:01,335 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ... [repeated 7x across cluster]
(WorkerDict pid=2224674) 2026-07-02 23:58:01,351 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends [repeated 7x across cluster]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 51%|█████ | 26/51 [00:01<00:01, 22.21it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 57%|█████▋ | 29/51 [00:01<00:00, 22.31it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 63%|██████▎ | 32/51 [00:01<00:00, 22.42it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 69%|██████▊ | 35/51 [00:01<00:00, 22.45it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 75%|███████▍ | 38/51 [00:01<00:00, 22.29it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 80%|████████ | 41/51 [00:01<00:00, 21.34it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 86%|████████▋ | 44/51 [00:02<00:00, 21.18it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 92%|█████████▏| 47/51 [00:02<00:00, 20.82it/s]
(WorkerDict pid=2224673) Capturing CUDA graphs (decode, FULL): 98%|█████████▊| 50/51 [00:02<00:00, 21.22it/s] Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:02<00:00, 21.51it/s]
(vLLMHttpServer pid=2225571) WARNING 07-02 23:58:09 [api_server.py:1358] LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development!
(WorkerDict pid=2224676) WARNING 07-02 23:57:47 [worker_base.py:301] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 7x across cluster]
(WorkerDict pid=2224679) NCCL version 2.27.5+cuda12.9 [repeated 2x across cluster]
(vLLMHttpServer pid=2225571) WARNING 07-02 23:58:09 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(TaskRunner pid=2224264) AgentLoopManager: ['198.19.44.212:42387', '198.19.44.212:39253', '198.19.44.212:42941', '198.19.44.212:38755']
(TaskRunner pid=2224264) wandb: Currently logged in as: seongryongjung (seongryongjung-chung-ang-university) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
(TaskRunner pid=2224264) wandb: setting up run ffovp19e
(TaskRunner pid=2224264) wandb: Tracking run with wandb version 0.23.1
(TaskRunner pid=2224264) wandb: Run data is saved locally in /mnt/mole/SDPO/L2T/wandb/run-20260702_235814-ffovp19e
(TaskRunner pid=2224264) wandb: Run `wandb offline` to turn off syncing.
(TaskRunner pid=2224264) wandb: Syncing run qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
(TaskRunner pid=2224264) wandb: ⭐️ View project at https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root
(TaskRunner pid=2224264) wandb: 🚀 View run at https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root/runs/ffovp19e
(TaskRunner pid=2224264) Checkpoint tracker file does not exist: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/latest_checkpointed_iteration.txt
(TaskRunner pid=2224264) Training from scratch
(vLLMHttpServer pid=2225569) WARNING 07-02 23:58:09 [api_server.py:1358] LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development! [repeated 3x across cluster]
(vLLMHttpServer pid=2225569) WARNING 07-02 23:58:10 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(TaskRunner pid=2224264) wandb: Detected [openai] in use.
(TaskRunner pid=2224264) wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
(TaskRunner pid=2224264) wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
(TaskRunner pid=2224264) Training Progress: 0%| | 0/100 [00:00<?, ?it/s]
(AgentLoopWorker pid=2226513) Using dataset class: RLHFDataset
(AgentLoopWorker pid=2226513) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(AgentLoopWorker pid=2226513) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(AgentLoopWorker pid=2226515) You're using a Qwen2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
(AgentLoopWorker pid=2226522) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(AgentLoopWorker pid=2226522) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(TaskRunner pid=2224264) step:1 - global_seqlen/min:19322 - global_seqlen/max:25315 - global_seqlen/minmax_diff:5993 - global_seqlen/balanced_min:22987 - global_seqlen/balanced_max:23220 - global_seqlen/mean:23039.25 - actor/entropy:0.5288177132606506 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.21512234210968018 - training/rollout_probs_diff_mean:0.004612532444298267 - training/rollout_probs_diff_std:0.009421703405678272 - training/rollout_actor_probs_pearson_corr:0.9995055794715881 - rollout_corr/rollout_is_mean:0.9999278783798218 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.5366545742144808e-05 - rollout_corr/rollout_is_max:1.599510669708252 - rollout_corr/rollout_is_min:0.4775051474571228 - rollout_corr/rollout_is_std:0.03466796875 - rollout_corr/rollout_is_eff_sample_size:0.9987994557723341 - rollout_corr/rollout_is_seq_mean:0.9999526739120483 - rollout_corr/rollout_is_seq_std:0.0017850211588665843 - rollout_corr/rollout_is_seq_max:1.0056862831115723 - rollout_corr/rollout_is_seq_min:0.9945246577262878 - rollout_corr/rollout_is_seq_max_deviation:0.005686283111572266 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7541180718690157) - rollout_corr/training_log_ppl:np.float64(0.5497869531391189) - rollout_corr/kl:np.float64(0.00068277636619829) - rollout_corr/k3_kl:np.float64(0.0006354950018021555) - rollout_corr/rollout_ppl:np.float64(1.7529067401774228) - rollout_corr/rollout_log_ppl:np.float64(0.5491041730856523) - rollout_corr/log_ppl_diff:np.float64(0.0006827800534665585) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015149706741794944) - rollout_corr/log_ppl_diff_max:np.float64(0.005993187427520752) - rollout_corr/log_ppl_diff_min:np.float64(-0.004957437515258789) - rollout_corr/ppl_ratio:np.float64(1.0006846021860838) - rollout_corr/chi2_token:np.float64(0.0011827307753264904) - rollout_corr/chi2_seq:np.float64(0.5518562712240964) - actor/pg_loss:np.float64(-0.00024519627913832664) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0) - actor/ppo_kl:np.float64(0.0) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3594253621995449) - perf/mfu/actor:np.float64(0.08964318930500031) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(136.935546875) - perf/cpu_memory_used_gb:np.float64(102.53416061401367) - actor/lr:np.float64(0.0) - training/global_step:1 - training/epoch:0 - critic/score/mean:0.48046875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.48046875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0064135086722671986 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0064135086722671986 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:461.9765625 - response_length/max:1716.0 - response_length/min:149.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:461.9765625 - response_length_non_aborted/max:1716.0 - response_length_non_aborted/min:149.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:258.0 - prompt_length/max:491.0 - prompt_length/min:161.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00023651309311389923 - timing_s/agent_loop/generate_sequences/min:np.float64(13.904150538146496) - timing_s/agent_loop/generate_sequences/max:np.float64(27.40781383961439) - timing_s/agent_loop/generate_sequences/mean:np.float64(18.873495450563496) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(27.40781383961439) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:280 - timing_s/agent_loop/slowest/response_length:1278 - timing_s/gen:38.962751153856516 - timing_s/reward:0.06675866805016994 - timing_s/old_log_prob:3.8775708489120007 - timing_s/adv:0.16563834808766842 - timing_s/update_actor:13.486334782093763 - timing_s/step:56.663143653422594 - timing_s/stop_profile:0.00013279728591442108 - timing_per_token_ms/adv:0.0008986748054280653 - timing_per_token_ms/gen:0.3294501475813549 - timing_per_token_ms/update_actor:0.07317043079795221 - perf/total_num_tokens:184314 - perf/time_per_step:56.663143653422594 - perf/throughput:406.60027867353193
(AgentLoopWorker pid=2226520) Using dataset class: RLHFDataset [repeated 7x across cluster]
(TaskRunner pid=2224264) Training Progress: 1%| | 1/100 [00:56<1:33:50, 56.88s/it]
(AgentLoopWorker pid=2226513) You're using a Qwen2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding. [repeated 7x across cluster]
(TaskRunner pid=2224264) step:2 - global_seqlen/min:18367 - global_seqlen/max:28812 - global_seqlen/minmax_diff:10445 - global_seqlen/balanced_min:24319 - global_seqlen/balanced_max:24347 - global_seqlen/mean:24339.125 - actor/entropy:0.5474216938018799 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5073403120040894 - training/rollout_probs_diff_mean:0.00468479422852397 - training/rollout_probs_diff_std:0.00983211025595665 - training/rollout_actor_probs_pearson_corr:0.9994499683380127 - rollout_corr/rollout_is_mean:1.00003981590271 - rollout_corr/rollout_is_ratio_fraction_high:8.855121450324077e-06 - rollout_corr/rollout_is_ratio_fraction_low:8.855121450324077e-06 - rollout_corr/rollout_is_max:2.8898818492889404 - rollout_corr/rollout_is_min:0.4623408317565918 - rollout_corr/rollout_is_std:0.03588784113526344 - rollout_corr/rollout_is_eff_sample_size:0.9987138383870012 - rollout_corr/rollout_is_seq_mean:1.0000025033950806 - rollout_corr/rollout_is_seq_std:0.002084122272208333 - rollout_corr/rollout_is_seq_max:1.0075422525405884 - rollout_corr/rollout_is_seq_min:0.9931289553642273 - rollout_corr/rollout_is_seq_max_deviation:0.007542252540588379 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.808347711339593) - rollout_corr/training_log_ppl:np.float64(0.5770619873364922) - rollout_corr/kl:np.float64(0.0005897953656575083) - rollout_corr/k3_kl:np.float64(0.000690597933726167) - rollout_corr/rollout_ppl:np.float64(1.8072920083068311) - rollout_corr/rollout_log_ppl:np.float64(0.5764721901505254) - rollout_corr/log_ppl_diff:np.float64(0.000589797185966745) - rollout_corr/log_ppl_abs_diff:np.float64(0.001534482667921111) - rollout_corr/log_ppl_diff_max:np.float64(0.00782063603401184) - rollout_corr/log_ppl_diff_min:np.float64(-0.004696309566497803) - rollout_corr/ppl_ratio:np.float64(1.0005918587557971) - rollout_corr/chi2_token:np.float64(0.0016041339840739965) - rollout_corr/chi2_seq:np.float64(0.7937973756343126) - actor/pg_loss:np.float64(1.557217910885811e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00031911871974443784) - actor/ppo_kl:np.float64(-0.00011317147449219078) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3977566659450531) - perf/mfu/actor:np.float64(0.09740370282378204) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(136.9375) - perf/cpu_memory_used_gb:np.float64(101.18880462646484) - actor/lr:np.float64(1e-07) - training/global_step:2 - training/epoch:0 - critic/score/mean:0.375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.014309875667095184 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:-0.014309875667095184 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:441.12890625 - response_length/max:1401.0 - response_length/min:107.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:441.12890625 - response_length_non_aborted/max:1401.0 - response_length_non_aborted/min:107.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:319.46875 - prompt_length/max:818.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001631416380405426 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2865754403173923) - timing_s/agent_loop/generate_sequences/max:np.float64(10.83000554330647) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.861061018127657) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.83000554330647) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:298 - timing_s/agent_loop/slowest/response_length:1401 - timing_s/gen:17.44803098961711 - timing_s/reward:0.0691465511918068 - timing_s/old_log_prob:3.25926635786891 - timing_s/adv:0.13879693113267422 - timing_s/update_actor:13.240625251084566 - timing_s/step:34.242017718032 - timing_s/stop_profile:0.00012666545808315277 - timing_per_token_ms/adv:0.0007128282710074531 - timing_per_token_ms/gen:0.15450443189629864 - timing_per_token_ms/update_actor:0.06800072543222366 - perf/total_num_tokens:194713 - perf/time_per_step:34.242017718032 - perf/throughput:710.7970447425738
(TaskRunner pid=2224264) Training Progress: 2%|▏ | 2/100 [01:31<1:11:14, 43.61s/it]
(TaskRunner pid=2224264) step:3 - global_seqlen/min:19797 - global_seqlen/max:28335 - global_seqlen/minmax_diff:8538 - global_seqlen/balanced_min:24282 - global_seqlen/balanced_max:24340 - global_seqlen/mean:24297.5 - actor/entropy:0.5263562202453613 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.13323187828063965 - training/rollout_probs_diff_mean:0.004719756543636322 - training/rollout_probs_diff_std:0.009652746841311455 - training/rollout_actor_probs_pearson_corr:0.9994633197784424 - rollout_corr/rollout_is_mean:0.9999251961708069 - rollout_corr/rollout_is_ratio_fraction_high:8.550809070584364e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.3356199264526367 - rollout_corr/rollout_is_min:0.5226753354072571 - rollout_corr/rollout_is_std:0.03476754575967789 - rollout_corr/rollout_is_eff_sample_size:0.998792439351094 - rollout_corr/rollout_is_seq_mean:0.9998748302459717 - rollout_corr/rollout_is_seq_std:0.001937155844643712 - rollout_corr/rollout_is_seq_max:1.0051332712173462 - rollout_corr/rollout_is_seq_min:0.994463324546814 - rollout_corr/rollout_is_seq_max_deviation:0.005536675453186035 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7754999860189855) - rollout_corr/training_log_ppl:np.float64(0.5597236861940473) - rollout_corr/kl:np.float64(0.000738800011342633) - rollout_corr/k3_kl:np.float64(0.000668586852611952) - rollout_corr/rollout_ppl:np.float64(1.7741392091847956) - rollout_corr/rollout_log_ppl:np.float64(0.5589848817326128) - rollout_corr/log_ppl_diff:np.float64(0.0007388044614344835) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017042472027242184) - rollout_corr/log_ppl_diff_max:np.float64(0.0062329769134521484) - rollout_corr/log_ppl_diff_min:np.float64(-0.005337923765182495) - rollout_corr/ppl_ratio:np.float64(1.0007410887628794) - rollout_corr/chi2_token:np.float64(0.0012068876530975103) - rollout_corr/chi2_seq:np.float64(1.152974521741271) - actor/pg_loss:np.float64(-0.00012802286073565483) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003878075890497712) - actor/ppo_kl:np.float64(-4.74486657751072e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4122833386063576) - perf/mfu/actor:np.float64(0.09657854222960446) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(136.9375) - perf/cpu_memory_used_gb:np.float64(100.6053695678711) - actor/lr:np.float64(2e-07) - training/global_step:3 - training/epoch:0 - critic/score/mean:0.44921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.44921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00122917874250561 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00122917874250561 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:456.828125 - response_length/max:1459.0 - response_length/min:112.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:456.828125 - response_length_non_aborted/max:1459.0 - response_length_non_aborted/min:112.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.46875 - prompt_length/max:557.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.80570912361145e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4546770006418228) - timing_s/agent_loop/generate_sequences/max:np.float64(11.504085250198841) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.162285889819032) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.504085250198841) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:301 - timing_s/agent_loop/slowest/response_length:1459 - timing_s/gen:18.921971820294857 - timing_s/reward:0.0675499476492405 - timing_s/old_log_prob:3.2109978441148996 - timing_s/adv:0.14413173869252205 - timing_s/update_actor:13.062051558867097 - timing_s/step:35.49862107075751 - timing_s/stop_profile:0.00011751241981983185 - timing_per_token_ms/adv:0.0007414946943745346 - timing_per_token_ms/gen:0.1617981651699461 - timing_per_token_ms/update_actor:0.06719853667490018 - perf/total_num_tokens:194380 - perf/time_per_step:35.49862107075751 - perf/throughput:684.4632063755122
(TaskRunner pid=2224264) Training Progress: 3%|▎ | 3/100 [02:06<1:04:33, 39.93s/it]
(TaskRunner pid=2224264) step:4 - global_seqlen/min:22442 - global_seqlen/max:26662 - global_seqlen/minmax_diff:4220 - global_seqlen/balanced_min:24730 - global_seqlen/balanced_max:24744 - global_seqlen/mean:24738.75 - actor/entropy:0.5523660778999329 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.6050522327423096 - training/rollout_probs_diff_mean:0.004622706677764654 - training/rollout_probs_diff_std:0.009672068059444427 - training/rollout_actor_probs_pearson_corr:0.9995031356811523 - rollout_corr/rollout_is_mean:1.0001981258392334 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.5127732442342676e-05 - rollout_corr/rollout_is_max:1.8165526390075684 - rollout_corr/rollout_is_min:0.13162213563919067 - rollout_corr/rollout_is_std:0.03505437821149826 - rollout_corr/rollout_is_eff_sample_size:0.998773174430212 - rollout_corr/rollout_is_seq_mean:1.0002080202102661 - rollout_corr/rollout_is_seq_std:0.0018218032782897353 - rollout_corr/rollout_is_seq_max:1.005950689315796 - rollout_corr/rollout_is_seq_min:0.9943360090255737 - rollout_corr/rollout_is_seq_max_deviation:0.0059506893157958984 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8033743798732758) - rollout_corr/training_log_ppl:np.float64(0.5734791937866248) - rollout_corr/kl:np.float64(0.0005666991043140612) - rollout_corr/k3_kl:np.float64(0.0006585608642808438) - rollout_corr/rollout_ppl:np.float64(1.8023865297436714) - rollout_corr/rollout_log_ppl:np.float64(0.5729124903446063) - rollout_corr/log_ppl_diff:np.float64(0.0005667034420184791) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015296083292923868) - rollout_corr/log_ppl_diff_max:np.float64(0.006748795509338379) - rollout_corr/log_ppl_diff_min:np.float64(-0.005822420120239258) - rollout_corr/ppl_ratio:np.float64(1.000568695133552) - rollout_corr/chi2_token:np.float64(0.001501972321420908) - rollout_corr/chi2_seq:np.float64(1.0207945266738534) - actor/pg_loss:np.float64(2.6238849386572838e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014007428308104863) - actor/ppo_kl:np.float64(0.00010998321266719735) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3790104538202286) - perf/mfu/actor:np.float64(0.10117752971397821) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(136.9375) - perf/cpu_memory_used_gb:np.float64(99.83869171142578) - actor/lr:np.float64(3e-07) - training/global_step:4 - training/epoch:0 - critic/score/mean:0.42578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.42578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.008165465667843819 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.008165465667843819 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:466.3671875 - response_length/max:1192.0 - response_length/min:113.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:466.3671875 - response_length_non_aborted/max:1192.0 - response_length_non_aborted/min:113.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:306.71875 - prompt_length/max:604.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015220977365970612 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4423759039491415) - timing_s/agent_loop/generate_sequences/max:np.float64(10.007848922163248) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.2785777673416305) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.007848922163248) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:289 - timing_s/agent_loop/slowest/response_length:1192 - timing_s/gen:16.54260015115142 - timing_s/reward:0.0658042561262846 - timing_s/old_log_prob:3.2218400053679943 - timing_s/adv:0.15534020774066448 - timing_s/update_actor:12.887455742806196 - timing_s/step:32.961092529818416 - timing_s/stop_profile:0.00016492418944835663 - timing_per_token_ms/adv:0.0007849032779579833 - timing_per_token_ms/gen:0.1385593445946178 - timing_per_token_ms/update_actor:0.06511775929870242 - perf/total_num_tokens:197910 - perf/time_per_step:32.961092529818416 - perf/throughput:750.543992970499
(TaskRunner pid=2224264) Training Progress: 4%|▍ | 4/100 [02:39<59:31, 37.20s/it]
(TaskRunner pid=2224264) step:5 - global_seqlen/min:18776 - global_seqlen/max:28441 - global_seqlen/minmax_diff:9665 - global_seqlen/balanced_min:22325 - global_seqlen/balanced_max:29119 - global_seqlen/mean:23183.375 - actor/entropy:0.5180944204330444 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2772417664527893 - training/rollout_probs_diff_mean:0.004665252286940813 - training/rollout_probs_diff_std:0.009622659534215927 - training/rollout_actor_probs_pearson_corr:0.9994248151779175 - rollout_corr/rollout_is_mean:0.9999130964279175 - rollout_corr/rollout_is_ratio_fraction_high:8.664685083203949e-06 - rollout_corr/rollout_is_ratio_fraction_low:8.664685083203949e-06 - rollout_corr/rollout_is_max:2.8200924396514893 - rollout_corr/rollout_is_min:0.4454187750816345 - rollout_corr/rollout_is_std:0.03461463004350662 - rollout_corr/rollout_is_eff_sample_size:0.9988030234819574 - rollout_corr/rollout_is_seq_mean:0.9998463988304138 - rollout_corr/rollout_is_seq_std:0.001816977746784687 - rollout_corr/rollout_is_seq_max:1.0054938793182373 - rollout_corr/rollout_is_seq_min:0.9929466843605042 - rollout_corr/rollout_is_seq_max_deviation:0.00705331563949585 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7746925223618746) - rollout_corr/training_log_ppl:np.float64(0.5630445900387713) - rollout_corr/kl:np.float64(0.0009524658903821148) - rollout_corr/k3_kl:np.float64(0.0006796251157652478) - rollout_corr/rollout_ppl:np.float64(1.7729679476469755) - rollout_corr/rollout_log_ppl:np.float64(0.5620921208319487) - rollout_corr/log_ppl_diff:np.float64(0.0009524692068225704) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016687612733221613) - rollout_corr/log_ppl_diff_max:np.float64(0.006459951400756836) - rollout_corr/log_ppl_diff_min:np.float64(-0.003880620002746582) - rollout_corr/ppl_ratio:np.float64(1.000954748596996) - rollout_corr/chi2_token:np.float64(0.0008346459362655878) - rollout_corr/chi2_seq:np.float64(0.2616440069396049) - actor/pg_loss:np.float64(-1.638184767216444e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003420987316076207) - actor/ppo_kl:np.float64(0.00012373590199388218) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.38139186054468155) - perf/mfu/actor:np.float64(0.08868156608274999) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(99.51568698883057) - actor/lr:np.float64(4e-07) - training/global_step:5 - training/epoch:0 - critic/score/mean:0.41015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.41015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0017578480765223503 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:-0.0017578480765223503 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:450.82421875 - response_length/max:8192.0 - response_length/min:93.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:450.82421875 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:93.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:273.65625 - prompt_length/max:520.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010846368968486786 - timing_s/agent_loop/generate_sequences/min:np.float64(1.193764079362154) - timing_s/agent_loop/generate_sequences/max:np.float64(59.1775236427784) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.851797078110394) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(59.1775236427784) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:347 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:65.853411315009 - timing_s/reward:0.063438905403018 - timing_s/old_log_prob:3.305457219481468 - timing_s/adv:0.151566743850708 - timing_s/update_actor:13.759135007858276 - timing_s/step:83.21836626902223 - timing_s/stop_profile:0.00011375546455383301 - timing_per_token_ms/adv:0.0008172167763036444 - timing_per_token_ms/gen:0.5705990877386817 - timing_per_token_ms/update_actor:0.07418643213001923 - perf/total_num_tokens:185467 - perf/time_per_step:83.21836626902223 - perf/throughput:278.5848369704169
(TaskRunner pid=2224264) Training Progress: 5%|▌ | 5/100 [04:03<1:25:12, 53.81s/it]
(TaskRunner pid=2224264) step:6 - global_seqlen/min:20366 - global_seqlen/max:28896 - global_seqlen/minmax_diff:8530 - global_seqlen/balanced_min:23779 - global_seqlen/balanced_max:24033 - global_seqlen/mean:23815.375 - actor/entropy:0.5188326835632324 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2553183436393738 - training/rollout_probs_diff_mean:0.004564165603369474 - training/rollout_probs_diff_std:0.00953100249171257 - training/rollout_actor_probs_pearson_corr:0.9994511604309082 - rollout_corr/rollout_is_mean:0.9998612403869629 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.322029316332191e-06 - rollout_corr/rollout_is_max:1.6952842473983765 - rollout_corr/rollout_is_min:0.40082022547721863 - rollout_corr/rollout_is_std:0.03409762680530548 - rollout_corr/rollout_is_eff_sample_size:0.998838464114971 - rollout_corr/rollout_is_seq_mean:0.9999702572822571 - rollout_corr/rollout_is_seq_std:0.0019584151450544596 - rollout_corr/rollout_is_seq_max:1.0068446397781372 - rollout_corr/rollout_is_seq_min:0.9937593340873718 - rollout_corr/rollout_is_seq_max_deviation:0.006844639778137207 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.75880296388641) - rollout_corr/training_log_ppl:np.float64(0.5500174012850039) - rollout_corr/kl:np.float64(0.0007605231542111568) - rollout_corr/k3_kl:np.float64(0.0006334543155617212) - rollout_corr/rollout_ppl:np.float64(1.7574654794298112) - rollout_corr/rollout_log_ppl:np.float64(0.5492568750632927) - rollout_corr/log_ppl_diff:np.float64(0.0007605262217111886) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016004611388780177) - rollout_corr/log_ppl_diff_max:np.float64(0.006067186594009399) - rollout_corr/log_ppl_diff_min:np.float64(-0.005928128957748413) - rollout_corr/ppl_ratio:np.float64(1.000762639567256) - rollout_corr/chi2_token:np.float64(0.0010106135159730911) - rollout_corr/chi2_seq:np.float64(0.4114875444211066) - actor/pg_loss:np.float64(-0.000129641848616302) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000321736315072485) - actor/ppo_kl:np.float64(9.994746215014061e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.36745208501815796) - perf/mfu/actor:np.float64(0.09673023323806183) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(99.67207098007202) - actor/lr:np.float64(5e-07) - training/global_step:6 - training/epoch:0 - critic/score/mean:0.4140625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4140625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0014771601418033242 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0014771601418033242 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:469.38671875 - response_length/max:1729.0 - response_length/min:119.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:469.38671875 - response_length_non_aborted/max:1729.0 - response_length_non_aborted/min:119.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:274.84375 - prompt_length/max:450.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010452978312969208 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3636087086051702) - timing_s/agent_loop/generate_sequences/max:np.float64(12.901104992255569) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.271141361357877) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.901104992255569) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:266 - timing_s/agent_loop/slowest/response_length:1729 - timing_s/gen:20.175010323524475 - timing_s/reward:0.06442111358046532 - timing_s/old_log_prob:3.271257309243083 - timing_s/adv:0.14838975109159946 - timing_s/update_actor:12.736947799101472 - timing_s/step:36.482897439971566 - timing_s/stop_profile:0.00020318478345870972 - timing_per_token_ms/adv:0.0007788547896663367 - timing_per_token_ms/gen:0.1678970259025197 - timing_per_token_ms/update_actor:0.06685254693187422 - perf/total_num_tokens:190523 - perf/time_per_step:36.482897439971566 - perf/throughput:652.7818970295732
(TaskRunner pid=2224264) Training Progress: 6%|▌ | 6/100 [04:39<1:15:06, 47.94s/it]
(TaskRunner pid=2224264) step:7 - global_seqlen/min:20657 - global_seqlen/max:26337 - global_seqlen/minmax_diff:5680 - global_seqlen/balanced_min:23234 - global_seqlen/balanced_max:23251 - global_seqlen/mean:23239.0 - actor/entropy:0.5476059913635254 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24462002515792847 - training/rollout_probs_diff_mean:0.004715604707598686 - training/rollout_probs_diff_std:0.009818820282816887 - training/rollout_actor_probs_pearson_corr:0.9994401335716248 - rollout_corr/rollout_is_mean:1.0001124143600464 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.5858569145202637 - rollout_corr/rollout_is_min:0.5473191738128662 - rollout_corr/rollout_is_std:0.03527304157614708 - rollout_corr/rollout_is_eff_sample_size:0.9987575965499879 - rollout_corr/rollout_is_seq_mean:1.0001611709594727 - rollout_corr/rollout_is_seq_std:0.0020254836417734623 - rollout_corr/rollout_is_seq_max:1.0101655721664429 - rollout_corr/rollout_is_seq_min:0.9944638013839722 - rollout_corr/rollout_is_seq_max_deviation:0.010165572166442871 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7864349940791726) - rollout_corr/training_log_ppl:np.float64(0.5668349982588552) - rollout_corr/kl:np.float64(0.0004977610002434574) - rollout_corr/k3_kl:np.float64(0.000654406503088012) - rollout_corr/rollout_ppl:np.float64(1.7855724152177572) - rollout_corr/rollout_log_ppl:np.float64(0.5663372349808924) - rollout_corr/log_ppl_diff:np.float64(0.0004977632779628038) - rollout_corr/log_ppl_abs_diff:np.float64(0.001586579019203782) - rollout_corr/log_ppl_diff_max:np.float64(0.006533026695251465) - rollout_corr/log_ppl_diff_min:np.float64(-0.008994758129119873) - rollout_corr/ppl_ratio:np.float64(1.0005000154487789) - rollout_corr/chi2_token:np.float64(0.001633110223338008) - rollout_corr/chi2_seq:np.float64(1.2687437802087516) - actor/pg_loss:np.float64(0.000499273301102221) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00025790440895434585) - actor/ppo_kl:np.float64(-1.094456836625568e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.37454719841480255) - perf/mfu/actor:np.float64(0.09487462357860618) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(99.51276779174805) - actor/lr:np.float64(6e-07) - training/global_step:7 - training/epoch:0 - critic/score/mean:0.28125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.28125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004087449051439762 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:-0.004087449051439762 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:437.21875 - response_length/max:1408.0 - response_length/min:119.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:437.21875 - response_length_non_aborted/max:1408.0 - response_length_non_aborted/min:119.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:289.0 - prompt_length/max:662.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010518915951251984 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2608289290219545) - timing_s/agent_loop/generate_sequences/max:np.float64(11.097910234704614) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.5351159983474645) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.097910234704614) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:230 - timing_s/agent_loop/slowest/response_length:1408 - timing_s/gen:17.775591058656573 - timing_s/reward:0.06724509596824646 - timing_s/old_log_prob:3.1036592219024897 - timing_s/adv:0.15197663940489292 - timing_s/update_actor:12.948086498305202 - timing_s/step:34.13282694108784 - timing_s/stop_profile:0.0001219138503074646 - timing_per_token_ms/adv:0.0008174654643320115 - timing_per_token_ms/gen:0.15881272834908666 - timing_per_token_ms/update_actor:0.06964631921718448 - perf/total_num_tokens:185912 - perf/time_per_step:34.13282694108784 - perf/throughput:680.840178872666
(TaskRunner pid=2224264) Training Progress: 7%|▋ | 7/100 [05:13<1:07:20, 43.45s/it]
(TaskRunner pid=2224264) step:8 - global_seqlen/min:18447 - global_seqlen/max:32287 - global_seqlen/minmax_diff:13840 - global_seqlen/balanced_min:25765 - global_seqlen/balanced_max:25781 - global_seqlen/mean:25775.5 - actor/entropy:0.5252702832221985 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.31757450103759766 - training/rollout_probs_diff_mean:0.004371982999145985 - training/rollout_probs_diff_std:0.009270070120692253 - training/rollout_actor_probs_pearson_corr:0.9994732737541199 - rollout_corr/rollout_is_mean:1.0000965595245361 - rollout_corr/rollout_is_ratio_fraction_high:7.615218237333465e-06 - rollout_corr/rollout_is_ratio_fraction_low:7.615218237333465e-06 - rollout_corr/rollout_is_max:2.049854278564453 - rollout_corr/rollout_is_min:0.36485040187835693 - rollout_corr/rollout_is_std:0.03379738703370094 - rollout_corr/rollout_is_eff_sample_size:0.9988592777394362 - rollout_corr/rollout_is_seq_mean:1.0001250505447388 - rollout_corr/rollout_is_seq_std:0.0016558379866182804 - rollout_corr/rollout_is_seq_max:1.005513310432434 - rollout_corr/rollout_is_seq_min:0.9954277276992798 - rollout_corr/rollout_is_seq_max_deviation:0.005513310432434082 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7645341814495623) - rollout_corr/training_log_ppl:np.float64(0.5513433935120702) - rollout_corr/kl:np.float64(0.0004816958155657858) - rollout_corr/k3_kl:np.float64(0.0006227592039635965) - rollout_corr/rollout_ppl:np.float64(1.763689087703824) - rollout_corr/rollout_log_ppl:np.float64(0.5508616954903118) - rollout_corr/log_ppl_diff:np.float64(0.00048169802175834775) - rollout_corr/log_ppl_abs_diff:np.float64(0.001352956343907863) - rollout_corr/log_ppl_diff_max:np.float64(0.005591928958892822) - rollout_corr/log_ppl_diff_min:np.float64(-0.0054190754890441895) - rollout_corr/ppl_ratio:np.float64(1.0004832921549678) - rollout_corr/chi2_token:np.float64(0.0015316253993660212) - rollout_corr/chi2_seq:np.float64(0.7357629749458283) - actor/pg_loss:np.float64(-0.0002484882716089487) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003153111326810176) - actor/ppo_kl:np.float64(-1.135815940678242e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.33920033276081085) - perf/mfu/actor:np.float64(0.10468805371453549) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(99.58919906616211) - actor/lr:np.float64(7e-07) - training/global_step:8 - training/epoch:0 - critic/score/mean:0.48828125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.48828125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0034925295040011406 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0034925295040011406 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:512.953125 - response_length/max:1337.0 - response_length/min:140.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:512.953125 - response_length_non_aborted/max:1337.0 - response_length_non_aborted/min:140.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:292.53125 - prompt_length/max:720.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.801797568798065e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7887007519602776) - timing_s/agent_loop/generate_sequences/max:np.float64(11.159991431981325) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.707926993221918) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.159991431981325) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:202 - timing_s/agent_loop/slowest/response_length:1337 - timing_s/gen:17.992310052737594 - timing_s/reward:0.06701129302382469 - timing_s/old_log_prob:3.249571204185486 - timing_s/adv:0.13843823038041592 - timing_s/update_actor:12.995383325964212 - timing_s/step:34.52710779942572 - timing_s/stop_profile:0.00011713989078998566 - timing_per_token_ms/adv:0.0006713653972785005 - timing_per_token_ms/gen:0.13701536791204114 - timing_per_token_ms/update_actor:0.06302197496636444 - perf/total_num_tokens:206204 - perf/time_per_step:34.52710779942572 - perf/throughput:746.5293690318514
(TaskRunner pid=2224264) Training Progress: 8%|▊ | 8/100 [05:48<1:02:17, 40.63s/it]
(TaskRunner pid=2224264) step:9 - global_seqlen/min:21454 - global_seqlen/max:27246 - global_seqlen/minmax_diff:5792 - global_seqlen/balanced_min:24307 - global_seqlen/balanced_max:24315 - global_seqlen/mean:24311.875 - actor/entropy:0.5553610920906067 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.6999252438545227 - training/rollout_probs_diff_mean:0.004598262254148722 - training/rollout_probs_diff_std:0.009674612432718277 - training/rollout_actor_probs_pearson_corr:0.9994522333145142 - rollout_corr/rollout_is_mean:0.999959409236908 - rollout_corr/rollout_is_ratio_fraction_high:8.295244697364978e-06 - rollout_corr/rollout_is_ratio_fraction_low:3.3180978789459914e-05 - rollout_corr/rollout_is_max:2.1562492847442627 - rollout_corr/rollout_is_min:0.04011661186814308 - rollout_corr/rollout_is_std:0.034607741981744766 - rollout_corr/rollout_is_eff_sample_size:0.9988037370269406 - rollout_corr/rollout_is_seq_mean:0.9998873472213745 - rollout_corr/rollout_is_seq_std:0.001812683418393135 - rollout_corr/rollout_is_seq_max:1.006127119064331 - rollout_corr/rollout_is_seq_min:0.9938367009162903 - rollout_corr/rollout_is_seq_max_deviation:0.006163299083709717 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8204427626915276) - rollout_corr/training_log_ppl:np.float64(0.579765857430175) - rollout_corr/kl:np.float64(0.0006254475316893604) - rollout_corr/k3_kl:np.float64(0.0006774885101776817) - rollout_corr/rollout_ppl:np.float64(1.8192923632450402) - rollout_corr/rollout_log_ppl:np.float64(0.5791404079063796) - rollout_corr/log_ppl_diff:np.float64(0.0006254495237953961) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016019002650864422) - rollout_corr/log_ppl_diff_max:np.float64(0.007795929908752441) - rollout_corr/log_ppl_diff_min:np.float64(-0.004667341709136963) - rollout_corr/ppl_ratio:np.float64(1.0006277253851295) - rollout_corr/chi2_token:np.float64(0.001466952497139573) - rollout_corr/chi2_seq:np.float64(0.8160945861600339) - actor/pg_loss:np.float64(0.00012180034536868334) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00034742705497592397) - actor/ppo_kl:np.float64(-0.0001504568158097186) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.38103173673152924) - perf/mfu/actor:np.float64(0.09934898847820164) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(99.49935913085938) - actor/lr:np.float64(8e-07) - training/global_step:9 - training/epoch:0 - critic/score/mean:0.35546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.35546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.003955794498324394 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.003955794498324394 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:470.90234375 - response_length/max:1203.0 - response_length/min:145.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:470.90234375 - response_length_non_aborted/max:1203.0 - response_length_non_aborted/min:145.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:288.84375 - prompt_length/max:735.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010239146649837494 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8441594671458006) - timing_s/agent_loop/generate_sequences/max:np.float64(10.078126607462764) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.1269525975803845) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.078126607462764) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:250 - timing_s/agent_loop/slowest/response_length:1203 - timing_s/gen:16.780712580308318 - timing_s/reward:0.06167248822748661 - timing_s/old_log_prob:3.1073175128549337 - timing_s/adv:0.14903088845312595 - timing_s/update_actor:12.87536489032209 - timing_s/step:33.06730374135077 - timing_s/stop_profile:0.00011834129691123962 - timing_per_token_ms/adv:0.0007662453453977015 - timing_per_token_ms/gen:0.13920011099292678 - timing_per_token_ms/update_actor:0.06619895056593789 - perf/total_num_tokens:194495 - perf/time_per_step:33.06730374135077 - perf/throughput:735.2239901434093
(TaskRunner pid=2224264) Training Progress: 9%|▉ | 9/100 [06:21<58:03, 38.28s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 10}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) To determine the correct reactant for the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1", we need to analyze the structure of the product and identify what functional groups or atoms are present that must be introduced in the synthesis.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) The product has the following key features:
(TaskRunner pid=2224264) - A nitrogen-containing ring (indole-like structure)
(TaskRunner pid=2224264) - A chlorine atom attached to a benzene ring
(TaskRunner pid=2224264) - A trifluoromethyl group (C(F)(F)F) attached to a benzene ring
(TaskRunner pid=2224264) - A substituent group that includes a nitrogen atom bonded to a carbon chain
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) We are given four options (A, B, C, D) and need to identify which one is the only correct reactant that can be used in the synthesis of this product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Let’s analyze each option:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option A:**
(TaskRunner pid=2224264) O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) This option includes the same ring structure as the product, but it has an extra oxygen and a double bond. It seems to be a more complex derivative and may not be a direct reactant for the synthesis of the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option B:**
(TaskRunner pid=2224264) O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) This option has a chlorine atom attached to a carbon that is also bonded to a nitrogen. It includes a ring structure and a chlorine atom, but it does not match the exact structure of the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option C:**
(TaskRunner pid=2224264) O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) This option includes a trifluoromethyl group (C(F)(F)F), a chlorine atom, and a ring structure. However, it has an oxygen atom attached to a carbon that is also bonded to a ring. This structure is more complex and seems to be a derivative of the product, not a direct reactant.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option D:**
(TaskRunner pid=2224264) O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) This option contains a ring structure, a trifluoromethyl group (C(F)(F)F), and a carbon chain with a double bond. It matches the necessary components of the product, including the trifluoromethyl group and the ring structure. It is the only option that contains the trifluoromethyl group and a carbon chain that could be used to build the product's structure.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Thus, the only correct reactant that matches the necessary components of the product is **Option D**.
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) D
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 0.0
(TaskRunner pid=2224264) [acc] 0.0
(TaskRunner pid=2224264) [pred] D
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_10
(TaskRunner pid=2224264) step:10 - global_seqlen/min:20418 - global_seqlen/max:28068 - global_seqlen/minmax_diff:7650 - global_seqlen/balanced_min:23873 - global_seqlen/balanced_max:23970 - global_seqlen/mean:23892.875 - actor/entropy:0.5462746024131775 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2553977370262146 - training/rollout_probs_diff_mean:0.004709707107394934 - training/rollout_probs_diff_std:0.009677772410213947 - training/rollout_actor_probs_pearson_corr:0.9994706511497498 - rollout_corr/rollout_is_mean:0.9998794198036194 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.18444459582679e-06 - rollout_corr/rollout_is_max:1.622496247291565 - rollout_corr/rollout_is_min:0.4166650176048279 - rollout_corr/rollout_is_std:0.034952208399772644 - rollout_corr/rollout_is_eff_sample_size:0.9987795959879308 - rollout_corr/rollout_is_seq_mean:0.9998565912246704 - rollout_corr/rollout_is_seq_std:0.0018025203607976437 - rollout_corr/rollout_is_seq_max:1.0046859979629517 - rollout_corr/rollout_is_seq_min:0.9944931268692017 - rollout_corr/rollout_is_seq_max_deviation:0.00550687313079834 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7906524906866252) - rollout_corr/training_log_ppl:np.float64(0.5681954241008498) - rollout_corr/kl:np.float64(0.000816460652924178) - rollout_corr/k3_kl:np.float64(0.0006859729254529157) - rollout_corr/rollout_ppl:np.float64(1.789125895127654) - rollout_corr/rollout_log_ppl:np.float64(0.5673789661959745) - rollout_corr/log_ppl_diff:np.float64(0.0008164579048752785) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015062049496918917) - rollout_corr/log_ppl_diff_max:np.float64(0.007841110229492188) - rollout_corr/log_ppl_diff_min:np.float64(-0.004483282566070557) - rollout_corr/ppl_ratio:np.float64(1.0008185552433133) - rollout_corr/chi2_token:np.float64(0.0011271920520812273) - rollout_corr/chi2_seq:np.float64(0.35812679631635547) - actor/pg_loss:np.float64(-9.637698531150818e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003609476652854937) - actor/ppo_kl:np.float64(1.5247296355269668e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.38365066051483154) - perf/mfu/actor:np.float64(0.09690410487686144) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(99.56629180908203) - actor/lr:np.float64(9e-07) - val-aux/sciknoweval/reward/mean@16:np.float64(0.42410714285714285) - val-aux/sciknoweval/reward/std@16:np.float64(0.3064350931481341) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.5519095238095238) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.2843083193630651) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.2967190476190476) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.23547573506551264) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.42388571428571425) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.3061951415792409) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.6715952380952382) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.229658540263386) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.21003333333333332) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.1505707856063924) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.43492857142857144) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.25799645877611216) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.765652380952381) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.1562366777957225) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.1566142857142857) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.09130497085868691) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.44051904761904753) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.20808730912319104) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8246428571428572) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.0841254257162172) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.12242857142857143) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.05650288713800214) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.44238571428571427) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.1575830385066246) - val-aux/sciknoweval/score/mean@16:np.float64(0.42410714285714285) - val-aux/sciknoweval/score/std@16:np.float64(0.3064350931481341) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.5519095238095238) - val-aux/sciknoweval/score/best@2/std:np.float64(0.2843083193630651) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.2967190476190476) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.23547573506551264) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.42388571428571425) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.3061951415792409) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.6715952380952382) - val-aux/sciknoweval/score/best@4/std:np.float64(0.229658540263386) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.21003333333333332) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.1505707856063924) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.43492857142857144) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.25799645877611216) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.765652380952381) - val-aux/sciknoweval/score/best@8/std:np.float64(0.1562366777957225) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.1566142857142857) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.09130497085868691) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.44051904761904753) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.20808730912319104) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8246428571428572) - val-aux/sciknoweval/score/best@16/std:np.float64(0.0841254257162172) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.12242857142857143) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.05650288713800214) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.44238571428571427) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.1575830385066246) - val-core/sciknoweval/acc/mean@16:np.float64(0.42410714285714285) - val-aux/sciknoweval/acc/std@16:np.float64(0.3064350931481341) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.5519095238095238) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.2843083193630651) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.2967190476190476) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.23547573506551264) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.42388571428571425) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.3061951415792409) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.6715952380952382) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.229658540263386) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.21003333333333332) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.1505707856063924) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.43492857142857144) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.25799645877611216) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.765652380952381) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.1562366777957225) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.1566142857142857) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.09130497085868691) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.44051904761904753) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.20808730912319104) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8246428571428572) - val-core/sciknoweval/acc/best@16/std:np.float64(0.0841254257162172) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.12242857142857143) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.05650288713800214) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.44238571428571427) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.1575830385066246) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9955357142857143) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.017290104224140252) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.999647619047619) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.004935643403728794) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9915619047619048) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.02303424670029897) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9958761904761906) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.016639024114376826) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9841142857142856) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.02968846984091712) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9983333333333334) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.010724673876175908) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9717047619047618) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.03493004126518059) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9995761904761905) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.005142725136555151) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9547523809523809) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.03440111225850822) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999714285714284) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0007119567591960759) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:10 - training/epoch:0 - critic/score/mean:0.35546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.35546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.003161241766065359 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.003161241766065359 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:477.27734375 - response_length/max:1573.0 - response_length/min:165.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:477.27734375 - response_length_non_aborted/max:1573.0 - response_length_non_aborted/min:165.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:269.375 - prompt_length/max:701.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015650317072868347 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0671779178082943) - timing_s/agent_loop/generate_sequences/max:np.float64(11.94185172766447) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.342340253017028) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.94185172766447) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:1573 - timing_s/gen:18.750355437397957 - timing_s/reward:0.06060847267508507 - timing_s/old_log_prob:3.169859381392598 - timing_s/adv:0.15408215299248695 - timing_s/update_actor:12.863298328593373 - timing_s/step:35.09146223217249 - timing_s/testing:148.6573334839195 - timing_s/save_checkpoint:12.811893673613667 - timing_s/stop_profile:0.00013393908739089966 - timing_per_token_ms/adv:0.0008061093160224908 - timing_per_token_ms/gen:0.1534612461422453 - timing_per_token_ms/update_actor:0.0672967272073441 - perf/total_num_tokens:191143 - perf/time_per_step:35.09146223217249 - perf/throughput:680.8743061750952
(TaskRunner pid=2224264) Training Progress: 10%|█ | 10/100 [09:38<2:10:46, 87.18s/it]
(TaskRunner pid=2224264) step:11 - global_seqlen/min:20294 - global_seqlen/max:26725 - global_seqlen/minmax_diff:6431 - global_seqlen/balanced_min:24058 - global_seqlen/balanced_max:24076 - global_seqlen/mean:24070.25 - actor/entropy:0.5913231372833252 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23963481187820435 - training/rollout_probs_diff_mean:0.004752492066472769 - training/rollout_probs_diff_std:0.009357922710478306 - training/rollout_actor_probs_pearson_corr:0.9995139241218567 - rollout_corr/rollout_is_mean:1.0000724792480469 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.7242271496797912e-05 - rollout_corr/rollout_is_max:1.639595627784729 - rollout_corr/rollout_is_min:0.41323140263557434 - rollout_corr/rollout_is_std:0.03530850633978844 - rollout_corr/rollout_is_eff_sample_size:0.9987550993777402 - rollout_corr/rollout_is_seq_mean:1.0001287460327148 - rollout_corr/rollout_is_seq_std:0.001979757333174348 - rollout_corr/rollout_is_seq_max:1.0079063177108765 - rollout_corr/rollout_is_seq_min:0.9954867959022522 - rollout_corr/rollout_is_seq_max_deviation:0.007906317710876465 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8813741342164576) - rollout_corr/training_log_ppl:np.float64(0.6172284478088841) - rollout_corr/kl:np.float64(0.0006792962190935725) - rollout_corr/k3_kl:np.float64(0.0007297945928712579) - rollout_corr/rollout_ppl:np.float64(1.8801453905180097) - rollout_corr/rollout_log_ppl:np.float64(0.6165491492720321) - rollout_corr/log_ppl_diff:np.float64(0.0006792985368520021) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017337198369204998) - rollout_corr/log_ppl_diff_max:np.float64(0.00989621877670288) - rollout_corr/log_ppl_diff_min:np.float64(-0.006999552249908447) - rollout_corr/ppl_ratio:np.float64(1.000681943492964) - rollout_corr/chi2_token:np.float64(0.0015983630437403917) - rollout_corr/chi2_seq:np.float64(0.8742716582491994) - actor/pg_loss:np.float64(-0.00019931478891521692) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003874804629049322) - actor/ppo_kl:np.float64(0.000144808165869037) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4361106902360916) - perf/mfu/actor:np.float64(0.09751332153344734) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(105.7747859954834) - actor/lr:np.float64(1e-06) - training/global_step:11 - training/epoch:0 - critic/score/mean:0.5078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005713657476007938 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005713657476007938 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:453.1015625 - response_length/max:1181.0 - response_length/min:128.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:453.1015625 - response_length_non_aborted/max:1181.0 - response_length_non_aborted/min:128.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.09375 - prompt_length/max:620.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.448345422744751e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.323524136096239) - timing_s/agent_loop/generate_sequences/max:np.float64(8.916998555883765) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.45590021499811) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.916998555883765) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:319 - timing_s/agent_loop/slowest/response_length:1181 - timing_s/gen:15.152160251513124 - timing_s/reward:0.057105498388409615 - timing_s/old_log_prob:3.2711402103304863 - timing_s/adv:0.15576941706240177 - timing_s/update_actor:13.000678863376379 - timing_s/step:31.738592451438308 - timing_s/stop_profile:0.00011883489787578583 - timing_per_token_ms/adv:0.0008089312380552849 - timing_per_token_ms/gen:0.13062882779724058 - timing_per_token_ms/update_actor:0.0675142492463538 - perf/total_num_tokens:192562 - perf/time_per_step:31.738592451438308 - perf/throughput:758.3905945680714
(TaskRunner pid=2224264) Training Progress: 11%|█ | 11/100 [10:09<1:44:10, 70.23s/it]
(TaskRunner pid=2224264) step:12 - global_seqlen/min:22279 - global_seqlen/max:28767 - global_seqlen/minmax_diff:6488 - global_seqlen/balanced_min:25313 - global_seqlen/balanced_max:25329 - global_seqlen/mean:25325.625 - actor/entropy:0.5468142628669739 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.30783188343048096 - training/rollout_probs_diff_mean:0.004490185528993607 - training/rollout_probs_diff_std:0.009273474104702473 - training/rollout_actor_probs_pearson_corr:0.9995062351226807 - rollout_corr/rollout_is_mean:1.0000845193862915 - rollout_corr/rollout_is_ratio_fraction_high:7.435773568431614e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.075246810913086 - rollout_corr/rollout_is_min:0.5100579261779785 - rollout_corr/rollout_is_std:0.03405739739537239 - rollout_corr/rollout_is_eff_sample_size:0.9988416753032981 - rollout_corr/rollout_is_seq_mean:1.0000801086425781 - rollout_corr/rollout_is_seq_std:0.0016866717487573624 - rollout_corr/rollout_is_seq_max:1.0045456886291504 - rollout_corr/rollout_is_seq_min:0.9936270713806152 - rollout_corr/rollout_is_seq_max_deviation:0.006372928619384766 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7922574547119439) - rollout_corr/training_log_ppl:np.float64(0.5712134827044792) - rollout_corr/kl:np.float64(0.0005522373108419742) - rollout_corr/k3_kl:np.float64(0.0006720679998579726) - rollout_corr/rollout_ppl:np.float64(1.7912592180073261) - rollout_corr/rollout_log_ppl:np.float64(0.5706612397334538) - rollout_corr/log_ppl_diff:np.float64(0.0005522429710254073) - rollout_corr/log_ppl_abs_diff:np.float64(0.001352141029201448) - rollout_corr/log_ppl_diff_max:np.float64(0.007192671298980713) - rollout_corr/log_ppl_diff_min:np.float64(-0.003842592239379883) - rollout_corr/ppl_ratio:np.float64(1.0005537504330277) - rollout_corr/chi2_token:np.float64(0.0016048443503677845) - rollout_corr/chi2_seq:np.float64(0.7803806539159268) - actor/pg_loss:np.float64(-5.2671064622700214e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002636122353578685) - actor/ppo_kl:np.float64(2.2634695866941e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3025449775159359) - perf/mfu/actor:np.float64(0.09925995879832478) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.3304214477539) - actor/lr:np.float64(1e-06) - training/global_step:12 - training/epoch:0 - critic/score/mean:0.44140625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.44140625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.003494813572615385 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.003494813572615385 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:525.33203125 - response_length/max:1340.0 - response_length/min:219.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:525.33203125 - response_length_non_aborted/max:1340.0 - response_length_non_aborted/min:219.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.09375 - prompt_length/max:665.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001250188797712326 - timing_s/agent_loop/generate_sequences/min:np.float64(2.742388065904379) - timing_s/agent_loop/generate_sequences/max:np.float64(11.409071335569024) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.0465768424546695) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.409071335569024) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:264 - timing_s/agent_loop/slowest/response_length:1340 - timing_s/gen:18.710343282669783 - timing_s/reward:0.06669484823942184 - timing_s/old_log_prob:3.1704280357807875 - timing_s/adv:0.14182846806943417 - timing_s/update_actor:13.454730957746506 - timing_s/step:35.63072715699673 - timing_s/stop_profile:0.0001154579222202301 - timing_per_token_ms/adv:0.0007000245209616454 - timing_per_token_ms/gen:0.1391258748757838 - timing_per_token_ms/update_actor:0.06640868170946672 - perf/total_num_tokens:202605 - perf/time_per_step:35.63072715699673 - perf/throughput:710.7804701377491
(TaskRunner pid=2224264) Training Progress: 12%|█▏ | 12/100 [10:45<1:27:35, 59.72s/it]
(TaskRunner pid=2224264) step:13 - global_seqlen/min:22139 - global_seqlen/max:26070 - global_seqlen/minmax_diff:3931 - global_seqlen/balanced_min:23703 - global_seqlen/balanced_max:23955 - global_seqlen/mean:23768.875 - actor/entropy:0.6082448959350586 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8117851614952087 - training/rollout_probs_diff_mean:0.004782064817845821 - training/rollout_probs_diff_std:0.009643850848078728 - training/rollout_actor_probs_pearson_corr:0.9995208978652954 - rollout_corr/rollout_is_mean:0.9998765587806702 - rollout_corr/rollout_is_ratio_fraction_high:8.387642992602196e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.677528598520439e-05 - rollout_corr/rollout_is_max:2.2560038566589355 - rollout_corr/rollout_is_min:0.1834239512681961 - rollout_corr/rollout_is_std:0.03520030155777931 - rollout_corr/rollout_is_eff_sample_size:0.9987622341887205 - rollout_corr/rollout_is_seq_mean:0.9998448491096497 - rollout_corr/rollout_is_seq_std:0.0017469775630161166 - rollout_corr/rollout_is_seq_max:1.0048439502716064 - rollout_corr/rollout_is_seq_min:0.9949353933334351 - rollout_corr/rollout_is_seq_max_deviation:0.005064606666564941 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8870359733700752) - rollout_corr/training_log_ppl:np.float64(0.6238076707813889) - rollout_corr/kl:np.float64(0.0008689244228179405) - rollout_corr/k3_kl:np.float64(0.0006925447481762603) - rollout_corr/rollout_ppl:np.float64(1.8853362454101443) - rollout_corr/rollout_log_ppl:np.float64(0.6229387456551194) - rollout_corr/log_ppl_diff:np.float64(0.0008689251262694597) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016076937317848206) - rollout_corr/log_ppl_diff_max:np.float64(0.007483184337615967) - rollout_corr/log_ppl_diff_min:np.float64(-0.0051155686378479) - rollout_corr/ppl_ratio:np.float64(1.00087097636424) - rollout_corr/chi2_token:np.float64(0.001052779145538807) - rollout_corr/chi2_seq:np.float64(0.7514310674741864) - actor/pg_loss:np.float64(7.94560182839632e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002786977966025006) - actor/ppo_kl:np.float64(6.474458803040761e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.397162064909935) - perf/mfu/actor:np.float64(0.09482056304606831) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(102.62731170654297) - actor/lr:np.float64(1e-06) - training/global_step:13 - training/epoch:0 - critic/score/mean:0.37109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.37109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0025341168511658907 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0025341168511658907 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:465.71484375 - response_length/max:1517.0 - response_length/min:174.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:465.71484375 - response_length_non_aborted/max:1517.0 - response_length_non_aborted/min:174.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.0625 - prompt_length/max:612.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010670162737369537 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1883809324353933) - timing_s/agent_loop/generate_sequences/max:np.float64(11.769639683887362) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.302525144354149) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.769639683887362) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:232 - timing_s/agent_loop/slowest/response_length:1517 - timing_s/gen:18.91310457699001 - timing_s/reward:0.0668586827814579 - timing_s/old_log_prob:3.0675366707146168 - timing_s/adv:0.13432571291923523 - timing_s/update_actor:13.279257319867611 - timing_s/step:35.54763768427074 - timing_s/stop_profile:0.00011998414993286133 - timing_per_token_ms/adv:0.0007064160215788254 - timing_per_token_ms/gen:0.15863637533856728 - timing_per_token_ms/update_actor:0.06983532729182393 - perf/total_num_tokens:190151 - perf/time_per_step:35.54763768427074 - perf/throughput:668.6485107987174
(TaskRunner pid=2224264) Training Progress: 13%|█▎ | 13/100 [11:21<1:15:59, 52.41s/it]
(TaskRunner pid=2224264) step:14 - global_seqlen/min:21108 - global_seqlen/max:28388 - global_seqlen/minmax_diff:7280 - global_seqlen/balanced_min:24288 - global_seqlen/balanced_max:24367 - global_seqlen/mean:24304.0 - actor/entropy:0.5927061438560486 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2932337522506714 - training/rollout_probs_diff_mean:0.004692592658102512 - training/rollout_probs_diff_std:0.00925843883305788 - training/rollout_actor_probs_pearson_corr:0.9995315670967102 - rollout_corr/rollout_is_mean:0.9999940991401672 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.629692554473877 - rollout_corr/rollout_is_min:0.5129373669624329 - rollout_corr/rollout_is_std:0.03493344411253929 - rollout_corr/rollout_is_eff_sample_size:0.998781141930825 - rollout_corr/rollout_is_seq_mean:0.9999703168869019 - rollout_corr/rollout_is_seq_std:0.0018831780180335045 - rollout_corr/rollout_is_seq_max:1.007323145866394 - rollout_corr/rollout_is_seq_min:0.9952291250228882 - rollout_corr/rollout_is_seq_max_deviation:0.007323145866394043 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8519614487886429) - rollout_corr/training_log_ppl:np.float64(0.6055547567666508) - rollout_corr/kl:np.float64(0.0006922458704323731) - rollout_corr/k3_kl:np.float64(0.0006847217604217803) - rollout_corr/rollout_ppl:np.float64(1.850690824445337) - rollout_corr/rollout_log_ppl:np.float64(0.604862505744677) - rollout_corr/log_ppl_diff:np.float64(0.0006922510219737887) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016473830910399556) - rollout_corr/log_ppl_diff_max:np.float64(0.005706071853637695) - rollout_corr/log_ppl_diff_min:np.float64(-0.006285727024078369) - rollout_corr/ppl_ratio:np.float64(1.00069451588206) - rollout_corr/chi2_token:np.float64(0.0013829697854816914) - rollout_corr/chi2_seq:np.float64(1.670509943505749) - actor/pg_loss:np.float64(4.7211069613695145e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002791377244193427) - actor/ppo_kl:np.float64(2.9361409389139226e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.39511965960264206) - perf/mfu/actor:np.float64(0.09905766411058545) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(102.6309814453125) - actor/lr:np.float64(1e-06) - training/global_step:14 - training/epoch:0 - critic/score/mean:0.39453125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.39453125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.001839753007516265 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:-0.001839753007516265 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:492.0625 - response_length/max:1506.0 - response_length/min:184.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:492.0625 - response_length_non_aborted/max:1506.0 - response_length_non_aborted/min:184.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:267.4375 - prompt_length/max:666.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010376237332820892 - timing_s/agent_loop/generate_sequences/min:np.float64(2.323818039149046) - timing_s/agent_loop/generate_sequences/max:np.float64(11.8414571788162) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.66649279979174) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.8414571788162) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:274 - timing_s/agent_loop/slowest/response_length:1506 - timing_s/gen:18.678927469998598 - timing_s/reward:0.09569268859922886 - timing_s/old_log_prob:3.05848809145391 - timing_s/adv:0.16165275499224663 - timing_s/update_actor:12.951634736731648 - timing_s/step:35.03177675232291 - timing_s/stop_profile:0.00011714547872543335 - timing_per_token_ms/adv:0.0008314102359295107 - timing_per_token_ms/gen:0.1482831153149895 - timing_per_token_ms/update_actor:0.06661267042838447 - perf/total_num_tokens:194432 - perf/time_per_step:35.03177675232291 - perf/throughput:693.7701211055027
(TaskRunner pid=2224264) Training Progress: 14%|█▍ | 14/100 [11:56<1:07:37, 47.17s/it]
(TaskRunner pid=2224264) step:15 - global_seqlen/min:22401 - global_seqlen/max:28174 - global_seqlen/minmax_diff:5773 - global_seqlen/balanced_min:25422 - global_seqlen/balanced_max:25432 - global_seqlen/mean:25428.5 - actor/entropy:0.644464373588562 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.21453610062599182 - training/rollout_probs_diff_mean:0.004607322160154581 - training/rollout_probs_diff_std:0.008982905186712742 - training/rollout_actor_probs_pearson_corr:0.9995591640472412 - rollout_corr/rollout_is_mean:0.9999948143959045 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:7.693491170357447e-06 - rollout_corr/rollout_is_max:1.5525251626968384 - rollout_corr/rollout_is_min:0.4232032895088196 - rollout_corr/rollout_is_std:0.0347040556371212 - rollout_corr/rollout_is_eff_sample_size:0.9987969583907628 - rollout_corr/rollout_is_seq_mean:1.000001072883606 - rollout_corr/rollout_is_seq_std:0.001659586327150464 - rollout_corr/rollout_is_seq_max:1.0077872276306152 - rollout_corr/rollout_is_seq_min:0.994013249874115 - rollout_corr/rollout_is_seq_max_deviation:0.007787227630615234 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.001432800665498) - rollout_corr/training_log_ppl:np.float64(0.6742313308641315) - rollout_corr/kl:np.float64(0.0006003224210004987) - rollout_corr/k3_kl:np.float64(0.0006980515448731239) - rollout_corr/rollout_ppl:np.float64(2.0002612890675664) - rollout_corr/rollout_log_ppl:np.float64(0.673631009995006) - rollout_corr/log_ppl_diff:np.float64(0.0006003208691254258) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014080120017752051) - rollout_corr/log_ppl_diff_max:np.float64(0.00482177734375) - rollout_corr/log_ppl_diff_min:np.float64(-0.007164597511291504) - rollout_corr/ppl_ratio:np.float64(1.0006020197179168) - rollout_corr/chi2_token:np.float64(0.001624844502657652) - rollout_corr/chi2_seq:np.float64(1.5461109012831002) - actor/pg_loss:np.float64(0.00015092291869223118) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00017774101524992147) - actor/ppo_kl:np.float64(-3.1671785084341764e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3353484757244587) - perf/mfu/actor:np.float64(0.10308922077230062) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(102.59129333496094) - actor/lr:np.float64(1e-06) - training/global_step:15 - training/epoch:0 - critic/score/mean:0.453125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.453125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0006222110823728144 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0006222110823728144 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:507.734375 - response_length/max:1102.0 - response_length/min:186.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:507.734375 - response_length_non_aborted/max:1102.0 - response_length_non_aborted/min:186.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.90625 - prompt_length/max:494.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.000116739422082901 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3703849464654922) - timing_s/agent_loop/generate_sequences/max:np.float64(9.983016459271312) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.813961479689169) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.983016459271312) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:353 - timing_s/agent_loop/slowest/response_length:1093 - timing_s/gen:17.158777246251702 - timing_s/reward:0.06794460862874985 - timing_s/old_log_prob:3.08405601978302 - timing_s/adv:0.12691123597323895 - timing_s/update_actor:12.908528231084347 - timing_s/step:33.43038131669164 - timing_s/stop_profile:2.7742236852645874e-05 - timing_per_token_ms/adv:0.0006238631652144196 - timing_per_token_ms/gen:0.13201090357171644 - timing_per_token_ms/update_actor:0.06345502207702158 - perf/total_num_tokens:203428 - perf/time_per_step:33.43038131669164 - perf/throughput:760.6404413731191
(TaskRunner pid=2224264) Training Progress: 15%|█▌ | 15/100 [12:29<1:00:58, 43.04s/it]
(TaskRunner pid=2224264) step:16 - global_seqlen/min:21410 - global_seqlen/max:32573 - global_seqlen/minmax_diff:11163 - global_seqlen/balanced_min:26094 - global_seqlen/balanced_max:26108 - global_seqlen/mean:26103.625 - actor/entropy:0.6026089787483215 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1841939091682434 - training/rollout_probs_diff_mean:0.004535277374088764 - training/rollout_probs_diff_std:0.008988428860902786 - training/rollout_actor_probs_pearson_corr:0.9995559453964233 - rollout_corr/rollout_is_mean:1.000097632408142 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6370495557785034 - rollout_corr/rollout_is_min:0.5366882681846619 - rollout_corr/rollout_is_std:0.034516338258981705 - rollout_corr/rollout_is_eff_sample_size:0.9988102779034601 - rollout_corr/rollout_is_seq_mean:1.0000861883163452 - rollout_corr/rollout_is_seq_std:0.00168060592841357 - rollout_corr/rollout_is_seq_max:1.0056310892105103 - rollout_corr/rollout_is_seq_min:0.9956023097038269 - rollout_corr/rollout_is_seq_max_deviation:0.005631089210510254 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8813290460966527) - rollout_corr/training_log_ppl:np.float64(0.6181044758995995) - rollout_corr/kl:np.float64(0.0006577574873949388) - rollout_corr/k3_kl:np.float64(0.0006780388332572329) - rollout_corr/rollout_ppl:np.float64(1.880119965877384) - rollout_corr/rollout_log_ppl:np.float64(0.617446719086729) - rollout_corr/log_ppl_diff:np.float64(0.0006577568128705025) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014887393917888403) - rollout_corr/log_ppl_diff_max:np.float64(0.009718984365463257) - rollout_corr/log_ppl_diff_min:np.float64(-0.005425512790679932) - rollout_corr/ppl_ratio:np.float64(1.0006597633473575) - rollout_corr/chi2_token:np.float64(0.0014098691754043102) - rollout_corr/chi2_seq:np.float64(2.303532388759777) - actor/pg_loss:np.float64(-0.00024759862571954727) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005523854347302404) - actor/ppo_kl:np.float64(0.00011259556639942048) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.4002230614423752) - perf/mfu/actor:np.float64(0.10440002990848575) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(102.71824264526367) - actor/lr:np.float64(1e-06) - training/global_step:16 - training/epoch:0 - critic/score/mean:0.48828125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.48828125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004673639312386513 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004673639312386513 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:535.01953125 - response_length/max:1208.0 - response_length/min:161.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:535.01953125 - response_length_non_aborted/max:1208.0 - response_length_non_aborted/min:161.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.71875 - prompt_length/max:859.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:3.0385330319404602e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0918461363762617) - timing_s/agent_loop/generate_sequences/max:np.float64(10.83283576183021) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.100973397937196) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.83283576183021) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:203 - timing_s/agent_loop/slowest/response_length:1163 - timing_s/gen:17.82087113149464 - timing_s/reward:0.06884737312793732 - timing_s/old_log_prob:3.0925582088530064 - timing_s/adv:0.11866064369678497 - timing_s/update_actor:13.111870618537068 - timing_s/step:34.29907225072384 - timing_s/stop_profile:0.00011087954044342041 - timing_per_token_ms/adv:0.0005682191826651709 - timing_per_token_ms/gen:0.1301125917679308 - timing_per_token_ms/update_actor:0.0627875947236115 - perf/total_num_tokens:208829 - perf/time_per_step:34.29907225072384 - perf/throughput:761.059214931072
(TaskRunner pid=2224264) Training Progress: 16%|█▌ | 16/100 [13:04<56:35, 40.42s/it]
(TaskRunner pid=2224264) step:17 - global_seqlen/min:25658 - global_seqlen/max:31580 - global_seqlen/minmax_diff:5922 - global_seqlen/balanced_min:28427 - global_seqlen/balanced_max:28619 - global_seqlen/mean:28480.125 - actor/entropy:0.6307497024536133 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24447917938232422 - training/rollout_probs_diff_mean:0.004340663086622953 - training/rollout_probs_diff_std:0.008799507282674313 - training/rollout_actor_probs_pearson_corr:0.9995622634887695 - rollout_corr/rollout_is_mean:1.0000449419021606 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:7.015722076175734e-06 - rollout_corr/rollout_is_max:1.5821566581726074 - rollout_corr/rollout_is_min:0.4949151575565338 - rollout_corr/rollout_is_std:0.03397679701447487 - rollout_corr/rollout_is_eff_sample_size:0.998847027329727 - rollout_corr/rollout_is_seq_mean:1.0000169277191162 - rollout_corr/rollout_is_seq_std:0.0016468465328216553 - rollout_corr/rollout_is_seq_max:1.006962537765503 - rollout_corr/rollout_is_seq_min:0.9948194622993469 - rollout_corr/rollout_is_seq_max_deviation:0.00696253776550293 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0382777974009514) - rollout_corr/training_log_ppl:np.float64(0.6885525062098168) - rollout_corr/kl:np.float64(0.0007926652352825414) - rollout_corr/k3_kl:np.float64(0.000665645356832556) - rollout_corr/rollout_ppl:np.float64(2.0366465840488672) - rollout_corr/rollout_log_ppl:np.float64(0.687759839813225) - rollout_corr/log_ppl_diff:np.float64(0.0007926663965918124) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014466078137047589) - rollout_corr/log_ppl_diff_max:np.float64(0.0072629451751708984) - rollout_corr/log_ppl_diff_min:np.float64(-0.004837632179260254) - rollout_corr/ppl_ratio:np.float64(1.0007944458629936) - rollout_corr/chi2_token:np.float64(0.0010911454446613789) - rollout_corr/chi2_seq:np.float64(0.8875894017983228) - actor/pg_loss:np.float64(4.744785837829113e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019340229709996493) - actor/ppo_kl:np.float64(0.00017584012368843105) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3084394074976444) - perf/mfu/actor:np.float64(0.1127631964826109) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(102.56375503540039) - actor/lr:np.float64(1e-06) - training/global_step:17 - training/epoch:0 - critic/score/mean:0.3984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.3984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005952840205281973 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005952840205281973 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:556.78515625 - response_length/max:1943.0 - response_length/min:141.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:556.78515625 - response_length_non_aborted/max:1943.0 - response_length_non_aborted/min:141.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:333.21875 - prompt_length/max:617.0 - prompt_length/min:189.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012286566197872162 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8024302516132593) - timing_s/agent_loop/generate_sequences/max:np.float64(15.304179856553674) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.37867253353761) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.304179856553674) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:297 - timing_s/agent_loop/slowest/response_length:1943 - timing_s/gen:21.94722494482994 - timing_s/reward:0.07538571581244469 - timing_s/old_log_prob:3.1670491863042116 - timing_s/adv:0.15483485721051693 - timing_s/update_actor:13.154093807563186 - timing_s/step:38.58247912488878 - timing_s/stop_profile:0.00011283159255981445 - timing_per_token_ms/adv:0.0006795741644853952 - timing_per_token_ms/gen:0.15397563400962516 - timing_per_token_ms/update_actor:0.057733655521013276 - perf/total_num_tokens:227841 - perf/time_per_step:38.58247912488878 - perf/throughput:738.1621307384586
(TaskRunner pid=2224264) Training Progress: 17%|█▋ | 17/100 [13:42<55:10, 39.88s/it]
(TaskRunner pid=2224264) step:18 - global_seqlen/min:21346 - global_seqlen/max:28688 - global_seqlen/minmax_diff:7342 - global_seqlen/balanced_min:24408 - global_seqlen/balanced_max:24420 - global_seqlen/mean:24413.25 - actor/entropy:0.6875190734863281 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4119926691055298 - training/rollout_probs_diff_mean:0.0047507998533546925 - training/rollout_probs_diff_std:0.009137631393969059 - training/rollout_actor_probs_pearson_corr:0.9995434284210205 - rollout_corr/rollout_is_mean:0.9999079704284668 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.3933751435833983e-05 - rollout_corr/rollout_is_max:1.649957537651062 - rollout_corr/rollout_is_min:0.2244005799293518 - rollout_corr/rollout_is_std:0.03564453125 - rollout_corr/rollout_is_eff_sample_size:0.9987308417828792 - rollout_corr/rollout_is_seq_mean:0.9998347759246826 - rollout_corr/rollout_is_seq_std:0.0020229120273143053 - rollout_corr/rollout_is_seq_max:1.005630373954773 - rollout_corr/rollout_is_seq_min:0.9910240769386292 - rollout_corr/rollout_is_seq_max_deviation:0.00897592306137085 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0345947919413447) - rollout_corr/training_log_ppl:np.float64(0.6928353062248789) - rollout_corr/kl:np.float64(0.0008856774063996653) - rollout_corr/k3_kl:np.float64(0.0006838213192850162) - rollout_corr/rollout_ppl:np.float64(2.0328480429016054) - rollout_corr/rollout_log_ppl:np.float64(0.6919496294576675) - rollout_corr/log_ppl_diff:np.float64(0.0008856767672114074) - rollout_corr/log_ppl_abs_diff:np.float64(0.001704056456219405) - rollout_corr/log_ppl_diff_max:np.float64(0.00990024209022522) - rollout_corr/log_ppl_diff_min:np.float64(-0.004864394664764404) - rollout_corr/ppl_ratio:np.float64(1.0008882319089025) - rollout_corr/chi2_token:np.float64(0.0009777550585567951) - rollout_corr/chi2_seq:np.float64(0.8223780991975218) - actor/pg_loss:np.float64(0.00021487544290721416) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000207112968837464) - actor/ppo_kl:np.float64(5.649118667616637e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.38337405771017075) - perf/mfu/actor:np.float64(0.09893896312961671) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(102.62813949584961) - actor/lr:np.float64(1e-06) - training/global_step:18 - training/epoch:0 - critic/score/mean:0.4609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0014918704982846975 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0014918704982846975 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:489.6328125 - response_length/max:1187.0 - response_length/min:151.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:489.6328125 - response_length_non_aborted/max:1187.0 - response_length_non_aborted/min:151.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.28125 - prompt_length/max:540.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001486111432313919 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7897587660700083) - timing_s/agent_loop/generate_sequences/max:np.float64(10.173580570146441) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.444523438432952) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.173580570146441) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:330 - timing_s/agent_loop/slowest/response_length:1187 - timing_s/gen:17.149094123393297 - timing_s/reward:0.10750813968479633 - timing_s/old_log_prob:3.131292313337326 - timing_s/adv:0.1582884006202221 - timing_s/update_actor:12.998078908771276 - timing_s/step:33.6287423055619 - timing_s/stop_profile:0.00010932423174381256 - timing_per_token_ms/adv:0.0008104635834035928 - timing_per_token_ms/gen:0.13681405169206276 - timing_per_token_ms/update_actor:0.06655237887607793 - perf/total_num_tokens:195306 - perf/time_per_step:33.6287423055619 - perf/throughput:725.9638132813031
(TaskRunner pid=2224264) Training Progress: 18%|█▊ | 18/100 [14:16<51:58, 38.03s/it]
(TaskRunner pid=2224264) step:19 - global_seqlen/min:21987 - global_seqlen/max:30303 - global_seqlen/minmax_diff:8316 - global_seqlen/balanced_min:26121 - global_seqlen/balanced_max:26132 - global_seqlen/mean:26128.5 - actor/entropy:0.6984733939170837 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2413516640663147 - training/rollout_probs_diff_mean:0.004623448941856623 - training/rollout_probs_diff_std:0.008824565447866917 - training/rollout_actor_probs_pearson_corr:0.9995649456977844 - rollout_corr/rollout_is_mean:0.9998148083686829 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:7.593936970806681e-06 - rollout_corr/rollout_is_max:1.9665265083312988 - rollout_corr/rollout_is_min:0.4158303737640381 - rollout_corr/rollout_is_std:0.035356584936380386 - rollout_corr/rollout_is_eff_sample_size:0.9987510563634333 - rollout_corr/rollout_is_seq_mean:0.9998022317886353 - rollout_corr/rollout_is_seq_std:0.001719683175906539 - rollout_corr/rollout_is_seq_max:1.005003571510315 - rollout_corr/rollout_is_seq_min:0.9942381978034973 - rollout_corr/rollout_is_seq_max_deviation:0.0057618021965026855 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1011477317661047) - rollout_corr/training_log_ppl:np.float64(0.7186182939331047) - rollout_corr/kl:np.float64(0.000854842338526618) - rollout_corr/k3_kl:np.float64(0.0006763917490957283) - rollout_corr/rollout_ppl:np.float64(2.0992823550477624) - rollout_corr/rollout_log_ppl:np.float64(0.7177634529653005) - rollout_corr/log_ppl_diff:np.float64(0.0008548409678041935) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015009507769718766) - rollout_corr/log_ppl_diff_max:np.float64(0.0058748722076416016) - rollout_corr/log_ppl_diff_min:np.float64(-0.0035433173179626465) - rollout_corr/ppl_ratio:np.float64(1.000856626080349) - rollout_corr/chi2_token:np.float64(0.0010158277582377195) - rollout_corr/chi2_seq:np.float64(0.722900487948209) - actor/pg_loss:np.float64(0.00023403007071465254) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00021148656901459617) - actor/ppo_kl:np.float64(1.1429972420096135e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.30054471269249916) - perf/mfu/actor:np.float64(0.10550384895416316) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(102.6054801940918) - actor/lr:np.float64(1e-06) - training/global_step:19 - training/epoch:0 - critic/score/mean:0.4921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0012871723156422377 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0012871723156422377 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:514.390625 - response_length/max:1276.0 - response_length/min:169.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:514.390625 - response_length_non_aborted/max:1276.0 - response_length_non_aborted/min:169.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.125 - prompt_length/max:836.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001385621726512909 - timing_s/agent_loop/generate_sequences/min:np.float64(1.989379409700632) - timing_s/agent_loop/generate_sequences/max:np.float64(10.74332383275032) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.9351765791507205) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.74332383275032) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:213 - timing_s/agent_loop/slowest/response_length:1276 - timing_s/gen:17.415069840848446 - timing_s/reward:0.06816395185887814 - timing_s/old_log_prob:3.1230949088931084 - timing_s/adv:0.12103297747671604 - timing_s/update_actor:13.004424184560776 - timing_s/step:33.81597386114299 - timing_s/stop_profile:3.315135836601257e-05 - timing_per_token_ms/adv:0.0005790275823177567 - timing_per_token_ms/gen:0.13224894323417002 - timing_per_token_ms/update_actor:0.062213790423104924 - perf/total_num_tokens:209028 - perf/time_per_step:33.81597386114299 - perf/throughput:772.667382204939
(TaskRunner pid=2224264) Training Progress: 19%|█▉ | 19/100 [14:50<49:38, 36.77s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 20}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) To determine which molecule is the only correct reactant used in the synthesis of the product **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**, we need to analyze the structure and functional groups of the product and match it with the structure of the reactant.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Step 1: Analyze the product
(TaskRunner pid=2224264) The product is:
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) This can be interpreted as:
(TaskRunner pid=2224264) - A nitrogen atom (N) is attached to a ring.
(TaskRunner pid=2224264) - The ring is a pyrrole-like structure (N in the ring).
(TaskRunner pid=2224264) - The ring is substituted with a group: `Cc2ccc(Cl)cc2C(F)(F)F`, which is a benzene ring with a chlorine atom and a trifluoromethyl group.
(TaskRunner pid=2224264) - The structure is a pyrrole derivative with a substituent on the ring.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Step 2: Analyze the options
(TaskRunner pid=2224264) Let’s examine the options one by one to see which one matches the structure of the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) #### Option A:
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) This structure contains:
(TaskRunner pid=2224264) - A carbonyl group (O=C), which is not in the product.
(TaskRunner pid=2224264) - Two carbonyl groups (C=O), which are not in the product.
(TaskRunner pid=2224264) - The part `c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1` matches the pyrrole-like structure of the product, but the presence of two carbonyl groups makes this a different compound.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) #### Option B:
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) This contains:
(TaskRunner pid=2224264) - A carbonyl group (O=C), which is not in the product.
(TaskRunner pid=2224264) - A chlorine atom (Cl) attached to the ring.
(TaskRunner pid=2224264) - The rest of the structure does not match the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) #### Option C:
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) This contains:
(TaskRunner pid=2224264) - Two carbonyl groups (C=O), which are not in the product.
(TaskRunner pid=2224264) - A hydroxyl group (O), which is not in the product.
(TaskRunner pid=2224264) - The part `c2ccc(Cl)c(C(F)(F)F)c2` is similar to the product, but the overall structure is not matching the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) #### Option D:
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) ```
(TaskRunner pid=2224264) This contains:
(TaskRunner pid=2224264) - Two carbonyl groups (C=O), which are not in the product.
(TaskRunner pid=2224264) - A fluorinated group (C(F)(F)F), which is present in the product.
(TaskRunner pid=2224264) - The structure is not matching the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Step 3: Conclusion
(TaskRunner pid=2224264) None of the options exactly match the structure of the product. However, **Option A** is the only one that contains the core structure of the product: `c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1`. The other groups in Option A (carbonyl groups) are not part of the product, but the rest of the structure is the closest match.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Thus, **Option A** is the best candidate as the correct reactant.
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_20
(TaskRunner pid=2224264) step:20 - global_seqlen/min:22848 - global_seqlen/max:30898 - global_seqlen/minmax_diff:8050 - global_seqlen/balanced_min:26847 - global_seqlen/balanced_max:26876 - global_seqlen/mean:26867.0 - actor/entropy:0.6798826456069946 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4967540204524994 - training/rollout_probs_diff_mean:0.004492460284382105 - training/rollout_probs_diff_std:0.008851698599755764 - training/rollout_actor_probs_pearson_corr:0.9995761513710022 - rollout_corr/rollout_is_mean:0.9998908638954163 - rollout_corr/rollout_is_ratio_fraction_high:1.4051259313418996e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.54838228225708 - rollout_corr/rollout_is_min:0.5467735528945923 - rollout_corr/rollout_is_std:0.034748680889606476 - rollout_corr/rollout_is_eff_sample_size:0.9987937474899505 - rollout_corr/rollout_is_seq_mean:0.9998890161514282 - rollout_corr/rollout_is_seq_std:0.0017418391071259975 - rollout_corr/rollout_is_seq_max:1.0073792934417725 - rollout_corr/rollout_is_seq_min:0.9951199293136597 - rollout_corr/rollout_is_seq_max_deviation:0.007379293441772461 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.033839966636151) - rollout_corr/training_log_ppl:np.float64(0.6936833750805818) - rollout_corr/kl:np.float64(0.0007494871477833343) - rollout_corr/k3_kl:np.float64(0.0006487431917321373) - rollout_corr/rollout_ppl:np.float64(2.0322980885393918) - rollout_corr/rollout_log_ppl:np.float64(0.6929338824702427) - rollout_corr/log_ppl_diff:np.float64(0.0007494926103390753) - rollout_corr/log_ppl_abs_diff:np.float64(0.001494129013735801) - rollout_corr/log_ppl_diff_max:np.float64(0.005956470966339111) - rollout_corr/log_ppl_diff_min:np.float64(-0.006281614303588867) - rollout_corr/ppl_ratio:np.float64(1.000751372659579) - rollout_corr/chi2_token:np.float64(0.0011060182005167007) - rollout_corr/chi2_seq:np.float64(0.7747026174329221) - actor/pg_loss:np.float64(-0.00010517542250454426) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014331985744320264) - actor/ppo_kl:np.float64(1.3654966693366077e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.31467055529356003) - perf/mfu/actor:np.float64(0.10809592356334449) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(102.8171272277832) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.48392857142857143) - val-aux/sciknoweval/reward/std@16:np.float64(0.3171121217029239) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.6190095238095239) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.2728610239160176) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.3498904761904762) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.2656159472576298) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.48431904761904765) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.3171144172517289) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.729252380952381) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.20041976709665926) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.24425714285714284) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.1861647179153658) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.5091) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.2646122540406198) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8068619047619048) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.12410804929329204) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.17493809523809525) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.11508084398834663) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.5237238095238095) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.20562351058186512) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8522714285714285) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.06491422684529762) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.13282380952380954) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.06387633477899576) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.5326761904761904) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.15389547364520514) - val-aux/sciknoweval/score/mean@16:np.float64(0.48392857142857143) - val-aux/sciknoweval/score/std@16:np.float64(0.3171121217029239) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.6190095238095239) - val-aux/sciknoweval/score/best@2/std:np.float64(0.2728610239160176) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.3498904761904762) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.2656159472576298) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.48431904761904765) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.3171144172517289) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.729252380952381) - val-aux/sciknoweval/score/best@4/std:np.float64(0.20041976709665926) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.24425714285714284) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.1861647179153658) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.5091) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.2646122540406198) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8068619047619048) - val-aux/sciknoweval/score/best@8/std:np.float64(0.12410804929329204) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.17493809523809525) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.11508084398834663) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.5237238095238095) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.20562351058186512) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8522714285714285) - val-aux/sciknoweval/score/best@16/std:np.float64(0.06491422684529762) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.13282380952380954) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.06387633477899576) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.5326761904761904) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.15389547364520514) - val-core/sciknoweval/acc/mean@16:np.float64(0.48392857142857143) - val-aux/sciknoweval/acc/std@16:np.float64(0.3171121217029239) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.6190095238095239) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.2728610239160176) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.3498904761904762) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.2656159472576298) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.48431904761904765) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.3171144172517289) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.729252380952381) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.20041976709665926) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.24425714285714284) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.1861647179153658) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.5091) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.2646122540406198) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8068619047619048) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.12410804929329204) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.17493809523809525) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.11508084398834663) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.5237238095238095) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.20562351058186512) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8522714285714285) - val-core/sciknoweval/acc/best@16/std:np.float64(0.06491422684529762) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.13282380952380954) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.06387633477899576) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.5326761904761904) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.15389547364520514) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9931547619047619) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.023312712167358005) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9993809523809524) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.006485809156183406) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9871333333333333) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.030787750802297818) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9933857142857144) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.02299957497117341) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9999904761904761) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0002127457895589398) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9763380952380952) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.03858134164350246) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9975285714285714) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.014190325800172833) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9601904761904761) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.043352249329006456) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9995333333333334) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.005554643730532354) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9392047619047619) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.04061806930568887) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999666666666667) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0009652924861912784) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:20 - training/epoch:0 - critic/score/mean:0.56640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.56640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0017669458175078034 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0017669458175078034 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:556.0 - response_length/max:1546.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:556.0 - response_length_non_aborted/max:1546.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:283.59375 - prompt_length/max:392.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.37429004907608e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.663187799975276) - timing_s/agent_loop/generate_sequences/max:np.float64(12.704742550849915) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.302203726881999) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.704742550849915) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:336 - timing_s/agent_loop/slowest/response_length:1546 - timing_s/gen:19.35746957361698 - timing_s/reward:0.06838529743254185 - timing_s/old_log_prob:3.1675357930362225 - timing_s/adv:0.14193121530115604 - timing_s/update_actor:13.063340313732624 - timing_s/step:35.8843690212816 - timing_s/testing:160.607242802158 - timing_s/save_checkpoint:12.75176297314465 - timing_s/stop_profile:0.0001437067985534668 - timing_per_token_ms/adv:0.0006603417542950275 - timing_per_token_ms/gen:0.1359984092121247 - timing_per_token_ms/update_actor:0.06077781439001668 - perf/total_num_tokens:214936 - perf/time_per_step:35.8843690212816 - perf/throughput:748.710392094849
(TaskRunner pid=2224264) Training Progress: 20%|██ | 20/100 [18:19<1:58:06, 88.58s/it]
(TaskRunner pid=2224264) step:21 - global_seqlen/min:21817 - global_seqlen/max:33018 - global_seqlen/minmax_diff:11201 - global_seqlen/balanced_min:27703 - global_seqlen/balanced_max:27729 - global_seqlen/mean:27721.25 - actor/entropy:0.6702685952186584 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.30163779854774475 - training/rollout_probs_diff_mean:0.004457509610801935 - training/rollout_probs_diff_std:0.00877729244530201 - training/rollout_actor_probs_pearson_corr:0.9996225833892822 - rollout_corr/rollout_is_mean:1.0001065731048584 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.7989544868469238 - rollout_corr/rollout_is_min:0.6091647744178772 - rollout_corr/rollout_is_std:0.03465593233704567 - rollout_corr/rollout_is_eff_sample_size:0.9988006450060433 - rollout_corr/rollout_is_seq_mean:1.0000840425491333 - rollout_corr/rollout_is_seq_std:0.001599725685082376 - rollout_corr/rollout_is_seq_max:1.0058624744415283 - rollout_corr/rollout_is_seq_min:0.9920676350593567 - rollout_corr/rollout_is_seq_max_deviation:0.00793236494064331 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.009537471458316) - rollout_corr/training_log_ppl:np.float64(0.6828376608900726) - rollout_corr/kl:np.float64(0.000724996415337742) - rollout_corr/k3_kl:np.float64(0.0006654245569848172) - rollout_corr/rollout_ppl:np.float64(2.008096869569272) - rollout_corr/rollout_log_ppl:np.float64(0.6821126618888229) - rollout_corr/log_ppl_diff:np.float64(0.000724999001249671) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013606392312794924) - rollout_corr/log_ppl_diff_max:np.float64(0.009048700332641602) - rollout_corr/log_ppl_diff_min:np.float64(-0.004755973815917969) - rollout_corr/ppl_ratio:np.float64(1.0007265841122717) - rollout_corr/chi2_token:np.float64(0.001218994613736868) - rollout_corr/chi2_seq:np.float64(0.7947670572903007) - actor/pg_loss:np.float64(-0.00015212141443043947) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00016183609022846213) - actor/ppo_kl:np.float64(0.00017459750991832834) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3394359387457371) - perf/mfu/actor:np.float64(0.10795117855645653) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(104.55794906616211) - actor/lr:np.float64(1e-06) - training/global_step:21 - training/epoch:0 - critic/score/mean:0.35546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.35546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.003770750481635332 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.003770750481635332 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:583.1015625 - response_length/max:1506.0 - response_length/min:126.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:583.1015625 - response_length_non_aborted/max:1506.0 - response_length_non_aborted/min:126.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:283.1875 - prompt_length/max:669.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011318549513816833 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5702421497553587) - timing_s/agent_loop/generate_sequences/max:np.float64(12.702512258663774) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.579333163579577) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.702512258663774) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:321 - timing_s/agent_loop/slowest/response_length:1501 - timing_s/gen:18.926135268062353 - timing_s/reward:0.06378746405243874 - timing_s/old_log_prob:3.246299659833312 - timing_s/adv:0.12370392866432667 - timing_s/update_actor:13.3603654820472 - timing_s/step:35.82531424425542 - timing_s/stop_profile:0.0001258011907339096 - timing_per_token_ms/adv:0.0005578028077031459 - timing_per_token_ms/gen:0.12678788850075937 - timing_per_token_ms/update_actor:0.060244241701074085 - perf/total_num_tokens:221770 - perf/time_per_step:35.82531424425542 - perf/throughput:773.7894442738933
(TaskRunner pid=2224264) Training Progress: 21%|██ | 21/100 [18:55<1:35:47, 72.76s/it]
(TaskRunner pid=2224264) step:22 - global_seqlen/min:20941 - global_seqlen/max:26032 - global_seqlen/minmax_diff:5091 - global_seqlen/balanced_min:23791 - global_seqlen/balanced_max:27438 - global_seqlen/mean:24253.875 - actor/entropy:0.6328612565994263 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18033698201179504 - training/rollout_probs_diff_mean:0.004449577536433935 - training/rollout_probs_diff_std:0.00884386245161295 - training/rollout_actor_probs_pearson_corr:0.9996188879013062 - rollout_corr/rollout_is_mean:0.9999265074729919 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.831578016281128 - rollout_corr/rollout_is_min:0.5145671367645264 - rollout_corr/rollout_is_std:0.0340801440179348 - rollout_corr/rollout_is_eff_sample_size:0.9988397723743896 - rollout_corr/rollout_is_seq_mean:0.9999299049377441 - rollout_corr/rollout_is_seq_std:0.0016637897351756692 - rollout_corr/rollout_is_seq_max:1.00577974319458 - rollout_corr/rollout_is_seq_min:0.9933801293373108 - rollout_corr/rollout_is_seq_max_deviation:0.006619870662689209 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.9823644189164042) - rollout_corr/training_log_ppl:np.float64(0.6632540230639279) - rollout_corr/kl:np.float64(0.0008493472823345449) - rollout_corr/k3_kl:np.float64(0.000727977444228145) - rollout_corr/rollout_ppl:np.float64(1.9806274939328432) - rollout_corr/rollout_log_ppl:np.float64(0.6624046702054329) - rollout_corr/log_ppl_diff:np.float64(0.0008493528584949672) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016184973646886647) - rollout_corr/log_ppl_diff_max:np.float64(0.007628798484802246) - rollout_corr/log_ppl_diff_min:np.float64(-0.00557255744934082) - rollout_corr/ppl_ratio:np.float64(1.000851531745866) - rollout_corr/chi2_token:np.float64(0.0012481804005801678) - rollout_corr/chi2_seq:np.float64(0.8115007842425257) - actor/pg_loss:np.float64(0.0003393040969967842) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005306760692747048) - actor/ppo_kl:np.float64(0.00014998488964579337) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.35529135540127754) - perf/mfu/actor:np.float64(0.09394398118409722) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.99365997314453) - actor/lr:np.float64(1e-06) - training/global_step:22 - training/epoch:0 - critic/score/mean:0.44921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.44921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.01673314906656742 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.01673314906656742 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:502.37109375 - response_length/max:4901.0 - response_length/min:131.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:502.37109375 - response_length_non_aborted/max:4901.0 - response_length_non_aborted/min:131.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:255.5625 - prompt_length/max:592.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013022124767303467 - timing_s/agent_loop/generate_sequences/min:np.float64(1.669778237119317) - timing_s/agent_loop/generate_sequences/max:np.float64(30.597046691924334) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.5186303148948355) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(30.597046691924334) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:447 - timing_s/agent_loop/slowest/response_length:4901 - timing_s/gen:37.434364441782236 - timing_s/reward:0.06758200004696846 - timing_s/old_log_prob:3.2071466892957687 - timing_s/adv:0.11781474389135838 - timing_s/update_actor:13.525701014325023 - timing_s/step:54.440226985141635 - timing_s/stop_profile:0.00011406838893890381 - timing_per_token_ms/adv:0.0006071954682053815 - timing_per_token_ms/gen:0.2910756369543045 - timing_per_token_ms/update_actor:0.0697089692591649 - perf/total_num_tokens:194031 - perf/time_per_step:54.440226985141635 - perf/throughput:445.51384781366926
(TaskRunner pid=2224264) Training Progress: 22%|██▏ | 22/100 [19:49<1:27:27, 67.28s/it]
(TaskRunner pid=2224264) step:23 - global_seqlen/min:20573 - global_seqlen/max:31570 - global_seqlen/minmax_diff:10997 - global_seqlen/balanced_min:28209 - global_seqlen/balanced_max:28226 - global_seqlen/mean:28218.75 - actor/entropy:0.6778538823127747 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7014397382736206 - training/rollout_probs_diff_mean:0.0045140404254198074 - training/rollout_probs_diff_std:0.008873715065419674 - training/rollout_actor_probs_pearson_corr:0.9996016025543213 - rollout_corr/rollout_is_mean:0.999861478805542 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.2932093341078144e-05 - rollout_corr/rollout_is_max:1.5868637561798096 - rollout_corr/rollout_is_min:0.09759196639060974 - rollout_corr/rollout_is_std:0.0343623049557209 - rollout_corr/rollout_is_eff_sample_size:0.9988203866993153 - rollout_corr/rollout_is_seq_mean:0.9999100565910339 - rollout_corr/rollout_is_seq_std:0.0015114216366782784 - rollout_corr/rollout_is_seq_max:1.004460334777832 - rollout_corr/rollout_is_seq_min:0.9952100515365601 - rollout_corr/rollout_is_seq_max_deviation:0.004789948463439941 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.013377800118178) - rollout_corr/training_log_ppl:np.float64(0.6853225293452851) - rollout_corr/kl:np.float64(0.0008045039580757418) - rollout_corr/k3_kl:np.float64(0.0007065492574156451) - rollout_corr/rollout_ppl:np.float64(2.0117388791404665) - rollout_corr/rollout_log_ppl:np.float64(0.684518019144889) - rollout_corr/log_ppl_diff:np.float64(0.0008045102003961802) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014397446066141129) - rollout_corr/log_ppl_diff_max:np.float64(0.007155239582061768) - rollout_corr/log_ppl_diff_min:np.float64(-0.004554927349090576) - rollout_corr/ppl_ratio:np.float64(1.0008063388522714) - rollout_corr/chi2_token:np.float64(0.0012174241710454226) - rollout_corr/chi2_seq:np.float64(1.7728085950948298) - actor/pg_loss:np.float64(0.0003834259696304798) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000311835868842536) - actor/ppo_kl:np.float64(9.214319616290823e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.32652221247553825) - perf/mfu/actor:np.float64(0.11250225764410283) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.84423065185547) - actor/lr:np.float64(1e-06) - training/global_step:23 - training/epoch:0 - critic/score/mean:0.609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0047743674367666245 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0047743674367666245 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:604.1171875 - response_length/max:1391.0 - response_length/min:180.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:604.1171875 - response_length_non_aborted/max:1391.0 - response_length_non_aborted/min:180.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.71875 - prompt_length/max:766.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010936148464679718 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4273153580725193) - timing_s/agent_loop/generate_sequences/max:np.float64(12.947387943044305) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.094733837271633) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.947387943044305) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:266 - timing_s/agent_loop/slowest/response_length:1391 - timing_s/gen:19.754568798467517 - timing_s/reward:0.07217162102460861 - timing_s/old_log_prob:3.125368718057871 - timing_s/adv:0.1287905927747488 - timing_s/update_actor:13.216036595404148 - timing_s/step:36.38285103067756 - timing_s/stop_profile:0.00010989978909492493 - timing_per_token_ms/adv:0.0005705009646721985 - timing_per_token_ms/gen:0.12773396613386992 - timing_per_token_ms/update_actor:0.05854279776480243 - perf/total_num_tokens:225750 - perf/time_per_step:36.38285103067756 - perf/throughput:775.6057923060046
(TaskRunner pid=2224264) Training Progress: 23%|██▎ | 23/100 [20:26<1:14:27, 58.02s/it]
(TaskRunner pid=2224264) step:24 - global_seqlen/min:25634 - global_seqlen/max:31704 - global_seqlen/minmax_diff:6070 - global_seqlen/balanced_min:29137 - global_seqlen/balanced_max:29209 - global_seqlen/mean:29156.5 - actor/entropy:0.712544858455658 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3460633456707001 - training/rollout_probs_diff_mean:0.004510445520281792 - training/rollout_probs_diff_std:0.008606033399701118 - training/rollout_actor_probs_pearson_corr:0.9996117949485779 - rollout_corr/rollout_is_mean:0.9999745488166809 - rollout_corr/rollout_is_ratio_fraction_high:6.192870841914555e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.238574168382911e-05 - rollout_corr/rollout_is_max:2.087355136871338 - rollout_corr/rollout_is_min:0.47689226269721985 - rollout_corr/rollout_is_std:0.03445065766572952 - rollout_corr/rollout_is_eff_sample_size:0.9988145592508434 - rollout_corr/rollout_is_seq_mean:0.999997615814209 - rollout_corr/rollout_is_seq_std:0.0016162468818947673 - rollout_corr/rollout_is_seq_max:1.0054336786270142 - rollout_corr/rollout_is_seq_min:0.9961015582084656 - rollout_corr/rollout_is_seq_max_deviation:0.00543367862701416 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.082198773045093) - rollout_corr/training_log_ppl:np.float64(0.7211795629700646) - rollout_corr/kl:np.float64(0.0007120691560231762) - rollout_corr/k3_kl:np.float64(0.0006404822171930391) - rollout_corr/rollout_ppl:np.float64(2.0806773733347654) - rollout_corr/rollout_log_ppl:np.float64(0.7204674915410578) - rollout_corr/log_ppl_diff:np.float64(0.0007120714290067554) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014570994535461068) - rollout_corr/log_ppl_diff_max:np.float64(0.005695939064025879) - rollout_corr/log_ppl_diff_min:np.float64(-0.0035497546195983887) - rollout_corr/ppl_ratio:np.float64(1.000713731860742) - rollout_corr/chi2_token:np.float64(0.0011395812034606934) - rollout_corr/chi2_seq:np.float64(1.328517090762034) - actor/pg_loss:np.float64(-4.311627708375454e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001907837179260241) - actor/ppo_kl:np.float64(0.0001019216878037632) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3508700765669346) - perf/mfu/actor:np.float64(0.11424736196001828) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(104.2998194694519) - actor/lr:np.float64(1e-06) - training/global_step:24 - training/epoch:0 - critic/score/mean:0.49609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.49609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.002073063515126705 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.002073063515126705 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:630.765625 - response_length/max:1901.0 - response_length/min:207.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:630.765625 - response_length_non_aborted/max:1901.0 - response_length_non_aborted/min:207.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.375 - prompt_length/max:848.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011662021279335022 - timing_s/agent_loop/generate_sequences/min:np.float64(2.381055638194084) - timing_s/agent_loop/generate_sequences/max:np.float64(15.576675411313772) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.741719769590418) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.576675411313772) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:270 - timing_s/agent_loop/slowest/response_length:1901 - timing_s/gen:22.96907013654709 - timing_s/reward:0.07352139614522457 - timing_s/old_log_prob:3.155397677794099 - timing_s/adv:0.11861182004213333 - timing_s/update_actor:13.56319702602923 - timing_s/step:39.965819630771875 - timing_s/stop_profile:0.00010657869279384613 - timing_per_token_ms/adv:0.0005085136249298326 - timing_per_token_ms/gen:0.14224448299776493 - timing_per_token_ms/update_actor:0.05814825607510002 - perf/total_num_tokens:233252 - perf/time_per_step:39.965819630771875 - perf/throughput:729.5358951565405
(TaskRunner pid=2224264) Training Progress: 24%|██▍ | 24/100 [21:06<1:06:38, 52.62s/it]
(TaskRunner pid=2224264) step:25 - global_seqlen/min:23500 - global_seqlen/max:34274 - global_seqlen/minmax_diff:10774 - global_seqlen/balanced_min:27954 - global_seqlen/balanced_max:27980 - global_seqlen/mean:27971.25 - actor/entropy:0.7009389400482178 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18294647336006165 - training/rollout_probs_diff_mean:0.004573621321469545 - training/rollout_probs_diff_std:0.008723455481231213 - training/rollout_actor_probs_pearson_corr:0.999584436416626 - rollout_corr/rollout_is_mean:0.9999302625656128 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:6.836859029135667e-06 - rollout_corr/rollout_is_max:1.4950411319732666 - rollout_corr/rollout_is_min:0.4878106117248535 - rollout_corr/rollout_is_std:0.03484973683953285 - rollout_corr/rollout_is_eff_sample_size:0.9987866122287246 - rollout_corr/rollout_is_seq_mean:0.9999464154243469 - rollout_corr/rollout_is_seq_std:0.001561047974973917 - rollout_corr/rollout_is_seq_max:1.004056453704834 - rollout_corr/rollout_is_seq_min:0.9949167370796204 - rollout_corr/rollout_is_seq_max_deviation:0.005083262920379639 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1113617275841534) - rollout_corr/training_log_ppl:np.float64(0.7249491217080504) - rollout_corr/kl:np.float64(0.0006290591016089309) - rollout_corr/k3_kl:np.float64(0.0006558882103604446) - rollout_corr/rollout_ppl:np.float64(2.1101012001745403) - rollout_corr/rollout_log_ppl:np.float64(0.7243200563825667) - rollout_corr/log_ppl_diff:np.float64(0.0006290653254836798) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012821299023926258) - rollout_corr/log_ppl_diff_max:np.float64(0.005804777145385742) - rollout_corr/log_ppl_diff_min:np.float64(-0.004645586013793945) - rollout_corr/ppl_ratio:np.float64(1.0006304802373052) - rollout_corr/chi2_token:np.float64(0.00137246772646904) - rollout_corr/chi2_seq:np.float64(0.4346559832338244) - actor/pg_loss:np.float64(-0.00020399875938892365) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024524242849111033) - actor/ppo_kl:np.float64(-6.53956497700392e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.31579897925257683) - perf/mfu/actor:np.float64(0.10990411377383731) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(108.17280960083008) - actor/lr:np.float64(1e-06) - training/global_step:25 - training/epoch:0 - critic/score/mean:0.5 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002174121094867587 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002174121094867587 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:571.3515625 - response_length/max:1596.0 - response_length/min:161.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:571.3515625 - response_length_non_aborted/max:1596.0 - response_length_non_aborted/min:161.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.75 - prompt_length/max:646.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010717101395130157 - timing_s/agent_loop/generate_sequences/min:np.float64(3.3154151998460293) - timing_s/agent_loop/generate_sequences/max:np.float64(15.293862147256732) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.137247967235453) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.293862147256732) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:202 - timing_s/agent_loop/slowest/response_length:1596 - timing_s/gen:22.198232408612967 - timing_s/reward:0.06951954215765 - timing_s/old_log_prob:3.2000021431595087 - timing_s/adv:0.11978011205792427 - timing_s/update_actor:13.237928776070476 - timing_s/step:38.91434215568006 - timing_s/stop_profile:0.00012552551925182343 - timing_per_token_ms/adv:0.000535282263296797 - timing_per_token_ms/gen:0.15176618222015345 - timing_per_token_ms/update_actor:0.05915863956772791 - perf/total_num_tokens:223770 - perf/time_per_step:38.91434215568006 - perf/throughput:718.7902570239705
(TaskRunner pid=2224264) Training Progress: 25%|██▌ | 25/100 [21:45<1:00:38, 48.52s/it]
(TaskRunner pid=2224264) step:26 - global_seqlen/min:22426 - global_seqlen/max:29951 - global_seqlen/minmax_diff:7525 - global_seqlen/balanced_min:26952 - global_seqlen/balanced_max:26964 - global_seqlen/mean:26960.375 - actor/entropy:0.7369614243507385 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.31862038373947144 - training/rollout_probs_diff_mean:0.004758655559271574 - training/rollout_probs_diff_std:0.00890500657260418 - training/rollout_actor_probs_pearson_corr:0.9995867609977722 - rollout_corr/rollout_is_mean:0.9999259114265442 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.4305845070339274e-05 - rollout_corr/rollout_is_max:1.8682441711425781 - rollout_corr/rollout_is_min:0.27226293087005615 - rollout_corr/rollout_is_std:0.03548027575016022 - rollout_corr/rollout_is_eff_sample_size:0.9987426137038081 - rollout_corr/rollout_is_seq_mean:0.9999041557312012 - rollout_corr/rollout_is_seq_std:0.001717878389172256 - rollout_corr/rollout_is_seq_max:1.005202293395996 - rollout_corr/rollout_is_seq_min:0.9952166080474854 - rollout_corr/rollout_is_seq_max_deviation:0.005202293395996094 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1379281943663955) - rollout_corr/training_log_ppl:np.float64(0.7447412704350427) - rollout_corr/kl:np.float64(0.0007289938001822094) - rollout_corr/k3_kl:np.float64(0.0007074658968804215) - rollout_corr/rollout_ppl:np.float64(2.136324817314744) - rollout_corr/rollout_log_ppl:np.float64(0.7440122718689963) - rollout_corr/log_ppl_diff:np.float64(0.0007289985660463572) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015073230024427176) - rollout_corr/log_ppl_diff_max:np.float64(0.005815982818603516) - rollout_corr/log_ppl_diff_min:np.float64(-0.004653453826904297) - rollout_corr/ppl_ratio:np.float64(1.00073087005876) - rollout_corr/chi2_token:np.float64(0.0013713838998228312) - rollout_corr/chi2_seq:np.float64(0.9597642624285072) - actor/pg_loss:np.float64(0.00014449155423790216) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00012465850477383356) - actor/ppo_kl:np.float64(-1.70074631995476e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2908688709139824) - perf/mfu/actor:np.float64(0.105823819821739) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(104.64182186126709) - actor/lr:np.float64(1e-06) - training/global_step:26 - training/epoch:0 - critic/score/mean:0.5546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0029881333466619253 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0029881333466619253 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:546.10546875 - response_length/max:1208.0 - response_length/min:146.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:546.10546875 - response_length_non_aborted/max:1208.0 - response_length_non_aborted/min:146.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:296.40625 - prompt_length/max:709.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015214644372463226 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7672749813646078) - timing_s/agent_loop/generate_sequences/max:np.float64(10.418701702728868) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.990022340542055) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.418701702728868) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:329 - timing_s/agent_loop/slowest/response_length:1208 - timing_s/gen:18.300879038870335 - timing_s/reward:0.07066521607339382 - timing_s/old_log_prob:3.2463080435991287 - timing_s/adv:0.16146212071180344 - timing_s/update_actor:13.367159321904182 - timing_s/step:35.227726478129625 - timing_s/stop_profile:2.6691704988479614e-05 - timing_per_token_ms/adv:0.0007486084703560477 - timing_per_token_ms/gen:0.13090476627018258 - timing_per_token_ms/update_actor:0.06197595230919536 - perf/total_num_tokens:215683 - perf/time_per_step:35.227726478129625 - perf/throughput:765.3169163992961
(TaskRunner pid=2224264) Training Progress: 26%|██▌ | 26/100 [22:20<54:56, 44.54s/it]
(TaskRunner pid=2224264) step:27 - global_seqlen/min:23952 - global_seqlen/max:34841 - global_seqlen/minmax_diff:10889 - global_seqlen/balanced_min:28921 - global_seqlen/balanced_max:28949 - global_seqlen/mean:28940.875 - actor/entropy:0.701714038848877 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.17479756474494934 - training/rollout_probs_diff_mean:0.004477211274206638 - training/rollout_probs_diff_std:0.008556351996958256 - training/rollout_actor_probs_pearson_corr:0.9996283054351807 - rollout_corr/rollout_is_mean:0.9999887943267822 - rollout_corr/rollout_is_ratio_fraction_high:6.0401425798772834e-06 - rollout_corr/rollout_is_ratio_fraction_low:6.0401425798772834e-06 - rollout_corr/rollout_is_max:2.401174545288086 - rollout_corr/rollout_is_min:0.40530839562416077 - rollout_corr/rollout_is_std:0.034324124455451965 - rollout_corr/rollout_is_eff_sample_size:0.9988231220557214 - rollout_corr/rollout_is_seq_mean:0.9999889135360718 - rollout_corr/rollout_is_seq_std:0.0014754357980564237 - rollout_corr/rollout_is_seq_max:1.0065122842788696 - rollout_corr/rollout_is_seq_min:0.9951512813568115 - rollout_corr/rollout_is_seq_max_deviation:0.006512284278869629 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.07450102083385) - rollout_corr/training_log_ppl:np.float64(0.7144390830071643) - rollout_corr/kl:np.float64(0.0007389031823423764) - rollout_corr/k3_kl:np.float64(0.0006497853174778356) - rollout_corr/rollout_ppl:np.float64(2.0729061695747077) - rollout_corr/rollout_log_ppl:np.float64(0.713700178428553) - rollout_corr/log_ppl_diff:np.float64(0.0007389045786112547) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013226103037595749) - rollout_corr/log_ppl_diff_max:np.float64(0.0057825446128845215) - rollout_corr/log_ppl_diff_min:np.float64(-0.005777716636657715) - rollout_corr/ppl_ratio:np.float64(1.0007404030766338) - rollout_corr/chi2_token:np.float64(0.001133262412622571) - rollout_corr/chi2_seq:np.float64(0.8787905650679022) - actor/pg_loss:np.float64(-2.4800654500722885e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00022700108638673555) - actor/ppo_kl:np.float64(0.00011755238243083) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.32230544835329056) - perf/mfu/actor:np.float64(0.11335348091929753) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(104.77265930175781) - actor/lr:np.float64(1e-06) - training/global_step:27 - training/epoch:0 - critic/score/mean:0.515625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.515625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006488623563200235 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006488623563200235 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:646.71484375 - response_length/max:1758.0 - response_length/min:222.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:646.71484375 - response_length_non_aborted/max:1758.0 - response_length_non_aborted/min:222.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:257.6875 - prompt_length/max:380.0 - prompt_length/min:160.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.4581090807914734e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.452285857871175) - timing_s/agent_loop/generate_sequences/max:np.float64(14.636356772854924) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.195729885730543) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.636356772854924) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:194 - timing_s/agent_loop/slowest/response_length:1758 - timing_s/gen:21.458807596936822 - timing_s/reward:0.06992848217487335 - timing_s/old_log_prob:3.163326393812895 - timing_s/adv:0.12315551564097404 - timing_s/update_actor:13.37716430425644 - timing_s/step:38.279962316155434 - timing_s/stop_profile:0.00011939182877540588 - timing_per_token_ms/adv:0.0005319272293986189 - timing_per_token_ms/gen:0.1296142619666513 - timing_per_token_ms/update_actor:0.057777988330762454 - perf/total_num_tokens:231527 - perf/time_per_step:38.279962316155434 - perf/throughput:756.0319616037337
(TaskRunner pid=2224264) Training Progress: 27%|██▋ | 27/100 [22:58<51:55, 42.68s/it]
(TaskRunner pid=2224264) step:28 - global_seqlen/min:25216 - global_seqlen/max:34360 - global_seqlen/minmax_diff:9144 - global_seqlen/balanced_min:29787 - global_seqlen/balanced_max:30195 - global_seqlen/mean:29843.75 - actor/entropy:0.7055036425590515 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1455976366996765 - training/rollout_probs_diff_mean:0.004539164714515209 - training/rollout_probs_diff_std:0.008678900077939034 - training/rollout_actor_probs_pearson_corr:0.9996199011802673 - rollout_corr/rollout_is_mean:1.0000463724136353 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.916669579164591e-06 - rollout_corr/rollout_is_max:1.6682544946670532 - rollout_corr/rollout_is_min:0.4844903349876404 - rollout_corr/rollout_is_std:0.0349300317466259 - rollout_corr/rollout_is_eff_sample_size:0.9987814986875572 - rollout_corr/rollout_is_seq_mean:1.0001145601272583 - rollout_corr/rollout_is_seq_std:0.0015147364465519786 - rollout_corr/rollout_is_seq_max:1.0062413215637207 - rollout_corr/rollout_is_seq_min:0.9959177374839783 - rollout_corr/rollout_is_seq_max_deviation:0.006241321563720703 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.081616838928312) - rollout_corr/training_log_ppl:np.float64(0.716681465273723) - rollout_corr/kl:np.float64(0.0007082077879232429) - rollout_corr/k3_kl:np.float64(0.0006868325160667155) - rollout_corr/rollout_ppl:np.float64(2.0801399582996964) - rollout_corr/rollout_log_ppl:np.float64(0.7159732520813122) - rollout_corr/log_ppl_diff:np.float64(0.0007082131924107671) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013121537631377578) - rollout_corr/log_ppl_diff_max:np.float64(0.00504302978515625) - rollout_corr/log_ppl_diff_min:np.float64(-0.0041460394859313965) - rollout_corr/ppl_ratio:np.float64(1.0007096340414137) - rollout_corr/chi2_token:np.float64(0.0013347528874874115) - rollout_corr/chi2_seq:np.float64(4.891642920905724) - actor/pg_loss:np.float64(-6.637291517108679e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002813626040278905) - actor/ppo_kl:np.float64(0.00019132641280006357) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.33150169998407364) - perf/mfu/actor:np.float64(0.11585717187378697) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.45846176147461) - actor/lr:np.float64(1e-06) - training/global_step:28 - training/epoch:0 - critic/score/mean:0.5390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.007486805785447359 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:-0.007486805785447359 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:660.2109375 - response_length/max:2519.0 - response_length/min:204.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:660.2109375 - response_length_non_aborted/max:2519.0 - response_length_non_aborted/min:204.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:272.40625 - prompt_length/max:443.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011588819324970245 - timing_s/agent_loop/generate_sequences/min:np.float64(2.6954549103975296) - timing_s/agent_loop/generate_sequences/max:np.float64(18.969659883528948) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.485852991718275) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(18.969659883528948) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:203 - timing_s/agent_loop/slowest/response_length:2519 - timing_s/gen:25.569389276206493 - timing_s/reward:0.08602294698357582 - timing_s/old_log_prob:3.1977268513292074 - timing_s/adv:0.1315357405692339 - timing_s/update_actor:13.538917873054743 - timing_s/step:42.61257563531399 - timing_s/stop_profile:0.00011502020061016083 - timing_per_token_ms/adv:0.0005509350390334404 - timing_per_token_ms/gen:0.1512856288603695 - timing_per_token_ms/update_actor:0.056707509415936094 - perf/total_num_tokens:238750 - perf/time_per_step:42.61257563531399 - perf/throughput:700.3507662950987
(TaskRunner pid=2224264) Training Progress: 28%|██▊ | 28/100 [23:41<51:12, 42.67s/it]
(TaskRunner pid=2224264) step:29 - global_seqlen/min:26637 - global_seqlen/max:40228 - global_seqlen/minmax_diff:13591 - global_seqlen/balanced_min:33532 - global_seqlen/balanced_max:33558 - global_seqlen/mean:33551.125 - actor/entropy:0.6751730442047119 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.30162957310676575 - training/rollout_probs_diff_mean:0.004225059412419796 - training/rollout_probs_diff_std:0.008393223397433758 - training/rollout_actor_probs_pearson_corr:0.9996880292892456 - rollout_corr/rollout_is_mean:0.9999781847000122 - rollout_corr/rollout_is_ratio_fraction_high:5.061727733846055e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.012345546769211e-05 - rollout_corr/rollout_is_max:2.110851764678955 - rollout_corr/rollout_is_min:0.42815104126930237 - rollout_corr/rollout_is_std:0.033269479870796204 - rollout_corr/rollout_is_eff_sample_size:0.9988943655271626 - rollout_corr/rollout_is_seq_mean:1.0000176429748535 - rollout_corr/rollout_is_seq_std:0.001319664646871388 - rollout_corr/rollout_is_seq_max:1.0051316022872925 - rollout_corr/rollout_is_seq_min:0.9961720108985901 - rollout_corr/rollout_is_seq_max_deviation:0.0051316022872924805 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.010434144642204) - rollout_corr/training_log_ppl:np.float64(0.6835667735431343) - rollout_corr/kl:np.float64(0.0005298591240814332) - rollout_corr/k3_kl:np.float64(0.0006146891930143283) - rollout_corr/rollout_ppl:np.float64(2.009353395085782) - rollout_corr/rollout_log_ppl:np.float64(0.6830369132512715) - rollout_corr/log_ppl_diff:np.float64(0.0005298602918628603) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011505558213684708) - rollout_corr/log_ppl_diff_max:np.float64(0.0041373372077941895) - rollout_corr/log_ppl_diff_min:np.float64(-0.005634099245071411) - rollout_corr/ppl_ratio:np.float64(1.0005309600383043) - rollout_corr/chi2_token:np.float64(0.0014236378483474255) - rollout_corr/chi2_seq:np.float64(0.9517913018353283) - actor/pg_loss:np.float64(7.968046702444553e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001298733541261754) - actor/ppo_kl:np.float64(-2.9945098688699545e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.21987130120396614) - perf/mfu/actor:np.float64(0.11725733884439407) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(109.95436000823975) - actor/lr:np.float64(1e-06) - training/global_step:29 - training/epoch:0 - critic/score/mean:0.46484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00392979895696044 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00392979895696044 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:771.72265625 - response_length/max:1889.0 - response_length/min:208.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:771.72265625 - response_length_non_aborted/max:1889.0 - response_length_non_aborted/min:208.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:276.75 - prompt_length/max:492.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.993037045001984e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.6769527215510607) - timing_s/agent_loop/generate_sequences/max:np.float64(16.69622828066349) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.844669617952604) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.69622828066349) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:330 - timing_s/agent_loop/slowest/response_length:1889 - timing_s/gen:23.39403291605413 - timing_s/reward:0.07856784015893936 - timing_s/old_log_prob:4.369897283613682 - timing_s/adv:0.31469788402318954 - timing_s/update_actor:15.082241535186768 - timing_s/step:43.32600663602352 - timing_s/stop_profile:0.0001105237752199173 - timing_per_token_ms/adv:0.001172456527252028 - timing_per_token_ms/gen:0.11841422606715966 - timing_per_token_ms/update_actor:0.05619126607225081 - perf/total_num_tokens:268409 - perf/time_per_step:43.32600663602352 - perf/throughput:774.387662400066
(TaskRunner pid=2224264) Training Progress: 29%|██▉ | 29/100 [24:24<50:44, 42.88s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 30}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) To determine which of the given options is the correct reactant for the synthesis of the product **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**, we need to analyze the structure and functional groups involved.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) The target product is:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This can be interpreted as:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - A **nitrile group (–CN)** attached to a ring (likely a pyrrole or similar heterocycle).
(TaskRunner pid=2224264) - The ring has a **chloro group (–Cl)** on a carbon and a **trifluoromethyl group (–CF₃)** on another carbon.
(TaskRunner pid=2224264) - The structure suggests that the molecule is a **pyrrole derivative** with a **chloro-substituted benzene ring** and a **trifluoromethyl group** attached to the same ring.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Now, let's examine the options one by one to see which one could be a **reactant** that leads to this product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option A:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This is a **complex molecule**. It contains:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - A **carbonyl group (C=O)** and a **nitrile group (–CN)**.
(TaskRunner pid=2224264) - A **benzene ring** with a **chloro group (–Cl)** and a **trifluoromethyl group (–CF₃)**.
(TaskRunner pid=2224264) - A **pyrrole-like ring** (as suggested by the nitrile group and the ring structure).
(TaskRunner pid=2224264) - The **nitrile group** is attached to the ring in a way that matches the target product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This option **matches the structure** of the target molecule. It is **likely the correct reactant** that could be used in the synthesis of the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option B:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This structure contains:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - A **carbonyl group (C=O)**.
(TaskRunner pid=2224264) - A **chloro group (–Cl)**.
(TaskRunner pid=2224264) - A **pyrrole-like ring**.
(TaskRunner pid=2224264) - However, the **trifluoromethyl group (–CF₃)** is **missing**.
(TaskRunner pid=2224264) - The **nitrile group (–CN)** is **not clearly present**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This molecule is **not a good match** for the target product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option C:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This structure contains:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - A **carbonyl group (C=O)**.
(TaskRunner pid=2224264) - A **trifluoromethyl group (–CF₃)**.
(TaskRunner pid=2224264) - A **benzene ring** with a **chloro group (–Cl)**.
(TaskRunner pid=2224264) - A **pyrrole-like ring**.
(TaskRunner pid=2224264) - However, the **nitrile group (–CN)** is **missing**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This molecule is **not a good match** for the target product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option D:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This structure contains:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - A **carbonyl group (C=O)**.
(TaskRunner pid=2224264) - A **trifluoromethyl group (–CF₃)**.
(TaskRunner pid=2224264) - A **benzene ring**.
(TaskRunner pid=2224264) - A **pyrrole-like ring**.
(TaskRunner pid=2224264) - However, the **nitrile group (–CN)** is **missing**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Again, this molecule is **not a good match** for the target product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Final Analysis:
(TaskRunner pid=2224264) Only **Option A** contains all the necessary functional groups and structure to be a **valid reactant** for the synthesis of the target molecule **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_30
(TaskRunner pid=2224264) step:30 - global_seqlen/min:28624 - global_seqlen/max:34380 - global_seqlen/minmax_diff:5756 - global_seqlen/balanced_min:31823 - global_seqlen/balanced_max:31938 - global_seqlen/mean:31860.75 - actor/entropy:0.7115141153335571 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5642241835594177 - training/rollout_probs_diff_mean:0.004424138925969601 - training/rollout_probs_diff_std:0.008687857538461685 - training/rollout_actor_probs_pearson_corr:0.999598503112793 - rollout_corr/rollout_is_mean:0.9999087452888489 - rollout_corr/rollout_is_ratio_fraction_high:5.690936632163357e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.1381873264326714e-05 - rollout_corr/rollout_is_max:2.0644965171813965 - rollout_corr/rollout_is_min:0.22554616630077362 - rollout_corr/rollout_is_std:0.034556034952402115 - rollout_corr/rollout_is_eff_sample_size:0.9988071858420651 - rollout_corr/rollout_is_seq_mean:0.9999098777770996 - rollout_corr/rollout_is_seq_std:0.0014933171914890409 - rollout_corr/rollout_is_seq_max:1.0051254034042358 - rollout_corr/rollout_is_seq_min:0.9926964044570923 - rollout_corr/rollout_is_seq_max_deviation:0.007303595542907715 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1091170106083155) - rollout_corr/training_log_ppl:np.float64(0.7282795129576698) - rollout_corr/kl:np.float64(0.0007562485593410884) - rollout_corr/k3_kl:np.float64(0.0006585642515801737) - rollout_corr/rollout_ppl:np.float64(2.107493511401117) - rollout_corr/rollout_log_ppl:np.float64(0.7275232608662918) - rollout_corr/log_ppl_diff:np.float64(0.0007562520913779736) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013768195640295744) - rollout_corr/log_ppl_diff_max:np.float64(0.005323231220245361) - rollout_corr/log_ppl_diff_min:np.float64(-0.0032753348350524902) - rollout_corr/ppl_ratio:np.float64(1.0007577680516988) - rollout_corr/chi2_token:np.float64(0.0011402126401662827) - rollout_corr/chi2_seq:np.float64(0.9514149494934827) - actor/pg_loss:np.float64(-0.00013444386422634125) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002210548966559145) - actor/ppo_kl:np.float64(4.4027712672978225e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2887660190463066) - perf/mfu/actor:np.float64(0.12035666503727542) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(107.33295822143555) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.549404761904762) - val-aux/sciknoweval/reward/std@16:np.float64(0.31608937376624846) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.6822666666666667) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.25471779568655717) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.41630476190476196) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.28182671788474917) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.5503333333333333) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.3158685881058942) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7810047619047619) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.1712884300404119) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.30183333333333334) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.21622513075595678) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.587) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.2597657274415112) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8442666666666666) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.10407977320648662) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.21562857142857142) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.14281142327250002) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.609352380952381) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.1968004873596571) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8824571428571428) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.056375240990812564) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.16108571428571428) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.07902405626412515) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6248190476190476) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.1453890445797554) - val-aux/sciknoweval/score/mean@16:np.float64(0.549404761904762) - val-aux/sciknoweval/score/std@16:np.float64(0.31608937376624846) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.6822666666666667) - val-aux/sciknoweval/score/best@2/std:np.float64(0.25471779568655717) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.41630476190476196) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.28182671788474917) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.5503333333333333) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.3158685881058942) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7810047619047619) - val-aux/sciknoweval/score/best@4/std:np.float64(0.1712884300404119) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.30183333333333334) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.21622513075595678) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.587) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.2597657274415112) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8442666666666666) - val-aux/sciknoweval/score/best@8/std:np.float64(0.10407977320648662) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.21562857142857142) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.14281142327250002) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.609352380952381) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.1968004873596571) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8824571428571428) - val-aux/sciknoweval/score/best@16/std:np.float64(0.056375240990812564) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.16108571428571428) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.07902405626412515) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6248190476190476) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.1453890445797554) - val-core/sciknoweval/acc/mean@16:np.float64(0.549404761904762) - val-aux/sciknoweval/acc/std@16:np.float64(0.31608937376624846) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.6822666666666667) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.25471779568655717) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.41630476190476196) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.28182671788474917) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.5503333333333333) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.3158685881058942) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7810047619047619) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.1712884300404119) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.30183333333333334) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.21622513075595678) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.587) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.2597657274415112) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8442666666666666) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.10407977320648662) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.21562857142857142) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.14281142327250002) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.609352380952381) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.1968004873596571) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8824571428571428) - val-core/sciknoweval/acc/best@16/std:np.float64(0.056375240990812564) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.16108571428571428) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.07902405626412515) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6248190476190476) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.1453890445797554) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9892857142857143) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.03624438335470682) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9989476190476191) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.010879631942208362) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9796380952380952) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.04790227517330985) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9895285714285714) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.03580241391826494) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0001505093393264677) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9626380952380952) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.05990016890311522) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9957904761904761) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0229613769030037) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9365666666666667) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.06682360250957863) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9989619047619047) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.011013375581110648) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9040238095238093) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.061018043502578855) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999142857142858) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0015394818506189996) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:30 - training/epoch:0 - critic/score/mean:0.41796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.41796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002168958308175206 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002168958308175206 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:686.3984375 - response_length/max:2205.0 - response_length/min:164.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:686.3984375 - response_length_non_aborted/max:2205.0 - response_length_non_aborted/min:164.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:309.25 - prompt_length/max:516.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.668455898761749e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4918374791741371) - timing_s/agent_loop/generate_sequences/max:np.float64(17.408627146854997) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.319493488706939) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(17.408627146854997) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:349 - timing_s/agent_loop/slowest/response_length:2205 - timing_s/gen:24.27543739043176 - timing_s/reward:0.07385276816785336 - timing_s/old_log_prob:3.287810454145074 - timing_s/adv:0.10759958252310753 - timing_s/update_actor:13.824590772390366 - timing_s/step:41.648208467289805 - timing_s/testing:186.15690745972097 - timing_s/save_checkpoint:15.06649879179895 - timing_s/stop_profile:0.00015422143042087555 - timing_per_token_ms/adv:0.00042214787208048904 - timing_per_token_ms/gen:0.13814997547452032 - timing_per_token_ms/update_actor:0.05423832918398957 - perf/total_num_tokens:254886 - perf/time_per_step:41.648208467289805 - perf/throughput:764.9968911633546
(TaskRunner pid=2224264) Training Progress: 30%|███ | 30/100 [28:27<2:00:03, 102.91s/it]
(TaskRunner pid=2224264) step:31 - global_seqlen/min:30423 - global_seqlen/max:35721 - global_seqlen/minmax_diff:5298 - global_seqlen/balanced_min:33243 - global_seqlen/balanced_max:33352 - global_seqlen/mean:33261.875 - actor/entropy:0.7014114856719971 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.39983245730400085 - training/rollout_probs_diff_mean:0.004444092512130737 - training/rollout_probs_diff_std:0.008645395748317242 - training/rollout_actor_probs_pearson_corr:0.9995979070663452 - rollout_corr/rollout_is_mean:1.0000203847885132 - rollout_corr/rollout_is_ratio_fraction_high:5.1425781748548616e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.1425781748548616e-06 - rollout_corr/rollout_is_max:3.193486452102661 - rollout_corr/rollout_is_min:0.4723663926124573 - rollout_corr/rollout_is_std:0.03425285220146179 - rollout_corr/rollout_is_eff_sample_size:0.998828117093035 - rollout_corr/rollout_is_seq_mean:1.0000666379928589 - rollout_corr/rollout_is_seq_std:0.0014698212035000324 - rollout_corr/rollout_is_seq_max:1.0053932666778564 - rollout_corr/rollout_is_seq_min:0.9955719709396362 - rollout_corr/rollout_is_seq_max_deviation:0.005393266677856445 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.08949265582487) - rollout_corr/training_log_ppl:np.float64(0.7218374547082931) - rollout_corr/kl:np.float64(0.0005589085460151466) - rollout_corr/k3_kl:np.float64(0.0006561674455269895) - rollout_corr/rollout_ppl:np.float64(2.088360784109682) - rollout_corr/rollout_log_ppl:np.float64(0.7212785450974479) - rollout_corr/log_ppl_diff:np.float64(0.0005589096108451486) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012190231354907155) - rollout_corr/log_ppl_diff_max:np.float64(0.004900336265563965) - rollout_corr/log_ppl_diff_min:np.float64(-0.0036498308181762695) - rollout_corr/ppl_ratio:np.float64(1.0005601237062365) - rollout_corr/chi2_token:np.float64(0.0015174625441432) - rollout_corr/chi2_seq:np.float64(2.2517903256230056) - actor/pg_loss:np.float64(-9.307335130870342e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00020832591962971492) - actor/ppo_kl:np.float64(1.1285746243316908e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2606422230601311) - perf/mfu/actor:np.float64(0.12484098624463014) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(113.83588361740112) - actor/lr:np.float64(1e-06) - training/global_step:31 - training/epoch:0 - critic/score/mean:0.51171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.009717543609440327 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.009717543609440327 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:759.58984375 - response_length/max:2514.0 - response_length/min:234.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:759.58984375 - response_length_non_aborted/max:2514.0 - response_length_non_aborted/min:234.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.84375 - prompt_length/max:635.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014492124319076538 - timing_s/agent_loop/generate_sequences/min:np.float64(3.011936265975237) - timing_s/agent_loop/generate_sequences/max:np.float64(20.60571327060461) - timing_s/agent_loop/generate_sequences/mean:np.float64(10.018827380772564) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(20.60571327060461) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:198 - timing_s/agent_loop/slowest/response_length:2514 - timing_s/gen:27.48937368579209 - timing_s/reward:0.07477623783051968 - timing_s/old_log_prob:3.7115073055028915 - timing_s/adv:0.17105644941329956 - timing_s/update_actor:13.997227132320404 - timing_s/step:45.54523225687444 - timing_s/stop_profile:0.00011219456791877747 - timing_per_token_ms/adv:0.0006428397730633779 - timing_per_token_ms/gen:0.14136624764491573 - timing_per_token_ms/update_actor:0.05260236807275749 - perf/total_num_tokens:266095 - perf/time_per_step:45.54523225687444 - perf/throughput:730.3042130162717
(TaskRunner pid=2224264) Training Progress: 31%|███ | 31/100 [29:13<1:38:34, 85.71s/it]
(TaskRunner pid=2224264) step:32 - global_seqlen/min:25482 - global_seqlen/max:38028 - global_seqlen/minmax_diff:12546 - global_seqlen/balanced_min:31743 - global_seqlen/balanced_max:31780 - global_seqlen/mean:31769.5 - actor/entropy:0.6990305185317993 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2773066759109497 - training/rollout_probs_diff_mean:0.004490962252020836 - training/rollout_probs_diff_std:0.00860312208533287 - training/rollout_actor_probs_pearson_corr:0.9996287226676941 - rollout_corr/rollout_is_mean:1.0000104904174805 - rollout_corr/rollout_is_ratio_fraction_high:1.0702741747081745e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:3.417339324951172 - rollout_corr/rollout_is_min:0.6783297061920166 - rollout_corr/rollout_is_std:0.03459395840764046 - rollout_corr/rollout_is_eff_sample_size:0.9988045694973763 - rollout_corr/rollout_is_seq_mean:0.9999669790267944 - rollout_corr/rollout_is_seq_std:0.0015490534715354443 - rollout_corr/rollout_is_seq_max:1.004462480545044 - rollout_corr/rollout_is_seq_min:0.9947811365127563 - rollout_corr/rollout_is_seq_max_deviation:0.005218863487243652 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0522580044344068) - rollout_corr/training_log_ppl:np.float64(0.7055484902230091) - rollout_corr/kl:np.float64(0.0007327849301645983) - rollout_corr/k3_kl:np.float64(0.0006532650920689775) - rollout_corr/rollout_ppl:np.float64(2.0508041838184) - rollout_corr/rollout_log_ppl:np.float64(0.7048157050157897) - rollout_corr/log_ppl_diff:np.float64(0.0007327852072194219) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013509184354916215) - rollout_corr/log_ppl_diff_max:np.float64(0.006094217300415039) - rollout_corr/log_ppl_diff_min:np.float64(-0.0032320618629455566) - rollout_corr/ppl_ratio:np.float64(1.0007343096658587) - rollout_corr/chi2_token:np.float64(0.0011649802327156067) - rollout_corr/chi2_seq:np.float64(0.8535998552106321) - actor/pg_loss:np.float64(-7.09133455529809e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00026227998409922293) - actor/ppo_kl:np.float64(7.257984136543882e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.28863443806767464) - perf/mfu/actor:np.float64(0.12142939169953111) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(106.63142013549805) - actor/lr:np.float64(1e-06) - training/global_step:32 - training/epoch:0 - critic/score/mean:0.45703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.45703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.010431160219013691 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.010431160219013691 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:729.953125 - response_length/max:1851.0 - response_length/min:205.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:729.953125 - response_length_non_aborted/max:1851.0 - response_length_non_aborted/min:205.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.84375 - prompt_length/max:404.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010944157838821411 - timing_s/agent_loop/generate_sequences/min:np.float64(2.5309949908405542) - timing_s/agent_loop/generate_sequences/max:np.float64(15.264747710898519) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.378220335514925) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.264747710898519) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:312 - timing_s/agent_loop/slowest/response_length:1851 - timing_s/gen:22.15535177104175 - timing_s/reward:0.06814277172088623 - timing_s/old_log_prob:3.451232196763158 - timing_s/adv:0.157595532014966 - timing_s/update_actor:13.804202174767852 - timing_s/step:39.72742600180209 - timing_s/stop_profile:0.00011063739657402039 - timing_per_token_ms/adv:0.0006200740175914242 - timing_per_token_ms/gen:0.11856150743327777 - timing_per_token_ms/update_actor:0.054313894516627 - perf/total_num_tokens:254156 - perf/time_per_step:39.72742600180209 - perf/throughput:799.6868460231703
(TaskRunner pid=2224264) Training Progress: 32%|███▏ | 32/100 [29:53<1:21:31, 71.93s/it]
(TaskRunner pid=2224264) step:33 - global_seqlen/min:26534 - global_seqlen/max:38525 - global_seqlen/minmax_diff:11991 - global_seqlen/balanced_min:31307 - global_seqlen/balanced_max:31349 - global_seqlen/mean:31335.875 - actor/entropy:0.7530014514923096 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.40599530935287476 - training/rollout_probs_diff_mean:0.004578310996294022 - training/rollout_probs_diff_std:0.00858127698302269 - training/rollout_actor_probs_pearson_corr:0.9995956420898438 - rollout_corr/rollout_is_mean:0.9998608231544495 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.69842768527451e-06 - rollout_corr/rollout_is_max:1.6179550886154175 - rollout_corr/rollout_is_min:0.4878464341163635 - rollout_corr/rollout_is_std:0.03484460711479187 - rollout_corr/rollout_is_eff_sample_size:0.9987870879096347 - rollout_corr/rollout_is_seq_mean:0.9998835921287537 - rollout_corr/rollout_is_seq_std:0.0015449043130502105 - rollout_corr/rollout_is_seq_max:1.003616213798523 - rollout_corr/rollout_is_seq_min:0.9941043257713318 - rollout_corr/rollout_is_seq_max_deviation:0.005895674228668213 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1979672331362963) - rollout_corr/training_log_ppl:np.float64(0.7705774857313372) - rollout_corr/kl:np.float64(0.0007750543829923018) - rollout_corr/k3_kl:np.float64(0.0006546921346171075) - rollout_corr/rollout_ppl:np.float64(2.1962045463733375) - rollout_corr/rollout_log_ppl:np.float64(0.7698024294222705) - rollout_corr/log_ppl_diff:np.float64(0.0007750563090667129) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013818222796544433) - rollout_corr/log_ppl_diff_max:np.float64(0.006634533405303955) - rollout_corr/log_ppl_diff_min:np.float64(-0.004233121871948242) - rollout_corr/ppl_ratio:np.float64(1.0007766336202621) - rollout_corr/chi2_token:np.float64(0.0010767022613435984) - rollout_corr/chi2_seq:np.float64(0.8498944872990251) - actor/pg_loss:np.float64(-0.0003114178543910384) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014040332393960853) - actor/ppo_kl:np.float64(3.355153801543054e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2854798920452595) - perf/mfu/actor:np.float64(0.12016219905694546) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(104.70193099975586) - actor/lr:np.float64(1e-06) - training/global_step:33 - training/epoch:0 - critic/score/mean:0.57421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.57421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.013369936496019363 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.013369936496019363 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:685.49609375 - response_length/max:1829.0 - response_length/min:195.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:685.49609375 - response_length_non_aborted/max:1829.0 - response_length_non_aborted/min:195.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:293.75 - prompt_length/max:474.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012979470193386078 - timing_s/agent_loop/generate_sequences/min:np.float64(2.423111606389284) - timing_s/agent_loop/generate_sequences/max:np.float64(15.327023109421134) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.73543391644489) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.327023109421134) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:198 - timing_s/agent_loop/slowest/response_length:1829 - timing_s/gen:22.37846451252699 - timing_s/reward:0.07666251063346863 - timing_s/old_log_prob:3.2687183786183596 - timing_s/adv:0.12963519990444183 - timing_s/update_actor:13.795639680698514 - timing_s/step:39.736858043819666 - timing_s/stop_profile:0.00010845810174942017 - timing_per_token_ms/adv:0.0005171197545323125 - timing_per_token_ms/gen:0.12752206438384034 - timing_per_token_ms/update_actor:0.05503133262075223 - perf/total_num_tokens:250687 - perf/time_per_step:39.736858043819666 - perf/throughput:788.5846174713785
(TaskRunner pid=2224264) Training Progress: 33%|███▎ | 33/100 [30:33<1:09:33, 62.29s/it]
(TaskRunner pid=2224264) step:34 - global_seqlen/min:31672 - global_seqlen/max:34621 - global_seqlen/minmax_diff:2949 - global_seqlen/balanced_min:33347 - global_seqlen/balanced_max:33381 - global_seqlen/mean:33365.5 - actor/entropy:0.7147052884101868 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.26073455810546875 - training/rollout_probs_diff_mean:0.004393857438117266 - training/rollout_probs_diff_std:0.008422989398241043 - training/rollout_actor_probs_pearson_corr:0.9996627569198608 - rollout_corr/rollout_is_mean:1.0000426769256592 - rollout_corr/rollout_is_ratio_fraction_high:1.0432097042212263e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:5.339309215545654 - rollout_corr/rollout_is_min:0.5086534023284912 - rollout_corr/rollout_is_std:0.03433627635240555 - rollout_corr/rollout_is_eff_sample_size:0.99882252741175 - rollout_corr/rollout_is_seq_mean:1.0001329183578491 - rollout_corr/rollout_is_seq_std:0.001656638691201806 - rollout_corr/rollout_is_seq_max:1.0120186805725098 - rollout_corr/rollout_is_seq_min:0.9953494668006897 - rollout_corr/rollout_is_seq_max_deviation:0.012018680572509766 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.108749763574451) - rollout_corr/training_log_ppl:np.float64(0.7289996044710279) - rollout_corr/kl:np.float64(0.0005081289243759057) - rollout_corr/k3_kl:np.float64(0.0006798729952492977) - rollout_corr/rollout_ppl:np.float64(2.107720549684018) - rollout_corr/rollout_log_ppl:np.float64(0.7284914773190394) - rollout_corr/log_ppl_diff:np.float64(0.0005081271519884467) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012719499645754695) - rollout_corr/log_ppl_diff_max:np.float64(0.0048035383224487305) - rollout_corr/log_ppl_diff_min:np.float64(-0.006281495094299316) - rollout_corr/ppl_ratio:np.float64(1.0005095447413623) - rollout_corr/chi2_token:np.float64(0.0017949678003787994) - rollout_corr/chi2_seq:np.float64(4.14957369142212) - actor/pg_loss:np.float64(-5.571101792156696e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00020079502564840368) - actor/ppo_kl:np.float64(8.411469075042532e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3010113313794136) - perf/mfu/actor:np.float64(0.12801561879083337) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(104.21452951431274) - actor/lr:np.float64(1e-06) - training/global_step:34 - training/epoch:0 - critic/score/mean:0.56640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.56640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.007644771598279476 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.007644771598279476 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:748.890625 - response_length/max:2008.0 - response_length/min:165.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:748.890625 - response_length_non_aborted/max:2008.0 - response_length_non_aborted/min:165.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:293.78125 - prompt_length/max:621.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014275126159191132 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1188270319253206) - timing_s/agent_loop/generate_sequences/max:np.float64(16.865083750337362) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.53478015192377) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.865083750337362) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:201 - timing_s/agent_loop/slowest/response_length:2008 - timing_s/gen:23.755706263706088 - timing_s/reward:0.08198409341275692 - timing_s/old_log_prob:3.3104164469987154 - timing_s/adv:0.14668767154216766 - timing_s/update_actor:13.706282814964652 - timing_s/step:41.08737375959754 - timing_s/stop_profile:0.00011901184916496277 - timing_per_token_ms/adv:0.0005495484540249946 - timing_per_token_ms/gen:0.12391092169514327 - timing_per_token_ms/update_actor:0.05134900876266148 - perf/total_num_tokens:266924 - perf/time_per_step:41.08737375959754 - perf/throughput:812.0621238831601
(TaskRunner pid=2224264) Training Progress: 34%|███▍ | 34/100 [31:14<1:01:32, 55.95s/it]
(TaskRunner pid=2224264) step:35 - global_seqlen/min:28121 - global_seqlen/max:32901 - global_seqlen/minmax_diff:4780 - global_seqlen/balanced_min:30268 - global_seqlen/balanced_max:30287 - global_seqlen/mean:30280.375 - actor/entropy:0.7235326170921326 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.21305981278419495 - training/rollout_probs_diff_mean:0.00439543928951025 - training/rollout_probs_diff_std:0.00852300226688385 - training/rollout_actor_probs_pearson_corr:0.9996048808097839 - rollout_corr/rollout_is_mean:1.0000625848770142 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.8470300245971885e-06 - rollout_corr/rollout_is_max:1.7555011510849 - rollout_corr/rollout_is_min:0.4614633619785309 - rollout_corr/rollout_is_std:0.03465937077999115 - rollout_corr/rollout_is_eff_sample_size:0.9988002882356332 - rollout_corr/rollout_is_seq_mean:1.0000625848770142 - rollout_corr/rollout_is_seq_std:0.001464530942030251 - rollout_corr/rollout_is_seq_max:1.0040720701217651 - rollout_corr/rollout_is_seq_min:0.9944659471511841 - rollout_corr/rollout_is_seq_max_deviation:0.005534052848815918 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0981541997753084) - rollout_corr/training_log_ppl:np.float64(0.7273534100968391) - rollout_corr/kl:np.float64(0.0005946585815870264) - rollout_corr/k3_kl:np.float64(0.0006194554455305479) - rollout_corr/rollout_ppl:np.float64(2.0969019741751254) - rollout_corr/rollout_log_ppl:np.float64(0.7267587499227375) - rollout_corr/log_ppl_diff:np.float64(0.0005946601741015911) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012836407404392958) - rollout_corr/log_ppl_diff_max:np.float64(0.005986064672470093) - rollout_corr/log_ppl_diff_min:np.float64(-0.003438234329223633) - rollout_corr/ppl_ratio:np.float64(1.000595951685682) - rollout_corr/chi2_token:np.float64(0.001301520736888051) - rollout_corr/chi2_seq:np.float64(1.3097328948788345) - actor/pg_loss:np.float64(0.00015541678294539452) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001370309196317976) - actor/ppo_kl:np.float64(4.7729094319493015e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2635045759379864) - perf/mfu/actor:np.float64(0.11788960362270617) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.97077560424805) - actor/lr:np.float64(1e-06) - training/global_step:35 - training/epoch:0 - critic/score/mean:0.4921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0020808118861168623 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0020808118861168623 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:668.07421875 - response_length/max:1568.0 - response_length/min:256.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:668.07421875 - response_length_non_aborted/max:1568.0 - response_length_non_aborted/min:256.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.1875 - prompt_length/max:514.0 - prompt_length/min:181.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001488998532295227 - timing_s/agent_loop/generate_sequences/min:np.float64(3.3336482755839825) - timing_s/agent_loop/generate_sequences/max:np.float64(13.82805041782558) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.910963898211776) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(13.82805041782558) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:218 - timing_s/agent_loop/slowest/response_length:1568 - timing_s/gen:20.717208752408624 - timing_s/reward:0.07455052435398102 - timing_s/old_log_prob:3.3136789835989475 - timing_s/adv:0.15318860113620758 - timing_s/update_actor:13.571898808702826 - timing_s/step:37.9156019538641 - timing_s/stop_profile:0.00011204928159713745 - timing_per_token_ms/adv:0.0006323757596141378 - timing_per_token_ms/gen:0.12113414111461128 - timing_per_token_ms/update_actor:0.056025969000973515 - perf/total_num_tokens:242243 - perf/time_per_step:37.9156019538641 - perf/throughput:798.6257223832373
(TaskRunner pid=2224264) Training Progress: 35%|███▌ | 35/100 [31:52<54:46, 50.56s/it]
(TaskRunner pid=2224264) step:36 - global_seqlen/min:23484 - global_seqlen/max:37627 - global_seqlen/minmax_diff:14143 - global_seqlen/balanced_min:32177 - global_seqlen/balanced_max:32196 - global_seqlen/mean:32190.375 - actor/entropy:0.7190540432929993 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9983694553375244 - training/rollout_probs_diff_mean:0.0044546606950461864 - training/rollout_probs_diff_std:0.008799256756901741 - training/rollout_actor_probs_pearson_corr:0.999652624130249 - rollout_corr/rollout_is_mean:0.9999595880508423 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.36610241397284e-06 - rollout_corr/rollout_is_max:1.631447434425354 - rollout_corr/rollout_is_min:0.00012358058302197605 - rollout_corr/rollout_is_std:0.03424588963389397 - rollout_corr/rollout_is_eff_sample_size:0.9988282360231038 - rollout_corr/rollout_is_seq_mean:0.9999497532844543 - rollout_corr/rollout_is_seq_std:0.0013012057170271873 - rollout_corr/rollout_is_seq_max:1.005060076713562 - rollout_corr/rollout_is_seq_min:0.9961336851119995 - rollout_corr/rollout_is_seq_max_deviation:0.005060076713562012 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0887983799912035) - rollout_corr/training_log_ppl:np.float64(0.7234805303742178) - rollout_corr/kl:np.float64(0.0007222782085847257) - rollout_corr/k3_kl:np.float64(0.000665171026241751) - rollout_corr/rollout_ppl:np.float64(2.0872840681113303) - rollout_corr/rollout_log_ppl:np.float64(0.7227582528139465) - rollout_corr/log_ppl_diff:np.float64(0.0007222775602713227) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012800676049664617) - rollout_corr/log_ppl_diff_max:np.float64(0.014325737953186035) - rollout_corr/log_ppl_diff_min:np.float64(-0.004422605037689209) - rollout_corr/ppl_ratio:np.float64(1.0007238842081279) - rollout_corr/chi2_token:np.float64(0.001139611704275012) - rollout_corr/chi2_seq:np.float64(2.33724519261159) - actor/pg_loss:np.float64(7.578812073916197e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00016148608483490534) - actor/ppo_kl:np.float64(3.428134950134165e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3038923889398575) - perf/mfu/actor:np.float64(0.12476615328489614) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.85765075683594) - actor/lr:np.float64(1e-06) - training/global_step:36 - training/epoch:0 - critic/score/mean:0.55859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0017614230746403337 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0017614230746403337 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:727.94921875 - response_length/max:1574.0 - response_length/min:181.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:727.94921875 - response_length_non_aborted/max:1574.0 - response_length_non_aborted/min:181.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.0 - prompt_length/max:704.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010665319859981537 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2873131558299065) - timing_s/agent_loop/generate_sequences/max:np.float64(14.295806081965566) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.435247752218856) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.295806081965566) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:193 - timing_s/agent_loop/slowest/response_length:1574 - timing_s/gen:20.98940148949623 - timing_s/reward:0.07632862776517868 - timing_s/old_log_prob:3.21400635689497 - timing_s/adv:0.11980516463518143 - timing_s/update_actor:13.706334153190255 - timing_s/step:38.190680116415024 - timing_s/stop_profile:0.0001220405101776123 - timing_per_token_ms/adv:0.0004652212215420814 - timing_per_token_ms/gen:0.11263127627107526 - timing_per_token_ms/update_actor:0.05322372818424084 - perf/total_num_tokens:257523 - perf/time_per_step:38.190680116415024 - perf/throughput:842.8856176919461
(TaskRunner pid=2224264) Training Progress: 36%|███▌ | 36/100 [32:30<49:59, 46.86s/it]
(TaskRunner pid=2224264) step:37 - global_seqlen/min:27444 - global_seqlen/max:33696 - global_seqlen/minmax_diff:6252 - global_seqlen/balanced_min:31074 - global_seqlen/balanced_max:31120 - global_seqlen/mean:31108.25 - actor/entropy:0.749553382396698 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1951146274805069 - training/rollout_probs_diff_mean:0.004549640696495771 - training/rollout_probs_diff_std:0.008537894114851952 - training/rollout_actor_probs_pearson_corr:0.9996463060379028 - rollout_corr/rollout_is_mean:0.9998870491981506 - rollout_corr/rollout_is_ratio_fraction_high:5.924381184740923e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.0695595741271973 - rollout_corr/rollout_is_min:0.5410090088844299 - rollout_corr/rollout_is_std:0.035030562430620193 - rollout_corr/rollout_is_eff_sample_size:0.9987741257668857 - rollout_corr/rollout_is_seq_mean:0.9999449849128723 - rollout_corr/rollout_is_seq_std:0.001592667424120009 - rollout_corr/rollout_is_seq_max:1.0082616806030273 - rollout_corr/rollout_is_seq_min:0.9951016306877136 - rollout_corr/rollout_is_seq_max_deviation:0.008261680603027344 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2040511020459235) - rollout_corr/training_log_ppl:np.float64(0.7697590510360897) - rollout_corr/kl:np.float64(0.0007292848855877665) - rollout_corr/k3_kl:np.float64(0.0006632836551716537) - rollout_corr/rollout_ppl:np.float64(2.202401583082974) - rollout_corr/rollout_log_ppl:np.float64(0.769029765157029) - rollout_corr/log_ppl_diff:np.float64(0.000729285879060626) - rollout_corr/log_ppl_abs_diff:np.float64(0.001381295034661889) - rollout_corr/log_ppl_diff_max:np.float64(0.005849003791809082) - rollout_corr/log_ppl_diff_min:np.float64(-0.005055129528045654) - rollout_corr/ppl_ratio:np.float64(1.0007309019565582) - rollout_corr/chi2_token:np.float64(0.001207549823448062) - rollout_corr/chi2_seq:np.float64(0.7918412457220256) - actor/pg_loss:np.float64(-0.0002318680053576827) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019654674611047085) - actor/ppo_kl:np.float64(3.777838127683175e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3199927434325218) - perf/mfu/actor:np.float64(0.11945996157211544) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.98107528686523) - actor/lr:np.float64(1e-06) - training/global_step:37 - training/epoch:0 - critic/score/mean:0.46484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.011995390988886356 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.011995390988886356 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:659.3515625 - response_length/max:2064.0 - response_length/min:147.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:659.3515625 - response_length_non_aborted/max:2064.0 - response_length_non_aborted/min:147.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.78125 - prompt_length/max:1009.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015878677368164062 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0059274714440107) - timing_s/agent_loop/generate_sequences/max:np.float64(16.902138613164425) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.578634077712195) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.902138613164425) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:171 - timing_s/agent_loop/slowest/response_length:2064 - timing_s/gen:23.681104212999344 - timing_s/reward:0.08328941836953163 - timing_s/old_log_prob:3.3095543030649424 - timing_s/adv:0.13988415524363518 - timing_s/update_actor:13.724779484793544 - timing_s/step:41.0210815500468 - timing_s/stop_profile:0.0001230333000421524 - timing_per_token_ms/adv:0.0005620862441781327 - timing_per_token_ms/gen:0.14029588855646138 - timing_per_token_ms/update_actor:0.05514927505080462 - perf/total_num_tokens:248866 - perf/time_per_step:41.0210815500468 - perf/throughput:758.3478744227432
(TaskRunner pid=2224264) Training Progress: 37%|███▋ | 37/100 [33:11<47:22, 45.12s/it]
(TaskRunner pid=2224264) step:38 - global_seqlen/min:22559 - global_seqlen/max:30637 - global_seqlen/minmax_diff:8078 - global_seqlen/balanced_min:27684 - global_seqlen/balanced_max:27696 - global_seqlen/mean:27690.75 - actor/entropy:0.7535861730575562 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.36917543411254883 - training/rollout_probs_diff_mean:0.004689306020736694 - training/rollout_probs_diff_std:0.008696562610566616 - training/rollout_actor_probs_pearson_corr:0.9996323585510254 - rollout_corr/rollout_is_mean:1.0000420808792114 - rollout_corr/rollout_is_ratio_fraction_high:6.4868513618421275e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.0322699546813965 - rollout_corr/rollout_is_min:0.5430662631988525 - rollout_corr/rollout_is_std:0.03519861027598381 - rollout_corr/rollout_is_eff_sample_size:0.9987627098464102 - rollout_corr/rollout_is_seq_mean:1.0000241994857788 - rollout_corr/rollout_is_seq_std:0.0014864748809486628 - rollout_corr/rollout_is_seq_max:1.0049471855163574 - rollout_corr/rollout_is_seq_min:0.995693564414978 - rollout_corr/rollout_is_seq_max_deviation:0.004947185516357422 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1352647584863007) - rollout_corr/training_log_ppl:np.float64(0.7448823321610689) - rollout_corr/kl:np.float64(0.0005967357578882115) - rollout_corr/k3_kl:np.float64(0.0006872201306578063) - rollout_corr/rollout_ppl:np.float64(2.1338966507464647) - rollout_corr/rollout_log_ppl:np.float64(0.7442855968838558) - rollout_corr/log_ppl_diff:np.float64(0.0005967352772131562) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013833154225721955) - rollout_corr/log_ppl_diff_max:np.float64(0.0049326419830322266) - rollout_corr/log_ppl_diff_min:np.float64(-0.004489302635192871) - rollout_corr/ppl_ratio:np.float64(1.0005982399452478) - rollout_corr/chi2_token:np.float64(0.0015692522283643484) - rollout_corr/chi2_seq:np.float64(1.3063512451481074) - actor/pg_loss:np.float64(-0.00012253155000507832) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00015133037595660426) - actor/ppo_kl:np.float64(-1.1263918693771302e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.27262789383530617) - perf/mfu/actor:np.float64(0.10989664398239996) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.94292449951172) - actor/lr:np.float64(1e-06) - training/global_step:38 - training/epoch:0 - critic/score/mean:0.58203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0031607181299477816 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0031607181299477816 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:602.1796875 - response_length/max:1274.0 - response_length/min:154.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:602.1796875 - response_length_non_aborted/max:1274.0 - response_length_non_aborted/min:154.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.15625 - prompt_length/max:500.0 - prompt_length/min:164.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.338022649288177e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9933321122080088) - timing_s/agent_loop/generate_sequences/max:np.float64(11.328456040471792) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.766713866491045) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.328456040471792) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:221 - timing_s/agent_loop/slowest/response_length:1274 - timing_s/gen:17.93396664969623 - timing_s/reward:0.07525189034640789 - timing_s/old_log_prob:3.114859651774168 - timing_s/adv:0.12139961495995522 - timing_s/update_actor:12.985273517668247 - timing_s/step:34.3180456738919 - timing_s/stop_profile:0.00011270120739936829 - timing_per_token_ms/adv:0.0005480151989380714 - timing_per_token_ms/gen:0.11633497223430656 - timing_per_token_ms/update_actor:0.05861737907815898 - perf/total_num_tokens:221526 - perf/time_per_step:34.3180456738919 - perf/throughput:806.8859824691666
(TaskRunner pid=2224264) Training Progress: 38%|███▊ | 38/100 [33:45<43:17, 41.90s/it]
(TaskRunner pid=2224264) step:39 - global_seqlen/min:24673 - global_seqlen/max:36395 - global_seqlen/minmax_diff:11722 - global_seqlen/balanced_min:29976 - global_seqlen/balanced_max:30014 - global_seqlen/mean:30001.0 - actor/entropy:0.7436038255691528 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.39769014716148376 - training/rollout_probs_diff_mean:0.004428871441632509 - training/rollout_probs_diff_std:0.008462125435471535 - training/rollout_actor_probs_pearson_corr:0.9996308088302612 - rollout_corr/rollout_is_mean:0.9998753666877747 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.558502197265625 - rollout_corr/rollout_is_min:0.5394280552864075 - rollout_corr/rollout_is_std:0.03459912911057472 - rollout_corr/rollout_is_eff_sample_size:0.9988039748754949 - rollout_corr/rollout_is_seq_mean:0.9998376965522766 - rollout_corr/rollout_is_seq_std:0.0016168290749192238 - rollout_corr/rollout_is_seq_max:1.0040711164474487 - rollout_corr/rollout_is_seq_min:0.9940407276153564 - rollout_corr/rollout_is_seq_max_deviation:0.005959272384643555 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.18053255463019) - rollout_corr/training_log_ppl:np.float64(0.7624029302969575) - rollout_corr/kl:np.float64(0.0008376638515699142) - rollout_corr/k3_kl:np.float64(0.0006715801446830483) - rollout_corr/rollout_ppl:np.float64(2.178553344681859) - rollout_corr/rollout_log_ppl:np.float64(0.7615652666427195) - rollout_corr/log_ppl_diff:np.float64(0.0008376636542379856) - rollout_corr/log_ppl_abs_diff:np.float64(0.001551654888316989) - rollout_corr/log_ppl_diff_max:np.float64(0.00657200813293457) - rollout_corr/log_ppl_diff_min:np.float64(-0.004390835762023926) - rollout_corr/ppl_ratio:np.float64(1.0008396471384913) - rollout_corr/chi2_token:np.float64(0.0010091052390635014) - rollout_corr/chi2_seq:np.float64(2.287320557516068) - actor/pg_loss:np.float64(-9.888317435979843e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001992107047499303) - actor/ppo_kl:np.float64(5.7301584355684554e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.29527168720960617) - perf/mfu/actor:np.float64(0.11556263551770317) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.95692443847656) - actor/lr:np.float64(1e-06) - training/global_step:39 - training/epoch:0 - critic/score/mean:0.47265625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.47265625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002959122182801366 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002959122182801366 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:655.25 - response_length/max:1746.0 - response_length/min:175.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:655.25 - response_length_non_aborted/max:1746.0 - response_length_non_aborted/min:175.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:282.28125 - prompt_length/max:452.0 - prompt_length/min:181.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001086648553609848 - timing_s/agent_loop/generate_sequences/min:np.float64(2.099978694692254) - timing_s/agent_loop/generate_sequences/max:np.float64(14.446734165772796) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.490971767700103) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.446734165772796) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:296 - timing_s/agent_loop/slowest/response_length:1746 - timing_s/gen:21.59979004599154 - timing_s/reward:0.07381308265030384 - timing_s/old_log_prob:3.1907936204224825 - timing_s/adv:0.1207788810133934 - timing_s/update_actor:13.672080148011446 - timing_s/step:38.74228004924953 - timing_s/stop_profile:0.00011139921844005585 - timing_per_token_ms/adv:0.0005032285632703635 - timing_per_token_ms/gen:0.12876639430317352 - timing_per_token_ms/update_actor:0.056965101779988356 - perf/total_num_tokens:240008 - perf/time_per_step:38.74228004924953 - perf/throughput:774.3736290652606
(TaskRunner pid=2224264) Training Progress: 39%|███▉ | 39/100 [34:24<41:38, 40.96s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 40}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) We are given a target product:
(TaskRunner pid=2224264) **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**
(TaskRunner pid=2224264) This is a nitrogen-containing compound with a specific substituent pattern. Let's break down the structure:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - The core is a **pyrrole ring** (N connected to a carbon ring).
(TaskRunner pid=2224264) - The nitrogen is attached to a **carbon chain** that leads to:
(TaskRunner pid=2224264) - A **chloro-substituted benzene ring** (from "Cc2ccc(Cl)cc2C(F)(F)F")
(TaskRunner pid=2224264) - A **trifluoromethyl group** (C(F)(F)F)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) So the structure is:
(TaskRunner pid=2224264) **N–C–(–CH₂–C(Cl)–C(F)(F)F–)**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Now, we are to identify the **correct reactant** that could be used in the synthesis of this compound. Let's evaluate each option.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option A:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This appears to be a **pyrrole derivative** with a **carbonyl group** (O=C) and a **trifluoromethyl group**. The structure seems to match the target product. The **"c1ccn(...)"** part resembles the pyrrole ring.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This is a **strong candidate** for the correct reactant.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option B:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This has a **chlorine atom** on the ring and a **trifluoromethyl** group? Not directly. The structure is more complex and does not clearly match the target.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option C:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This includes a **trifluoromethyl group** and a **chloro-substituted benzene**, but the **N** is not directly attached to the **chloro-substituted benzene**. The structure is more complex than the target.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option D:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This has a **trifluoromethyl group** and a **benzene ring**, but the **N** is not directly attached to the **chloro-substituted benzene**. The structure is also more complex than the target.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Final Evaluation:
(TaskRunner pid=2224264) - **Option A** contains a **pyrrole ring** with a **chloro-substituted benzene** and a **trifluoromethyl group**, which matches the structure of the target product.
(TaskRunner pid=2224264) - The other options either lack the correct substituent pattern or have more complex structures that do not align with the target.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### ✅ **Correct Answer: A**
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_40
(TaskRunner pid=2224264) step:40 - global_seqlen/min:26780 - global_seqlen/max:38560 - global_seqlen/minmax_diff:11780 - global_seqlen/balanced_min:31398 - global_seqlen/balanced_max:31614 - global_seqlen/mean:31430.375 - actor/entropy:0.7148814797401428 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.353511780500412 - training/rollout_probs_diff_mean:0.004250292666256428 - training/rollout_probs_diff_std:0.008334573358297348 - training/rollout_actor_probs_pearson_corr:0.9996918439865112 - rollout_corr/rollout_is_mean:0.9999884963035583 - rollout_corr/rollout_is_ratio_fraction_high:5.834475814481266e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.834475814481266e-06 - rollout_corr/rollout_is_max:2.314491033554077 - rollout_corr/rollout_is_min:0.4158317446708679 - rollout_corr/rollout_is_std:0.034074895083904266 - rollout_corr/rollout_is_eff_sample_size:0.9988401291730077 - rollout_corr/rollout_is_seq_mean:0.9999082088470459 - rollout_corr/rollout_is_seq_std:0.0015310312155634165 - rollout_corr/rollout_is_seq_max:1.0046111345291138 - rollout_corr/rollout_is_seq_min:0.9931868314743042 - rollout_corr/rollout_is_seq_max_deviation:0.006813168525695801 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0881334473378956) - rollout_corr/training_log_ppl:np.float64(0.7168666583602317) - rollout_corr/kl:np.float64(0.0009168515699862212) - rollout_corr/k3_kl:np.float64(0.0007126086813400434) - rollout_corr/rollout_ppl:np.float64(2.0860950271598995) - rollout_corr/rollout_log_ppl:np.float64(0.7159498034743592) - rollout_corr/log_ppl_diff:np.float64(0.0009168548858724535) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013491588761098683) - rollout_corr/log_ppl_diff_max:np.float64(0.006253480911254883) - rollout_corr/log_ppl_diff_min:np.float64(-0.003960609436035156) - rollout_corr/ppl_ratio:np.float64(1.0009184894151986) - rollout_corr/chi2_token:np.float64(0.0010311168152838945) - rollout_corr/chi2_seq:np.float64(0.5281695865560323) - actor/pg_loss:np.float64(0.0002642440376803279) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00030195120598364156) - actor/ppo_kl:np.float64(0.00023282111740385858) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.26611069589853287) - perf/mfu/actor:np.float64(0.12153132959867792) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(103.97843170166016) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6053571428571428) - val-aux/sciknoweval/reward/std@16:np.float64(0.28468084712859354) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7211619047619049) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.22140257031009367) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.4908571428571428) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.25871458688057813) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6054666666666666) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.28424403856079056) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8044809523809524) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.15305264517858658) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.3869142857142857) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.2100138722335839) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6357761904761906) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.22516406707067232) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8634285714285714) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.09899214217926788) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.3012904761904762) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.15568733164559595) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6521095238095237) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.15983092148992065) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.9010238095238096) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.056177528972713935) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.23781428571428573) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.09852453096057993) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6605380952380953) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.11372781200461213) - val-aux/sciknoweval/score/mean@16:np.float64(0.6053571428571428) - val-aux/sciknoweval/score/std@16:np.float64(0.28468084712859354) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7211619047619049) - val-aux/sciknoweval/score/best@2/std:np.float64(0.22140257031009367) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.4908571428571428) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.25871458688057813) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6054666666666666) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.28424403856079056) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8044809523809524) - val-aux/sciknoweval/score/best@4/std:np.float64(0.15305264517858658) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.3869142857142857) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.2100138722335839) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6357761904761906) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.22516406707067232) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8634285714285714) - val-aux/sciknoweval/score/best@8/std:np.float64(0.09899214217926788) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.3012904761904762) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.15568733164559595) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6521095238095237) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.15983092148992065) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.9010238095238096) - val-aux/sciknoweval/score/best@16/std:np.float64(0.056177528972713935) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.23781428571428573) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.09852453096057993) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6605380952380953) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.11372781200461213) - val-core/sciknoweval/acc/mean@16:np.float64(0.6053571428571428) - val-aux/sciknoweval/acc/std@16:np.float64(0.28468084712859354) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7211619047619049) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.22140257031009367) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.4908571428571428) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.25871458688057813) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6054666666666666) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.28424403856079056) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8044809523809524) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.15305264517858658) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.3869142857142857) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.2100138722335839) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6357761904761906) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.22516406707067232) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8634285714285714) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.09899214217926788) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.3012904761904762) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.15568733164559595) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6521095238095237) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.15983092148992065) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.9010238095238096) - val-core/sciknoweval/acc/best@16/std:np.float64(0.056177528972713935) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.23781428571428573) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.09852453096057993) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6605380952380953) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.11372781200461213) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.986904761904762) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.036439037613581124) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9981857142857142) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0128589514254664) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9762047619047618) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.04654134805312809) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9871476190476189) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.036109563011539794) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9999333333333333) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0014234611985593974) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9578047619047619) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.05537427217213782) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9950380952380953) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.022826156373882658) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9323095238095238) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.05709074252726499) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9987190476190476) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.010923127179237949) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9055857142857143) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.04723161312584812) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9998666666666666) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.002542800942671283) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:40 - training/epoch:0 - critic/score/mean:0.55859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0004572770558297634 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0004572770558297634 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:669.51171875 - response_length/max:1961.0 - response_length/min:161.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:669.51171875 - response_length_non_aborted/max:1961.0 - response_length_non_aborted/min:161.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.6875 - prompt_length/max:501.0 - prompt_length/min:159.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011765025556087494 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0561186894774437) - timing_s/agent_loop/generate_sequences/max:np.float64(16.069326899945736) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.7381278641551035) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.069326899945736) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:423 - timing_s/agent_loop/slowest/response_length:1961 - timing_s/gen:22.699881782755256 - timing_s/reward:0.07818994671106339 - timing_s/old_log_prob:3.2518085297197104 - timing_s/adv:0.13994908705353737 - timing_s/update_actor:13.636622594669461 - timing_s/step:39.892101377248764 - timing_s/testing:188.02605186402798 - timing_s/save_checkpoint:13.08927645906806 - timing_s/stop_profile:0.00019207783043384552 - timing_per_token_ms/adv:0.0005565837468274613 - timing_per_token_ms/gen:0.1324419136074871 - timing_per_token_ms/update_actor:0.054233454877127066 - perf/total_num_tokens:251443 - perf/time_per_step:39.892101377248764 - perf/throughput:787.8846667607576
(TaskRunner pid=2224264) Training Progress: 40%|████ | 40/100 [38:25<1:41:00, 101.00s/it]
(TaskRunner pid=2224264) step:41 - global_seqlen/min:25184 - global_seqlen/max:36473 - global_seqlen/minmax_diff:11289 - global_seqlen/balanced_min:30315 - global_seqlen/balanced_max:30362 - global_seqlen/mean:30349.125 - actor/entropy:0.7456910610198975 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1853349208831787 - training/rollout_probs_diff_mean:0.004585789050906897 - training/rollout_probs_diff_std:0.008587997406721115 - training/rollout_actor_probs_pearson_corr:0.9996112585067749 - rollout_corr/rollout_is_mean:1.000074863433838 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1413375432312023e-05 - rollout_corr/rollout_is_max:1.7971372604370117 - rollout_corr/rollout_is_min:0.38024628162384033 - rollout_corr/rollout_is_std:0.0351342037320137 - rollout_corr/rollout_is_eff_sample_size:0.9987673475326293 - rollout_corr/rollout_is_seq_mean:1.000067949295044 - rollout_corr/rollout_is_seq_std:0.0015586130321025848 - rollout_corr/rollout_is_seq_max:1.005883812904358 - rollout_corr/rollout_is_seq_min:0.993992805480957 - rollout_corr/rollout_is_seq_max_deviation:0.006007194519042969 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1489445390179753) - rollout_corr/training_log_ppl:np.float64(0.7491948831593618) - rollout_corr/kl:np.float64(0.0007140677866042111) - rollout_corr/k3_kl:np.float64(0.000693655981081065) - rollout_corr/rollout_ppl:np.float64(2.147350816987455) - rollout_corr/rollout_log_ppl:np.float64(0.7484808203298599) - rollout_corr/log_ppl_diff:np.float64(0.0007140628295019269) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013865955406799912) - rollout_corr/log_ppl_diff_max:np.float64(0.00584721565246582) - rollout_corr/log_ppl_diff_min:np.float64(-0.004841506481170654) - rollout_corr/ppl_ratio:np.float64(1.0007157281506807) - rollout_corr/chi2_token:np.float64(0.0013592832256108522) - rollout_corr/chi2_seq:np.float64(1.1909501447807997) - actor/pg_loss:np.float64(0.000211457721889019) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024292130228786846) - actor/ppo_kl:np.float64(0.00014254798117097067) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2715475298464298) - perf/mfu/actor:np.float64(0.11821237532599987) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(105.00201988220215) - actor/lr:np.float64(1e-06) - training/global_step:41 - training/epoch:0 - critic/score/mean:0.5625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00029318107408471406 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00029318107408471406 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:684.50390625 - response_length/max:1922.0 - response_length/min:186.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:684.50390625 - response_length_non_aborted/max:1922.0 - response_length_non_aborted/min:186.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.90625 - prompt_length/max:631.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.857250213623047e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0997980404645205) - timing_s/agent_loop/generate_sequences/max:np.float64(15.434562433511019) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.771742777309555) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.434562433511019) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:245 - timing_s/agent_loop/slowest/response_length:1922 - timing_s/gen:22.271852983161807 - timing_s/reward:0.07384450361132622 - timing_s/old_log_prob:3.3064878918230534 - timing_s/adv:0.13560799323022366 - timing_s/update_actor:13.53317659534514 - timing_s/step:39.414824433624744 - timing_s/stop_profile:0.00011513754725456238 - timing_per_token_ms/adv:0.0005585333729976715 - timing_per_token_ms/gen:0.12709850874642223 - timing_per_token_ms/update_actor:0.055739566607542805 - perf/total_num_tokens:242793 - perf/time_per_step:39.414824433624744 - perf/throughput:769.9926470840549
(TaskRunner pid=2224264) Training Progress: 41%|████ | 41/100 [39:05<1:21:09, 82.54s/it]
(TaskRunner pid=2224264) step:42 - global_seqlen/min:23559 - global_seqlen/max:32248 - global_seqlen/minmax_diff:8689 - global_seqlen/balanced_min:29140 - global_seqlen/balanced_max:29158 - global_seqlen/mean:29150.625 - actor/entropy:0.7418811321258545 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8146259784698486 - training/rollout_probs_diff_mean:0.004636453464627266 - training/rollout_probs_diff_std:0.008857984095811844 - training/rollout_actor_probs_pearson_corr:0.9996300339698792 - rollout_corr/rollout_is_mean:0.9999567270278931 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.9698286349885166e-06 - rollout_corr/rollout_is_max:1.5637112855911255 - rollout_corr/rollout_is_min:0.1537874937057495 - rollout_corr/rollout_is_std:0.03467312455177307 - rollout_corr/rollout_is_eff_sample_size:0.9987990990027735 - rollout_corr/rollout_is_seq_mean:0.9999449849128723 - rollout_corr/rollout_is_seq_std:0.0014816136099398136 - rollout_corr/rollout_is_seq_max:1.006678581237793 - rollout_corr/rollout_is_seq_min:0.9932494759559631 - rollout_corr/rollout_is_seq_max_deviation:0.006750524044036865 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.149658269248903) - rollout_corr/training_log_ppl:np.float64(0.750017037265934) - rollout_corr/kl:np.float64(0.0007465042794017052) - rollout_corr/k3_kl:np.float64(0.0006643551571414719) - rollout_corr/rollout_ppl:np.float64(2.148035450838506) - rollout_corr/rollout_log_ppl:np.float64(0.7492705312324688) - rollout_corr/log_ppl_diff:np.float64(0.0007465060334652662) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012925199698656797) - rollout_corr/log_ppl_diff_max:np.float64(0.005987584590911865) - rollout_corr/log_ppl_diff_min:np.float64(-0.00581049919128418) - rollout_corr/ppl_ratio:np.float64(1.0007479563355446) - rollout_corr/chi2_token:np.float64(0.0011864088010042906) - rollout_corr/chi2_seq:np.float64(0.5256321090273559) - actor/pg_loss:np.float64(-3.93550144508481e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024687639711373777) - actor/ppo_kl:np.float64(6.906396298589357e-05) - actor/pg_clipfrac_lower:np.float64(8.510348379786592e-06) - actor/grad_norm:np.float64(0.31687242165207863) - perf/mfu/actor:np.float64(0.11438361389153799) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(104.63388013839722) - actor/lr:np.float64(1e-06) - training/global_step:42 - training/epoch:0 - critic/score/mean:0.55859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007985391654074192 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007985391654074192 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:654.33203125 - response_length/max:1379.0 - response_length/min:161.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:654.33203125 - response_length_non_aborted/max:1379.0 - response_length_non_aborted/min:161.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:256.625 - prompt_length/max:557.0 - prompt_length/min:175.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015127286314964294 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8712534438818693) - timing_s/agent_loop/generate_sequences/max:np.float64(12.603244848549366) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.466689185537689) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.603244848549366) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:1379 - timing_s/gen:20.526346927508712 - timing_s/reward:0.07452544383704662 - timing_s/old_log_prob:3.2816093750298023 - timing_s/adv:0.14070634730160236 - timing_s/update_actor:13.285876266658306 - timing_s/step:37.395900616422296 - timing_s/stop_profile:0.00011482276022434235 - timing_per_token_ms/adv:0.0006033590501987623 - timing_per_token_ms/gen:0.12253877061834714 - timing_per_token_ms/update_actor:0.056970803656260824 - perf/total_num_tokens:233205 - perf/time_per_step:37.395900616422296 - perf/throughput:779.5139178222811
(TaskRunner pid=2224264) Training Progress: 42%|████▏ | 42/100 [39:42<1:06:43, 69.02s/it]
(TaskRunner pid=2224264) step:43 - global_seqlen/min:28157 - global_seqlen/max:35791 - global_seqlen/minmax_diff:7634 - global_seqlen/balanced_min:32488 - global_seqlen/balanced_max:32514 - global_seqlen/mean:32503.125 - actor/entropy:0.7455630302429199 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4102365970611572 - training/rollout_probs_diff_mean:0.0044556111097335815 - training/rollout_probs_diff_std:0.008428295142948627 - training/rollout_actor_probs_pearson_corr:0.9996055960655212 - rollout_corr/rollout_is_mean:1.0000288486480713 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.1921050574746914e-05 - rollout_corr/rollout_is_max:1.6338140964508057 - rollout_corr/rollout_is_min:0.3978634774684906 - rollout_corr/rollout_is_std:0.03456293046474457 - rollout_corr/rollout_is_eff_sample_size:0.9988067101420113 - rollout_corr/rollout_is_seq_mean:1.000025749206543 - rollout_corr/rollout_is_seq_std:0.0014121269341558218 - rollout_corr/rollout_is_seq_max:1.0047554969787598 - rollout_corr/rollout_is_seq_min:0.9933763146400452 - rollout_corr/rollout_is_seq_max_deviation:0.006623685359954834 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1632221047766507) - rollout_corr/training_log_ppl:np.float64(0.7551403474062681) - rollout_corr/kl:np.float64(0.0007046414169096238) - rollout_corr/k3_kl:np.float64(0.0006673236663345961) - rollout_corr/rollout_ppl:np.float64(2.16164415422827) - rollout_corr/rollout_log_ppl:np.float64(0.7544357032165863) - rollout_corr/log_ppl_diff:np.float64(0.0007046441896818578) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013581206439994276) - rollout_corr/log_ppl_diff_max:np.float64(0.013478577136993408) - rollout_corr/log_ppl_diff_min:np.float64(-0.0035314559936523438) - rollout_corr/ppl_ratio:np.float64(1.0007065150421113) - rollout_corr/chi2_token:np.float64(0.0012414606753736734) - rollout_corr/chi2_seq:np.float64(1.941952116554603) - actor/pg_loss:np.float64(6.042851600795984e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00017739249074111285) - actor/ppo_kl:np.float64(0.00010932787842676817) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.27033093199133873) - perf/mfu/actor:np.float64(0.12426556577467283) - perf/max_memory_allocated_gb:np.float64(131.90583896636963) - perf/max_memory_reserved_gb:np.float64(144.6015625) - perf/cpu_memory_used_gb:np.float64(104.3560562133789) - actor/lr:np.float64(1e-06) - training/global_step:43 - training/epoch:0 - critic/score/mean:0.5859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005291193723678589 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005291193723678589 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:712.78515625 - response_length/max:1722.0 - response_length/min:175.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:712.78515625 - response_length_non_aborted/max:1722.0 - response_length_non_aborted/min:175.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.9375 - prompt_length/max:813.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010757148265838623 - timing_s/agent_loop/generate_sequences/min:np.float64(2.216030417010188) - timing_s/agent_loop/generate_sequences/max:np.float64(14.333171149715781) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.09952915172471) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.333171149715781) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:211 - timing_s/agent_loop/slowest/response_length:1722 - timing_s/gen:21.01793995499611 - timing_s/reward:0.07846580259501934 - timing_s/old_log_prob:3.292416427284479 - timing_s/adv:0.14018669165670872 - timing_s/update_actor:13.660967510193586 - timing_s/step:38.27972553297877 - timing_s/stop_profile:0.00010936520993709564 - timing_per_token_ms/adv:0.0005391277440888712 - timing_per_token_ms/gen:0.11518383517011344 - timing_per_token_ms/update_actor:0.05253713108429415 - perf/total_num_tokens:260025 - perf/time_per_step:38.27972553297877 - perf/throughput:849.0950378418437
(TaskRunner pid=2224264) Training Progress: 43%|████▎ | 43/100 [40:21<56:49, 59.81s/it]
(TaskRunner pid=2224264) step:44 - global_seqlen/min:28261 - global_seqlen/max:39735 - global_seqlen/minmax_diff:11474 - global_seqlen/balanced_min:33071 - global_seqlen/balanced_max:39250 - global_seqlen/mean:33860.375 - actor/entropy:0.7164559364318848 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3860778212547302 - training/rollout_probs_diff_mean:0.0043455152772367 - training/rollout_probs_diff_std:0.008481818251311779 - training/rollout_actor_probs_pearson_corr:0.9996494650840759 - rollout_corr/rollout_is_mean:1.0000731945037842 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.569669802847784e-05 - rollout_corr/rollout_is_max:1.6253478527069092 - rollout_corr/rollout_is_min:0.3264278769493103 - rollout_corr/rollout_is_std:0.03381501883268356 - rollout_corr/rollout_is_eff_sample_size:0.9988579694289216 - rollout_corr/rollout_is_seq_mean:1.0000576972961426 - rollout_corr/rollout_is_seq_std:0.0014711449621245265 - rollout_corr/rollout_is_seq_max:1.0052176713943481 - rollout_corr/rollout_is_seq_min:0.9947458505630493 - rollout_corr/rollout_is_seq_max_deviation:0.005254149436950684 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1722452212125063) - rollout_corr/training_log_ppl:np.float64(0.7589113183676091) - rollout_corr/kl:np.float64(0.0006052850321509595) - rollout_corr/k3_kl:np.float64(0.0006508081007154942) - rollout_corr/rollout_ppl:np.float64(2.170979785732925) - rollout_corr/rollout_log_ppl:np.float64(0.7583060346714774) - rollout_corr/log_ppl_diff:np.float64(0.0006052836961316643) - rollout_corr/log_ppl_abs_diff:np.float64(0.001306259231569129) - rollout_corr/log_ppl_diff_max:np.float64(0.005559146404266357) - rollout_corr/log_ppl_diff_min:np.float64(-0.00449979305267334) - rollout_corr/ppl_ratio:np.float64(1.0006066909991205) - rollout_corr/chi2_token:np.float64(0.001387621508911252) - rollout_corr/chi2_seq:np.float64(1.5838370565325022) - actor/pg_loss:np.float64(8.792639710009098e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00022318573451229895) - actor/ppo_kl:np.float64(5.0495547061046864e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.303693313151598) - perf/mfu/actor:np.float64(0.121345258264149) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.35682678222656) - actor/lr:np.float64(1e-06) - training/global_step:44 - training/epoch:0 - critic/score/mean:0.60546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.60546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.03973030298948288 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.03973030298948288 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:746.57421875 - response_length/max:8192.0 - response_length/min:178.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:746.57421875 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:178.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:311.5625 - prompt_length/max:701.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.71539455652237e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3765512369573116) - timing_s/agent_loop/generate_sequences/max:np.float64(61.594920651987195) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.441452034945542) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(61.594920651987195) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:384 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:68.32033305615187 - timing_s/reward:0.07946610450744629 - timing_s/old_log_prob:3.648138355463743 - timing_s/adv:0.15895853005349636 - timing_s/update_actor:14.574448632076383 - timing_s/step:86.86667879670858 - timing_s/stop_profile:0.00011816807091236115 - timing_per_token_ms/adv:0.0005868161902131044 - timing_per_token_ms/gen:0.35746787700146954 - timing_per_token_ms/update_actor:0.0538034820644942 - perf/total_num_tokens:270883 - perf/time_per_step:86.86667879670858 - perf/throughput:389.7970484084282
(TaskRunner pid=2224264) Training Progress: 44%|████▍ | 44/100 [41:47<1:03:24, 67.94s/it]
(TaskRunner pid=2224264) step:45 - global_seqlen/min:30046 - global_seqlen/max:36319 - global_seqlen/minmax_diff:6273 - global_seqlen/balanced_min:33079 - global_seqlen/balanced_max:33119 - global_seqlen/mean:33104.0 - actor/entropy:0.7044548392295837 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.456476628780365 - training/rollout_probs_diff_mean:0.004338444676250219 - training/rollout_probs_diff_std:0.008459679782390594 - training/rollout_actor_probs_pearson_corr:0.999611496925354 - rollout_corr/rollout_is_mean:1.0000234842300415 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.426760253612883e-06 - rollout_corr/rollout_is_max:1.7352091073989868 - rollout_corr/rollout_is_min:0.412738174200058 - rollout_corr/rollout_is_std:0.03374973684549332 - rollout_corr/rollout_is_eff_sample_size:0.9988624890615934 - rollout_corr/rollout_is_seq_mean:0.9999771118164062 - rollout_corr/rollout_is_seq_std:0.0013954972382634878 - rollout_corr/rollout_is_seq_max:1.0044217109680176 - rollout_corr/rollout_is_seq_min:0.9954630732536316 - rollout_corr/rollout_is_seq_max_deviation:0.004536926746368408 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1041139638982713) - rollout_corr/training_log_ppl:np.float64(0.7206308286986314) - rollout_corr/kl:np.float64(0.0006204138266534542) - rollout_corr/k3_kl:np.float64(0.0006203657341075086) - rollout_corr/rollout_ppl:np.float64(2.102761774789542) - rollout_corr/rollout_log_ppl:np.float64(0.7200104069779627) - rollout_corr/log_ppl_diff:np.float64(0.0006204217206686735) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011971264611929655) - rollout_corr/log_ppl_diff_max:np.float64(0.005265474319458008) - rollout_corr/log_ppl_diff_min:np.float64(-0.0036600232124328613) - rollout_corr/ppl_ratio:np.float64(1.0006216508336365) - rollout_corr/chi2_token:np.float64(0.0012450276408344507) - rollout_corr/chi2_seq:np.float64(1.1021782979369164) - actor/pg_loss:np.float64(-7.425120566040277e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00015113081622075697) - actor/ppo_kl:np.float64(1.2432504128412347e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.29480820521712303) - perf/mfu/actor:np.float64(0.1272612688514589) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.27553939819336) - actor/lr:np.float64(1e-06) - training/global_step:45 - training/epoch:0 - critic/score/mean:0.58984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005770002957433462 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005770002957433462 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:719.8125 - response_length/max:1962.0 - response_length/min:191.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:719.8125 - response_length_non_aborted/max:1962.0 - response_length_non_aborted/min:191.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:314.6875 - prompt_length/max:845.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.384138345718384e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.28562774322927) - timing_s/agent_loop/generate_sequences/max:np.float64(16.55846000276506) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.204205383684894) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.55846000276506) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:201 - timing_s/agent_loop/slowest/response_length:1962 - timing_s/gen:23.205180127173662 - timing_s/reward:0.0775289349257946 - timing_s/old_log_prob:3.335056768730283 - timing_s/adv:0.130776172503829 - timing_s/update_actor:13.470531040802598 - timing_s/step:40.30686186812818 - timing_s/stop_profile:0.00012123957276344299 - timing_per_token_ms/adv:0.0004938080462475418 - timing_per_token_ms/gen:0.1259289535424463 - timing_per_token_ms/update_actor:0.05086443874155162 - perf/total_num_tokens:264832 - perf/time_per_step:40.30686186812818 - perf/throughput:821.2993635750221
(TaskRunner pid=2224264) Training Progress: 45%|████▌ | 45/100 [42:28<54:41, 59.66s/it]
(TaskRunner pid=2224264) step:46 - global_seqlen/min:25951 - global_seqlen/max:37896 - global_seqlen/minmax_diff:11945 - global_seqlen/balanced_min:31356 - global_seqlen/balanced_max:31791 - global_seqlen/mean:31428.0 - actor/entropy:0.7460911870002747 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.456478089094162 - training/rollout_probs_diff_mean:0.004542972426861525 - training/rollout_probs_diff_std:0.008595590479671955 - training/rollout_actor_probs_pearson_corr:0.9995923638343811 - rollout_corr/rollout_is_mean:1.0000001192092896 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1051189176214393e-05 - rollout_corr/rollout_is_max:1.62330961227417 - rollout_corr/rollout_is_min:0.39613932371139526 - rollout_corr/rollout_is_std:0.03452324494719505 - rollout_corr/rollout_is_eff_sample_size:0.9988095643491313 - rollout_corr/rollout_is_seq_mean:0.9999744892120361 - rollout_corr/rollout_is_seq_std:0.0015889799688011408 - rollout_corr/rollout_is_seq_max:1.0049422979354858 - rollout_corr/rollout_is_seq_min:0.9941216111183167 - rollout_corr/rollout_is_seq_max_deviation:0.00587838888168335 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2059524417854846) - rollout_corr/training_log_ppl:np.float64(0.7630574661307037) - rollout_corr/kl:np.float64(0.0006121348877812238) - rollout_corr/k3_kl:np.float64(0.000635629686428274) - rollout_corr/rollout_ppl:np.float64(2.204429803881794) - rollout_corr/rollout_log_ppl:np.float64(0.7624453329481184) - rollout_corr/log_ppl_diff:np.float64(0.0006121331825852394) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012938708532601595) - rollout_corr/log_ppl_diff_max:np.float64(0.006876647472381592) - rollout_corr/log_ppl_diff_min:np.float64(-0.004626870155334473) - rollout_corr/ppl_ratio:np.float64(1.0006135429721326) - rollout_corr/chi2_token:np.float64(0.0013200147077441216) - rollout_corr/chi2_seq:np.float64(1.2292231675237417) - actor/pg_loss:np.float64(-5.2339513786137104e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001717434528245576) - actor/ppo_kl:np.float64(-3.4712943731474155e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.27383991330862045) - perf/mfu/actor:np.float64(0.11895889859672076) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.13016891479492) - actor/lr:np.float64(1e-06) - training/global_step:46 - training/epoch:0 - critic/score/mean:0.46484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.014861086383461952 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.014861086383461952 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:706.9375 - response_length/max:2609.0 - response_length/min:176.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:706.9375 - response_length_non_aborted/max:2609.0 - response_length_non_aborted/min:176.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:275.1875 - prompt_length/max:595.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.24672931432724e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7639765553176403) - timing_s/agent_loop/generate_sequences/max:np.float64(19.902556966990232) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.572092931586667) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(19.902556966990232) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:177 - timing_s/agent_loop/slowest/response_length:2609 - timing_s/gen:27.38719269633293 - timing_s/reward:0.0768952015787363 - timing_s/old_log_prob:3.2372986152768135 - timing_s/adv:0.13518765196204185 - timing_s/update_actor:14.003670601174235 - timing_s/step:44.92860447615385 - timing_s/stop_profile:0.00011640042066574097 - timing_per_token_ms/adv:0.0005376879373569821 - timing_per_token_ms/gen:0.15133052281149395 - timing_per_token_ms/update_actor:0.055697429844303786 - perf/total_num_tokens:251424 - perf/time_per_step:44.92860447615385 - perf/throughput:699.5098193330401
(TaskRunner pid=2224264) Training Progress: 46%|████▌ | 46/100 [43:13<49:43, 55.26s/it]
(TaskRunner pid=2224264) step:47 - global_seqlen/min:28796 - global_seqlen/max:40193 - global_seqlen/minmax_diff:11397 - global_seqlen/balanced_min:34771 - global_seqlen/balanced_max:34835 - global_seqlen/mean:34821.25 - actor/entropy:0.7023751139640808 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.34761902689933777 - training/rollout_probs_diff_mean:0.004359257873147726 - training/rollout_probs_diff_std:0.008405433967709541 - training/rollout_actor_probs_pearson_corr:0.9996216297149658 - rollout_corr/rollout_is_mean:0.9998831152915955 - rollout_corr/rollout_is_ratio_fraction_high:4.914246346743312e-06 - rollout_corr/rollout_is_ratio_fraction_low:4.914246346743312e-06 - rollout_corr/rollout_is_max:2.393986701965332 - rollout_corr/rollout_is_min:0.404596209526062 - rollout_corr/rollout_is_std:0.03408888727426529 - rollout_corr/rollout_is_eff_sample_size:0.9988389398452722 - rollout_corr/rollout_is_seq_mean:0.9998857975006104 - rollout_corr/rollout_is_seq_std:0.0014544243458658457 - rollout_corr/rollout_is_seq_max:1.0058720111846924 - rollout_corr/rollout_is_seq_min:0.9959080219268799 - rollout_corr/rollout_is_seq_max_deviation:0.005872011184692383 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.088344756513834) - rollout_corr/training_log_ppl:np.float64(0.71887747416622) - rollout_corr/kl:np.float64(0.0006769733809974809) - rollout_corr/k3_kl:np.float64(0.0006363334275647503) - rollout_corr/rollout_ppl:np.float64(2.086956723127514) - rollout_corr/rollout_log_ppl:np.float64(0.7182004956412129) - rollout_corr/log_ppl_diff:np.float64(0.0006769785250071436) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012075564882252365) - rollout_corr/log_ppl_diff_max:np.float64(0.006249547004699707) - rollout_corr/log_ppl_diff_min:np.float64(-0.004652619361877441) - rollout_corr/ppl_ratio:np.float64(1.0006782109849155) - rollout_corr/chi2_token:np.float64(0.0012049346696585417) - rollout_corr/chi2_seq:np.float64(0.8489019055850804) - actor/pg_loss:np.float64(0.00018669036217033863) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019566034120543918) - actor/ppo_kl:np.float64(-4.604769495308858e-05) - actor/pg_clipfrac_lower:np.float64(4.4389203139871825e-06) - actor/grad_norm:np.float64(0.2251530885696411) - perf/mfu/actor:np.float64(0.1320186966648972) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.04901123046875) - actor/lr:np.float64(1e-06) - training/global_step:47 - training/epoch:0 - critic/score/mean:0.5625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006613961420953274 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006613961420953274 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:794.8828125 - response_length/max:2347.0 - response_length/min:195.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:794.8828125 - response_length_non_aborted/max:2347.0 - response_length_non_aborted/min:195.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:293.28125 - prompt_length/max:950.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010444037616252899 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4994081798940897) - timing_s/agent_loop/generate_sequences/max:np.float64(18.286521969363093) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.572983121870493) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(18.286521969363093) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:309 - timing_s/agent_loop/slowest/response_length:2347 - timing_s/gen:25.14034183882177 - timing_s/reward:0.07981901802122593 - timing_s/old_log_prob:3.3178899493068457 - timing_s/adv:0.1422868575900793 - timing_s/update_actor:13.743424396961927 - timing_s/step:42.512125335633755 - timing_s/stop_profile:0.00011054240167140961 - timing_per_token_ms/adv:0.0005107759543026145 - timing_per_token_ms/gen:0.12354583438410618 - timing_per_token_ms/update_actor:0.04933562263331273 - perf/total_num_tokens:278570 - perf/time_per_step:42.512125335633755 - perf/throughput:819.0898414296109
(TaskRunner pid=2224264) Training Progress: 47%|████▋ | 47/100 [43:55<45:26, 51.45s/it]
(TaskRunner pid=2224264) step:48 - global_seqlen/min:21182 - global_seqlen/max:34146 - global_seqlen/minmax_diff:12964 - global_seqlen/balanced_min:29830 - global_seqlen/balanced_max:29848 - global_seqlen/mean:29841.125 - actor/entropy:0.7442194819450378 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4606970250606537 - training/rollout_probs_diff_mean:0.004590690601617098 - training/rollout_probs_diff_std:0.008653250522911549 - training/rollout_actor_probs_pearson_corr:0.999643087387085 - rollout_corr/rollout_is_mean:0.9999340772628784 - rollout_corr/rollout_is_ratio_fraction_high:1.1716118024196476e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.565861463546753 - rollout_corr/rollout_is_min:0.6271895170211792 - rollout_corr/rollout_is_std:0.03491979092359543 - rollout_corr/rollout_is_eff_sample_size:0.9987818554445442 - rollout_corr/rollout_is_seq_mean:0.9998950362205505 - rollout_corr/rollout_is_seq_std:0.0015674727037549019 - rollout_corr/rollout_is_seq_max:1.0053834915161133 - rollout_corr/rollout_is_seq_min:0.9940774440765381 - rollout_corr/rollout_is_seq_max_deviation:0.005922555923461914 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.153257429599762) - rollout_corr/training_log_ppl:np.float64(0.7468871437013149) - rollout_corr/kl:np.float64(0.0006051990699234011) - rollout_corr/k3_kl:np.float64(0.0006705579712615872) - rollout_corr/rollout_ppl:np.float64(2.1520667281001806) - rollout_corr/rollout_log_ppl:np.float64(0.7462819424690679) - rollout_corr/log_ppl_diff:np.float64(0.0006052012322470546) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013254067162051797) - rollout_corr/log_ppl_diff_max:np.float64(0.006841838359832764) - rollout_corr/log_ppl_diff_min:np.float64(-0.004772186279296875) - rollout_corr/ppl_ratio:np.float64(1.0006067838985473) - rollout_corr/chi2_token:np.float64(0.0014893666375428438) - rollout_corr/chi2_seq:np.float64(1.7015499353874475) - actor/pg_loss:np.float64(-4.240649286657572e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019128616168018198) - actor/ppo_kl:np.float64(-0.00013246856422000075) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3110559768974781) - perf/mfu/actor:np.float64(0.11562384315159505) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.0516357421875) - actor/lr:np.float64(1e-06) - training/global_step:48 - training/epoch:0 - critic/score/mean:0.58203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004874637350440025 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004874637350440025 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:666.81640625 - response_length/max:1727.0 - response_length/min:162.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:666.81640625 - response_length_non_aborted/max:1727.0 - response_length_non_aborted/min:162.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.71875 - prompt_length/max:563.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011752359569072723 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0456913970410824) - timing_s/agent_loop/generate_sequences/max:np.float64(14.015774877741933) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.352618356359017) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.015774877741933) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:311 - timing_s/agent_loop/slowest/response_length:1727 - timing_s/gen:21.136593686416745 - timing_s/reward:0.07471263594925404 - timing_s/old_log_prob:3.2577999029308558 - timing_s/adv:0.11815616860985756 - timing_s/update_actor:13.682143926620483 - timing_s/step:38.35702735558152 - timing_s/stop_profile:0.00011106766760349274 - timing_per_token_ms/adv:0.0004949384809129078 - timing_per_token_ms/gen:0.12381941762934152 - timing_per_token_ms/update_actor:0.05731245021183218 - perf/total_num_tokens:238729 - perf/time_per_step:38.35702735558152 - perf/throughput:777.9832551507063
(TaskRunner pid=2224264) Training Progress: 48%|████▊ | 48/100 [44:34<41:11, 47.54s/it]
(TaskRunner pid=2224264) step:49 - global_seqlen/min:27565 - global_seqlen/max:33967 - global_seqlen/minmax_diff:6402 - global_seqlen/balanced_min:31660 - global_seqlen/balanced_max:31677 - global_seqlen/mean:31668.625 - actor/entropy:0.7485659122467041 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8766915202140808 - training/rollout_probs_diff_mean:0.004531657788902521 - training/rollout_probs_diff_std:0.008790009655058384 - training/rollout_actor_probs_pearson_corr:0.9996523857116699 - rollout_corr/rollout_is_mean:1.0000706911087036 - rollout_corr/rollout_is_ratio_fraction_high:5.495320692716632e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.495320692716632e-06 - rollout_corr/rollout_is_max:8.354974746704102 - rollout_corr/rollout_is_min:0.0006339381216093898 - rollout_corr/rollout_is_std:0.03499140590429306 - rollout_corr/rollout_is_eff_sample_size:0.99877733654154 - rollout_corr/rollout_is_seq_mean:1.0000840425491333 - rollout_corr/rollout_is_seq_std:0.0015765647403895855 - rollout_corr/rollout_is_seq_max:1.0120781660079956 - rollout_corr/rollout_is_seq_min:0.9965959787368774 - rollout_corr/rollout_is_seq_max_deviation:0.012078166007995605 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.135033040307462) - rollout_corr/training_log_ppl:np.float64(0.7435363659169525) - rollout_corr/kl:np.float64(0.000531884074939093) - rollout_corr/k3_kl:np.float64(0.0007136939013889787) - rollout_corr/rollout_ppl:np.float64(2.1338718575425446) - rollout_corr/rollout_log_ppl:np.float64(0.7430044782813638) - rollout_corr/log_ppl_diff:np.float64(0.0005318876355886459) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011775929015129805) - rollout_corr/log_ppl_diff_max:np.float64(0.011757910251617432) - rollout_corr/log_ppl_diff_min:np.float64(-0.0038299560546875) - rollout_corr/ppl_ratio:np.float64(1.0005332818254828) - rollout_corr/chi2_token:np.float64(0.00199099606834352) - rollout_corr/chi2_seq:np.float64(1.2359290518797934) - actor/pg_loss:np.float64(-0.00026974163483828306) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014913909058122954) - actor/ppo_kl:np.float64(-6.670435310951461e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.24813814461231232) - perf/mfu/actor:np.float64(0.12164250497550971) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.1574478149414) - actor/lr:np.float64(1e-06) - training/global_step:49 - training/epoch:0 - critic/score/mean:0.6484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.006872585508972406 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.006872585508972406 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:710.83203125 - response_length/max:1348.0 - response_length/min:200.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:710.83203125 - response_length_non_aborted/max:1348.0 - response_length_non_aborted/min:200.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.8125 - prompt_length/max:418.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015538372099399567 - timing_s/agent_loop/generate_sequences/min:np.float64(2.5202769823372364) - timing_s/agent_loop/generate_sequences/max:np.float64(13.002393383532763) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.453353088778385) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(13.002393383532763) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:191 - timing_s/agent_loop/slowest/response_length:1348 - timing_s/gen:20.19887711107731 - timing_s/reward:0.077494902536273 - timing_s/old_log_prob:3.275420231744647 - timing_s/adv:0.1729813516139984 - timing_s/update_actor:13.726705972105265 - timing_s/step:37.536205399781466 - timing_s/stop_profile:0.00011497922241687775 - timing_per_token_ms/adv:0.0006827789003074747 - timing_per_token_ms/gen:0.1109993081999929 - timing_per_token_ms/update_actor:0.05418101501132929 - perf/total_num_tokens:253349 - perf/time_per_step:37.536205399781466 - perf/throughput:843.6821107171471
(TaskRunner pid=2224264) Training Progress: 49%|████▉ | 49/100 [45:11<37:52, 44.56s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 50}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) We are given a product molecule:
(TaskRunner pid=2224264) **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**
(TaskRunner pid=2224264) This is a **nitrogen-containing heterocyclic compound** (likely a pyrrole or similar), with a substituent on the nitrogen:
(TaskRunner pid=2224264) - A **benzene ring** with a **chloro group** on one carbon
(TaskRunner pid=2224264) - A **trifluoromethyl group** (C(F)(F)F) on another carbon
(TaskRunner pid=2224264) - A **nitrile group** (the N in the ring is bonded to a carbon chain)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Let's break it down:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Product structure:
(TaskRunner pid=2224264) - The core is a **pyrrole-like ring** (N in ring)
(TaskRunner pid=2224264) - Attached to the N is a **benzene ring** with:
(TaskRunner pid=2224264) - A **chloro group** (Cl) on one carbon
(TaskRunner pid=2224264) - A **trifluoromethyl group** (C(F)(F)F) on another carbon
(TaskRunner pid=2224264) - The ring also has a **nitrile group** (C≡N)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) We are to determine which of the four options is the **only correct reactant** in the synthesis of this product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Let’s analyze each option:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### **Option A:**
(TaskRunner pid=2224264) `O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1`
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - Contains a **nitrile group** (C≡N)
(TaskRunner pid=2224264) - Contains a **benzene ring** with Cl and C(F)(F)F
(TaskRunner pid=2224264) - The structure matches the **product** closely
(TaskRunner pid=2224264) - The **nitrogen ring** is connected to the same substituents
(TaskRunner pid=2224264) - Seems to be a **correct structure** for a possible reactant
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### **Option B:**
(TaskRunner pid=2224264) `O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1`
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - Contains a **Cl group** on the ring
(TaskRunner pid=2224264) - The ring has **nitrogen** and **chlorine**
(TaskRunner pid=2224264) - However, the **trifluoromethyl group** is missing
(TaskRunner pid=2224264) - Structure does **not match** the product
(TaskRunner pid=2224264) - Likely incorrect
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### **Option C:**
(TaskRunner pid=2224264) `O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1`
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - Contains a **benzene ring** with Cl and C(F)(F)F
(TaskRunner pid=2224264) - Contains **nitrogen and oxygen** in the ring
(TaskRunner pid=2224264) - However, the **nitrile group** is not present
(TaskRunner pid=2224264) - The **structure is not the same** as the product
(TaskRunner pid=2224264) - Likely incorrect
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### **Option D:**
(TaskRunner pid=2224264) `O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F`
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - Contains a **trifluoromethyl group**
(TaskRunner pid=2224264) - Contains **double bonds and rings**
(TaskRunner pid=2224264) - But **no benzene ring with Cl**
(TaskRunner pid=2224264) - No **nitrogen ring** with the correct substituents
(TaskRunner pid=2224264) - Does not match the product
(TaskRunner pid=2224264) - Incorrect
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### **Conclusion:**
(TaskRunner pid=2224264) Only **Option A** contains all the necessary **functional groups and substituents** to be the correct reactant in the synthesis of the given product. It matches the **structure and substituents** of the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_50
(TaskRunner pid=2224264) step:50 - global_seqlen/min:25774 - global_seqlen/max:40010 - global_seqlen/minmax_diff:14236 - global_seqlen/balanced_min:30199 - global_seqlen/balanced_max:30236 - global_seqlen/mean:30221.125 - actor/entropy:0.6831239461898804 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18094828724861145 - training/rollout_probs_diff_mean:0.00435651233419776 - training/rollout_probs_diff_std:0.008434131741523743 - training/rollout_actor_probs_pearson_corr:0.9996002316474915 - rollout_corr/rollout_is_mean:0.9999532699584961 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.877063358639134e-06 - rollout_corr/rollout_is_max:1.8264771699905396 - rollout_corr/rollout_is_min:0.4736720621585846 - rollout_corr/rollout_is_std:0.03403113782405853 - rollout_corr/rollout_is_eff_sample_size:0.998842864637548 - rollout_corr/rollout_is_seq_mean:0.9999036192893982 - rollout_corr/rollout_is_seq_std:0.0016001641051843762 - rollout_corr/rollout_is_seq_max:1.0056898593902588 - rollout_corr/rollout_is_seq_min:0.9943985939025879 - rollout_corr/rollout_is_seq_max_deviation:0.005689859390258789 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0370463258586824) - rollout_corr/training_log_ppl:np.float64(0.6895827453117818) - rollout_corr/kl:np.float64(0.0007133520028830276) - rollout_corr/k3_kl:np.float64(0.0006415439404463541) - rollout_corr/rollout_ppl:np.float64(2.035565957427025) - rollout_corr/rollout_log_ppl:np.float64(0.6888693935470656) - rollout_corr/log_ppl_diff:np.float64(0.000713351764716208) - rollout_corr/log_ppl_abs_diff:np.float64(0.001375065534375608) - rollout_corr/log_ppl_diff_max:np.float64(0.010087192058563232) - rollout_corr/log_ppl_diff_min:np.float64(-0.004953622817993164) - rollout_corr/ppl_ratio:np.float64(1.0007150897290558) - rollout_corr/chi2_token:np.float64(0.0011441940441727638) - rollout_corr/chi2_seq:np.float64(1.4973946234676987) - actor/pg_loss:np.float64(0.00014200282748788595) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00030153528336995805) - actor/ppo_kl:np.float64(1.1964239081230232e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3060915023088455) - perf/mfu/actor:np.float64(0.11810264114100524) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.11758041381836) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6223214285714286) - val-aux/sciknoweval/reward/std@16:np.float64(0.2647595134884485) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7291095238095238) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.19877560636838412) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5156142857142857) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.24700525701136392) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6215809523809525) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.2629386719576279) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8044857142857144) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.12973275443729954) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.4167428571428572) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.20684059335831737) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6544095238095238) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.20967583799172174) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8527809523809524) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.07656020696044112) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.3329095238095238) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.16042430484607695) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6712285714285714) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.15275032646284742) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8799809523809523) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.041193057541303105) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.2648619047619048) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.10865600566624517) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6785476190476191) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.11458915286528892) - val-aux/sciknoweval/score/mean@16:np.float64(0.6223214285714286) - val-aux/sciknoweval/score/std@16:np.float64(0.2647595134884485) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7291095238095238) - val-aux/sciknoweval/score/best@2/std:np.float64(0.19877560636838412) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5156142857142857) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.24700525701136392) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6215809523809525) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.2629386719576279) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8044857142857144) - val-aux/sciknoweval/score/best@4/std:np.float64(0.12973275443729954) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.4167428571428572) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.20684059335831737) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6544095238095238) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.20967583799172174) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8527809523809524) - val-aux/sciknoweval/score/best@8/std:np.float64(0.07656020696044112) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.3329095238095238) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.16042430484607695) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6712285714285714) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.15275032646284742) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8799809523809523) - val-aux/sciknoweval/score/best@16/std:np.float64(0.041193057541303105) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.2648619047619048) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.10865600566624517) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6785476190476191) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.11458915286528892) - val-core/sciknoweval/acc/mean@16:np.float64(0.6223214285714286) - val-aux/sciknoweval/acc/std@16:np.float64(0.2647595134884485) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7291095238095238) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.19877560636838412) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5156142857142857) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.24700525701136392) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6215809523809525) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.2629386719576279) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8044857142857144) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.12973275443729954) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.4167428571428572) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.20684059335831737) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6544095238095238) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.20967583799172174) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8527809523809524) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.07656020696044112) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.3329095238095238) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.16042430484607695) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6712285714285714) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.15275032646284742) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8799809523809523) - val-core/sciknoweval/acc/best@16/std:np.float64(0.041193057541303105) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.2648619047619048) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.10865600566624517) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6785476190476191) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.11458915286528892) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.986904761904762) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.03936087691755317) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.998252380952381) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.013004708934596106) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9755047619047619) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0512654983800644) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9868380952380952) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.039356229004372134) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9999285714285715) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0010041883502657082) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9567476190476191) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.06208854001798514) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9947333333333334) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.024839640974003308) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9297809523809525) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.06686724116120492) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9986714285714285) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.011269482933237476) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.8981190476190476) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.05941598517810314) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9998619047619047) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0028135293122891663) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:50 - training/epoch:0 - critic/score/mean:0.51171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0002585907932370901 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0002585907932370901 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:664.66015625 - response_length/max:1839.0 - response_length/min:161.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:664.66015625 - response_length_non_aborted/max:1839.0 - response_length_non_aborted/min:161.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.75 - prompt_length/max:900.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00018392875790596008 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0269889794290066) - timing_s/agent_loop/generate_sequences/max:np.float64(14.812706418335438) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.506310426571872) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.812706418335438) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:165 - timing_s/agent_loop/slowest/response_length:1763 - timing_s/gen:21.724913926795125 - timing_s/reward:0.07297280617058277 - timing_s/old_log_prob:3.25269278883934 - timing_s/adv:0.11958634853363037 - timing_s/update_actor:13.451122742146254 - timing_s/step:38.707119492813945 - timing_s/testing:191.03151337988675 - timing_s/save_checkpoint:12.945183955132961 - timing_s/stop_profile:0.00014170818030834198 - timing_per_token_ms/adv:0.000494630612417764 - timing_per_token_ms/gen:0.1276787005036357 - timing_per_token_ms/update_actor:0.05563625916534483 - perf/total_num_tokens:241769 - perf/time_per_step:38.707119492813945 - perf/throughput:780.7639885373184
(TaskRunner pid=2224264) Training Progress: 50%|█████ | 50/100 [49:14<1:26:40, 104.01s/it]
(TaskRunner pid=2224264) step:51 - global_seqlen/min:30736 - global_seqlen/max:37371 - global_seqlen/minmax_diff:6635 - global_seqlen/balanced_min:34292 - global_seqlen/balanced_max:34302 - global_seqlen/mean:34297.75 - actor/entropy:0.762580394744873 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2294635772705078 - training/rollout_probs_diff_mean:0.004454846493899822 - training/rollout_probs_diff_std:0.008279886096715927 - training/rollout_actor_probs_pearson_corr:0.9996731281280518 - rollout_corr/rollout_is_mean:1.0000529289245605 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.811339259147644 - rollout_corr/rollout_is_min:0.5012558698654175 - rollout_corr/rollout_is_std:0.03488221764564514 - rollout_corr/rollout_is_eff_sample_size:0.9987848284293472 - rollout_corr/rollout_is_seq_mean:1.0000650882720947 - rollout_corr/rollout_is_seq_std:0.0014846812700852752 - rollout_corr/rollout_is_seq_max:1.0048807859420776 - rollout_corr/rollout_is_seq_min:0.9954461455345154 - rollout_corr/rollout_is_seq_max_deviation:0.004880785942077637 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.208350053522736) - rollout_corr/training_log_ppl:np.float64(0.7750773588195443) - rollout_corr/kl:np.float64(0.0005686312180488429) - rollout_corr/k3_kl:np.float64(0.0006700464483628821) - rollout_corr/rollout_ppl:np.float64(2.207024629227817) - rollout_corr/rollout_log_ppl:np.float64(0.7745087213115767) - rollout_corr/log_ppl_diff:np.float64(0.0005686375079676509) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012317007640376687) - rollout_corr/log_ppl_diff_max:np.float64(0.006791532039642334) - rollout_corr/log_ppl_diff_min:np.float64(-0.006045103073120117) - rollout_corr/ppl_ratio:np.float64(1.000570009695366) - rollout_corr/chi2_token:np.float64(0.0015524374321103096) - rollout_corr/chi2_seq:np.float64(2.341128587955609) - actor/pg_loss:np.float64(-0.00025634258054196835) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00016806365397314948) - actor/ppo_kl:np.float64(7.1047582039796e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.24387626722455025) - perf/mfu/actor:np.float64(0.13125870218702868) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.71975231170654) - actor/lr:np.float64(1e-06) - training/global_step:51 - training/epoch:0 - critic/score/mean:0.59375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0033099849242717028 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0033099849242717028 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:759.2734375 - response_length/max:1708.0 - response_length/min:153.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:759.2734375 - response_length_non_aborted/max:1708.0 - response_length_non_aborted/min:153.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.53125 - prompt_length/max:608.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.316764771938324e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.95208290964365) - timing_s/agent_loop/generate_sequences/max:np.float64(15.170884735882282) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.576293181155052) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.170884735882282) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:260 - timing_s/agent_loop/slowest/response_length:1708 - timing_s/gen:21.975822178646922 - timing_s/reward:0.07666116580367088 - timing_s/old_log_prob:3.373812895268202 - timing_s/adv:0.12022814154624939 - timing_s/update_actor:13.922087775543332 - timing_s/step:39.56273279339075 - timing_s/stop_profile:0.00014520063996315002 - timing_per_token_ms/adv:0.0004381779473371044 - timing_per_token_ms/gen:0.11305947389386915 - timing_per_token_ms/update_actor:0.05073979989774596 - perf/total_num_tokens:274382 - perf/time_per_step:39.56273279339075 - perf/throughput:866.920649266415
(TaskRunner pid=2224264) Training Progress: 51%|█████ | 51/100 [49:54<1:09:09, 84.69s/it]
(TaskRunner pid=2224264) step:52 - global_seqlen/min:25163 - global_seqlen/max:39354 - global_seqlen/minmax_diff:14191 - global_seqlen/balanced_min:31304 - global_seqlen/balanced_max:31326 - global_seqlen/mean:31315.625 - actor/entropy:0.7275306582450867 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1846359670162201 - training/rollout_probs_diff_mean:0.0044688270427286625 - training/rollout_probs_diff_std:0.008479300886392593 - training/rollout_actor_probs_pearson_corr:0.9996131658554077 - rollout_corr/rollout_is_mean:0.9999827146530151 - rollout_corr/rollout_is_ratio_fraction_high:5.457889528770465e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.091577905754093e-05 - rollout_corr/rollout_is_max:2.513453245162964 - rollout_corr/rollout_is_min:0.2490382194519043 - rollout_corr/rollout_is_std:0.03466796875 - rollout_corr/rollout_is_eff_sample_size:0.9987993368491189 - rollout_corr/rollout_is_seq_mean:0.9999496936798096 - rollout_corr/rollout_is_seq_std:0.0013713843654841185 - rollout_corr/rollout_is_seq_max:1.0041810274124146 - rollout_corr/rollout_is_seq_min:0.9949362277984619 - rollout_corr/rollout_is_seq_max_deviation:0.005063772201538086 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0971961123868823) - rollout_corr/training_log_ppl:np.float64(0.7249851016094908) - rollout_corr/kl:np.float64(0.000624122314107467) - rollout_corr/k3_kl:np.float64(0.0006732126169026742) - rollout_corr/rollout_ppl:np.float64(2.095865353010595) - rollout_corr/rollout_log_ppl:np.float64(0.7243609796278179) - rollout_corr/log_ppl_diff:np.float64(0.0006241219816729426) - rollout_corr/log_ppl_abs_diff:np.float64(0.001236821641214192) - rollout_corr/log_ppl_diff_max:np.float64(0.00571441650390625) - rollout_corr/log_ppl_diff_min:np.float64(-0.005221307277679443) - rollout_corr/ppl_ratio:np.float64(1.0006253765895963) - rollout_corr/chi2_token:np.float64(0.0014572420623153448) - rollout_corr/chi2_seq:np.float64(2.134200472617522) - actor/pg_loss:np.float64(0.00022022065240889788) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00021542626041082258) - actor/ppo_kl:np.float64(-4.695473637639225e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.25628167018294334) - perf/mfu/actor:np.float64(0.1214559238638076) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.17374038696289) - actor/lr:np.float64(1e-06) - training/global_step:52 - training/epoch:0 - critic/score/mean:0.5625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.011097254231572151 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.011097254231572151 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:715.70703125 - response_length/max:1611.0 - response_length/min:184.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:715.70703125 - response_length_non_aborted/max:1611.0 - response_length_non_aborted/min:184.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.90625 - prompt_length/max:453.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.15229320526123e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3523097801953554) - timing_s/agent_loop/generate_sequences/max:np.float64(14.437729807570577) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.303848396521062) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.437729807570577) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:1611 - timing_s/gen:21.366347413510084 - timing_s/reward:0.07527691684663296 - timing_s/old_log_prob:3.179901920258999 - timing_s/adv:0.12300918810069561 - timing_s/update_actor:13.513995960354805 - timing_s/step:38.342299580574036 - timing_s/stop_profile:0.00011920370161533356 - timing_per_token_ms/adv:0.0004910056405576115 - timing_per_token_ms/gen:0.1166151664575026 - timing_per_token_ms/update_actor:0.053942704162677595 - perf/total_num_tokens:250525 - perf/time_per_step:38.342299580574036 - perf/throughput:816.7383110184119
(TaskRunner pid=2224264) Training Progress: 52%|█████▏ | 52/100 [50:32<56:38, 70.80s/it]
(TaskRunner pid=2224264) step:53 - global_seqlen/min:29023 - global_seqlen/max:38170 - global_seqlen/minmax_diff:9147 - global_seqlen/balanced_min:32089 - global_seqlen/balanced_max:32125 - global_seqlen/mean:32112.5 - actor/entropy:0.7673122882843018 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2821834683418274 - training/rollout_probs_diff_mean:0.004418913275003433 - training/rollout_probs_diff_std:0.008327017538249493 - training/rollout_actor_probs_pearson_corr:0.9996277093887329 - rollout_corr/rollout_is_mean:1.0000674724578857 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6528507471084595 - rollout_corr/rollout_is_min:0.6141930818557739 - rollout_corr/rollout_is_std:0.035042472183704376 - rollout_corr/rollout_is_eff_sample_size:0.9987737690154207 - rollout_corr/rollout_is_seq_mean:1.0000889301300049 - rollout_corr/rollout_is_seq_std:0.0013838375452905893 - rollout_corr/rollout_is_seq_max:1.005210518836975 - rollout_corr/rollout_is_seq_min:0.9962599873542786 - rollout_corr/rollout_is_seq_max_deviation:0.005210518836975098 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2401722352951765) - rollout_corr/training_log_ppl:np.float64(0.7886160378111526) - rollout_corr/kl:np.float64(0.0006792797128305494) - rollout_corr/k3_kl:np.float64(0.0006657197256458858) - rollout_corr/rollout_ppl:np.float64(2.238586582709104) - rollout_corr/rollout_log_ppl:np.float64(0.7879367553978227) - rollout_corr/log_ppl_diff:np.float64(0.0006792824133299291) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012466523912735283) - rollout_corr/log_ppl_diff_max:np.float64(0.0049315690994262695) - rollout_corr/log_ppl_diff_min:np.float64(-0.004511535167694092) - rollout_corr/ppl_ratio:np.float64(1.000680540688336) - rollout_corr/chi2_token:np.float64(0.00131767219863832) - rollout_corr/chi2_seq:np.float64(0.9375523473136127) - actor/pg_loss:np.float64(-5.056627560406923e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00015928275297483196) - actor/ppo_kl:np.float64(0.00014284484250026708) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2610822059214115) - perf/mfu/actor:np.float64(0.12400191776249886) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.9232177734375) - actor/lr:np.float64(1e-06) - training/global_step:53 - training/epoch:0 - critic/score/mean:0.61328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.61328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.009051043540239334 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.009051043540239334 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:707.953125 - response_length/max:1922.0 - response_length/min:129.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:707.953125 - response_length_non_aborted/max:1922.0 - response_length_non_aborted/min:129.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:295.5625 - prompt_length/max:532.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.531714022159576e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6737753115594387) - timing_s/agent_loop/generate_sequences/max:np.float64(16.309401037171483) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.246375748174614) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.309401037171483) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:272 - timing_s/agent_loop/slowest/response_length:1922 - timing_s/gen:22.928913297131658 - timing_s/reward:0.07421708479523659 - timing_s/old_log_prob:3.3302101399749517 - timing_s/adv:0.1337327789515257 - timing_s/update_actor:13.578472383320332 - timing_s/step:40.13115260377526 - timing_s/stop_profile:0.0001122206449508667 - timing_per_token_ms/adv:0.000520563561508469 - timing_per_token_ms/gen:0.12651412135078935 - timing_per_token_ms/update_actor:0.0528550890748164 - perf/total_num_tokens:256900 - perf/time_per_step:40.13115260377526 - perf/throughput:800.1888287898084
(TaskRunner pid=2224264) Training Progress: 53%|█████▎ | 53/100 [51:12<48:15, 61.61s/it]
(TaskRunner pid=2224264) step:54 - global_seqlen/min:24046 - global_seqlen/max:37658 - global_seqlen/minmax_diff:13612 - global_seqlen/balanced_min:31667 - global_seqlen/balanced_max:31732 - global_seqlen/mean:31711.0 - actor/entropy:0.7221550345420837 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4564734995365143 - training/rollout_probs_diff_mean:0.0042567262426018715 - training/rollout_probs_diff_std:0.008278296329081059 - training/rollout_actor_probs_pearson_corr:0.9996740221977234 - rollout_corr/rollout_is_mean:0.9999973773956299 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.423934908321826e-06 - rollout_corr/rollout_is_max:1.8484896421432495 - rollout_corr/rollout_is_min:0.4127388894557953 - rollout_corr/rollout_is_std:0.03383616358041763 - rollout_corr/rollout_is_eff_sample_size:0.9988563043114053 - rollout_corr/rollout_is_seq_mean:0.9999293088912964 - rollout_corr/rollout_is_seq_std:0.0013600483071058989 - rollout_corr/rollout_is_seq_max:1.003616213798523 - rollout_corr/rollout_is_seq_min:0.9944389462471008 - rollout_corr/rollout_is_seq_max_deviation:0.00556105375289917 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1455902410671115) - rollout_corr/training_log_ppl:np.float64(0.7426831858465448) - rollout_corr/kl:np.float64(0.0007347915104665148) - rollout_corr/k3_kl:np.float64(0.0006544318975443275) - rollout_corr/rollout_ppl:np.float64(2.14404917974025) - rollout_corr/rollout_log_ppl:np.float64(0.7419483949197456) - rollout_corr/log_ppl_diff:np.float64(0.0007347909267991781) - rollout_corr/log_ppl_abs_diff:np.float64(0.00128568010404706) - rollout_corr/log_ppl_diff_max:np.float64(0.007453739643096924) - rollout_corr/log_ppl_diff_min:np.float64(-0.0039052367210388184) - rollout_corr/ppl_ratio:np.float64(1.000736182089895) - rollout_corr/chi2_token:np.float64(0.0011647003702819347) - rollout_corr/chi2_seq:np.float64(1.3221846474334598) - actor/pg_loss:np.float64(-0.0001485362881794572) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00030488177571896813) - actor/ppo_kl:np.float64(6.0147084293804554e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.3080219365656376) - perf/mfu/actor:np.float64(0.12202071329811175) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.63494873046875) - actor/lr:np.float64(1e-06) - training/global_step:54 - training/epoch:0 - critic/score/mean:0.515625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.515625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.016614189371466637 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.016614189371466637 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:720.1875 - response_length/max:2286.0 - response_length/min:162.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:720.1875 - response_length_non_aborted/max:2286.0 - response_length_non_aborted/min:162.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:270.78125 - prompt_length/max:597.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.760074317455292e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9534575715661049) - timing_s/agent_loop/generate_sequences/max:np.float64(17.89197232015431) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.935718771463144) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(17.89197232015431) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:275 - timing_s/agent_loop/slowest/response_length:2286 - timing_s/gen:25.277695801109076 - timing_s/reward:0.08129468001425266 - timing_s/old_log_prob:3.2715712431818247 - timing_s/adv:0.1201015543192625 - timing_s/update_actor:13.721273077651858 - timing_s/step:42.558674819767475 - timing_s/stop_profile:0.00013141706585884094 - timing_per_token_ms/adv:0.0004734222916309108 - timing_per_token_ms/gen:0.13710457238300072 - timing_per_token_ms/update_actor:0.05408719796620991 - perf/total_num_tokens:253688 - perf/time_per_step:42.558674819767475 - perf/throughput:745.1124860981575
(TaskRunner pid=2224264) Training Progress: 54%|█████▍ | 54/100 [51:55<42:51, 55.91s/it]
(TaskRunner pid=2224264) step:55 - global_seqlen/min:29461 - global_seqlen/max:36701 - global_seqlen/minmax_diff:7240 - global_seqlen/balanced_min:33720 - global_seqlen/balanced_max:34509 - global_seqlen/mean:33827.375 - actor/entropy:0.8024973273277283 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23547464609146118 - training/rollout_probs_diff_mean:0.0043955049477517605 - training/rollout_probs_diff_std:0.008106191642582417 - training/rollout_actor_probs_pearson_corr:0.999698281288147 - rollout_corr/rollout_is_mean:1.0000178813934326 - rollout_corr/rollout_is_ratio_fraction_high:5.190302545088343e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.0640006065368652 - rollout_corr/rollout_is_min:0.5909958481788635 - rollout_corr/rollout_is_std:0.03487367555499077 - rollout_corr/rollout_is_eff_sample_size:0.9987854230284318 - rollout_corr/rollout_is_seq_mean:0.999980092048645 - rollout_corr/rollout_is_seq_std:0.0013782148016616702 - rollout_corr/rollout_is_seq_max:1.0047253370285034 - rollout_corr/rollout_is_seq_min:0.9960294961929321 - rollout_corr/rollout_is_seq_max_deviation:0.004725337028503418 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.3131857896223664) - rollout_corr/training_log_ppl:np.float64(0.8200820172205567) - rollout_corr/kl:np.float64(0.0007159894126873567) - rollout_corr/k3_kl:np.float64(0.0006568527235231159) - rollout_corr/rollout_ppl:np.float64(2.311495006084442) - rollout_corr/rollout_log_ppl:np.float64(0.8193660224787891) - rollout_corr/log_ppl_diff:np.float64(0.0007159947417676449) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012078655418008566) - rollout_corr/log_ppl_diff_max:np.float64(0.004928708076477051) - rollout_corr/log_ppl_diff_min:np.float64(-0.0037838220596313477) - rollout_corr/ppl_ratio:np.float64(1.0007172094192356) - rollout_corr/chi2_token:np.float64(0.0011943182907998562) - rollout_corr/chi2_seq:np.float64(2.1422022855840623) - actor/pg_loss:np.float64(6.403832230716944e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014861452268633002) - actor/ppo_kl:np.float64(7.01219851411139e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2634842284023762) - perf/mfu/actor:np.float64(0.12720097629374041) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.60932922363281) - actor/lr:np.float64(1e-06) - training/global_step:55 - training/epoch:0 - critic/score/mean:0.61328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.61328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.000431443884735927 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.000431443884735927 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:752.60546875 - response_length/max:2800.0 - response_length/min:187.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:752.60546875 - response_length_non_aborted/max:2800.0 - response_length_non_aborted/min:187.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:304.5 - prompt_length/max:625.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.310990571975708e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7985909432172775) - timing_s/agent_loop/generate_sequences/max:np.float64(20.643996415659785) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.059685343854653) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(20.643996415659785) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:236 - timing_s/agent_loop/slowest/response_length:2800 - timing_s/gen:27.551894007250667 - timing_s/reward:0.07855446077883244 - timing_s/old_log_prob:3.331155611202121 - timing_s/adv:0.1180018950253725 - timing_s/update_actor:14.12481613457203 - timing_s/step:45.289689207449555 - timing_s/stop_profile:3.606826066970825e-05 - timing_per_token_ms/adv:0.0004360443835258149 - timing_per_token_ms/gen:0.14300266266278433 - timing_per_token_ms/update_actor:0.05219447316918631 - perf/total_num_tokens:270619 - perf/time_per_step:45.289689207449555 - perf/throughput:746.9111754124345
(TaskRunner pid=2224264) Training Progress: 55%|█████▌ | 55/100 [52:40<39:32, 52.73s/it]
(TaskRunner pid=2224264) step:56 - global_seqlen/min:25792 - global_seqlen/max:39171 - global_seqlen/minmax_diff:13379 - global_seqlen/balanced_min:31985 - global_seqlen/balanced_max:31996 - global_seqlen/mean:31991.875 - actor/entropy:0.8143277764320374 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2929895520210266 - training/rollout_probs_diff_mean:0.004465053323656321 - training/rollout_probs_diff_std:0.008296534419059753 - training/rollout_actor_probs_pearson_corr:0.9996805787086487 - rollout_corr/rollout_is_mean:0.9999870657920837 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.5952778994687833e-05 - rollout_corr/rollout_is_max:1.629705786705017 - rollout_corr/rollout_is_min:0.2926250994205475 - rollout_corr/rollout_is_std:0.034940268844366074 - rollout_corr/rollout_is_eff_sample_size:0.9987806662555786 - rollout_corr/rollout_is_seq_mean:1.0000478029251099 - rollout_corr/rollout_is_seq_std:0.0013941883808001876 - rollout_corr/rollout_is_seq_max:1.0048588514328003 - rollout_corr/rollout_is_seq_min:0.9960804581642151 - rollout_corr/rollout_is_seq_max_deviation:0.004858851432800293 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.281407651025802) - rollout_corr/training_log_ppl:np.float64(0.804402201261837) - rollout_corr/kl:np.float64(0.0005219981214015235) - rollout_corr/k3_kl:np.float64(0.0006556217045954327) - rollout_corr/rollout_ppl:np.float64(2.2802593959495425) - rollout_corr/rollout_log_ppl:np.float64(0.803880195133388) - rollout_corr/log_ppl_diff:np.float64(0.000522006128448993) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012433496885932982) - rollout_corr/log_ppl_diff_max:np.float64(0.0066702961921691895) - rollout_corr/log_ppl_diff_min:np.float64(-0.0035393238067626953) - rollout_corr/ppl_ratio:np.float64(1.0005232789553702) - rollout_corr/chi2_token:np.float64(0.0015740657690912485) - rollout_corr/chi2_seq:np.float64(5.73605796508491) - actor/pg_loss:np.float64(-4.861317574977875e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00018069532825393253) - actor/ppo_kl:np.float64(-5.679699522431747e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2360125184059143) - perf/mfu/actor:np.float64(0.12372654611791109) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.54317474365234) - actor/lr:np.float64(1e-06) - training/global_step:56 - training/epoch:0 - critic/score/mean:0.5859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002341070445254445 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002341070445254445 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:734.58984375 - response_length/max:1632.0 - response_length/min:182.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:734.58984375 - response_length_non_aborted/max:1632.0 - response_length_non_aborted/min:182.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.15625 - prompt_length/max:458.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.9824674129486084e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.308333108201623) - timing_s/agent_loop/generate_sequences/max:np.float64(14.604689434170723) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.345092790725175) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.604689434170723) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:308 - timing_s/agent_loop/slowest/response_length:1632 - timing_s/gen:21.219629768282175 - timing_s/reward:0.07743704505264759 - timing_s/old_log_prob:3.259379044175148 - timing_s/adv:0.11571713909506798 - timing_s/update_actor:13.64204434491694 - timing_s/step:38.399653954431415 - timing_s/stop_profile:0.00011725537478923798 - timing_per_token_ms/adv:0.0004521348744605778 - timing_per_token_ms/gen:0.11283736017804459 - timing_per_token_ms/update_actor:0.05330276962868283 - perf/total_num_tokens:255935 - perf/time_per_step:38.399653954431415 - perf/throughput:833.1292526220294
(TaskRunner pid=2224264) Training Progress: 56%|█████▌ | 56/100 [53:19<35:31, 48.45s/it]
(TaskRunner pid=2224264) step:57 - global_seqlen/min:27687 - global_seqlen/max:34593 - global_seqlen/minmax_diff:6906 - global_seqlen/balanced_min:31298 - global_seqlen/balanced_max:31311 - global_seqlen/mean:31305.125 - actor/entropy:0.7938913106918335 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.680421769618988 - training/rollout_probs_diff_mean:0.004562178626656532 - training/rollout_probs_diff_std:0.008708939887583256 - training/rollout_actor_probs_pearson_corr:0.9995999932289124 - rollout_corr/rollout_is_mean:0.9999580979347229 - rollout_corr/rollout_is_ratio_fraction_high:5.648919795930851e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.6946758478297852e-05 - rollout_corr/rollout_is_max:2.2787086963653564 - rollout_corr/rollout_is_min:0.24652595818042755 - rollout_corr/rollout_is_std:0.035597678273916245 - rollout_corr/rollout_is_eff_sample_size:0.9987344090023405 - rollout_corr/rollout_is_seq_mean:0.9999576807022095 - rollout_corr/rollout_is_seq_std:0.001491630100645125 - rollout_corr/rollout_is_seq_max:1.0051113367080688 - rollout_corr/rollout_is_seq_min:0.9956648349761963 - rollout_corr/rollout_is_seq_max_deviation:0.005111336708068848 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.283419963438064) - rollout_corr/training_log_ppl:np.float64(0.8058918532915413) - rollout_corr/kl:np.float64(0.0006797224436070337) - rollout_corr/k3_kl:np.float64(0.0007055759461991329) - rollout_corr/rollout_ppl:np.float64(2.281822293996811) - rollout_corr/rollout_log_ppl:np.float64(0.80521212273743) - rollout_corr/log_ppl_diff:np.float64(0.0006797305541113019) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013461158378049731) - rollout_corr/log_ppl_diff_max:np.float64(0.0058408379554748535) - rollout_corr/log_ppl_diff_min:np.float64(-0.004617094993591309) - rollout_corr/ppl_ratio:np.float64(1.0006813008803874) - rollout_corr/chi2_token:np.float64(0.0014507242012768984) - rollout_corr/chi2_seq:np.float64(1.6866442584432662) - actor/pg_loss:np.float64(1.2409058399498463e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002567762589933409) - actor/ppo_kl:np.float64(-1.4622721524304438e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2836088426411152) - perf/mfu/actor:np.float64(0.12280457724250629) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.51215362548828) - actor/lr:np.float64(1e-06) - training/global_step:57 - training/epoch:0 - critic/score/mean:0.60546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.60546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0012533540138974786 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0012533540138974786 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:691.50390625 - response_length/max:1421.0 - response_length/min:166.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:691.50390625 - response_length_non_aborted/max:1421.0 - response_length_non_aborted/min:166.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.78125 - prompt_length/max:660.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012706033885478973 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5593209099024534) - timing_s/agent_loop/generate_sequences/max:np.float64(12.716230429708958) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.830025155824842) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.716230429708958) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:270 - timing_s/agent_loop/slowest/response_length:1334 - timing_s/gen:19.76116858050227 - timing_s/reward:0.08693374507129192 - timing_s/old_log_prob:3.292963979765773 - timing_s/adv:0.1231906358152628 - timing_s/update_actor:13.490689365193248 - timing_s/step:36.84112096019089 - timing_s/stop_profile:0.00011323578655719757 - timing_per_token_ms/adv:0.0004918948407619471 - timing_per_token_ms/gen:0.11162925338512793 - timing_per_token_ms/update_actor:0.05386773477662702 - perf/total_num_tokens:250441 - perf/time_per_step:36.84112096019089 - perf/throughput:849.7332378628523
(TaskRunner pid=2224264) Training Progress: 57%|█████▋ | 57/100 [53:56<32:14, 44.98s/it]
(TaskRunner pid=2224264) step:58 - global_seqlen/min:26170 - global_seqlen/max:34255 - global_seqlen/minmax_diff:8085 - global_seqlen/balanced_min:30104 - global_seqlen/balanced_max:30420 - global_seqlen/mean:30161.5 - actor/entropy:0.7656847238540649 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.16260474920272827 - training/rollout_probs_diff_mean:0.00442152377218008 - training/rollout_probs_diff_std:0.00836502481251955 - training/rollout_actor_probs_pearson_corr:0.9997066259384155 - rollout_corr/rollout_is_mean:0.9998526573181152 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.867856089025736e-06 - rollout_corr/rollout_is_max:1.6352856159210205 - rollout_corr/rollout_is_min:0.49183452129364014 - rollout_corr/rollout_is_std:0.034748680889606476 - rollout_corr/rollout_is_eff_sample_size:0.9987937474899505 - rollout_corr/rollout_is_seq_mean:0.9999055862426758 - rollout_corr/rollout_is_seq_std:0.0014970327029004693 - rollout_corr/rollout_is_seq_max:1.0064585208892822 - rollout_corr/rollout_is_seq_min:0.9960758090019226 - rollout_corr/rollout_is_seq_max_deviation:0.0064585208892822266 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.18733112514019) - rollout_corr/training_log_ppl:np.float64(0.7628000770928338) - rollout_corr/kl:np.float64(0.0007255446162326429) - rollout_corr/k3_kl:np.float64(0.0006535338600883733) - rollout_corr/rollout_ppl:np.float64(2.185690295882523) - rollout_corr/rollout_log_ppl:np.float64(0.7620745294261724) - rollout_corr/log_ppl_diff:np.float64(0.0007255476666614413) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013074638554826379) - rollout_corr/log_ppl_diff_max:np.float64(0.005662679672241211) - rollout_corr/log_ppl_diff_min:np.float64(-0.00701594352722168) - rollout_corr/ppl_ratio:np.float64(1.000726986443624) - rollout_corr/chi2_token:np.float64(0.0011695893481373787) - rollout_corr/chi2_seq:np.float64(1.832958000479266) - actor/pg_loss:np.float64(1.464621163904667e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00012746554830300738) - actor/ppo_kl:np.float64(9.933237482329105e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2319943569600582) - perf/mfu/actor:np.float64(0.11497644796674522) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.57088470458984) - actor/lr:np.float64(1e-06) - training/global_step:58 - training/epoch:0 - critic/score/mean:0.63671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.63671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002008273731917143 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002008273731917143 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:665.703125 - response_length/max:2537.0 - response_length/min:155.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:665.703125 - response_length_non_aborted/max:2537.0 - response_length_non_aborted/min:155.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:276.84375 - prompt_length/max:598.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001398194581270218 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0110350400209427) - timing_s/agent_loop/generate_sequences/max:np.float64(19.426406839862466) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.346314342365076) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(19.426406839862466) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:199 - timing_s/agent_loop/slowest/response_length:2537 - timing_s/gen:26.27677822113037 - timing_s/reward:0.07973587512969971 - timing_s/old_log_prob:3.217642493546009 - timing_s/adv:0.12296506203711033 - timing_s/update_actor:13.712465612217784 - timing_s/step:43.495522832497954 - timing_s/stop_profile:0.00010489486157894135 - timing_per_token_ms/adv:0.0005096110191681047 - timing_per_token_ms/gen:0.15418834773577264 - timing_per_token_ms/update_actor:0.056829342092642045 - perf/total_num_tokens:241292 - perf/time_per_step:43.495522832497954 - perf/throughput:693.4391872044505
(TaskRunner pid=2224264) Training Progress: 58%|█████▊ | 58/100 [54:39<31:11, 44.55s/it]
(TaskRunner pid=2224264) step:59 - global_seqlen/min:27515 - global_seqlen/max:33648 - global_seqlen/minmax_diff:6133 - global_seqlen/balanced_min:30606 - global_seqlen/balanced_max:31185 - global_seqlen/mean:30683.625 - actor/entropy:0.7695986032485962 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4610300064086914 - training/rollout_probs_diff_mean:0.004400559701025486 - training/rollout_probs_diff_std:0.008376654237508774 - training/rollout_actor_probs_pearson_corr:0.9996274709701538 - rollout_corr/rollout_is_mean:1.0000059604644775 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.658765076077543e-06 - rollout_corr/rollout_is_max:1.8190242052078247 - rollout_corr/rollout_is_min:0.3681882917881012 - rollout_corr/rollout_is_std:0.03469890356063843 - rollout_corr/rollout_is_eff_sample_size:0.9987974340815278 - rollout_corr/rollout_is_seq_mean:0.9999397397041321 - rollout_corr/rollout_is_seq_std:0.0016430045943707228 - rollout_corr/rollout_is_seq_max:1.0059504508972168 - rollout_corr/rollout_is_seq_min:0.9930161237716675 - rollout_corr/rollout_is_seq_max_deviation:0.0069838762283325195 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2372520426288247) - rollout_corr/training_log_ppl:np.float64(0.7862228258745745) - rollout_corr/kl:np.float64(0.0008618131305837551) - rollout_corr/k3_kl:np.float64(0.0006665370408427407) - rollout_corr/rollout_ppl:np.float64(2.2352824225090444) - rollout_corr/rollout_log_ppl:np.float64(0.7853610130259767) - rollout_corr/log_ppl_diff:np.float64(0.000861812848597765) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014634081162512302) - rollout_corr/log_ppl_diff_max:np.float64(0.008165895938873291) - rollout_corr/log_ppl_diff_min:np.float64(-0.005269348621368408) - rollout_corr/ppl_ratio:np.float64(1.0008637136779726) - rollout_corr/chi2_token:np.float64(0.000947984866797924) - rollout_corr/chi2_seq:np.float64(1.44492331543006) - actor/pg_loss:np.float64(-2.2010761313140392e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00016003822292987024) - actor/ppo_kl:np.float64(0.00017181198663673847) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.20559313893318176) - perf/mfu/actor:np.float64(0.11867857354144504) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(104.51495361328125) - actor/lr:np.float64(1e-06) - training/global_step:59 - training/epoch:0 - critic/score/mean:0.50390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.50390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0009747223230078816 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0009747223230078816 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:690.30078125 - response_length/max:2398.0 - response_length/min:212.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:690.30078125 - response_length_non_aborted/max:2398.0 - response_length_non_aborted/min:212.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:268.5625 - prompt_length/max:615.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012865476310253143 - timing_s/agent_loop/generate_sequences/min:np.float64(2.7066904306411743) - timing_s/agent_loop/generate_sequences/max:np.float64(18.255667094141245) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.844594236710691) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(18.255667094141245) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:296 - timing_s/agent_loop/slowest/response_length:2398 - timing_s/gen:25.205973524600267 - timing_s/reward:0.07148762419819832 - timing_s/old_log_prob:3.309969963505864 - timing_s/adv:0.11456507630646229 - timing_s/update_actor:13.433156419545412 - timing_s/step:42.23662491515279 - timing_s/stop_profile:0.00011372379958629608 - timing_per_token_ms/adv:0.00046671912260392266 - timing_per_token_ms/gen:0.1426346844084059 - timing_per_token_ms/update_actor:0.054724451639699564 - perf/total_num_tokens:245469 - perf/time_per_step:42.23662491515279 - perf/throughput:726.4696234994847
(TaskRunner pid=2224264) Training Progress: 59%|█████▉ | 59/100 [55:21<29:58, 43.87s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 60}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) We are given a product molecule:
(TaskRunner pid=2224264) **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This is a condensed chemical formula. Let's interpret it step by step.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Step 1: Parse the product formula
(TaskRunner pid=2224264) The formula:
(TaskRunner pid=2224264) **Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This is a shorthand for a benzene ring with substituents. Let's break it down:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - **Nc1ccn(...)**: This is a nitrogen-containing ring.
(TaskRunner pid=2224264) - **Cc2ccc(Cl)cc2C(F)(F)F**: This is a side chain attached to the ring via the nitrogen. It contains a benzene ring with a Cl substituent and a trifluoromethyl group (C(F)(F)F).
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) So the product is likely a **nitrogen-containing heterocycle (e.g., pyridine or pyrrole)**, with a **benzene ring bearing a Cl and a CF₃ group** attached to it.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Step 2: Understand what we are looking for
(TaskRunner pid=2224264) We are to find the correct **reactant** that would be used in the synthesis of this product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) The options are all chemical formulas that may be reactants in a synthesis. We must find the one that matches the **structural features of the product**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Step 3: Analyze each option
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) #### Option A:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This is a **diacyl compound** (two C=O groups), and it includes a **nitrogen ring** with the same side chain as the product:
(TaskRunner pid=2224264) - `c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1`
(TaskRunner pid=2224264) This matches the **side chain** of the product.
(TaskRunner pid=2224264) - The `O=C1c2ccccc2C(=O)N1` part suggests a **diketone ring** (like a benzene ring with two ketone groups), which may be a **starting material** for forming the nitrogen ring.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This **looks like a good candidate** as it contains the same side chain and a structure that could be used to form the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) #### Option B:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This includes a Cl group on the ring and a nitrogen, but it **does not match the side chain** of the product. It also has a different structure and is not consistent with the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) #### Option C:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This has a **trifluoromethyl group** and a **Cl group**, but the **side chain is different**. It has a **hydroxyl group** that is not in the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) #### Option D:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This has a **CF₃ group** and a **benzene ring**, but the **side chain is different**. It does not match the structure of the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Step 4: Final determination
(TaskRunner pid=2224264) Only **Option A** includes the **same side chain** as the product and includes a **ring structure** that could be used in the synthesis of the nitrogen-containing ring in the product. It is the only one that matches the **structural features** of the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_60
(TaskRunner pid=2224264) step:60 - global_seqlen/min:28478 - global_seqlen/max:38447 - global_seqlen/minmax_diff:9969 - global_seqlen/balanced_min:32368 - global_seqlen/balanced_max:32372 - global_seqlen/mean:32370.0 - actor/entropy:0.8034039735794067 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4281436800956726 - training/rollout_probs_diff_mean:0.004418870899826288 - training/rollout_probs_diff_std:0.008324537426233292 - training/rollout_actor_probs_pearson_corr:0.9996195435523987 - rollout_corr/rollout_is_mean:1.0001076459884644 - rollout_corr/rollout_is_ratio_fraction_high:1.047778732754523e-05 - rollout_corr/rollout_is_ratio_fraction_low:5.238893663772615e-06 - rollout_corr/rollout_is_max:2.1436619758605957 - rollout_corr/rollout_is_min:0.026059480383992195 - rollout_corr/rollout_is_std:0.0351596400141716 - rollout_corr/rollout_is_eff_sample_size:0.9987655638020635 - rollout_corr/rollout_is_seq_mean:1.0000665187835693 - rollout_corr/rollout_is_seq_std:0.0014459878439083695 - rollout_corr/rollout_is_seq_max:1.005103349685669 - rollout_corr/rollout_is_seq_min:0.9960857629776001 - rollout_corr/rollout_is_seq_max_deviation:0.005103349685668945 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2919784113764763) - rollout_corr/training_log_ppl:np.float64(0.806903613905888) - rollout_corr/kl:np.float64(0.0006409502570816628) - rollout_corr/k3_kl:np.float64(0.0006509173431368254) - rollout_corr/rollout_ppl:np.float64(2.290479706134647) - rollout_corr/rollout_log_ppl:np.float64(0.8062626618193462) - rollout_corr/log_ppl_diff:np.float64(0.0006409520865418017) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012293330510146916) - rollout_corr/log_ppl_diff_max:np.float64(0.005213141441345215) - rollout_corr/log_ppl_diff_min:np.float64(-0.004137933254241943) - rollout_corr/ppl_ratio:np.float64(1.0006422218866646) - rollout_corr/chi2_token:np.float64(0.0013303321320563555) - rollout_corr/chi2_seq:np.float64(2.3223254224285483) - actor/pg_loss:np.float64(-0.0001239327248185873) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00011024115792679368) - actor/ppo_kl:np.float64(6.963595266995526e-05) - actor/pg_clipfrac_lower:np.float64(2.784212483675219e-06) - actor/grad_norm:np.float64(0.24239107966423035) - perf/mfu/actor:np.float64(0.12592590460533137) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.14717817306519) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6589285714285714) - val-aux/sciknoweval/reward/std@16:np.float64(0.26147699146732906) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7634047619047619) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.19852720557117806) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5552285714285714) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.2397040092225637) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6592000000000001) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.26073502970265483) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8421285714285713) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.13617544486794253) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.46132857142857137) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.20013008279957237) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6862619047619049) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.20619876382539531) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8954952380952381) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.08197180786429441) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.3817190476190476) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.1582460634427849) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6993809523809524) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.14750004177458542) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.9245333333333334) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.03779423823528531) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.31384761904761904) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.11509261056785722) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7040333333333333) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.108925053237339) - val-aux/sciknoweval/score/mean@16:np.float64(0.6589285714285714) - val-aux/sciknoweval/score/std@16:np.float64(0.26147699146732906) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7634047619047619) - val-aux/sciknoweval/score/best@2/std:np.float64(0.19852720557117806) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5552285714285714) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.2397040092225637) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6592000000000001) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.26073502970265483) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8421285714285713) - val-aux/sciknoweval/score/best@4/std:np.float64(0.13617544486794253) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.46132857142857137) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.20013008279957237) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6862619047619049) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.20619876382539531) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8954952380952381) - val-aux/sciknoweval/score/best@8/std:np.float64(0.08197180786429441) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.3817190476190476) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.1582460634427849) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6993809523809524) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.14750004177458542) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.9245333333333334) - val-aux/sciknoweval/score/best@16/std:np.float64(0.03779423823528531) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.31384761904761904) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.11509261056785722) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7040333333333333) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.108925053237339) - val-core/sciknoweval/acc/mean@16:np.float64(0.6589285714285714) - val-aux/sciknoweval/acc/std@16:np.float64(0.26147699146732906) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7634047619047619) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.19852720557117806) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5552285714285714) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.2397040092225637) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6592000000000001) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.26073502970265483) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8421285714285713) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.13617544486794253) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.46132857142857137) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.20013008279957237) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6862619047619049) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.20619876382539531) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8954952380952381) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.08197180786429441) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.3817190476190476) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.1582460634427849) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6993809523809524) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.14750004177458542) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.9245333333333334) - val-core/sciknoweval/acc/best@16/std:np.float64(0.03779423823528531) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.31384761904761904) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.11509261056785722) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7040333333333333) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.108925053237339) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.986904761904762) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.036673747734753556) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9979476190476191) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.012620842930515878) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9751857142857142) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.04786735705451152) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9864428571428573) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.037432914347073416) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9998619047619047) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0017967329312416017) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9577666666666667) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.05639443888290546) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9949285714285714) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.022714201842546834) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9332380952380953) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.05880143064108464) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9987619047619049) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.009403419069561486) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9058904761904761) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.050590370282386964) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9997761904761904) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.002838983070955884) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:60 - training/epoch:1 - critic/score/mean:0.5546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.001474093645811081 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.001474093645811081 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:745.625 - response_length/max:1629.0 - response_length/min:179.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:745.625 - response_length_non_aborted/max:1629.0 - response_length_non_aborted/min:179.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.9375 - prompt_length/max:451.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00029304809868335724 - timing_s/agent_loop/generate_sequences/min:np.float64(2.268099818378687) - timing_s/agent_loop/generate_sequences/max:np.float64(14.966608583927155) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.466599339502864) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.966608583927155) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:309 - timing_s/agent_loop/slowest/response_length:1629 - timing_s/gen:21.801297603175044 - timing_s/reward:0.08670702017843723 - timing_s/old_log_prob:3.37482875213027 - timing_s/adv:0.1377221718430519 - timing_s/update_actor:13.325021024793386 - timing_s/step:38.84823939576745 - timing_s/testing:189.37862921506166 - timing_s/save_checkpoint:12.842970063909888 - timing_s/stop_profile:0.00024105235934257507 - timing_per_token_ms/adv:0.0005318279728261195 - timing_per_token_ms/gen:0.11421467730079131 - timing_per_token_ms/update_actor:0.05145590448252003 - perf/total_num_tokens:258960 - perf/time_per_step:38.84823939576745 - perf/throughput:833.2423940820015
(TaskRunner pid=2224264) Training Progress: 60%|██████ | 60/100 [59:23<1:08:47, 103.19s/it]
(TaskRunner pid=2224264) step:61 - global_seqlen/min:26866 - global_seqlen/max:38557 - global_seqlen/minmax_diff:11691 - global_seqlen/balanced_min:32591 - global_seqlen/balanced_max:32615 - global_seqlen/mean:32607.625 - actor/entropy:0.7484701871871948 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.32022327184677124 - training/rollout_probs_diff_mean:0.004327286966145039 - training/rollout_probs_diff_std:0.008308346383273602 - training/rollout_actor_probs_pearson_corr:0.9996346235275269 - rollout_corr/rollout_is_mean:1.0000005960464478 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.5481393347727135e-05 - rollout_corr/rollout_is_max:1.9516299962997437 - rollout_corr/rollout_is_min:0.3159535825252533 - rollout_corr/rollout_is_std:0.03432586044073105 - rollout_corr/rollout_is_eff_sample_size:0.9988231220557214 - rollout_corr/rollout_is_seq_mean:1.0000786781311035 - rollout_corr/rollout_is_seq_std:0.001447982620447874 - rollout_corr/rollout_is_seq_max:1.0040830373764038 - rollout_corr/rollout_is_seq_min:0.9964959025382996 - rollout_corr/rollout_is_seq_max_deviation:0.004083037376403809 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.167487114202231) - rollout_corr/training_log_ppl:np.float64(0.7570779783418402) - rollout_corr/kl:np.float64(0.0007175489161515003) - rollout_corr/k3_kl:np.float64(0.0006208742135527245) - rollout_corr/rollout_ppl:np.float64(2.1659105462022126) - rollout_corr/rollout_log_ppl:np.float64(0.7563604272436351) - rollout_corr/log_ppl_diff:np.float64(0.0007175510982051492) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012418738333508372) - rollout_corr/log_ppl_diff_max:np.float64(0.007266104221343994) - rollout_corr/log_ppl_diff_min:np.float64(-0.003547847270965576) - rollout_corr/ppl_ratio:np.float64(1.0007189356256276) - rollout_corr/chi2_token:np.float64(0.0010508422274142504) - rollout_corr/chi2_seq:np.float64(1.2128563108853996) - actor/pg_loss:np.float64(7.9984194599092e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00012339052045717835) - actor/ppo_kl:np.float64(0.0001892128098228696) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.21570049226284027) - perf/mfu/actor:np.float64(0.1265814073644025) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.56413173675537) - actor/lr:np.float64(1e-06) - training/global_step:61 - training/epoch:1 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0024512207601219416 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0024512207601219416 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:756.95703125 - response_length/max:1627.0 - response_length/min:211.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:756.95703125 - response_length_non_aborted/max:1627.0 - response_length_non_aborted/min:211.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.03125 - prompt_length/max:462.0 - prompt_length/min:177.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00018923170864582062 - timing_s/agent_loop/generate_sequences/min:np.float64(2.607853963971138) - timing_s/agent_loop/generate_sequences/max:np.float64(14.494692988693714) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.566406984020432) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.494692988693714) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:312 - timing_s/agent_loop/slowest/response_length:1627 - timing_s/gen:20.789081823080778 - timing_s/reward:0.07305939309298992 - timing_s/old_log_prob:3.3369578402489424 - timing_s/adv:0.11469490267336369 - timing_s/update_actor:13.50451728142798 - timing_s/step:37.91396317072213 - timing_s/stop_profile:0.00010998733341693878 - timing_per_token_ms/adv:0.00043967822968310206 - timing_per_token_ms/gen:0.10728132181731324 - timing_per_token_ms/update_actor:0.05176901599483242 - perf/total_num_tokens:260861 - perf/time_per_step:37.91396317072213 - perf/throughput:860.0426405747057
(TaskRunner pid=2224264) Training Progress: 61%|██████ | 61/100 [1:00:01<54:21, 83.62s/it]
(TaskRunner pid=2224264) step:62 - global_seqlen/min:27396 - global_seqlen/max:35660 - global_seqlen/minmax_diff:8264 - global_seqlen/balanced_min:31880 - global_seqlen/balanced_max:31902 - global_seqlen/mean:31893.125 - actor/entropy:0.8587515950202942 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2571102976799011 - training/rollout_probs_diff_mean:0.004673279356211424 - training/rollout_probs_diff_std:0.008435208350419998 - training/rollout_actor_probs_pearson_corr:0.9996413588523865 - rollout_corr/rollout_is_mean:0.9998844861984253 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.9304524660110474 - rollout_corr/rollout_is_min:0.5585559010505676 - rollout_corr/rollout_is_std:0.03600391373038292 - rollout_corr/rollout_is_eff_sample_size:0.9987051585549779 - rollout_corr/rollout_is_seq_mean:0.9999316930770874 - rollout_corr/rollout_is_seq_std:0.0016182069666683674 - rollout_corr/rollout_is_seq_max:1.0045205354690552 - rollout_corr/rollout_is_seq_min:0.9909459948539734 - rollout_corr/rollout_is_seq_max_deviation:0.009054005146026611 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.4564432627521455) - rollout_corr/training_log_ppl:np.float64(0.8728480438003317) - rollout_corr/kl:np.float64(0.000761049760463095) - rollout_corr/k3_kl:np.float64(0.000696839270858618) - rollout_corr/rollout_ppl:np.float64(2.454547442961484) - rollout_corr/rollout_log_ppl:np.float64(0.8720869926037267) - rollout_corr/log_ppl_diff:np.float64(0.0007610511966049671) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013679289259016514) - rollout_corr/log_ppl_diff_max:np.float64(0.009840637445449829) - rollout_corr/log_ppl_diff_min:np.float64(-0.0033975839614868164) - rollout_corr/ppl_ratio:np.float64(1.0007626954466105) - rollout_corr/chi2_token:np.float64(0.001287585822865367) - rollout_corr/chi2_seq:np.float64(1.209890850353986) - actor/pg_loss:np.float64(-3.718864172697067e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001223060548909416) - actor/ppo_kl:np.float64(2.9713928825003677e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.29783353582024574) - perf/mfu/actor:np.float64(0.12234924411478575) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.59233093261719) - actor/lr:np.float64(1e-06) - training/global_step:62 - training/epoch:1 - critic/score/mean:0.63671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.63671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006781507283449173 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006781507283449173 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:697.41015625 - response_length/max:1847.0 - response_length/min:168.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:697.41015625 - response_length_non_aborted/max:1847.0 - response_length_non_aborted/min:168.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.25 - prompt_length/max:662.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00017420388758182526 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4971247743815184) - timing_s/agent_loop/generate_sequences/max:np.float64(15.592459442093968) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.807667781009513) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.592459442093968) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:171 - timing_s/agent_loop/slowest/response_length:1847 - timing_s/gen:22.337838161736727 - timing_s/reward:0.07594001106917858 - timing_s/old_log_prob:3.2577473018318415 - timing_s/adv:0.1236040536314249 - timing_s/update_actor:13.773704160004854 - timing_s/step:39.65648138895631 - timing_s/stop_profile:0.0001120399683713913 - timing_per_token_ms/adv:0.00048444630947666974 - timing_per_token_ms/gen:0.12511601607362466 - timing_per_token_ms/update_actor:0.05398382943034296 - perf/total_num_tokens:255145 - perf/time_per_step:39.65648138895631 - perf/throughput:804.2348661039232
(TaskRunner pid=2224264) Training Progress: 62%|██████▏ | 62/100 [1:00:41<44:37, 70.45s/it]
(TaskRunner pid=2224264) step:63 - global_seqlen/min:26919 - global_seqlen/max:40692 - global_seqlen/minmax_diff:13773 - global_seqlen/balanced_min:33436 - global_seqlen/balanced_max:33474 - global_seqlen/mean:33459.75 - actor/entropy:0.7692099809646606 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.39975038170814514 - training/rollout_probs_diff_mean:0.0043107387609779835 - training/rollout_probs_diff_std:0.008506524376571178 - training/rollout_actor_probs_pearson_corr:0.9996923804283142 - rollout_corr/rollout_is_mean:0.999955415725708 - rollout_corr/rollout_is_ratio_fraction_high:5.143874204804888e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.143874204804888e-06 - rollout_corr/rollout_is_max:2.5493264198303223 - rollout_corr/rollout_is_min:0.3577776551246643 - rollout_corr/rollout_is_std:0.03481551632285118 - rollout_corr/rollout_is_eff_sample_size:0.9987891095585573 - rollout_corr/rollout_is_seq_mean:1.000030755996704 - rollout_corr/rollout_is_seq_std:0.0014692960539832711 - rollout_corr/rollout_is_seq_max:1.0053269863128662 - rollout_corr/rollout_is_seq_min:0.9961836934089661 - rollout_corr/rollout_is_seq_max_deviation:0.005326986312866211 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.23876611655578) - rollout_corr/training_log_ppl:np.float64(0.7817316737491637) - rollout_corr/kl:np.float64(0.0004539784086303733) - rollout_corr/k3_kl:np.float64(0.0006596663926075053) - rollout_corr/rollout_ppl:np.float64(2.2376751606352627) - rollout_corr/rollout_log_ppl:np.float64(0.781277688452974) - rollout_corr/log_ppl_diff:np.float64(0.000453985296189785) - rollout_corr/log_ppl_abs_diff:np.float64(0.001180113758891821) - rollout_corr/log_ppl_diff_max:np.float64(0.005360126495361328) - rollout_corr/log_ppl_diff_min:np.float64(-0.004343867301940918) - rollout_corr/ppl_ratio:np.float64(1.0004551277961582) - rollout_corr/chi2_token:np.float64(0.001740993233397603) - rollout_corr/chi2_seq:np.float64(1.6487001434434205) - actor/pg_loss:np.float64(-0.00019339460413902998) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00018608193454383581) - actor/ppo_kl:np.float64(-0.00013630927301377938) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.25826945155858994) - perf/mfu/actor:np.float64(0.12769393570296111) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.44427490234375) - actor/lr:np.float64(1e-06) - training/global_step:63 - training/epoch:1 - critic/score/mean:0.46484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002252373844385147 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002252373844385147 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:759.3984375 - response_length/max:1951.0 - response_length/min:194.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:759.3984375 - response_length_non_aborted/max:1951.0 - response_length_non_aborted/min:194.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.21875 - prompt_length/max:532.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010393001139163971 - timing_s/agent_loop/generate_sequences/min:np.float64(2.39320345595479) - timing_s/agent_loop/generate_sequences/max:np.float64(15.86801179125905) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.712352399430529) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.86801179125905) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:200 - timing_s/agent_loop/slowest/response_length:1951 - timing_s/gen:23.052989764139056 - timing_s/reward:0.07871297188103199 - timing_s/old_log_prob:3.3043668009340763 - timing_s/adv:0.11869384348392487 - timing_s/update_actor:13.733131846413016 - timing_s/step:40.37277623265982 - timing_s/stop_profile:0.00011059269309043884 - timing_per_token_ms/adv:0.0004434202417977005 - timing_per_token_ms/gen:0.1185816783645518 - timing_per_token_ms/update_actor:0.05130467145754607 - perf/total_num_tokens:267678 - perf/time_per_step:40.37277623265982 - perf/throughput:828.7701050623444
(TaskRunner pid=2224264) Training Progress: 63%|██████▎ | 63/100 [1:01:21<37:53, 61.44s/it]
(TaskRunner pid=2224264) step:64 - global_seqlen/min:26306 - global_seqlen/max:34933 - global_seqlen/minmax_diff:8627 - global_seqlen/balanced_min:30904 - global_seqlen/balanced_max:30933 - global_seqlen/mean:30923.0 - actor/entropy:0.7835981845855713 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3175749182701111 - training/rollout_probs_diff_mean:0.004405513871461153 - training/rollout_probs_diff_std:0.008360867388546467 - training/rollout_actor_probs_pearson_corr:0.9996961355209351 - rollout_corr/rollout_is_mean:0.9999273419380188 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1598237506404985e-05 - rollout_corr/rollout_is_max:1.7908501625061035 - rollout_corr/rollout_is_min:0.2933122217655182 - rollout_corr/rollout_is_std:0.03481380268931389 - rollout_corr/rollout_is_eff_sample_size:0.9987892284793369 - rollout_corr/rollout_is_seq_mean:0.9998888373374939 - rollout_corr/rollout_is_seq_std:0.0014235756825655699 - rollout_corr/rollout_is_seq_max:1.0034641027450562 - rollout_corr/rollout_is_seq_min:0.9946786165237427 - rollout_corr/rollout_is_seq_max_deviation:0.005321383476257324 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.228744763880968) - rollout_corr/training_log_ppl:np.float64(0.7778443787246943) - rollout_corr/kl:np.float64(0.000828439747124321) - rollout_corr/k3_kl:np.float64(0.0006999520201134146) - rollout_corr/rollout_ppl:np.float64(2.2268494172021747) - rollout_corr/rollout_log_ppl:np.float64(0.7770159327192232) - rollout_corr/log_ppl_diff:np.float64(0.0008284460054710507) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013873394345864654) - rollout_corr/log_ppl_diff_max:np.float64(0.007387757301330566) - rollout_corr/log_ppl_diff_min:np.float64(-0.0033279061317443848) - rollout_corr/ppl_ratio:np.float64(1.0008300342597067) - rollout_corr/chi2_token:np.float64(0.0011283555068075657) - rollout_corr/chi2_seq:np.float64(1.5243452661670744) - actor/pg_loss:np.float64(-7.703946903347969e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002916122386977804) - actor/ppo_kl:np.float64(8.220684872384254e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.241146732121706) - perf/mfu/actor:np.float64(0.12080489767203606) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.29822540283203) - actor/lr:np.float64(1e-06) - training/global_step:64 - training/epoch:1 - critic/score/mean:0.73046875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.73046875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005282997153699398 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005282997153699398 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:673.59375 - response_length/max:1796.0 - response_length/min:176.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:673.59375 - response_length_non_aborted/max:1796.0 - response_length_non_aborted/min:176.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:292.75 - prompt_length/max:495.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012784451246261597 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1317031513899565) - timing_s/agent_loop/generate_sequences/max:np.float64(14.859888169914484) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.550355381361442) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.859888169914484) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:236 - timing_s/agent_loop/slowest/response_length:1796 - timing_s/gen:21.599143993109465 - timing_s/reward:0.07637070119380951 - timing_s/old_log_prob:3.1958085894584656 - timing_s/adv:0.11453193984925747 - timing_s/update_actor:13.29977210238576 - timing_s/step:38.375933611765504 - timing_s/stop_profile:0.00011328049004077911 - timing_per_token_ms/adv:0.0004629723015605596 - timing_per_token_ms/gen:0.12525599624860512 - timing_per_token_ms/update_actor:0.053761650318475566 - perf/total_num_tokens:247384 - perf/time_per_step:38.375933611765504 - perf/throughput:805.7914711036361
(TaskRunner pid=2224264) Training Progress: 64%|██████▍ | 64/100 [1:01:59<32:43, 54.53s/it]
(TaskRunner pid=2224264) step:65 - global_seqlen/min:24032 - global_seqlen/max:34111 - global_seqlen/minmax_diff:10079 - global_seqlen/balanced_min:29951 - global_seqlen/balanced_max:30559 - global_seqlen/mean:30032.625 - actor/entropy:0.7714234590530396 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.28636491298675537 - training/rollout_probs_diff_mean:0.004574572201818228 - training/rollout_probs_diff_std:0.008538489229977131 - training/rollout_actor_probs_pearson_corr:0.9996505379676819 - rollout_corr/rollout_is_mean:0.9998939633369446 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6290990114212036 - rollout_corr/rollout_is_min:0.5353134870529175 - rollout_corr/rollout_is_std:0.03545675054192543 - rollout_corr/rollout_is_eff_sample_size:0.9987440406221719 - rollout_corr/rollout_is_seq_mean:0.9999128580093384 - rollout_corr/rollout_is_seq_std:0.0015794334467500448 - rollout_corr/rollout_is_seq_max:1.0056021213531494 - rollout_corr/rollout_is_seq_min:0.9935817122459412 - rollout_corr/rollout_is_seq_max_deviation:0.006418287754058838 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.22273992234841) - rollout_corr/training_log_ppl:np.float64(0.7840376188978553) - rollout_corr/kl:np.float64(0.0006947023282570797) - rollout_corr/k3_kl:np.float64(0.0007119409705182989) - rollout_corr/rollout_ppl:np.float64(2.221208442468196) - rollout_corr/rollout_log_ppl:np.float64(0.7833429137244821) - rollout_corr/log_ppl_diff:np.float64(0.0006947051733732224) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013338688295334578) - rollout_corr/log_ppl_diff_max:np.float64(0.0072593092918396) - rollout_corr/log_ppl_diff_min:np.float64(-0.004377126693725586) - rollout_corr/ppl_ratio:np.float64(1.0006962057668716) - rollout_corr/chi2_token:np.float64(0.0014604993630200624) - rollout_corr/chi2_seq:np.float64(0.6011870829388499) - actor/pg_loss:np.float64(0.0001900325296446681) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003155937018846089) - actor/ppo_kl:np.float64(-3.965757715462814e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.28370653092861176) - perf/mfu/actor:np.float64(0.11707659877392018) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.31705093383789) - actor/lr:np.float64(1e-06) - training/global_step:65 - training/epoch:1 - critic/score/mean:0.66796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.66796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0018750634044408798 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0018750634044408798 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:654.92578125 - response_length/max:2451.0 - response_length/min:231.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:654.92578125 - response_length_non_aborted/max:2451.0 - response_length_non_aborted/min:231.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:283.59375 - prompt_length/max:818.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011583603918552399 - timing_s/agent_loop/generate_sequences/min:np.float64(2.9298688657581806) - timing_s/agent_loop/generate_sequences/max:np.float64(18.389138467609882) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.613169322678004) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(18.389138467609882) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:189 - timing_s/agent_loop/slowest/response_length:2451 - timing_s/gen:25.793642738834023 - timing_s/reward:0.07493023946881294 - timing_s/old_log_prob:3.18870216794312 - timing_s/adv:0.1171203050762415 - timing_s/update_actor:13.542755868285894 - timing_s/step:42.80827572196722 - timing_s/stop_profile:0.00011235475540161133 - timing_per_token_ms/adv:0.00048747114627942733 - timing_per_token_ms/gen:0.15384402299183486 - timing_per_token_ms/update_actor:0.0563668505012711 - perf/total_num_tokens:240261 - perf/time_per_step:42.80827572196722 - perf/throughput:701.5611933322662
(TaskRunner pid=2224264) Training Progress: 65%|██████▌ | 65/100 [1:02:42<29:46, 51.03s/it]
(TaskRunner pid=2224264) step:66 - global_seqlen/min:28697 - global_seqlen/max:35089 - global_seqlen/minmax_diff:6392 - global_seqlen/balanced_min:30450 - global_seqlen/balanced_max:30469 - global_seqlen/mean:30460.5 - actor/entropy:0.7473804354667664 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3424738645553589 - training/rollout_probs_diff_mean:0.004387002903968096 - training/rollout_probs_diff_std:0.008388126268982887 - training/rollout_actor_probs_pearson_corr:0.9996775388717651 - rollout_corr/rollout_is_mean:0.9999525547027588 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1807769624283537e-05 - rollout_corr/rollout_is_max:1.7755447626113892 - rollout_corr/rollout_is_min:0.3809736371040344 - rollout_corr/rollout_is_std:0.0348651260137558 - rollout_corr/rollout_is_eff_sample_size:0.9987857797882222 - rollout_corr/rollout_is_seq_mean:0.999984622001648 - rollout_corr/rollout_is_seq_std:0.001510056434199214 - rollout_corr/rollout_is_seq_max:1.0043302774429321 - rollout_corr/rollout_is_seq_min:0.9965217709541321 - rollout_corr/rollout_is_seq_max_deviation:0.004330277442932129 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1832842687144876) - rollout_corr/training_log_ppl:np.float64(0.7604161555645987) - rollout_corr/kl:np.float64(0.0006697535459316839) - rollout_corr/k3_kl:np.float64(0.0006770667089313065) - rollout_corr/rollout_ppl:np.float64(2.18177503393963) - rollout_corr/rollout_log_ppl:np.float64(0.7597463995916769) - rollout_corr/log_ppl_diff:np.float64(0.0006697559729218483) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012636978644877672) - rollout_corr/log_ppl_diff_max:np.float64(0.005221843719482422) - rollout_corr/log_ppl_diff_min:np.float64(-0.0034908056259155273) - rollout_corr/ppl_ratio:np.float64(1.0006710432935506) - rollout_corr/chi2_token:np.float64(0.001379573019221425) - rollout_corr/chi2_seq:np.float64(0.7963614012114704) - actor/pg_loss:np.float64(-9.32483235374093e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00021860310857846343) - actor/ppo_kl:np.float64(3.254798675556003e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2768869362771511) - perf/mfu/actor:np.float64(0.11916675434662388) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.34953784942627) - actor/lr:np.float64(1e-06) - training/global_step:66 - training/epoch:1 - critic/score/mean:0.60546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.60546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0009460975416004658 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0009460975416004658 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:661.640625 - response_length/max:1563.0 - response_length/min:181.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:661.640625 - response_length_non_aborted/max:1563.0 - response_length_non_aborted/min:181.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:290.25 - prompt_length/max:617.0 - prompt_length/min:184.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.000106053426861763 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3329896610230207) - timing_s/agent_loop/generate_sequences/max:np.float64(13.444917742162943) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.728698585859092) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(13.444917742162943) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:229 - timing_s/agent_loop/slowest/response_length:1563 - timing_s/gen:20.854996770620346 - timing_s/reward:0.0732195358723402 - timing_s/old_log_prob:3.1821087524294853 - timing_s/adv:0.1175671610981226 - timing_s/update_actor:13.4094133451581 - timing_s/step:37.7182090356946 - timing_s/stop_profile:0.00011660158634185791 - timing_per_token_ms/adv:0.0004824574493939799 - timing_per_token_ms/gen:0.12312549752403086 - timing_per_token_ms/update_actor:0.05502787768240057 - perf/total_num_tokens:243684 - perf/time_per_step:37.7182090356946 - perf/throughput:807.5807621505498
(TaskRunner pid=2224264) Training Progress: 66%|██████▌ | 66/100 [1:03:20<26:39, 47.05s/it]
(TaskRunner pid=2224264) step:67 - global_seqlen/min:21976 - global_seqlen/max:31921 - global_seqlen/minmax_diff:9945 - global_seqlen/balanced_min:28720 - global_seqlen/balanced_max:28745 - global_seqlen/mean:28734.25 - actor/entropy:0.7726788520812988 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3997594714164734 - training/rollout_probs_diff_mean:0.004599802196025848 - training/rollout_probs_diff_std:0.008748593740165234 - training/rollout_actor_probs_pearson_corr:0.9996384978294373 - rollout_corr/rollout_is_mean:1.0000046491622925 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.9334631360834464e-05 - rollout_corr/rollout_is_max:1.7132740020751953 - rollout_corr/rollout_is_min:0.35777419805526733 - rollout_corr/rollout_is_std:0.03550714626908302 - rollout_corr/rollout_is_eff_sample_size:0.9987409489708702 - rollout_corr/rollout_is_seq_mean:1.0000040531158447 - rollout_corr/rollout_is_seq_std:0.0016247251769527793 - rollout_corr/rollout_is_seq_max:1.005348563194275 - rollout_corr/rollout_is_seq_min:0.9943971037864685 - rollout_corr/rollout_is_seq_max_deviation:0.005602896213531494 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.254453372210264) - rollout_corr/training_log_ppl:np.float64(0.7888535652309656) - rollout_corr/kl:np.float64(0.0006227922752195525) - rollout_corr/k3_kl:np.float64(0.0006862304701940047) - rollout_corr/rollout_ppl:np.float64(2.2529089520685375) - rollout_corr/rollout_log_ppl:np.float64(0.7882307717809454) - rollout_corr/log_ppl_diff:np.float64(0.0006227934500202537) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014135913224890828) - rollout_corr/log_ppl_diff_max:np.float64(0.007272124290466309) - rollout_corr/log_ppl_diff_min:np.float64(-0.004362225532531738) - rollout_corr/ppl_ratio:np.float64(1.000624400563538) - rollout_corr/chi2_token:np.float64(0.001499387202784419) - rollout_corr/chi2_seq:np.float64(1.2466026986949146) - actor/pg_loss:np.float64(-2.0926236175000668e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024252940147562185) - actor/ppo_kl:np.float64(-2.8959387931237757e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.29370710253715515) - perf/mfu/actor:np.float64(0.11386412359551698) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.32535934448242) - actor/lr:np.float64(1e-06) - training/global_step:67 - training/epoch:1 - critic/score/mean:0.6484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.009934777393937111 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.009934777393937111 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:606.1015625 - response_length/max:1545.0 - response_length/min:146.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:606.1015625 - response_length_non_aborted/max:1545.0 - response_length_non_aborted/min:146.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:291.84375 - prompt_length/max:576.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00022127665579319 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8514165561646223) - timing_s/agent_loop/generate_sequences/max:np.float64(13.32896644063294) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.998300799677963) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(13.32896644063294) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:193 - timing_s/agent_loop/slowest/response_length:1545 - timing_s/gen:20.499601708725095 - timing_s/reward:0.07158185541629791 - timing_s/old_log_prob:3.1310275867581367 - timing_s/adv:0.12111156806349754 - timing_s/update_actor:13.336501782760024 - timing_s/step:37.24595209956169 - timing_s/stop_profile:0.00011993572115898132 - timing_per_token_ms/adv:0.0005268606630741082 - timing_per_token_ms/gen:0.1321174108913593 - timing_per_token_ms/update_actor:0.058016573352184346 - perf/total_num_tokens:229874 - perf/time_per_step:37.24595209956169 - perf/throughput:771.4730965445812
(TaskRunner pid=2224264) Training Progress: 67%|██████▋ | 67/100 [1:03:57<24:16, 44.12s/it]
(TaskRunner pid=2224264) step:68 - global_seqlen/min:23879 - global_seqlen/max:35240 - global_seqlen/minmax_diff:11361 - global_seqlen/balanced_min:29992 - global_seqlen/balanced_max:30029 - global_seqlen/mean:30017.25 - actor/entropy:0.7595304250717163 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7168671488761902 - training/rollout_probs_diff_mean:0.004404507111757994 - training/rollout_probs_diff_std:0.008633471094071865 - training/rollout_actor_probs_pearson_corr:0.9996846318244934 - rollout_corr/rollout_is_mean:0.999897301197052 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.8815619114320725e-05 - rollout_corr/rollout_is_max:1.5993539094924927 - rollout_corr/rollout_is_min:0.019239548593759537 - rollout_corr/rollout_is_std:0.03493344411253929 - rollout_corr/rollout_is_eff_sample_size:0.9987809040931451 - rollout_corr/rollout_is_seq_mean:0.9998229742050171 - rollout_corr/rollout_is_seq_std:0.0016775608528405428 - rollout_corr/rollout_is_seq_max:1.005442500114441 - rollout_corr/rollout_is_seq_min:0.9936239123344421 - rollout_corr/rollout_is_seq_max_deviation:0.006376087665557861 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.246089845895767) - rollout_corr/training_log_ppl:np.float64(0.7848675699206069) - rollout_corr/kl:np.float64(0.0009016538949424557) - rollout_corr/k3_kl:np.float64(0.0007493318355500378) - rollout_corr/rollout_ppl:np.float64(2.243979589547962) - rollout_corr/rollout_log_ppl:np.float64(0.7839659146266058) - rollout_corr/log_ppl_diff:np.float64(0.0009016552940011024) - rollout_corr/log_ppl_abs_diff:np.float64(0.001556948758661747) - rollout_corr/log_ppl_diff_max:np.float64(0.012223243713378906) - rollout_corr/log_ppl_diff_min:np.float64(-0.004822075366973877) - rollout_corr/ppl_ratio:np.float64(1.0009038699790835) - rollout_corr/chi2_token:np.float64(0.0011472513433545828) - rollout_corr/chi2_seq:np.float64(2.487598191248253) - actor/pg_loss:np.float64(-4.085572436451912e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00027236666869612236) - actor/ppo_kl:np.float64(5.479786341200388e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2867882438004017) - perf/mfu/actor:np.float64(0.11808746991936632) - perf/max_memory_allocated_gb:np.float64(131.9590311050415) - perf/max_memory_reserved_gb:np.float64(144.658203125) - perf/cpu_memory_used_gb:np.float64(105.29296493530273) - actor/lr:np.float64(1e-06) - training/global_step:68 - training/epoch:1 - critic/score/mean:0.41796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.41796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0006193474982865155 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0006193474982865155 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:622.8203125 - response_length/max:1786.0 - response_length/min:150.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:622.8203125 - response_length_non_aborted/max:1786.0 - response_length_non_aborted/min:150.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:315.21875 - prompt_length/max:704.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014729611575603485 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0367041416466236) - timing_s/agent_loop/generate_sequences/max:np.float64(14.751183833926916) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.172361494318466) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.751183833926916) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:229 - timing_s/agent_loop/slowest/response_length:1786 - timing_s/gen:22.098993657156825 - timing_s/reward:0.10232115164399147 - timing_s/old_log_prob:3.2531425300985575 - timing_s/adv:0.1175235714763403 - timing_s/update_actor:13.300975374877453 - timing_s/step:38.95930672250688 - timing_s/stop_profile:0.00012533552944660187 - timing_per_token_ms/adv:0.0004894001427360113 - timing_per_token_ms/gen:0.13860208512911795 - timing_per_token_ms/update_actor:0.05538888212143623 - perf/total_num_tokens:240138 - perf/time_per_step:38.95930672250688 - perf/throughput:770.4770060155862
(TaskRunner pid=2224264) Training Progress: 68%|██████▊ | 68/100 [1:04:36<22:42, 42.59s/it]
(TaskRunner pid=2224264) step:69 - global_seqlen/min:24189 - global_seqlen/max:45366 - global_seqlen/minmax_diff:21177 - global_seqlen/balanced_min:33420 - global_seqlen/balanced_max:39649 - global_seqlen/mean:34239.875 - actor/entropy:0.7071983814239502 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.38365769386291504 - training/rollout_probs_diff_mean:0.0040749297477304935 - training/rollout_probs_diff_std:0.00811900943517685 - training/rollout_actor_probs_pearson_corr:0.9996503591537476 - rollout_corr/rollout_is_mean:1.000025749206543 - rollout_corr/rollout_is_ratio_fraction_high:5.1385613915044814e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.1385613915044814e-06 - rollout_corr/rollout_is_max:2.4045958518981934 - rollout_corr/rollout_is_min:0.4110872149467468 - rollout_corr/rollout_is_std:0.03386257588863373 - rollout_corr/rollout_is_eff_sample_size:0.9988547581358254 - rollout_corr/rollout_is_seq_mean:1.0000370740890503 - rollout_corr/rollout_is_seq_std:0.0013557960046455264 - rollout_corr/rollout_is_seq_max:1.003797173500061 - rollout_corr/rollout_is_seq_min:0.9964458346366882 - rollout_corr/rollout_is_seq_max_deviation:0.003797173500061035 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.177241276949644) - rollout_corr/training_log_ppl:np.float64(0.7569285913486965) - rollout_corr/kl:np.float64(0.0004947985197105709) - rollout_corr/k3_kl:np.float64(0.0006430674019384242) - rollout_corr/rollout_ppl:np.float64(2.1761399055831134) - rollout_corr/rollout_log_ppl:np.float64(0.7564337924704887) - rollout_corr/log_ppl_diff:np.float64(0.0004947988782078028) - rollout_corr/log_ppl_abs_diff:np.float64(0.001128945965319872) - rollout_corr/log_ppl_diff_max:np.float64(0.004031240940093994) - rollout_corr/log_ppl_diff_min:np.float64(-0.004697918891906738) - rollout_corr/ppl_ratio:np.float64(1.00049584871158) - rollout_corr/chi2_token:np.float64(0.001590353436768055) - rollout_corr/chi2_seq:np.float64(0.8412239120807499) - actor/pg_loss:np.float64(-7.094477768987417e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019783686673235934) - actor/ppo_kl:np.float64(-9.313821129719457e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.25627975165843964) - perf/mfu/actor:np.float64(0.12214039553244277) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.25387191772461) - actor/lr:np.float64(1e-06) - training/global_step:69 - training/epoch:1 - critic/score/mean:0.5703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.010957982391119003 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.010957982391119003 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:760.18359375 - response_length/max:8192.0 - response_length/min:169.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:760.18359375 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:169.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:309.8125 - prompt_length/max:597.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012904033064842224 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1227447520941496) - timing_s/agent_loop/generate_sequences/max:np.float64(60.78352793678641) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.487850829478703) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(60.78352793678641) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:597 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:67.3937315158546 - timing_s/reward:0.07918412238359451 - timing_s/old_log_prob:3.5191430747509003 - timing_s/adv:0.11993566527962685 - timing_s/update_actor:14.636007921770215 - timing_s/step:85.83626089058816 - timing_s/stop_profile:0.00012475252151489258 - timing_per_token_ms/adv:0.00043785084378822515 - timing_per_token_ms/gen:0.3463068210077469 - timing_per_token_ms/update_actor:0.0534318828623433 - perf/total_num_tokens:273919 - perf/time_per_step:85.83626089058816 - perf/throughput:398.8975596647216
(TaskRunner pid=2224264) Training Progress: 69%|██████▉ | 69/100 [1:06:02<28:43, 55.58s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 70}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) We are given a product:
(TaskRunner pid=2224264) **Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This is a molecular structure with the following key features:
(TaskRunner pid=2224264) - A nitrogen atom at the end of a chain (Nc1...n1)
(TaskRunner pid=2224264) - A benzene ring with a substituent: `Cc2ccc(Cl)cc2C(F)(F)F` (i.e., a substituted benzene ring with a chloro group and a trifluoromethyl group)
(TaskRunner pid=2224264) - The overall structure appears to be a **pyrrole-like ring** (Nc1ccn...n1) with a substituent group on the nitrogen.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) We are to identify which of the four options is the **only correct reactant** that would be used in the synthesis of this product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Let’s analyze each option:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - This contains the core structure: `c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1` — which matches the product.
(TaskRunner pid=2224264) - It also has a ketone group `O=C1` and another ketone group `C(=O)N1` — which may be a part of a synthesis pathway (e.g., a diketone or acylated structure).
(TaskRunner pid=2224264) - This seems like a plausible reactant that could be used to form the product via a reaction like a **condensation** or **cyclization**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - This contains a `C(Cl)` group, not the `C(F)(F)F` group we see in the product.
(TaskRunner pid=2224264) - The rest of the structure is not matching the product's ring system or substituents.
(TaskRunner pid=2224264) - Therefore, this is **not a correct reactant**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - Contains a `C(F)(F)F` group (trifluoromethyl), which matches the product.
(TaskRunner pid=2224264) - It also has a `Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1` — which includes a benzene ring with a chloro and trifluoromethyl group.
(TaskRunner pid=2224264) - However, this structure is more of a **complex aromatic ring system**, not a direct match for the pyrrole-like structure in the product.
(TaskRunner pid=2224264) - It may be a reactant in a more complex multi-step synthesis, but it is **not the only correct one**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - Contains the `C(F)(F)F` group and a benzene ring.
(TaskRunner pid=2224264) - However, it does **not** match the nitrogen-containing ring structure of the product.
(TaskRunner pid=2224264) - The presence of a double bond in the side chain (CC=C) is not matching the pyrrole-like ring in the product.
(TaskRunner pid=2224264) - So this is **not the correct reactant**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Conclusion:
(TaskRunner pid=2224264) Only **Option A** contains the **exact ring structure** and **substituents** needed to form the product. It includes the pyrrole-like ring (`c1ccn(...n1`), the chloro-substituted benzene ring with a trifluoromethyl group, and has functional groups that are consistent with a synthetic pathway leading to the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_70
(TaskRunner pid=2224264) step:70 - global_seqlen/min:22032 - global_seqlen/max:35660 - global_seqlen/minmax_diff:13628 - global_seqlen/balanced_min:30368 - global_seqlen/balanced_max:30409 - global_seqlen/mean:30397.25 - actor/entropy:0.8035950064659119 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.22990918159484863 - training/rollout_probs_diff_mean:0.004655633121728897 - training/rollout_probs_diff_std:0.008662798441946507 - training/rollout_actor_probs_pearson_corr:0.9996584057807922 - rollout_corr/rollout_is_mean:1.0000817775726318 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.9939820857835e-06 - rollout_corr/rollout_is_max:1.8103430271148682 - rollout_corr/rollout_is_min:0.4562520384788513 - rollout_corr/rollout_is_std:0.035914406180381775 - rollout_corr/rollout_is_eff_sample_size:0.9987120548475574 - rollout_corr/rollout_is_seq_mean:1.0000861883163452 - rollout_corr/rollout_is_seq_std:0.0016128499992191792 - rollout_corr/rollout_is_seq_max:1.00638747215271 - rollout_corr/rollout_is_seq_min:0.9938380718231201 - rollout_corr/rollout_is_seq_max_deviation:0.006387472152709961 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.279810177627951) - rollout_corr/training_log_ppl:np.float64(0.8013737860601395) - rollout_corr/kl:np.float64(0.0007208169924197705) - rollout_corr/k3_kl:np.float64(0.0007489672824476656) - rollout_corr/rollout_ppl:np.float64(2.278083552606404) - rollout_corr/rollout_log_ppl:np.float64(0.8006529699778184) - rollout_corr/log_ppl_diff:np.float64(0.0007208160823211074) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014672434190288186) - rollout_corr/log_ppl_diff_max:np.float64(0.008777737617492676) - rollout_corr/log_ppl_diff_min:np.float64(-0.005332350730895996) - rollout_corr/ppl_ratio:np.float64(1.000722888391465) - rollout_corr/chi2_token:np.float64(0.0015351453330367804) - rollout_corr/chi2_seq:np.float64(2.378258903743699) - actor/pg_loss:np.float64(-0.000118100899271667) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00018792178138937743) - actor/ppo_kl:np.float64(0.00014457460659045296) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2343890555202961) - perf/mfu/actor:np.float64(0.11821289785724132) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.17000198364258) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6714285714285714) - val-aux/sciknoweval/reward/std@16:np.float64(0.2317466189895333) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7653619047619048) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.1782119464541067) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5797619047619048) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.2136060282647753) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6726619047619047) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.23202999913590713) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8335190476190476) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.11938512568929552) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.49611428571428573) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.1773846797198801) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6981047619047619) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.184153742497439) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8789999999999999) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.0730800489542387) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.42429047619047616) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.13809200842124467) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7115333333333334) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.13558200983257696) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.9054095238095239) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.03841748334380248) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.3662142857142857) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.09495676645373348) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7192904761904763) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.1010536286928432) - val-aux/sciknoweval/score/mean@16:np.float64(0.6714285714285714) - val-aux/sciknoweval/score/std@16:np.float64(0.2317466189895333) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7653619047619048) - val-aux/sciknoweval/score/best@2/std:np.float64(0.1782119464541067) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5797619047619048) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.2136060282647753) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6726619047619047) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.23202999913590713) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8335190476190476) - val-aux/sciknoweval/score/best@4/std:np.float64(0.11938512568929552) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.49611428571428573) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.1773846797198801) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6981047619047619) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.184153742497439) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8789999999999999) - val-aux/sciknoweval/score/best@8/std:np.float64(0.0730800489542387) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.42429047619047616) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.13809200842124467) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7115333333333334) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.13558200983257696) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.9054095238095239) - val-aux/sciknoweval/score/best@16/std:np.float64(0.03841748334380248) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.3662142857142857) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.09495676645373348) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7192904761904763) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.1010536286928432) - val-core/sciknoweval/acc/mean@16:np.float64(0.6714285714285714) - val-aux/sciknoweval/acc/std@16:np.float64(0.2317466189895333) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7653619047619048) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.1782119464541067) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5797619047619048) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.2136060282647753) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6726619047619047) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.23202999913590713) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8335190476190476) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.11938512568929552) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.49611428571428573) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.1773846797198801) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6981047619047619) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.184153742497439) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8789999999999999) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.0730800489542387) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.42429047619047616) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.13809200842124467) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7115333333333334) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.13558200983257696) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.9054095238095239) - val-core/sciknoweval/acc/best@16/std:np.float64(0.03841748334380248) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.3662142857142857) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.09495676645373348) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7192904761904763) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.1010536286928432) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9851190476190477) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.044118643209065715) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9977809523809523) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.015483180068832298) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9722142857142857) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.057096894053364385) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9851714285714287) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.04372250124311388) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9998476190476191) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.001196522485392713) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9513904761904762) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.0690690442763583) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9942000000000001) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.027176785822535302) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.920295238095238) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.07418068438997008) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9985380952380953) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.012138625400942288) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.8851476190476191) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.06448749000951351) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9998714285714286) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.002292661410582752) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:70 - training/epoch:1 - critic/score/mean:0.59765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.01194525696337223 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.01194525696337223 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:651.6953125 - response_length/max:1845.0 - response_length/min:145.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:651.6953125 - response_length_non_aborted/max:1845.0 - response_length_non_aborted/min:145.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:298.21875 - prompt_length/max:701.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011080317199230194 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8444180563092232) - timing_s/agent_loop/generate_sequences/max:np.float64(14.495014188811183) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.144595855919761) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.495014188811183) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:299 - timing_s/agent_loop/slowest/response_length:1676 - timing_s/gen:21.431634962558746 - timing_s/reward:0.07384154014289379 - timing_s/old_log_prob:3.262227326631546 - timing_s/adv:0.11903642117977142 - timing_s/update_actor:13.540440015494823 - timing_s/step:38.51284538023174 - timing_s/testing:189.72432128712535 - timing_s/save_checkpoint:13.012803422287107 - timing_s/stop_profile:0.00014807283878326416 - timing_per_token_ms/adv:0.0004895032493883963 - timing_per_token_ms/gen:0.12846083509691517 - timing_per_token_ms/update_actor:0.05568118832910388 - perf/total_num_tokens:243178 - perf/time_per_step:38.51284538023174 - perf/throughput:789.2756221954612
(TaskRunner pid=2224264) Training Progress: 70%|███████ | 70/100 [1:10:04<55:39, 111.31s/it]
(TaskRunner pid=2224264) step:71 - global_seqlen/min:26528 - global_seqlen/max:38219 - global_seqlen/minmax_diff:11691 - global_seqlen/balanced_min:31790 - global_seqlen/balanced_max:32177 - global_seqlen/mean:31863.5 - actor/entropy:0.7618645429611206 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.21302002668380737 - training/rollout_probs_diff_mean:0.004489706829190254 - training/rollout_probs_diff_std:0.008394083939492702 - training/rollout_actor_probs_pearson_corr:0.9996015429496765 - rollout_corr/rollout_is_mean:0.9997878670692444 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.690140925347805e-05 - rollout_corr/rollout_is_max:1.6874182224273682 - rollout_corr/rollout_is_min:0.4357602596282959 - rollout_corr/rollout_is_std:0.034910403192043304 - rollout_corr/rollout_is_eff_sample_size:0.9987822122017861 - rollout_corr/rollout_is_seq_mean:0.9997093677520752 - rollout_corr/rollout_is_seq_std:0.0016728744376450777 - rollout_corr/rollout_is_seq_max:1.005326747894287 - rollout_corr/rollout_is_seq_min:0.9935998916625977 - rollout_corr/rollout_is_seq_max_deviation:0.006400108337402344 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.216639978811145) - rollout_corr/training_log_ppl:np.float64(0.7745629880810156) - rollout_corr/kl:np.float64(0.000979817594189747) - rollout_corr/k3_kl:np.float64(0.0007290477265087247) - rollout_corr/rollout_ppl:np.float64(2.2143593542277813) - rollout_corr/rollout_log_ppl:np.float64(0.7735831652535126) - rollout_corr/log_ppl_diff:np.float64(0.0009798228275030851) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014419290237128735) - rollout_corr/log_ppl_diff_max:np.float64(0.008339285850524902) - rollout_corr/log_ppl_diff_min:np.float64(-0.0046994686126708984) - rollout_corr/ppl_ratio:np.float64(1.0009817637037486) - rollout_corr/chi2_token:np.float64(0.0009503676556050777) - rollout_corr/chi2_seq:np.float64(0.568886311724782) - actor/pg_loss:np.float64(5.502370186150074e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019256132691225503) - actor/ppo_kl:np.float64(4.591261260600277e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.20790256932377815) - perf/mfu/actor:np.float64(0.12276829314252474) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(109.04192352294922) - actor/lr:np.float64(1e-06) - training/global_step:71 - training/epoch:1 - critic/score/mean:0.59765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0034556337632238865 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0034556337632238865 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:693.359375 - response_length/max:2588.0 - response_length/min:171.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:693.359375 - response_length_non_aborted/max:2588.0 - response_length_non_aborted/min:171.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.375 - prompt_length/max:666.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.34839928150177e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.169468689709902) - timing_s/agent_loop/generate_sequences/max:np.float64(19.67856295593083) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.767557479441166) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(19.67856295593083) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:264 - timing_s/agent_loop/slowest/response_length:2588 - timing_s/gen:26.15543474815786 - timing_s/reward:0.07458089478313923 - timing_s/old_log_prob:3.3945595882833004 - timing_s/adv:0.11478821560740471 - timing_s/update_actor:13.66449068300426 - timing_s/step:43.516275111585855 - timing_s/stop_profile:3.4091994166374207e-05 - timing_per_token_ms/adv:0.0004503123307522899 - timing_per_token_ms/gen:0.14735456196145272 - timing_per_token_ms/update_actor:0.05360557802424506 - perf/total_num_tokens:254908 - perf/time_per_step:43.516275111585855 - perf/throughput:732.220299607321
(TaskRunner pid=2224264) Training Progress: 71%|███████ | 71/100 [1:10:47<43:58, 90.98s/it]
(TaskRunner pid=2224264) step:72 - global_seqlen/min:26255 - global_seqlen/max:35299 - global_seqlen/minmax_diff:9044 - global_seqlen/balanced_min:30687 - global_seqlen/balanced_max:30704 - global_seqlen/mean:30699.25 - actor/entropy:0.7660148739814758 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2309356927871704 - training/rollout_probs_diff_mean:0.0045791687443852425 - training/rollout_probs_diff_std:0.008554509840905666 - training/rollout_actor_probs_pearson_corr:0.999636709690094 - rollout_corr/rollout_is_mean:0.9999493360519409 - rollout_corr/rollout_is_ratio_fraction_high:5.8084829106519464e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.0826847553253174 - rollout_corr/rollout_is_min:0.5040711760520935 - rollout_corr/rollout_is_std:0.03556250035762787 - rollout_corr/rollout_is_eff_sample_size:0.9987369060711254 - rollout_corr/rollout_is_seq_mean:0.9999403953552246 - rollout_corr/rollout_is_seq_std:0.0016658001113682985 - rollout_corr/rollout_is_seq_max:1.006662368774414 - rollout_corr/rollout_is_seq_min:0.99498051404953 - rollout_corr/rollout_is_seq_max_deviation:0.0066623687744140625 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.172296098433435) - rollout_corr/training_log_ppl:np.float64(0.7561264133546501) - rollout_corr/kl:np.float64(0.0008874626383423845) - rollout_corr/k3_kl:np.float64(0.0007379499492117247) - rollout_corr/rollout_ppl:np.float64(2.170314473565668) - rollout_corr/rollout_log_ppl:np.float64(0.7552389547927305) - rollout_corr/log_ppl_diff:np.float64(0.0008874585619196296) - rollout_corr/log_ppl_abs_diff:np.float64(0.001500150072388351) - rollout_corr/log_ppl_diff_max:np.float64(0.00498625636100769) - rollout_corr/log_ppl_diff_min:np.float64(-0.0046173930168151855) - rollout_corr/ppl_ratio:np.float64(1.0008892305195332) - rollout_corr/chi2_token:np.float64(0.0011937415692955256) - rollout_corr/chi2_seq:np.float64(0.66025594715029) - actor/pg_loss:np.float64(3.53293726220727e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002114741264449549) - actor/ppo_kl:np.float64(0.00018003375489444862) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2194691300392151) - perf/mfu/actor:np.float64(0.1192681330197729) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.72951126098633) - actor/lr:np.float64(1e-06) - training/global_step:72 - training/epoch:1 - critic/score/mean:0.69921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.69921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007243903819471598 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007243903819471598 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:672.5078125 - response_length/max:1734.0 - response_length/min:186.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:672.5078125 - response_length_non_aborted/max:1734.0 - response_length_non_aborted/min:186.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.84375 - prompt_length/max:620.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.6632100343704224e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.31886431761086) - timing_s/agent_loop/generate_sequences/max:np.float64(14.253411900252104) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.534527560528659) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.253411900252104) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:244 - timing_s/agent_loop/slowest/response_length:1734 - timing_s/gen:21.547888031229377 - timing_s/reward:0.07392375729978085 - timing_s/old_log_prob:3.2580875027924776 - timing_s/adv:0.11826586164534092 - timing_s/update_actor:13.605389777570963 - timing_s/step:38.68964970111847 - timing_s/stop_profile:0.00017290934920310974 - timing_per_token_ms/adv:0.00048155028887245175 - timing_per_token_ms/gen:0.12516053502648306 - timing_per_token_ms/update_actor:0.055397891550978295 - perf/total_num_tokens:245594 - perf/time_per_step:38.68964970111847 - perf/throughput:793.4744883232304
(TaskRunner pid=2224264) Training Progress: 72%|███████▏ | 72/100 [1:11:26<35:08, 75.31s/it]
(TaskRunner pid=2224264) step:73 - global_seqlen/min:32171 - global_seqlen/max:38705 - global_seqlen/minmax_diff:6534 - global_seqlen/balanced_min:33645 - global_seqlen/balanced_max:39741 - global_seqlen/mean:34425.125 - actor/entropy:0.763630211353302 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.30194756388664246 - training/rollout_probs_diff_mean:0.004307533614337444 - training/rollout_probs_diff_std:0.008280006237328053 - training/rollout_actor_probs_pearson_corr:0.999650776386261 - rollout_corr/rollout_is_mean:0.9999016523361206 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.910024017590331e-06 - rollout_corr/rollout_is_max:1.6241782903671265 - rollout_corr/rollout_is_min:0.4921497702598572 - rollout_corr/rollout_is_std:0.034490425139665604 - rollout_corr/rollout_is_eff_sample_size:0.9988115860890442 - rollout_corr/rollout_is_seq_mean:0.9998833537101746 - rollout_corr/rollout_is_seq_std:0.0013762600719928741 - rollout_corr/rollout_is_seq_max:1.0045596361160278 - rollout_corr/rollout_is_seq_min:0.9957677125930786 - rollout_corr/rollout_is_seq_max_deviation:0.004559636116027832 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2688772692345083) - rollout_corr/training_log_ppl:np.float64(0.8037596073554596) - rollout_corr/kl:np.float64(0.0007620832186034932) - rollout_corr/k3_kl:np.float64(0.0006744360291293106) - rollout_corr/rollout_ppl:np.float64(2.267069778870791) - rollout_corr/rollout_log_ppl:np.float64(0.8029975256649777) - rollout_corr/log_ppl_diff:np.float64(0.0007620816904818639) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012150325201218948) - rollout_corr/log_ppl_diff_max:np.float64(0.006102800369262695) - rollout_corr/log_ppl_diff_min:np.float64(-0.003519415855407715) - rollout_corr/ppl_ratio:np.float64(1.0007633492350578) - rollout_corr/chi2_token:np.float64(0.0011786054819822311) - rollout_corr/chi2_seq:np.float64(0.6752500396687537) - actor/pg_loss:np.float64(7.905624806880951e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014205676188794314) - actor/ppo_kl:np.float64(1.4047067626421494e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2183140367269516) - perf/mfu/actor:np.float64(0.12278116839994453) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.55410766601562) - actor/lr:np.float64(1e-06) - training/global_step:73 - training/epoch:1 - critic/score/mean:0.578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005568580701947212 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005568580701947212 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:795.56640625 - response_length/max:8192.0 - response_length/min:236.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:795.56640625 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:236.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:280.21875 - prompt_length/max:557.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014955177903175354 - timing_s/agent_loop/generate_sequences/min:np.float64(2.818276045843959) - timing_s/agent_loop/generate_sequences/max:np.float64(61.54518798738718) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.143103411246557) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(61.54518798738718) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:314 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:68.7704401165247 - timing_s/reward:0.0806842241436243 - timing_s/old_log_prob:3.6475528068840504 - timing_s/adv:0.12522803619503975 - timing_s/update_actor:14.82589746080339 - timing_s/step:87.5357283577323 - timing_s/stop_profile:0.00011074915528297424 - timing_per_token_ms/adv:0.0004547116248490011 - timing_per_token_ms/gen:0.3376644986449547 - timing_per_token_ms/update_actor:0.053833854854569846 - perf/total_num_tokens:275401 - perf/time_per_step:87.5357283577323 - perf/throughput:393.26941862315726
(TaskRunner pid=2224264) Training Progress: 73%|███████▎ | 73/100 [1:12:54<35:32, 78.99s/it]
(TaskRunner pid=2224264) step:74 - global_seqlen/min:29986 - global_seqlen/max:43985 - global_seqlen/minmax_diff:13999 - global_seqlen/balanced_min:33665 - global_seqlen/balanced_max:33695 - global_seqlen/mean:33681.375 - actor/entropy:0.761839747428894 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.19873207807540894 - training/rollout_probs_diff_mean:0.004497237037867308 - training/rollout_probs_diff_std:0.008453729562461376 - training/rollout_actor_probs_pearson_corr:0.999616265296936 - rollout_corr/rollout_is_mean:0.9999936819076538 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6013860702514648 - rollout_corr/rollout_is_min:0.606749951839447 - rollout_corr/rollout_is_std:0.03470577299594879 - rollout_corr/rollout_is_eff_sample_size:0.9987970773134115 - rollout_corr/rollout_is_seq_mean:0.9999818801879883 - rollout_corr/rollout_is_seq_std:0.0015479137655347586 - rollout_corr/rollout_is_seq_max:1.0074827671051025 - rollout_corr/rollout_is_seq_min:0.9962047338485718 - rollout_corr/rollout_is_seq_max_deviation:0.007482767105102539 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2138938223943114) - rollout_corr/training_log_ppl:np.float64(0.7749610797036439) - rollout_corr/kl:np.float64(0.0006632767860637601) - rollout_corr/k3_kl:np.float64(0.0006568010118712664) - rollout_corr/rollout_ppl:np.float64(2.212330491282046) - rollout_corr/rollout_log_ppl:np.float64(0.7742978031164967) - rollout_corr/log_ppl_diff:np.float64(0.0006632765871472657) - rollout_corr/log_ppl_abs_diff:np.float64(0.001370829122606665) - rollout_corr/log_ppl_diff_max:np.float64(0.004554629325866699) - rollout_corr/log_ppl_diff_min:np.float64(-0.006599485874176025) - rollout_corr/ppl_ratio:np.float64(1.0006648083217442) - rollout_corr/chi2_token:np.float64(0.0013132649473845959) - rollout_corr/chi2_seq:np.float64(1.9479922475293279) - actor/pg_loss:np.float64(-5.3111580200493336e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014415476721296727) - actor/ppo_kl:np.float64(1.6327075574196215e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.20973877236247063) - perf/mfu/actor:np.float64(0.12778629328477945) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.42087173461914) - actor/lr:np.float64(1e-06) - training/global_step:74 - training/epoch:1 - critic/score/mean:0.6796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005659057758748531 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005659057758748531 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:779.13671875 - response_length/max:2249.0 - response_length/min:207.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:779.13671875 - response_length_non_aborted/max:2249.0 - response_length_non_aborted/min:207.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.40625 - prompt_length/max:665.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010413303971290588 - timing_s/agent_loop/generate_sequences/min:np.float64(2.6374386623501778) - timing_s/agent_loop/generate_sequences/max:np.float64(18.268491968512535) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.814450513171323) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(18.268491968512535) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:291 - timing_s/agent_loop/slowest/response_length:2249 - timing_s/gen:25.160439206287265 - timing_s/reward:0.07928537577390671 - timing_s/old_log_prob:3.3276401832699776 - timing_s/adv:0.12078331597149372 - timing_s/update_actor:13.96530625037849 - timing_s/step:42.73991168104112 - timing_s/stop_profile:0.00011014938354492188 - timing_per_token_ms/adv:0.0004482570707531006 - timing_per_token_ms/gen:0.12614341396621492 - timing_per_token_ms/update_actor:0.05182874159078456 - perf/total_num_tokens:269451 - perf/time_per_step:42.73991168104112 - perf/throughput:788.0543893341883
(TaskRunner pid=2224264) Training Progress: 74%|███████▍ | 74/100 [1:13:36<29:31, 68.13s/it]
(TaskRunner pid=2224264) step:75 - global_seqlen/min:24487 - global_seqlen/max:40274 - global_seqlen/minmax_diff:15787 - global_seqlen/balanced_min:34105 - global_seqlen/balanced_max:34135 - global_seqlen/mean:34124.625 - actor/entropy:0.8144033551216125 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.17917940020561218 - training/rollout_probs_diff_mean:0.00457530515268445 - training/rollout_probs_diff_std:0.008413956500589848 - training/rollout_actor_probs_pearson_corr:0.9996695518493652 - rollout_corr/rollout_is_mean:1.0000370740890503 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.187879196455469e-06 - rollout_corr/rollout_is_max:1.7824959754943848 - rollout_corr/rollout_is_min:0.4731596112251282 - rollout_corr/rollout_is_std:0.035698000341653824 - rollout_corr/rollout_is_eff_sample_size:0.9987273934949555 - rollout_corr/rollout_is_seq_mean:1.0000509023666382 - rollout_corr/rollout_is_seq_std:0.0015275044133886695 - rollout_corr/rollout_is_seq_max:1.004511833190918 - rollout_corr/rollout_is_seq_min:0.9952264428138733 - rollout_corr/rollout_is_seq_max_deviation:0.004773557186126709 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2711348473094404) - rollout_corr/training_log_ppl:np.float64(0.803941803984344) - rollout_corr/kl:np.float64(0.0005384247337580916) - rollout_corr/k3_kl:np.float64(0.000659077487966897) - rollout_corr/rollout_ppl:np.float64(2.2699187486432493) - rollout_corr/rollout_log_ppl:np.float64(0.803403376834467) - rollout_corr/log_ppl_diff:np.float64(0.0005384271498769522) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012698217760771513) - rollout_corr/log_ppl_diff_max:np.float64(0.0053713321685791016) - rollout_corr/log_ppl_diff_min:np.float64(-0.003410249948501587) - rollout_corr/ppl_ratio:np.float64(1.0005397242493927) - rollout_corr/chi2_token:np.float64(0.0015677101910114288) - rollout_corr/chi2_seq:np.float64(1.5271832724101841) - actor/pg_loss:np.float64(-4.3374369852244854e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(5.082515417598188e-05) - actor/ppo_kl:np.float64(-5.413583712154946e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.17228768952190876) - perf/mfu/actor:np.float64(0.13115778980210233) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.54808044433594) - actor/lr:np.float64(1e-06) - training/global_step:75 - training/epoch:1 - critic/score/mean:0.73828125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.73828125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0018546667415648699 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0018546667415648699 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:752.95703125 - response_length/max:1483.0 - response_length/min:199.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:752.95703125 - response_length_non_aborted/max:1483.0 - response_length_non_aborted/min:199.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:313.4375 - prompt_length/max:608.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011150166392326355 - timing_s/agent_loop/generate_sequences/min:np.float64(2.5929522551596165) - timing_s/agent_loop/generate_sequences/max:np.float64(14.319374568760395) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.897164289628563) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.319374568760395) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:277 - timing_s/agent_loop/slowest/response_length:1483 - timing_s/gen:21.023408722132444 - timing_s/reward:0.08007885701954365 - timing_s/old_log_prob:3.324468068778515 - timing_s/adv:0.12420106492936611 - timing_s/update_actor:13.739917827770114 - timing_s/step:38.37722675316036 - timing_s/stop_profile:0.00010982714593410492 - timing_per_token_ms/adv:0.0004549539552792379 - timing_per_token_ms/gen:0.109066901446549 - timing_per_token_ms/update_actor:0.050329922408561684 - perf/total_num_tokens:272997 - perf/time_per_step:38.37722675316036 - perf/throughput:889.189446113113
(TaskRunner pid=2224264) Training Progress: 75%|███████▌ | 75/100 [1:14:15<24:40, 59.22s/it]
(TaskRunner pid=2224264) step:76 - global_seqlen/min:31915 - global_seqlen/max:42520 - global_seqlen/minmax_diff:10605 - global_seqlen/balanced_min:37284 - global_seqlen/balanced_max:37334 - global_seqlen/mean:37307.375 - actor/entropy:0.746766984462738 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.48574504256248474 - training/rollout_probs_diff_mean:0.004121627192944288 - training/rollout_probs_diff_std:0.008096432313323021 - training/rollout_actor_probs_pearson_corr:0.999716579914093 - rollout_corr/rollout_is_mean:1.0002018213272095 - rollout_corr/rollout_is_ratio_fraction_high:4.438073119672481e-06 - rollout_corr/rollout_is_ratio_fraction_low:4.438073119672481e-06 - rollout_corr/rollout_is_max:2.379199504852295 - rollout_corr/rollout_is_min:0.4145638346672058 - rollout_corr/rollout_is_std:0.03402938321232796 - rollout_corr/rollout_is_eff_sample_size:0.9988438161069871 - rollout_corr/rollout_is_seq_mean:1.000188946723938 - rollout_corr/rollout_is_seq_std:0.0011295655276626348 - rollout_corr/rollout_is_seq_max:1.0031288862228394 - rollout_corr/rollout_is_seq_min:0.9967604875564575 - rollout_corr/rollout_is_seq_max_deviation:0.0032395124435424805 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.173908702097833) - rollout_corr/training_log_ppl:np.float64(0.7561200592317618) - rollout_corr/kl:np.float64(0.0004855001487325261) - rollout_corr/k3_kl:np.float64(0.0005892427249136745) - rollout_corr/rollout_ppl:np.float64(2.1727346065454185) - rollout_corr/rollout_log_ppl:np.float64(0.755634555243887) - rollout_corr/log_ppl_diff:np.float64(0.00048550398787483573) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009925910853780806) - rollout_corr/log_ppl_diff_max:np.float64(0.0035272836685180664) - rollout_corr/log_ppl_diff_min:np.float64(-0.002400696277618408) - rollout_corr/ppl_ratio:np.float64(1.0004863012582064) - rollout_corr/chi2_token:np.float64(0.0013905672822147608) - rollout_corr/chi2_seq:np.float64(0.9606744900811464) - actor/pg_loss:np.float64(-2.3183762095868587e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(9.338781410406227e-05) - actor/ppo_kl:np.float64(8.314622859639798e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.18520821258425713) - perf/mfu/actor:np.float64(0.13935883033564134) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.53511428833008) - actor/lr:np.float64(1e-06) - training/global_step:76 - training/epoch:1 - critic/score/mean:0.50390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.50390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00035671013756655157 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00035671013756655157 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:880.16796875 - response_length/max:2522.0 - response_length/min:204.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:880.16796875 - response_length_non_aborted/max:2522.0 - response_length_non_aborted/min:204.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:285.6875 - prompt_length/max:526.0 - prompt_length/min:182.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001479722559452057 - timing_s/agent_loop/generate_sequences/min:np.float64(2.7096176706254482) - timing_s/agent_loop/generate_sequences/max:np.float64(20.600152047351003) - timing_s/agent_loop/generate_sequences/mean:np.float64(10.33266173733864) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(20.600152047351003) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:196 - timing_s/agent_loop/slowest/response_length:2522 - timing_s/gen:27.267525793984532 - timing_s/reward:0.08974926918745041 - timing_s/old_log_prob:3.3883168008178473 - timing_s/adv:0.11706031486392021 - timing_s/update_actor:13.990974662825465 - timing_s/step:44.949168203398585 - timing_s/stop_profile:0.00010977871716022491 - timing_per_token_ms/adv:0.0003922157310180635 - timing_per_token_ms/gen:0.12101527937221027 - timing_per_token_ms/update_actor:0.0468773756624041 - perf/total_num_tokens:298459 - perf/time_per_step:44.949168203398585 - perf/throughput:829.990331104263
(TaskRunner pid=2224264) Training Progress: 76%|███████▌ | 76/100 [1:15:00<21:59, 54.96s/it]
(TaskRunner pid=2224264) step:77 - global_seqlen/min:27604 - global_seqlen/max:39041 - global_seqlen/minmax_diff:11437 - global_seqlen/balanced_min:34288 - global_seqlen/balanced_max:34332 - global_seqlen/mean:34317.375 - actor/entropy:0.8058213591575623 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.28673720359802246 - training/rollout_probs_diff_mean:0.004518347326666117 - training/rollout_probs_diff_std:0.008365931920707226 - training/rollout_actor_probs_pearson_corr:0.999616265296936 - rollout_corr/rollout_is_mean:0.9999557733535767 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6204079389572144 - rollout_corr/rollout_is_min:0.6416282057762146 - rollout_corr/rollout_is_std:0.03523077070713043 - rollout_corr/rollout_is_eff_sample_size:0.9987602126485932 - rollout_corr/rollout_is_seq_mean:0.9999356269836426 - rollout_corr/rollout_is_seq_std:0.0013528199633583426 - rollout_corr/rollout_is_seq_max:1.0050373077392578 - rollout_corr/rollout_is_seq_min:0.9947447180747986 - rollout_corr/rollout_is_seq_max_deviation:0.005255281925201416 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.28120540920645) - rollout_corr/training_log_ppl:np.float64(0.8050968096358702) - rollout_corr/kl:np.float64(0.0006265352302827765) - rollout_corr/k3_kl:np.float64(0.0006796217078317568) - rollout_corr/rollout_ppl:np.float64(2.279738489538431) - rollout_corr/rollout_log_ppl:np.float64(0.8044702711631544) - rollout_corr/log_ppl_diff:np.float64(0.0006265384727157652) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010923692607320845) - rollout_corr/log_ppl_diff_max:np.float64(0.005904078483581543) - rollout_corr/log_ppl_diff_min:np.float64(-0.0028671622276306152) - rollout_corr/ppl_ratio:np.float64(1.0006276031490415) - rollout_corr/chi2_token:np.float64(0.0014646041672676802) - rollout_corr/chi2_seq:np.float64(0.7338430834934115) - actor/pg_loss:np.float64(0.00012833683285862207) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00017851591564976843) - actor/ppo_kl:np.float64(-6.952557651018765e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2424900233745575) - perf/mfu/actor:np.float64(0.13009512944939702) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.56500244140625) - actor/lr:np.float64(1e-06) - training/global_step:77 - training/epoch:1 - critic/score/mean:0.671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007992738857865334 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007992738857865334 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:755.32421875 - response_length/max:2071.0 - response_length/min:186.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:755.32421875 - response_length_non_aborted/max:2071.0 - response_length_non_aborted/min:186.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:317.09375 - prompt_length/max:716.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013962015509605408 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1678621973842382) - timing_s/agent_loop/generate_sequences/max:np.float64(17.2276831548661) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.9278243403023225) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(17.2276831548661) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:272 - timing_s/agent_loop/slowest/response_length:2071 - timing_s/gen:24.56353418342769 - timing_s/reward:0.08003837987780571 - timing_s/old_log_prob:3.337373360991478 - timing_s/adv:0.12115020304918289 - timing_s/update_actor:13.919541602954268 - timing_s/step:42.11578721739352 - timing_s/stop_profile:0.000111408531665802 - timing_per_token_ms/adv:0.00044128594862363047 - timing_per_token_ms/gen:0.12703326998147366 - timing_per_token_ms/update_actor:0.05070150908597419 - perf/total_num_tokens:274539 - perf/time_per_step:42.11578721739352 - perf/throughput:814.8339914165767
(TaskRunner pid=2224264) Training Progress: 77%|███████▋ | 77/100 [1:15:42<19:35, 51.12s/it]
(TaskRunner pid=2224264) step:78 - global_seqlen/min:28938 - global_seqlen/max:37974 - global_seqlen/minmax_diff:9036 - global_seqlen/balanced_min:34229 - global_seqlen/balanced_max:34251 - global_seqlen/mean:34245.0 - actor/entropy:0.7469825148582458 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.48943376541137695 - training/rollout_probs_diff_mean:0.004324706271290779 - training/rollout_probs_diff_std:0.008407223038375378 - training/rollout_actor_probs_pearson_corr:0.9996141195297241 - rollout_corr/rollout_is_mean:1.0000590085983276 - rollout_corr/rollout_is_ratio_fraction_high:4.9409068196837325e-06 - rollout_corr/rollout_is_ratio_fraction_low:9.881813639367465e-06 - rollout_corr/rollout_is_max:4.681254863739014 - rollout_corr/rollout_is_min:0.4222910404205322 - rollout_corr/rollout_is_std:0.03437097743153572 - rollout_corr/rollout_is_eff_sample_size:0.9988201488429446 - rollout_corr/rollout_is_seq_mean:1.00004243850708 - rollout_corr/rollout_is_seq_std:0.001362150302156806 - rollout_corr/rollout_is_seq_max:1.0039869546890259 - rollout_corr/rollout_is_seq_min:0.9943282008171082 - rollout_corr/rollout_is_seq_max_deviation:0.005671799182891846 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1253380407579243) - rollout_corr/training_log_ppl:np.float64(0.7371736962813884) - rollout_corr/kl:np.float64(0.0005827662733111083) - rollout_corr/k3_kl:np.float64(0.0006460418212554941) - rollout_corr/rollout_ppl:np.float64(2.1241166265681386) - rollout_corr/rollout_log_ppl:np.float64(0.7365909329382703) - rollout_corr/log_ppl_diff:np.float64(0.0005827633431181312) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012038786662742496) - rollout_corr/log_ppl_diff_max:np.float64(0.006325364112854004) - rollout_corr/log_ppl_diff_min:np.float64(-0.003412604331970215) - rollout_corr/ppl_ratio:np.float64(1.0005839837249368) - rollout_corr/chi2_token:np.float64(0.0014792552683502436) - rollout_corr/chi2_seq:np.float64(2.1139756981283426) - actor/pg_loss:np.float64(0.0001295657129958272) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001085545256955811) - actor/ppo_kl:np.float64(1.3940017076219391e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.20791660994291306) - perf/mfu/actor:np.float64(0.1296498704894837) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.49341583251953) - actor/lr:np.float64(1e-06) - training/global_step:78 - training/epoch:1 - critic/score/mean:0.671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0018497519195079803 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0018497519195079803 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:790.59375 - response_length/max:1840.0 - response_length/min:181.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:790.59375 - response_length_non_aborted/max:1840.0 - response_length_non_aborted/min:181.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.5625 - prompt_length/max:449.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010417215526103973 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2816403061151505) - timing_s/agent_loop/generate_sequences/max:np.float64(16.65826540067792) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.189224266308884) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.65826540067792) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:189 - timing_s/agent_loop/slowest/response_length:1840 - timing_s/gen:23.61698125116527 - timing_s/reward:0.08289889059960842 - timing_s/old_log_prob:3.274001955986023 - timing_s/adv:0.12328475899994373 - timing_s/update_actor:13.82346528582275 - timing_s/step:41.00753286853433 - timing_s/stop_profile:0.00011822767555713654 - timing_per_token_ms/adv:0.000450010070813052 - timing_per_token_ms/gen:0.11668930220149645 - timing_per_token_ms/update_actor:0.050457969359843584 - perf/total_num_tokens:273960 - perf/time_per_step:41.00753286853433 - perf/throughput:835.0904725184452
(TaskRunner pid=2224264) Training Progress: 78%|███████▊ | 78/100 [1:16:23<17:38, 48.10s/it]
(TaskRunner pid=2224264) step:79 - global_seqlen/min:31052 - global_seqlen/max:40489 - global_seqlen/minmax_diff:9437 - global_seqlen/balanced_min:35267 - global_seqlen/balanced_max:35291 - global_seqlen/mean:35282.375 - actor/entropy:0.7419886589050293 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3267946243286133 - training/rollout_probs_diff_mean:0.004350726492702961 - training/rollout_probs_diff_std:0.008377747610211372 - training/rollout_actor_probs_pearson_corr:0.9996976256370544 - rollout_corr/rollout_is_mean:1.0000663995742798 - rollout_corr/rollout_is_ratio_fraction_high:4.6575100896006916e-06 - rollout_corr/rollout_is_ratio_fraction_low:9.315020179201383e-06 - rollout_corr/rollout_is_max:2.486323833465576 - rollout_corr/rollout_is_min:0.3386184871196747 - rollout_corr/rollout_is_std:0.034230221062898636 - rollout_corr/rollout_is_eff_sample_size:0.9988299010470403 - rollout_corr/rollout_is_seq_mean:1.0000535249710083 - rollout_corr/rollout_is_seq_std:0.001242220401763916 - rollout_corr/rollout_is_seq_max:1.0033190250396729 - rollout_corr/rollout_is_seq_min:0.9950276017189026 - rollout_corr/rollout_is_seq_max_deviation:0.004972398281097412 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1713191107846797) - rollout_corr/training_log_ppl:np.float64(0.7553618648089468) - rollout_corr/kl:np.float64(0.0005726922157927028) - rollout_corr/k3_kl:np.float64(0.0006373734220233018) - rollout_corr/rollout_ppl:np.float64(2.170036384370178) - rollout_corr/rollout_log_ppl:np.float64(0.754789168946445) - rollout_corr/log_ppl_diff:np.float64(0.0005726958625018597) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010784545447677374) - rollout_corr/log_ppl_diff_max:np.float64(0.004006564617156982) - rollout_corr/log_ppl_diff_min:np.float64(-0.00302046537399292) - rollout_corr/ppl_ratio:np.float64(1.0005736567545682) - rollout_corr/chi2_token:np.float64(0.001405506394803524) - rollout_corr/chi2_seq:np.float64(1.6988238336052746) - actor/pg_loss:np.float64(6.845139432698488e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00017011942372846534) - actor/ppo_kl:np.float64(1.6240256268673647e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.24053851515054703) - perf/mfu/actor:np.float64(0.13349716649984694) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.54805374145508) - actor/lr:np.float64(1e-06) - training/global_step:79 - training/epoch:1 - critic/score/mean:0.62109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.011310762725770473 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.011310762725770473 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:838.69921875 - response_length/max:2007.0 - response_length/min:283.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:838.69921875 - response_length_non_aborted/max:2007.0 - response_length_non_aborted/min:283.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.875 - prompt_length/max:766.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010603666305541992 - timing_s/agent_loop/generate_sequences/min:np.float64(3.655642168596387) - timing_s/agent_loop/generate_sequences/max:np.float64(17.38879804685712) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.424919297925953) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(17.38879804685712) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:295 - timing_s/agent_loop/slowest/response_length:2007 - timing_s/gen:24.167077971622348 - timing_s/reward:0.08212696015834808 - timing_s/old_log_prob:3.3000302650034428 - timing_s/adv:0.11685498803853989 - timing_s/update_actor:13.943561296910048 - timing_s/step:41.69688728451729 - timing_s/stop_profile:0.00011464767158031464 - timing_per_token_ms/adv:0.00041399915693933546 - timing_per_token_ms/gen:0.11255840737201092 - timing_per_token_ms/update_actor:0.04939988201230093 - perf/total_num_tokens:282259 - perf/time_per_step:41.69688728451729 - perf/throughput:846.1632821474639
(TaskRunner pid=2224264) Training Progress: 79%|███████▉ | 79/100 [1:17:05<16:10, 46.19s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 80}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) We are given a target product:
(TaskRunner pid=2224264) **Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264) This is a nitrogen-containing compound with a ring structure. Let's analyze it:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - The formula appears to represent a **pyrrole ring** (a 5-membered ring with one nitrogen atom).
(TaskRunner pid=2224264) - The nitrogen is at the first position (Nc1...), and there's a substituent at the third carbon: **Cc2ccc(Cl)cc2C(F)(F)F**, which implies a **benzene ring** with a **chlorine atom** and a **trifluoromethyl group (C(F)(F)F)** on the same carbon.
(TaskRunner pid=2224264) - So the full structure is: a **pyrrole ring** with a **benzene ring** substituted with **Cl** and **CF₃** at the same position, attached to the third carbon of the pyrrole.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Now we examine the options to find which one is the **correct reactant** that would lead to this product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option A:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264) - This has a **pyrrole ring** (from the N1 and the 5-membered ring).
(TaskRunner pid=2224264) - The ring is substituted with a **benzene ring** (c2ccccc2) with a **Cl** and **CF₃** at the same position.
(TaskRunner pid=2224264) - The structure matches the **target product** exactly.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option B:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1**
(TaskRunner pid=2224264) - This contains a **Cl** on the ring, but the ring structure is different (not a pyrrole, and the Cl is not in the correct position).
(TaskRunner pid=2224264) - The structure is not matching the target.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option C:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1**
(TaskRunner pid=2224264) - Contains a **benzene ring** with Cl and CF₃, but it's attached to an **oxygen** (Oc2...), not directly to the pyrrole.
(TaskRunner pid=2224264) - The structure is different from the target.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option D:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F**
(TaskRunner pid=2224264) - This has a **trifluoromethyl group** (C(F)(F)F), but it is attached to a **methylene group** (CC=C) and not directly to the ring.
(TaskRunner pid=2224264) - The structure is not matching the target.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Conclusion:
(TaskRunner pid=2224264) Only **Option A** has the correct **pyrrole ring** with the correct **benzene substituent** (Cl and CF₃) in the correct position, matching the structure of the given product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_80
(TaskRunner pid=2224264) step:80 - global_seqlen/min:28862 - global_seqlen/max:47477 - global_seqlen/minmax_diff:18615 - global_seqlen/balanced_min:36267 - global_seqlen/balanced_max:36835 - global_seqlen/mean:36375.375 - actor/entropy:0.7752931714057922 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4967353940010071 - training/rollout_probs_diff_mean:0.004443757236003876 - training/rollout_probs_diff_std:0.008464205078780651 - training/rollout_actor_probs_pearson_corr:0.9995999336242676 - rollout_corr/rollout_is_mean:1.0000269412994385 - rollout_corr/rollout_is_ratio_fraction_high:9.492214303463697e-06 - rollout_corr/rollout_is_ratio_fraction_low:9.492214303463697e-06 - rollout_corr/rollout_is_max:2.5489871501922607 - rollout_corr/rollout_is_min:0.2599738538265228 - rollout_corr/rollout_is_std:0.03524937480688095 - rollout_corr/rollout_is_eff_sample_size:0.99875914242478 - rollout_corr/rollout_is_seq_mean:0.9999639391899109 - rollout_corr/rollout_is_seq_std:0.0014692033873870969 - rollout_corr/rollout_is_seq_max:1.0044381618499756 - rollout_corr/rollout_is_seq_min:0.9949309825897217 - rollout_corr/rollout_is_seq_max_deviation:0.00506901741027832 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.2062102765776217) - rollout_corr/training_log_ppl:np.float64(0.7724762546131387) - rollout_corr/kl:np.float64(0.0008447698922706692) - rollout_corr/k3_kl:np.float64(0.0006712018573580281) - rollout_corr/rollout_ppl:np.float64(2.204339082352817) - rollout_corr/rollout_log_ppl:np.float64(0.7716314849676564) - rollout_corr/log_ppl_diff:np.float64(0.0008447696454823017) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013305179309099913) - rollout_corr/log_ppl_diff_max:np.float64(0.005142807960510254) - rollout_corr/log_ppl_diff_min:np.float64(-0.0035692453384399414) - rollout_corr/ppl_ratio:np.float64(1.0008462956175208) - rollout_corr/chi2_token:np.float64(0.0010051389690488577) - rollout_corr/chi2_seq:np.float64(3.9995412554126233) - actor/pg_loss:np.float64(6.170920096337795e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(9.559897284816543e-05) - actor/ppo_kl:np.float64(0.00016689712089679176) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.20688721537590027) - perf/mfu/actor:np.float64(0.1360888320089338) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(108.50061416625977) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6782738095238096) - val-aux/sciknoweval/reward/std@16:np.float64(0.2194428960706272) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7662047619047618) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.1707809803907824) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5911047619047619) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.19703442080325248) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6786285714285715) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.2187494224175815) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8316666666666666) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.1211692975832172) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.513647619047619) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.16233574997526817) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.7021999999999999) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.17154812598360286) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8789666666666667) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.08054198286562807) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.44790952380952387) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.12433382532071413) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7142047619047619) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.12568969808131827) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.9095333333333334) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.0468538841472672) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.3950142857142857) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.08487101067852891) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7184190476190476) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.09186432979311497) - val-aux/sciknoweval/score/mean@16:np.float64(0.6782738095238096) - val-aux/sciknoweval/score/std@16:np.float64(0.2194428960706272) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7662047619047618) - val-aux/sciknoweval/score/best@2/std:np.float64(0.1707809803907824) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5911047619047619) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.19703442080325248) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6786285714285715) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.2187494224175815) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8316666666666666) - val-aux/sciknoweval/score/best@4/std:np.float64(0.1211692975832172) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.513647619047619) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.16233574997526817) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.7021999999999999) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.17154812598360286) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8789666666666667) - val-aux/sciknoweval/score/best@8/std:np.float64(0.08054198286562807) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.44790952380952387) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.12433382532071413) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7142047619047619) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.12568969808131827) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.9095333333333334) - val-aux/sciknoweval/score/best@16/std:np.float64(0.0468538841472672) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.3950142857142857) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.08487101067852891) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7184190476190476) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.09186432979311497) - val-core/sciknoweval/acc/mean@16:np.float64(0.6782738095238096) - val-aux/sciknoweval/acc/std@16:np.float64(0.2194428960706272) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7662047619047618) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.1707809803907824) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5911047619047619) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.19703442080325248) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6786285714285715) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.2187494224175815) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8316666666666666) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.1211692975832172) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.513647619047619) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.16233574997526817) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.7021999999999999) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.17154812598360286) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8789666666666667) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.08054198286562807) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.44790952380952387) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.12433382532071413) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7142047619047619) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.12568969808131827) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.9095333333333334) - val-core/sciknoweval/acc/best@16/std:np.float64(0.0468538841472672) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.3950142857142857) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.08487101067852891) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7184190476190476) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.09186432979311497) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9797619047619047) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0529405396018874) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9968523809523809) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.02007926101605294) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9632904761904761) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.06706255313069127) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9803428571428572) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.052420251171557214) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9998619047619047) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.002377576095584018) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9359809523809526) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.07827091020757276) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9914190476190478) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.034868422157064094) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.8997619047619049) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.07810157852659988) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9974428571428571) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.017502498474107602) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.863804761904762) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.06109163301171088) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9996809523809523) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.004826434845870424) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:80 - training/epoch:1 - critic/score/mean:0.71484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.007080005016177893 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:-0.007080005016177893 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:823.04296875 - response_length/max:3037.0 - response_length/min:168.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:823.04296875 - response_length_non_aborted/max:3037.0 - response_length_non_aborted/min:168.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:313.6875 - prompt_length/max:950.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014653801918029785 - timing_s/agent_loop/generate_sequences/min:np.float64(2.194973863661289) - timing_s/agent_loop/generate_sequences/max:np.float64(22.72264883853495) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.372557892595069) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(22.72264883853495) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:286 - timing_s/agent_loop/slowest/response_length:3037 - timing_s/gen:30.101325387135148 - timing_s/reward:0.08319770731031895 - timing_s/old_log_prob:3.4729851465672255 - timing_s/adv:0.14867497608065605 - timing_s/update_actor:14.160021048039198 - timing_s/step:48.05162427574396 - timing_s/testing:207.67929325439036 - timing_s/save_checkpoint:12.710731318220496 - timing_s/stop_profile:0.00015132129192352295 - timing_per_token_ms/adv:0.0005109053036589178 - timing_per_token_ms/gen:0.14286411130159682 - timing_per_token_ms/update_actor:0.04865936450153159 - perf/total_num_tokens:291003 - perf/time_per_step:48.05162427574396 - perf/throughput:757.0061480390366
(TaskRunner pid=2224264) Training Progress: 80%|████████ | 80/100 [1:21:33<37:37, 112.89s/it]
(TaskRunner pid=2224264) step:81 - global_seqlen/min:23435 - global_seqlen/max:40414 - global_seqlen/minmax_diff:16979 - global_seqlen/balanced_min:31718 - global_seqlen/balanced_max:31758 - global_seqlen/mean:31742.875 - actor/entropy:0.728011965751648 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.25490427017211914 - training/rollout_probs_diff_mean:0.00432547414675355 - training/rollout_probs_diff_std:0.00843879021704197 - training/rollout_actor_probs_pearson_corr:0.9996644258499146 - rollout_corr/rollout_is_mean:0.9999246001243591 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.7127294995589182e-05 - rollout_corr/rollout_is_max:1.6446915864944458 - rollout_corr/rollout_is_min:0.3955734968185425 - rollout_corr/rollout_is_std:0.03461290895938873 - rollout_corr/rollout_is_eff_sample_size:0.9988031424060505 - rollout_corr/rollout_is_seq_mean:0.9999666810035706 - rollout_corr/rollout_is_seq_std:0.001528046908788383 - rollout_corr/rollout_is_seq_max:1.0054874420166016 - rollout_corr/rollout_is_seq_min:0.9947968125343323 - rollout_corr/rollout_is_seq_max_deviation:0.0054874420166015625 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.084522245451808) - rollout_corr/training_log_ppl:np.float64(0.7124685312155634) - rollout_corr/kl:np.float64(0.0007158686560133631) - rollout_corr/k3_kl:np.float64(0.000656772480510881) - rollout_corr/rollout_ppl:np.float64(2.083047821652144) - rollout_corr/rollout_log_ppl:np.float64(0.7117526578949764) - rollout_corr/log_ppl_diff:np.float64(0.0007158733205869794) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012598995817825198) - rollout_corr/log_ppl_diff_max:np.float64(0.005965232849121094) - rollout_corr/log_ppl_diff_min:np.float64(-0.0034725069999694824) - rollout_corr/ppl_ratio:np.float64(1.000717243179679) - rollout_corr/chi2_token:np.float64(0.0012002624571323395) - rollout_corr/chi2_seq:np.float64(0.9090896914713085) - actor/pg_loss:np.float64(-3.0948780477046967e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00011802908329627826) - actor/ppo_kl:np.float64(6.148918862525221e-05) - actor/pg_clipfrac_lower:np.float64(2.239822151750559e-06) - actor/grad_norm:np.float64(0.2090231031179428) - perf/mfu/actor:np.float64(0.12132654821351155) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(109.81432342529297) - actor/lr:np.float64(1e-06) - training/global_step:81 - training/epoch:1 - critic/score/mean:0.65625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.65625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0020438572391867638 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0020438572391867638 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:684.21484375 - response_length/max:1807.0 - response_length/min:189.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:684.21484375 - response_length_non_aborted/max:1807.0 - response_length_non_aborted/min:189.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:307.75 - prompt_length/max:735.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001064930111169815 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4930436480790377) - timing_s/agent_loop/generate_sequences/max:np.float64(14.782667718827724) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.092499443388078) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.782667718827724) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:266 - timing_s/agent_loop/slowest/response_length:1744 - timing_s/gen:21.074421510100365 - timing_s/reward:0.071615781635046 - timing_s/old_log_prob:3.5168352890759706 - timing_s/adv:0.12568366713821888 - timing_s/update_actor:13.775931548327208 - timing_s/step:38.66653628088534 - timing_s/stop_profile:0.00012343190610408783 - timing_per_token_ms/adv:0.0004949286538247516 - timing_per_token_ms/gen:0.12031595013730591 - timing_per_token_ms/update_actor:0.05424812476944514 - perf/total_num_tokens:253943 - perf/time_per_step:38.66653628088534 - perf/throughput:820.9391906585637
(TaskRunner pid=2224264) Training Progress: 81%|████████ | 81/100 [1:22:12<28:42, 90.64s/it]
(TaskRunner pid=2224264) step:82 - global_seqlen/min:30553 - global_seqlen/max:40093 - global_seqlen/minmax_diff:9540 - global_seqlen/balanced_min:34588 - global_seqlen/balanced_max:34791 - global_seqlen/mean:34631.0 - actor/entropy:0.7073292136192322 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.48784658312797546 - training/rollout_probs_diff_mean:0.004207909107208252 - training/rollout_probs_diff_std:0.008302947506308556 - training/rollout_actor_probs_pearson_corr:0.9996032118797302 - rollout_corr/rollout_is_mean:0.9999532103538513 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.4358463886310346e-05 - rollout_corr/rollout_is_max:1.861779808998108 - rollout_corr/rollout_is_min:0.3777696490287781 - rollout_corr/rollout_is_std:0.033776216208934784 - rollout_corr/rollout_is_eff_sample_size:0.998860348177861 - rollout_corr/rollout_is_seq_mean:0.9999899864196777 - rollout_corr/rollout_is_seq_std:0.0011922791600227356 - rollout_corr/rollout_is_seq_max:1.004441738128662 - rollout_corr/rollout_is_seq_min:0.9973618984222412 - rollout_corr/rollout_is_seq_max_deviation:0.004441738128662109 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0682692220434546) - rollout_corr/training_log_ppl:np.float64(0.7130721984431148) - rollout_corr/kl:np.float64(0.0005954480367265091) - rollout_corr/k3_kl:np.float64(0.0006088201273541927) - rollout_corr/rollout_ppl:np.float64(2.067028284072876) - rollout_corr/rollout_log_ppl:np.float64(0.712476747808978) - rollout_corr/log_ppl_diff:np.float64(0.000595450634136796) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011131800711154938) - rollout_corr/log_ppl_diff_max:np.float64(0.0041391849517822266) - rollout_corr/log_ppl_diff_min:np.float64(-0.00474703311920166) - rollout_corr/ppl_ratio:np.float64(1.0005964431911707) - rollout_corr/chi2_token:np.float64(0.0012475282419472933) - rollout_corr/chi2_seq:np.float64(2.9451562664471567) - actor/pg_loss:np.float64(0.00018601585179567337) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019039797803088732) - actor/ppo_kl:np.float64(-5.924717907390686e-07) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.25703465938568115) - perf/mfu/actor:np.float64(0.1307738033068337) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(109.3072280883789) - actor/lr:np.float64(1e-06) - training/global_step:82 - training/epoch:1 - critic/score/mean:0.58203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0006862149457447231 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0006862149457447231 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:816.15625 - response_length/max:2571.0 - response_length/min:362.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:816.15625 - response_length_non_aborted/max:2571.0 - response_length_non_aborted/min:362.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.0625 - prompt_length/max:408.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.565187454223633e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(4.610755139961839) - timing_s/agent_loop/generate_sequences/max:np.float64(20.74920883215964) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.369914709837758) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(20.74920883215964) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:202 - timing_s/agent_loop/slowest/response_length:2571 - timing_s/gen:27.471973603591323 - timing_s/reward:0.07828585989773273 - timing_s/old_log_prob:3.2765041179955006 - timing_s/adv:0.11844107136130333 - timing_s/update_actor:13.957106951624155 - timing_s/step:44.991200095042586 - timing_s/stop_profile:0.00011249631643295288 - timing_per_token_ms/adv:0.0004275110138362426 - timing_per_token_ms/gen:0.13148511316188366 - timing_per_token_ms/update_actor:0.05037793794441452 - perf/total_num_tokens:277048 - perf/time_per_step:44.991200095042586 - perf/throughput:769.7283007975567
(TaskRunner pid=2224264) Training Progress: 82%|████████▏ | 82/100 [1:22:57<23:05, 76.96s/it]
(TaskRunner pid=2224264) step:83 - global_seqlen/min:24830 - global_seqlen/max:36777 - global_seqlen/minmax_diff:11947 - global_seqlen/balanced_min:31744 - global_seqlen/balanced_max:31974 - global_seqlen/mean:31783.0 - actor/entropy:0.7531368136405945 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2553977966308594 - training/rollout_probs_diff_mean:0.0044290670193731785 - training/rollout_probs_diff_std:0.008538827300071716 - training/rollout_actor_probs_pearson_corr:0.999602735042572 - rollout_corr/rollout_is_mean:1.0000147819519043 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.6260687415488064e-06 - rollout_corr/rollout_is_max:1.5903000831604004 - rollout_corr/rollout_is_min:0.4172116219997406 - rollout_corr/rollout_is_std:0.03477611765265465 - rollout_corr/rollout_is_eff_sample_size:0.9987920825865458 - rollout_corr/rollout_is_seq_mean:0.9999656677246094 - rollout_corr/rollout_is_seq_std:0.0014977967366576195 - rollout_corr/rollout_is_seq_max:1.0056312084197998 - rollout_corr/rollout_is_seq_min:0.9933651685714722 - rollout_corr/rollout_is_seq_max_deviation:0.006634831428527832 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.162785124965012) - rollout_corr/training_log_ppl:np.float64(0.7481051302747801) - rollout_corr/kl:np.float64(0.0006437221610795518) - rollout_corr/k3_kl:np.float64(0.0006854262796878174) - rollout_corr/rollout_ppl:np.float64(2.161293275654316) - rollout_corr/rollout_log_ppl:np.float64(0.7474614095408469) - rollout_corr/log_ppl_diff:np.float64(0.0006437207339331508) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012794457143172622) - rollout_corr/log_ppl_diff_max:np.float64(0.0074920654296875) - rollout_corr/log_ppl_diff_min:np.float64(-0.004778653383255005) - rollout_corr/ppl_ratio:np.float64(1.000645110849291) - rollout_corr/chi2_token:np.float64(0.0014334383886307478) - rollout_corr/chi2_seq:np.float64(0.4927413531113416) - actor/pg_loss:np.float64(-0.00020378001499921083) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024108612069539959) - actor/ppo_kl:np.float64(-1.7973163554674265e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.23665034770965576) - perf/mfu/actor:np.float64(0.12241472211925314) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.91914367675781) - actor/lr:np.float64(1e-06) - training/global_step:83 - training/epoch:1 - critic/score/mean:0.55078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.008029807358980179 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.008029807358980179 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:694.3125 - response_length/max:2176.0 - response_length/min:179.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:694.3125 - response_length_non_aborted/max:2176.0 - response_length_non_aborted/min:179.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:298.90625 - prompt_length/max:563.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010167807340621948 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2234518099576235) - timing_s/agent_loop/generate_sequences/max:np.float64(17.420780858024955) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.779617345870065) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(17.420780858024955) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:353 - timing_s/agent_loop/slowest/response_length:2176 - timing_s/gen:23.951165637001395 - timing_s/reward:0.07695263624191284 - timing_s/old_log_prob:3.1909585148096085 - timing_s/adv:0.11885049380362034 - timing_s/update_actor:13.669648755341768 - timing_s/step:41.09482858143747 - timing_s/stop_profile:0.00010753236711025238 - timing_per_token_ms/adv:0.00046742949770168146 - timing_per_token_ms/gen:0.13475090938091522 - timing_per_token_ms/update_actor:0.05376163654839761 - perf/total_num_tokens:254264 - perf/time_per_step:41.09482858143747 - perf/throughput:773.4063164910336
(TaskRunner pid=2224264) Training Progress: 83%|████████▎ | 83/100 [1:23:38<18:45, 66.22s/it]
(TaskRunner pid=2224264) step:84 - global_seqlen/min:20654 - global_seqlen/max:34303 - global_seqlen/minmax_diff:13649 - global_seqlen/balanced_min:29574 - global_seqlen/balanced_max:29701 - global_seqlen/mean:29614.0 - actor/entropy:0.6809946894645691 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.43606680631637573 - training/rollout_probs_diff_mean:0.004484572447836399 - training/rollout_probs_diff_std:0.008784463629126549 - training/rollout_actor_probs_pearson_corr:0.999607264995575 - rollout_corr/rollout_is_mean:0.9998617172241211 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1636025192274246e-05 - rollout_corr/rollout_is_max:1.5042952299118042 - rollout_corr/rollout_is_min:0.4855743646621704 - rollout_corr/rollout_is_std:0.0344783253967762 - rollout_corr/rollout_is_eff_sample_size:0.9988122996462617 - rollout_corr/rollout_is_seq_mean:0.9998700618743896 - rollout_corr/rollout_is_seq_std:0.0014390589203685522 - rollout_corr/rollout_is_seq_max:1.0044506788253784 - rollout_corr/rollout_is_seq_min:0.9937525391578674 - rollout_corr/rollout_is_seq_max_deviation:0.006247460842132568 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.9907695450820029) - rollout_corr/training_log_ppl:np.float64(0.670324468635954) - rollout_corr/kl:np.float64(0.0006556954003826831) - rollout_corr/k3_kl:np.float64(0.000665820000790518) - rollout_corr/rollout_ppl:np.float64(1.9894441780634224) - rollout_corr/rollout_log_ppl:np.float64(0.6696687689982355) - rollout_corr/log_ppl_diff:np.float64(0.0006556996377184987) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012462410377338529) - rollout_corr/log_ppl_diff_max:np.float64(0.007571697235107422) - rollout_corr/log_ppl_diff_min:np.float64(-0.004120081663131714) - rollout_corr/ppl_ratio:np.float64(1.000657090684399) - rollout_corr/chi2_token:np.float64(0.0013479453045874834) - rollout_corr/chi2_seq:np.float64(0.807735110167414) - actor/pg_loss:np.float64(-0.00014557677786797285) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001528950670035556) - actor/ppo_kl:np.float64(-9.21530613155197e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.18504177778959274) - perf/mfu/actor:np.float64(0.1154202224364208) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.7387924194336) - actor/lr:np.float64(1e-06) - training/global_step:84 - training/epoch:1 - critic/score/mean:0.69140625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.69140625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.01133712474256754 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.01133712474256754 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:671.40625 - response_length/max:2226.0 - response_length/min:177.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:671.40625 - response_length_non_aborted/max:2226.0 - response_length_non_aborted/min:177.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:254.03125 - prompt_length/max:466.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00017830543220043182 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6220184583216906) - timing_s/agent_loop/generate_sequences/max:np.float64(16.999597314745188) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.399542242870666) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.999597314745188) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:177 - timing_s/agent_loop/slowest/response_length:2226 - timing_s/gen:24.433550149202347 - timing_s/reward:0.07261811941862106 - timing_s/old_log_prob:3.231289431452751 - timing_s/adv:0.12676293589174747 - timing_s/update_actor:13.54585151001811 - timing_s/step:41.49533977732062 - timing_s/stop_profile:0.00012038461863994598 - timing_per_token_ms/adv:0.0005350633817271707 - timing_per_token_ms/gen:0.14215470182221518 - timing_per_token_ms/update_actor:0.05717672177862713 - perf/total_num_tokens:236912 - perf/time_per_step:41.49533977732062 - perf/throughput:713.6705027340348
(TaskRunner pid=2224264) Training Progress: 84%|████████▍ | 84/100 [1:24:20<15:41, 58.82s/it]
(TaskRunner pid=2224264) step:85 - global_seqlen/min:27367 - global_seqlen/max:37161 - global_seqlen/minmax_diff:9794 - global_seqlen/balanced_min:32947 - global_seqlen/balanced_max:32956 - global_seqlen/mean:32953.5 - actor/entropy:0.735228419303894 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5948740243911743 - training/rollout_probs_diff_mean:0.004468359053134918 - training/rollout_probs_diff_std:0.008735636249184608 - training/rollout_actor_probs_pearson_corr:0.9996408224105835 - rollout_corr/rollout_is_mean:0.9999686479568481 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.0702284271246754e-05 - rollout_corr/rollout_is_max:1.7922857999801636 - rollout_corr/rollout_is_min:0.23469160497188568 - rollout_corr/rollout_is_std:0.03531694784760475 - rollout_corr/rollout_is_eff_sample_size:0.9987541480773114 - rollout_corr/rollout_is_seq_mean:1.0000033378601074 - rollout_corr/rollout_is_seq_std:0.0015298437792807817 - rollout_corr/rollout_is_seq_max:1.0047695636749268 - rollout_corr/rollout_is_seq_min:0.9963509440422058 - rollout_corr/rollout_is_seq_max_deviation:0.004769563674926758 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1323386910371482) - rollout_corr/training_log_ppl:np.float64(0.7435957858106121) - rollout_corr/kl:np.float64(0.0005658202565279602) - rollout_corr/k3_kl:np.float64(0.0006766391076098444) - rollout_corr/rollout_ppl:np.float64(2.1311398232355714) - rollout_corr/rollout_log_ppl:np.float64(0.7430299640400335) - rollout_corr/log_ppl_diff:np.float64(0.0005658217705786228) - rollout_corr/log_ppl_abs_diff:np.float64(0.00123508065007627) - rollout_corr/log_ppl_diff_max:np.float64(0.004446566104888916) - rollout_corr/log_ppl_diff_min:np.float64(-0.0032745003700256348) - rollout_corr/ppl_ratio:np.float64(1.000567066250369) - rollout_corr/chi2_token:np.float64(0.0015738557558506727) - rollout_corr/chi2_seq:np.float64(1.2473218482919037) - actor/pg_loss:np.float64(-0.00010735273826867342) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(9.374866226607992e-05) - actor/ppo_kl:np.float64(-8.093407468590108e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.18677664920687675) - perf/mfu/actor:np.float64(0.1273840230570504) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.69750595092773) - actor/lr:np.float64(1e-06) - training/global_step:85 - training/epoch:1 - critic/score/mean:0.71875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00654712226241827 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00654712226241827 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:729.984375 - response_length/max:1647.0 - response_length/min:212.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:729.984375 - response_length_non_aborted/max:1647.0 - response_length_non_aborted/min:212.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.8125 - prompt_length/max:706.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014764443039894104 - timing_s/agent_loop/generate_sequences/min:np.float64(2.8548108004033566) - timing_s/agent_loop/generate_sequences/max:np.float64(14.612083107233047) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.586479216020962) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.612083107233047) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:205 - timing_s/agent_loop/slowest/response_length:1647 - timing_s/gen:21.41356766782701 - timing_s/reward:0.08048001676797867 - timing_s/old_log_prob:3.2531582564115524 - timing_s/adv:0.11875008791685104 - timing_s/update_actor:13.537027467042208 - timing_s/step:38.487668914720416 - timing_s/stop_profile:0.00011523440480232239 - timing_per_token_ms/adv:0.00045044565796065303 - timing_per_token_ms/gen:0.11458703989718856 - timing_per_token_ms/update_actor:0.051348974566594624 - perf/total_num_tokens:263628 - perf/time_per_step:38.487668914720416 - perf/throughput:856.2092984383432
(TaskRunner pid=2224264) Training Progress: 85%|████████▌ | 85/100 [1:24:58<13:11, 52.74s/it]
(TaskRunner pid=2224264) step:86 - global_seqlen/min:24044 - global_seqlen/max:35362 - global_seqlen/minmax_diff:11318 - global_seqlen/balanced_min:30989 - global_seqlen/balanced_max:31001 - global_seqlen/mean:30998.375 - actor/entropy:0.6628086566925049 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4241427183151245 - training/rollout_probs_diff_mean:0.004253617953509092 - training/rollout_probs_diff_std:0.008793003857135773 - training/rollout_actor_probs_pearson_corr:0.9995636940002441 - rollout_corr/rollout_is_mean:1.00003182888031 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.650387720379513e-06 - rollout_corr/rollout_is_max:1.8955367803573608 - rollout_corr/rollout_is_min:0.47840172052383423 - rollout_corr/rollout_is_std:0.03403814136981964 - rollout_corr/rollout_is_eff_sample_size:0.998842864637548 - rollout_corr/rollout_is_seq_mean:0.999988317489624 - rollout_corr/rollout_is_seq_std:0.001431034179404378 - rollout_corr/rollout_is_seq_max:1.0039067268371582 - rollout_corr/rollout_is_seq_min:0.9941695332527161 - rollout_corr/rollout_is_seq_max_deviation:0.0058304667472839355 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.980902411043644) - rollout_corr/training_log_ppl:np.float64(0.6618329326156527) - rollout_corr/kl:np.float64(0.0007042251129403265) - rollout_corr/k3_kl:np.float64(0.0006341270048437764) - rollout_corr/rollout_ppl:np.float64(1.979446857701987) - rollout_corr/rollout_log_ppl:np.float64(0.6611287064151838) - rollout_corr/log_ppl_diff:np.float64(0.0007042262004688382) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012508378131315112) - rollout_corr/log_ppl_diff_max:np.float64(0.006446421146392822) - rollout_corr/log_ppl_diff_min:np.float64(-0.0046353936195373535) - rollout_corr/ppl_ratio:np.float64(1.0007055627647787) - rollout_corr/chi2_token:np.float64(0.0011301424819976091) - rollout_corr/chi2_seq:np.float64(0.7622500413563102) - actor/pg_loss:np.float64(0.00021916942205280066) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00026571475882519735) - actor/ppo_kl:np.float64(8.79584136299627e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2480062022805214) - perf/mfu/actor:np.float64(0.12020480007027705) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.7301025390625) - actor/lr:np.float64(1e-06) - training/global_step:86 - training/epoch:1 - critic/score/mean:0.609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0002775753091555089 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0002775753091555089 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:691.32421875 - response_length/max:1574.0 - response_length/min:164.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:691.32421875 - response_length_non_aborted/max:1574.0 - response_length_non_aborted/min:164.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.375 - prompt_length/max:503.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00017972663044929504 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9018322117626667) - timing_s/agent_loop/generate_sequences/max:np.float64(14.014412458986044) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.677346075157402) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.014412458986044) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:274 - timing_s/agent_loop/slowest/response_length:1521 - timing_s/gen:20.71387554332614 - timing_s/reward:0.0748683363199234 - timing_s/old_log_prob:3.207970952615142 - timing_s/adv:0.11068613082170486 - timing_s/update_actor:13.604356173425913 - timing_s/step:37.79681725241244 - timing_s/stop_profile:2.8118491172790527e-05 - timing_per_token_ms/adv:0.00044633844040899265 - timing_per_token_ms/gen:0.11704143171407987 - timing_per_token_ms/update_actor:0.05485915057412652 - perf/total_num_tokens:247987 - perf/time_per_step:37.79681725241244 - perf/throughput:820.1318855232839
(TaskRunner pid=2224264) Training Progress: 86%|████████▌ | 86/100 [1:25:36<11:15, 48.26s/it]
(TaskRunner pid=2224264) step:87 - global_seqlen/min:27376 - global_seqlen/max:34559 - global_seqlen/minmax_diff:7183 - global_seqlen/balanced_min:30988 - global_seqlen/balanced_max:31014 - global_seqlen/mean:31004.375 - actor/entropy:0.6844121217727661 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.45646268129348755 - training/rollout_probs_diff_mean:0.004233994521200657 - training/rollout_probs_diff_std:0.008540390990674496 - training/rollout_actor_probs_pearson_corr:0.9996417760848999 - rollout_corr/rollout_is_mean:1.000025749206543 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1601264304772485e-05 - rollout_corr/rollout_is_max:1.5985411405563354 - rollout_corr/rollout_is_min:0.412740021944046 - rollout_corr/rollout_is_std:0.033871378749608994 - rollout_corr/rollout_is_eff_sample_size:0.998854282390456 - rollout_corr/rollout_is_seq_mean:1.0000381469726562 - rollout_corr/rollout_is_seq_std:0.0015245388494804502 - rollout_corr/rollout_is_seq_max:1.0053566694259644 - rollout_corr/rollout_is_seq_min:0.9960179328918457 - rollout_corr/rollout_is_seq_max_deviation:0.0053566694259643555 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0139922401867807) - rollout_corr/training_log_ppl:np.float64(0.6768635550397448) - rollout_corr/kl:np.float64(0.0006538237156501481) - rollout_corr/k3_kl:np.float64(0.0006354092881792894) - rollout_corr/rollout_ppl:np.float64(2.0125835882499814) - rollout_corr/rollout_log_ppl:np.float64(0.6762097296887077) - rollout_corr/log_ppl_diff:np.float64(0.0006538253510370851) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013526437105610967) - rollout_corr/log_ppl_diff_max:np.float64(0.00598597526550293) - rollout_corr/log_ppl_diff_min:np.float64(-0.005899667739868164) - rollout_corr/ppl_ratio:np.float64(1.0006554168649018) - rollout_corr/chi2_token:np.float64(0.0012414318043738604) - rollout_corr/chi2_seq:np.float64(0.7429793914780021) - actor/pg_loss:np.float64(-8.32361401990056e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00030786197271481797) - actor/ppo_kl:np.float64(9.98133720919725e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2665778249502182) - perf/mfu/actor:np.float64(0.12025084389863877) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.70088958740234) - actor/lr:np.float64(1e-06) - training/global_step:87 - training/epoch:1 - critic/score/mean:0.59375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005423591006547213 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005423591006547213 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:673.41796875 - response_length/max:1719.0 - response_length/min:153.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:673.41796875 - response_length_non_aborted/max:1719.0 - response_length_non_aborted/min:153.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:295.46875 - prompt_length/max:845.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.614036202430725e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9507676176726818) - timing_s/agent_loop/generate_sequences/max:np.float64(14.615875907242298) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.613317727045796) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.615875907242298) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:198 - timing_s/agent_loop/slowest/response_length:1719 - timing_s/gen:21.28308396600187 - timing_s/reward:0.0755326934158802 - timing_s/old_log_prob:3.1504952367395163 - timing_s/adv:0.11150607839226723 - timing_s/update_actor:13.641991836950183 - timing_s/step:38.34868148341775 - timing_s/stop_profile:3.3913180232048035e-05 - timing_per_token_ms/adv:0.00044955783817714125 - timing_per_token_ms/gen:0.12345534363526708 - timing_per_token_ms/update_actor:0.055000269465801936 - perf/total_num_tokens:248035 - perf/time_per_step:38.34868148341775 - perf/throughput:808.4860756791995
(TaskRunner pid=2224264) Training Progress: 87%|████████▋ | 87/100 [1:26:14<09:48, 45.29s/it]
(TaskRunner pid=2224264) step:88 - global_seqlen/min:25663 - global_seqlen/max:37476 - global_seqlen/minmax_diff:11813 - global_seqlen/balanced_min:30723 - global_seqlen/balanced_max:30761 - global_seqlen/mean:30750.875 - actor/entropy:0.6911125183105469 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.729590892791748 - training/rollout_probs_diff_mean:0.004324982408434153 - training/rollout_probs_diff_std:0.0088273286819458 - training/rollout_actor_probs_pearson_corr:0.9995751976966858 - rollout_corr/rollout_is_mean:0.9999687671661377 - rollout_corr/rollout_is_ratio_fraction_high:5.643627446261235e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.128725489252247e-05 - rollout_corr/rollout_is_max:3.966233491897583 - rollout_corr/rollout_is_min:0.06101220101118088 - rollout_corr/rollout_is_std:0.03435536473989487 - rollout_corr/rollout_is_eff_sample_size:0.9988211002691073 - rollout_corr/rollout_is_seq_mean:0.9998989105224609 - rollout_corr/rollout_is_seq_std:0.0017780093476176262 - rollout_corr/rollout_is_seq_max:1.012873888015747 - rollout_corr/rollout_is_seq_min:0.9941136837005615 - rollout_corr/rollout_is_seq_max_deviation:0.01287388801574707 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0185752864927053) - rollout_corr/training_log_ppl:np.float64(0.6845818213769235) - rollout_corr/kl:np.float64(0.0008537062007487606) - rollout_corr/k3_kl:np.float64(0.0007372877449256521) - rollout_corr/rollout_ppl:np.float64(2.0169089869596064) - rollout_corr/rollout_log_ppl:np.float64(0.6837281162734143) - rollout_corr/log_ppl_diff:np.float64(0.0008537051035091281) - rollout_corr/log_ppl_abs_diff:np.float64(0.0014192980015650392) - rollout_corr/log_ppl_diff_max:np.float64(0.015641063451766968) - rollout_corr/log_ppl_diff_min:np.float64(-0.0030567049980163574) - rollout_corr/ppl_ratio:np.float64(1.0008557450491935) - rollout_corr/chi2_token:np.float64(0.0012827848549932241) - rollout_corr/chi2_seq:np.float64(1.2393478201702237) - actor/pg_loss:np.float64(-2.0556850358843803e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00026034799202534487) - actor/ppo_kl:np.float64(7.98297634219125e-05) - actor/pg_clipfrac_lower:np.float64(1.187310044770129e-05) - actor/grad_norm:np.float64(0.25147897377610207) - perf/mfu/actor:np.float64(0.11886404252645519) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.61107635498047) - actor/lr:np.float64(1e-06) - training/global_step:88 - training/epoch:1 - critic/score/mean:0.5546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0010137365898117423 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0010137365898117423 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:692.15234375 - response_length/max:1935.0 - response_length/min:145.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:692.15234375 - response_length_non_aborted/max:1935.0 - response_length_non_aborted/min:145.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:268.8125 - prompt_length/max:482.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.903677523136139e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8212964609265327) - timing_s/agent_loop/generate_sequences/max:np.float64(16.11359540745616) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.880805795379274) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.11359540745616) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:371 - timing_s/agent_loop/slowest/response_length:1935 - timing_s/gen:22.885224971920252 - timing_s/reward:0.0695581566542387 - timing_s/old_log_prob:3.2812732346355915 - timing_s/adv:0.15897942706942558 - timing_s/update_actor:13.396265247836709 - timing_s/step:39.86858329549432 - timing_s/stop_profile:0.00011085905134677887 - timing_per_token_ms/adv:0.0006462394446882633 - timing_per_token_ms/gen:0.1291556849496885 - timing_per_token_ms/update_actor:0.05445481326887734 - perf/total_num_tokens:246007 - perf/time_per_step:39.86858329549432 - perf/throughput:771.3059371105182
(TaskRunner pid=2224264) Training Progress: 88%|████████▊ | 88/100 [1:26:54<08:44, 43.68s/it]
(TaskRunner pid=2224264) step:89 - global_seqlen/min:30256 - global_seqlen/max:40801 - global_seqlen/minmax_diff:10545 - global_seqlen/balanced_min:34912 - global_seqlen/balanced_max:34937 - global_seqlen/mean:34929.25 - actor/entropy:0.7014224529266357 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5826672315597534 - training/rollout_probs_diff_mean:0.004312381613999605 - training/rollout_probs_diff_std:0.008647686801850796 - training/rollout_actor_probs_pearson_corr:0.9996581673622131 - rollout_corr/rollout_is_mean:1.0000931024551392 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.952648199221585e-05 - rollout_corr/rollout_is_max:1.9627834558486938 - rollout_corr/rollout_is_min:0.3244985342025757 - rollout_corr/rollout_is_std:0.03435710072517395 - rollout_corr/rollout_is_eff_sample_size:0.9988213381259312 - rollout_corr/rollout_is_seq_mean:1.000161051750183 - rollout_corr/rollout_is_seq_std:0.0014398518251255155 - rollout_corr/rollout_is_seq_max:1.0050846338272095 - rollout_corr/rollout_is_seq_min:0.9958447813987732 - rollout_corr/rollout_is_seq_max_deviation:0.005084633827209473 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.095038112718612) - rollout_corr/training_log_ppl:np.float64(0.7223394099273719) - rollout_corr/kl:np.float64(0.0004870060852217861) - rollout_corr/k3_kl:np.float64(0.0006450895384659816) - rollout_corr/rollout_ppl:np.float64(2.094029687810689) - rollout_corr/rollout_log_ppl:np.float64(0.7218523986521177) - rollout_corr/log_ppl_diff:np.float64(0.00048701127525418997) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011224675690755248) - rollout_corr/log_ppl_diff_max:np.float64(0.004818141460418701) - rollout_corr/log_ppl_diff_min:np.float64(-0.003881394863128662) - rollout_corr/ppl_ratio:np.float64(1.0004880586639047) - rollout_corr/chi2_token:np.float64(0.0016097698826342821) - rollout_corr/chi2_seq:np.float64(1.3918255360331386) - actor/pg_loss:np.float64(7.724866736680269e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001636164154206199) - actor/ppo_kl:np.float64(3.664902180844365e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.23648764193058014) - perf/mfu/actor:np.float64(0.1338068099945285) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.57841873168945) - actor/lr:np.float64(1e-06) - training/global_step:89 - training/epoch:1 - critic/score/mean:0.546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0012478643329814076 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:0.0012478643329814076 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:800.1953125 - response_length/max:1701.0 - response_length/min:290.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:800.1953125 - response_length_non_aborted/max:1701.0 - response_length_non_aborted/min:290.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:291.34375 - prompt_length/max:720.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.920712232589722e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(3.7672608997672796) - timing_s/agent_loop/generate_sequences/max:np.float64(15.327744403854012) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.27638334871881) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.327744403854012) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:223 - timing_s/agent_loop/slowest/response_length:1701 - timing_s/gen:21.85275011509657 - timing_s/reward:0.08627365715801716 - timing_s/old_log_prob:3.273784726858139 - timing_s/adv:0.1331963650882244 - timing_s/update_actor:13.806754915043712 - timing_s/step:39.24401411600411 - timing_s/stop_profile:0.00011419318616390228 - timing_per_token_ms/adv:0.0004766648478289128 - timing_per_token_ms/gen:0.10667683727164544 - timing_per_token_ms/update_actor:0.04940971719634587 - perf/total_num_tokens:279434 - perf/time_per_step:39.24401411600411 - perf/throughput:890.0529363981524
(TaskRunner pid=2224264) Training Progress: 89%|████████▉ | 89/100 [1:27:34<07:45, 42.36s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 90}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) We are given a product molecule:
(TaskRunner pid=2224264) **Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This is a nitrogen-containing heterocyclic compound with the following structural features:
(TaskRunner pid=2224264) - A ring with the formula Nc1ccn(...)n1, suggesting a pyrrole-like ring (a five-membered ring with a nitrogen).
(TaskRunner pid=2224264) - A substituent on the nitrogen: **Cc2ccc(Cl)cc2C(F)(F)F**, which is a benzene ring with a chlorine and a trifluoromethyl group attached to the same carbon.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) We are to determine which of the four options is the **only correct reactant** used in the synthesis of this product. Let's analyze the options one by one.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option A:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This molecule has a **benzene ring** (c2ccccc2), a **carbonyl group** (O=C), and a **trifluoromethyl group** (C(F)(F)F) attached to a carbon that is also connected to a **chlorine-substituted benzene ring**. Additionally, it contains a **pyrrole-like ring** with a nitrogen (N1) and a substituent **Cc2ccc(Cl)cc2C(F)(F)F**, which is **identical to the substituent in the product**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This matches the **product** almost exactly, and it appears to be a **precursor** or **reactant** that could form the product via a reaction such as a **condensation or cyclization**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option B:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This molecule has a **benzene ring** and a **trifluoromethyl group is missing**, and the nitrogen is attached to a **chlorine** (C(Cl)N1), not a **trifluoromethyl** group. It also has a **different structure** (e.g., a different substituent on the nitrogen), and the ring structure doesn't match the product's pyrrole-like ring.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Not a match.**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option C:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This has a **benzene ring**, a **trifluoromethyl group** (C(F)(F)F), and a **chlorine** (on a benzene ring). However, the **substituent on the nitrogen is different** (it has an **ether oxygen** and a different aromatic ring), and the **structure of the ring is different** from the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Not a match.**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Option D:
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) This has a **trifluoromethyl group** and a **benzene ring**, but the **substituent on the nitrogen is not the same** as in the product (it has an ethyl group, not the same benzene ring with a chlorine). The **ring structure is also different**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Not a match.**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Final Analysis:
(TaskRunner pid=2224264) Only **Option A** has:
(TaskRunner pid=2224264) - A **pyrrole-like ring** with a nitrogen.
(TaskRunner pid=2224264) - A **trifluoromethyl group** (C(F)(F)F).
(TaskRunner pid=2224264) - A **chlorine-substituted benzene ring** (Cc2ccc(Cl)cc2C(F)(F)F).
(TaskRunner pid=2224264) - The **same substituent as in the product**.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Therefore, **Option A** is the only molecule that is a **valid reactant** for the synthesis of the given product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_90
(TaskRunner pid=2224264) step:90 - global_seqlen/min:26503 - global_seqlen/max:35349 - global_seqlen/minmax_diff:8846 - global_seqlen/balanced_min:32189 - global_seqlen/balanced_max:32202 - global_seqlen/mean:32197.5 - actor/entropy:0.6767091751098633 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24459987878799438 - training/rollout_probs_diff_mean:0.004204816184937954 - training/rollout_probs_diff_std:0.008432798087596893 - training/rollout_actor_probs_pearson_corr:0.999668538570404 - rollout_corr/rollout_is_mean:1.000195026397705 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.363540822145296e-06 - rollout_corr/rollout_is_max:1.6443707942962646 - rollout_corr/rollout_is_min:0.3406277298927307 - rollout_corr/rollout_is_std:0.033881932497024536 - rollout_corr/rollout_is_eff_sample_size:0.9988539255817266 - rollout_corr/rollout_is_seq_mean:1.0002124309539795 - rollout_corr/rollout_is_seq_std:0.0013589308364316821 - rollout_corr/rollout_is_seq_max:1.0058317184448242 - rollout_corr/rollout_is_seq_min:0.9959684610366821 - rollout_corr/rollout_is_seq_max_deviation:0.005831718444824219 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.031145083718002) - rollout_corr/training_log_ppl:np.float64(0.6887705723638646) - rollout_corr/kl:np.float64(0.0006024212293984088) - rollout_corr/k3_kl:np.float64(0.0006607583559343766) - rollout_corr/rollout_ppl:np.float64(2.0299026197753847) - rollout_corr/rollout_log_ppl:np.float64(0.6881681511295028) - rollout_corr/log_ppl_diff:np.float64(0.0006024212343618274) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012092863908037543) - rollout_corr/log_ppl_diff_max:np.float64(0.008667349815368652) - rollout_corr/log_ppl_diff_min:np.float64(-0.0036773681640625) - rollout_corr/ppl_ratio:np.float64(1.000603698194027) - rollout_corr/chi2_token:np.float64(0.0014522264245897532) - rollout_corr/chi2_seq:np.float64(0.6241840105503798) - actor/pg_loss:np.float64(-5.9257843531668186e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00015337120976255392) - actor/ppo_kl:np.float64(0.00021126134707571964) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.18526915088295937) - perf/mfu/actor:np.float64(0.11914403018889852) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.8291244506836) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6815476190476191) - val-aux/sciknoweval/reward/std@16:np.float64(0.2156221846747797) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7688476190476189) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.18144705579138373) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5926619047619047) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.1824273460834292) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6801857142857143) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.2154753727607372) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8427380952380952) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.13509620467575745) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5238857142857143) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.13671434851677694) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.7026523809523809) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.1762779201364988) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.895695238095238) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.08638656437772123) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.47133333333333327) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.09762932212980469) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7126714285714286) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.13634398316845442) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.9269000000000001) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.048177977724428236) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.42947619047619046) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.06802419225042791) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7179285714285714) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.10690355682237213) - val-aux/sciknoweval/score/mean@16:np.float64(0.6815476190476191) - val-aux/sciknoweval/score/std@16:np.float64(0.2156221846747797) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7688476190476189) - val-aux/sciknoweval/score/best@2/std:np.float64(0.18144705579138373) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5926619047619047) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.1824273460834292) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6801857142857143) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.2154753727607372) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8427380952380952) - val-aux/sciknoweval/score/best@4/std:np.float64(0.13509620467575745) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5238857142857143) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.13671434851677694) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.7026523809523809) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.1762779201364988) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.895695238095238) - val-aux/sciknoweval/score/best@8/std:np.float64(0.08638656437772123) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.47133333333333327) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.09762932212980469) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7126714285714286) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.13634398316845442) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.9269000000000001) - val-aux/sciknoweval/score/best@16/std:np.float64(0.048177977724428236) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.42947619047619046) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.06802419225042791) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7179285714285714) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.10690355682237213) - val-core/sciknoweval/acc/mean@16:np.float64(0.6815476190476191) - val-aux/sciknoweval/acc/std@16:np.float64(0.2156221846747797) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7688476190476189) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.18144705579138373) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5926619047619047) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.1824273460834292) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6801857142857143) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.2154753727607372) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8427380952380952) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.13509620467575745) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5238857142857143) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.13671434851677694) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.7026523809523809) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.1762779201364988) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.895695238095238) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.08638656437772123) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.47133333333333327) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.09762932212980469) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7126714285714286) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.13634398316845442) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.9269000000000001) - val-core/sciknoweval/acc/best@16/std:np.float64(0.048177977724428236) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.42947619047619046) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.06802419225042791) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7179285714285714) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.10690355682237213) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9767857142857143) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0671951594177618) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9968238095238093) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.022641622754649007) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9569095238095239) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.08670860027074634) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9766666666666668) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.06717160756445564) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9998523809523809) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0023879182972022734) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9241857142857143) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.10421760940450413) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9911857142857143) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.04162781171103064) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.8778714285714286) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.11040129964532797) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9977285714285715) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.020137431304512083) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.8244380952380953) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.09488164505205152) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9997952380952381) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.00450120576193258) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:90 - training/epoch:1 - critic/score/mean:0.59375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0026113740168511868 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0026113740168511868 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:728.296875 - response_length/max:1524.0 - response_length/min:189.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:728.296875 - response_length_non_aborted/max:1524.0 - response_length_non_aborted/min:189.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.875 - prompt_length/max:621.0 - prompt_length/min:180.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011400692164897919 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4655274618417025) - timing_s/agent_loop/generate_sequences/max:np.float64(14.063222439959645) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.401196123711998) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.063222439959645) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:371 - timing_s/agent_loop/slowest/response_length:1524 - timing_s/gen:20.9882550816983 - timing_s/reward:0.07745247334241867 - timing_s/old_log_prob:3.1980783604085445 - timing_s/adv:0.11822893470525742 - timing_s/update_actor:14.292423715814948 - timing_s/step:38.767167622223496 - timing_s/testing:206.41960541717708 - timing_s/save_checkpoint:12.91796376183629 - timing_s/stop_profile:0.00014068372547626495 - timing_per_token_ms/adv:0.00045899889240335977 - timing_per_token_ms/gen:0.11257136234847084 - timing_per_token_ms/update_actor:0.05548731934084536 - perf/total_num_tokens:257580 - perf/time_per_step:38.767167622223496 - perf/throughput:830.5352692710675
(TaskRunner pid=2224264) Training Progress: 90%|█████████ | 90/100 [1:31:52<17:51, 107.11s/it]
(TaskRunner pid=2224264) step:91 - global_seqlen/min:27956 - global_seqlen/max:38739 - global_seqlen/minmax_diff:10783 - global_seqlen/balanced_min:31665 - global_seqlen/balanced_max:31695 - global_seqlen/mean:31683.5 - actor/entropy:0.7154297232627869 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9744127988815308 - training/rollout_probs_diff_mean:0.004367751069366932 - training/rollout_probs_diff_std:0.00891902670264244 - training/rollout_actor_probs_pearson_corr:0.9995865821838379 - rollout_corr/rollout_is_mean:0.9999821186065674 - rollout_corr/rollout_is_ratio_fraction_high:5.548773515329231e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.1097547030658461e-05 - rollout_corr/rollout_is_max:2.104722261428833 - rollout_corr/rollout_is_min:0.00197325786575675 - rollout_corr/rollout_is_std:0.03477783128619194 - rollout_corr/rollout_is_eff_sample_size:0.9987919636650865 - rollout_corr/rollout_is_seq_mean:0.9999819993972778 - rollout_corr/rollout_is_seq_std:0.0012956903083249927 - rollout_corr/rollout_is_seq_max:1.00333833694458 - rollout_corr/rollout_is_seq_min:0.9953454732894897 - rollout_corr/rollout_is_seq_max_deviation:0.004654526710510254 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.104408088605851) - rollout_corr/training_log_ppl:np.float64(0.7243244127603248) - rollout_corr/kl:np.float64(0.0006612393046498966) - rollout_corr/k3_kl:np.float64(0.0006583722558843874) - rollout_corr/rollout_ppl:np.float64(2.102920087054372) - rollout_corr/rollout_log_ppl:np.float64(0.7236631693085656) - rollout_corr/log_ppl_diff:np.float64(0.0006612434517592192) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012168984394520521) - rollout_corr/log_ppl_diff_max:np.float64(0.00693821907043457) - rollout_corr/log_ppl_diff_min:np.float64(-0.0027844905853271484) - rollout_corr/ppl_ratio:np.float64(1.0006624751258641) - rollout_corr/chi2_token:np.float64(0.001317617017775774) - rollout_corr/chi2_seq:np.float64(1.2634925008751452) - actor/pg_loss:np.float64(0.0001394905848428607) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00015355198502220446) - actor/ppo_kl:np.float64(-4.81430525312021e-06) - actor/pg_clipfrac_lower:np.float64(4.3841191654792055e-06) - actor/grad_norm:np.float64(0.22441749274730682) - perf/mfu/actor:np.float64(0.1235221491000339) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(107.04840850830078) - actor/lr:np.float64(1e-06) - training/global_step:91 - training/epoch:1 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0006852735532447696 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0006852735532447696 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:703.984375 - response_length/max:1848.0 - response_length/min:173.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:703.984375 - response_length_non_aborted/max:1848.0 - response_length_non_aborted/min:173.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.125 - prompt_length/max:1009.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.789582014083862e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.103301614522934) - timing_s/agent_loop/generate_sequences/max:np.float64(15.148089375346899) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.013559678431193) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.148089375346899) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:191 - timing_s/agent_loop/slowest/response_length:1848 - timing_s/gen:21.60629509203136 - timing_s/reward:0.07231813855469227 - timing_s/old_log_prob:3.4730070438236 - timing_s/adv:0.13708950020372868 - timing_s/update_actor:13.167173700407147 - timing_s/step:38.57017913646996 - timing_s/stop_profile:0.00017870403826236725 - timing_per_token_ms/adv:0.0005408552566940548 - timing_per_token_ms/gen:0.11988844241500034 - timing_per_token_ms/update_actor:0.051948071158517635 - perf/total_num_tokens:253468 - perf/time_per_step:38.57017913646996 - perf/throughput:821.4506831274145
(TaskRunner pid=2224264) Training Progress: 91%|█████████ | 91/100 [1:32:30<12:59, 86.56s/it]
(TaskRunner pid=2224264) step:92 - global_seqlen/min:28907 - global_seqlen/max:37134 - global_seqlen/minmax_diff:8227 - global_seqlen/balanced_min:32432 - global_seqlen/balanced_max:32477 - global_seqlen/mean:32465.625 - actor/entropy:0.7178817987442017 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2975451946258545 - training/rollout_probs_diff_mean:0.004235805943608284 - training/rollout_probs_diff_std:0.008502726443111897 - training/rollout_actor_probs_pearson_corr:0.9996208548545837 - rollout_corr/rollout_is_mean:0.9999630451202393 - rollout_corr/rollout_is_ratio_fraction_high:5.372733539843466e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.1497116088867188 - rollout_corr/rollout_is_min:0.550010621547699 - rollout_corr/rollout_is_std:0.03401536867022514 - rollout_corr/rollout_is_eff_sample_size:0.9988440539746302 - rollout_corr/rollout_is_seq_mean:0.9999423623085022 - rollout_corr/rollout_is_seq_std:0.0013055680319666862 - rollout_corr/rollout_is_seq_max:1.004227876663208 - rollout_corr/rollout_is_seq_min:0.9956101775169373 - rollout_corr/rollout_is_seq_max_deviation:0.004389822483062744 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.064156681764871) - rollout_corr/training_log_ppl:np.float64(0.7071324991993606) - rollout_corr/kl:np.float64(0.0008166503134958703) - rollout_corr/k3_kl:np.float64(0.0006099401913388647) - rollout_corr/rollout_ppl:np.float64(2.0623925644904375) - rollout_corr/rollout_log_ppl:np.float64(0.7063158513628878) - rollout_corr/log_ppl_diff:np.float64(0.0008166478364728391) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012665537069551647) - rollout_corr/log_ppl_diff_max:np.float64(0.007030129432678223) - rollout_corr/log_ppl_diff_min:np.float64(-0.0036208629608154297) - rollout_corr/ppl_ratio:np.float64(1.0008179140277207) - rollout_corr/chi2_token:np.float64(0.0008041651453822851) - rollout_corr/chi2_seq:np.float64(0.6329832167830318) - actor/pg_loss:np.float64(6.987596862018108e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00012492035489231057) - actor/ppo_kl:np.float64(0.0001699282564437965) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.19231200218200684) - perf/mfu/actor:np.float64(0.12487351766630518) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(106.44847011566162) - actor/lr:np.float64(1e-06) - training/global_step:92 - training/epoch:1 - critic/score/mean:0.62109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004429147113114595 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004429147113114595 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:727.05078125 - response_length/max:1984.0 - response_length/min:182.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:727.05078125 - response_length_non_aborted/max:1984.0 - response_length_non_aborted/min:182.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:287.5 - prompt_length/max:436.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00019785761833190918 - timing_s/agent_loop/generate_sequences/min:np.float64(2.329558404162526) - timing_s/agent_loop/generate_sequences/max:np.float64(16.53738614730537) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.280053526119445) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.53738614730537) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:309 - timing_s/agent_loop/slowest/response_length:1984 - timing_s/gen:23.22112664207816 - timing_s/reward:0.07667875476181507 - timing_s/old_log_prob:3.3302839752286673 - timing_s/adv:0.12147887982428074 - timing_s/update_actor:13.518468294292688 - timing_s/step:40.354144191369414 - timing_s/stop_profile:0.00012041814625263214 - timing_per_token_ms/adv:0.00046772116594197993 - timing_per_token_ms/gen:0.12476092218712242 - timing_per_token_ms/update_actor:0.0520491608212251 - perf/total_num_tokens:259725 - perf/time_per_step:40.354144191369414 - perf/throughput:804.5177428627878
(TaskRunner pid=2224264) Training Progress: 92%|█████████▏| 92/100 [1:33:11<09:41, 72.72s/it]
(TaskRunner pid=2224264) step:93 - global_seqlen/min:23013 - global_seqlen/max:34946 - global_seqlen/minmax_diff:11933 - global_seqlen/balanced_min:27908 - global_seqlen/balanced_max:27913 - global_seqlen/mean:27910.875 - actor/entropy:0.7500928044319153 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1930849552154541 - training/rollout_probs_diff_mean:0.004663003142923117 - training/rollout_probs_diff_std:0.008796103298664093 - training/rollout_actor_probs_pearson_corr:0.9996201992034912 - rollout_corr/rollout_is_mean:0.999956488609314 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6296987533569336 - rollout_corr/rollout_is_min:0.6042289137840271 - rollout_corr/rollout_is_std:0.035632822662591934 - rollout_corr/rollout_is_eff_sample_size:0.9987316741318075 - rollout_corr/rollout_is_seq_mean:0.9998897910118103 - rollout_corr/rollout_is_seq_std:0.0015105620259419084 - rollout_corr/rollout_is_seq_max:1.0058465003967285 - rollout_corr/rollout_is_seq_min:0.9952189922332764 - rollout_corr/rollout_is_seq_max_deviation:0.005846500396728516 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1263672169297934) - rollout_corr/training_log_ppl:np.float64(0.7347684952546842) - rollout_corr/kl:np.float64(0.0007506872085336624) - rollout_corr/k3_kl:np.float64(0.0006735208150985272) - rollout_corr/rollout_ppl:np.float64(2.124785815831274) - rollout_corr/rollout_log_ppl:np.float64(0.7340178099111654) - rollout_corr/log_ppl_diff:np.float64(0.0007506853435188532) - rollout_corr/log_ppl_abs_diff:np.float64(0.001359031768515706) - rollout_corr/log_ppl_diff_max:np.float64(0.005597174167633057) - rollout_corr/log_ppl_diff_min:np.float64(-0.003313422203063965) - rollout_corr/ppl_ratio:np.float64(1.000752320047468) - rollout_corr/chi2_token:np.float64(0.0011984859593212605) - rollout_corr/chi2_seq:np.float64(0.6925807157531381) - actor/pg_loss:np.float64(-4.779419396072626e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(8.594229848313262e-05) - actor/ppo_kl:np.float64(-7.275951552143667e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.18249434977769852) - perf/mfu/actor:np.float64(0.11041731672624022) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(106.24386596679688) - actor/lr:np.float64(1e-06) - training/global_step:93 - training/epoch:1 - critic/score/mean:0.8515625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.8515625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.007838345132768154 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.007838345132768154 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:605.74609375 - response_length/max:1369.0 - response_length/min:190.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:605.74609375 - response_length_non_aborted/max:1369.0 - response_length_non_aborted/min:190.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.46875 - prompt_length/max:709.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00018016062676906586 - timing_s/agent_loop/generate_sequences/min:np.float64(2.355575904250145) - timing_s/agent_loop/generate_sequences/max:np.float64(11.529610631987453) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.815270844308543) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.529610631987453) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:200 - timing_s/agent_loop/slowest/response_length:1369 - timing_s/gen:18.318352431058884 - timing_s/reward:0.07145974412560463 - timing_s/old_log_prob:3.1092032585293055 - timing_s/adv:0.12323503755033016 - timing_s/update_actor:13.111440777778625 - timing_s/step:34.81759689189494 - timing_s/stop_profile:2.6732683181762695e-05 - timing_per_token_ms/adv:0.0005519131769889432 - timing_per_token_ms/gen:0.11812880829464492 - timing_per_token_ms/update_actor:0.05872012601619721 - perf/total_num_tokens:223287 - perf/time_per_step:34.81759689189494 - perf/throughput:801.6312868076565
(TaskRunner pid=2224264) Training Progress: 93%|█████████▎| 93/100 [1:33:46<07:09, 61.36s/it]
(TaskRunner pid=2224264) step:94 - global_seqlen/min:24960 - global_seqlen/max:37338 - global_seqlen/minmax_diff:12378 - global_seqlen/balanced_min:30359 - global_seqlen/balanced_max:30372 - global_seqlen/mean:30366.25 - actor/entropy:0.7063392400741577 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3128166198730469 - training/rollout_probs_diff_mean:0.004224883858114481 - training/rollout_probs_diff_std:0.008449923247098923 - training/rollout_actor_probs_pearson_corr:0.9996349811553955 - rollout_corr/rollout_is_mean:0.9998794198036194 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.6113572100002784e-06 - rollout_corr/rollout_is_max:1.5920956134796143 - rollout_corr/rollout_is_min:0.19120460748672485 - rollout_corr/rollout_is_std:0.03371969982981682 - rollout_corr/rollout_is_eff_sample_size:0.99886415419973 - rollout_corr/rollout_is_seq_mean:0.9998659491539001 - rollout_corr/rollout_is_seq_std:0.0013268847251310945 - rollout_corr/rollout_is_seq_max:1.0026715993881226 - rollout_corr/rollout_is_seq_min:0.9956828355789185 - rollout_corr/rollout_is_seq_max_deviation:0.004317164421081543 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.097409463953227) - rollout_corr/training_log_ppl:np.float64(0.7194324630545452) - rollout_corr/kl:np.float64(0.0006619715687250505) - rollout_corr/k3_kl:np.float64(0.0006161111765550231) - rollout_corr/rollout_ppl:np.float64(2.0959334070794284) - rollout_corr/rollout_log_ppl:np.float64(0.7187704919488169) - rollout_corr/log_ppl_diff:np.float64(0.0006619711057282984) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012567761004902422) - rollout_corr/log_ppl_diff_max:np.float64(0.004305779933929443) - rollout_corr/log_ppl_diff_min:np.float64(-0.004627227783203125) - rollout_corr/ppl_ratio:np.float64(1.0006632735021412) - rollout_corr/chi2_token:np.float64(0.00113762472756207) - rollout_corr/chi2_seq:np.float64(2.2418907159008086) - actor/pg_loss:np.float64(9.933649562299252e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(4.07081788580399e-05) - actor/ppo_kl:np.float64(-6.808040846806307e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1753209587186575) - perf/mfu/actor:np.float64(0.11873190434052154) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(106.19572830200195) - actor/lr:np.float64(1e-06) - training/global_step:94 - training/epoch:1 - critic/score/mean:0.72265625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.72265625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006172493100166321 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006172493100166321 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:696.1328125 - response_length/max:1462.0 - response_length/min:227.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:696.1328125 - response_length_non_aborted/max:1462.0 - response_length_non_aborted/min:227.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:252.8125 - prompt_length/max:384.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.6177377700805664e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.743150135502219) - timing_s/agent_loop/generate_sequences/max:np.float64(13.47325548902154) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.079402086157643) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(13.47325548902154) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:272 - timing_s/agent_loop/slowest/response_length:1462 - timing_s/gen:20.203036999329925 - timing_s/reward:0.07774028927087784 - timing_s/old_log_prob:3.2072019651532173 - timing_s/adv:0.12383914738893509 - timing_s/update_actor:13.169512040913105 - timing_s/step:36.86546222306788 - timing_s/stop_profile:0.00011588446795940399 - timing_per_token_ms/adv:0.0005097729691225254 - timing_per_token_ms/gen:0.11336646091313576 - timing_per_token_ms/update_actor:0.054211139179653006 - perf/total_num_tokens:242930 - perf/time_per_step:36.86546222306788 - perf/throughput:823.7045779124636
(TaskRunner pid=2224264) Training Progress: 94%|█████████▍| 94/100 [1:34:23<05:24, 54.02s/it]
(TaskRunner pid=2224264) step:95 - global_seqlen/min:20976 - global_seqlen/max:34422 - global_seqlen/minmax_diff:13446 - global_seqlen/balanced_min:29028 - global_seqlen/balanced_max:29035 - global_seqlen/mean:29031.75 - actor/entropy:0.712871789932251 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2599616050720215 - training/rollout_probs_diff_mean:0.004302844405174255 - training/rollout_probs_diff_std:0.008476506918668747 - training/rollout_actor_probs_pearson_corr:0.9996457099914551 - rollout_corr/rollout_is_mean:1.0001041889190674 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:6.065090474294266e-06 - rollout_corr/rollout_is_max:1.6687859296798706 - rollout_corr/rollout_is_min:0.4844895303249359 - rollout_corr/rollout_is_std:0.03429459035396576 - rollout_corr/rollout_is_eff_sample_size:0.9988256195681333 - rollout_corr/rollout_is_seq_mean:1.0001366138458252 - rollout_corr/rollout_is_seq_std:0.0015432714717462659 - rollout_corr/rollout_is_seq_max:1.005467414855957 - rollout_corr/rollout_is_seq_min:0.9945797324180603 - rollout_corr/rollout_is_seq_max_deviation:0.005467414855957031 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0434681265614927) - rollout_corr/training_log_ppl:np.float64(0.6912423392059281) - rollout_corr/kl:np.float64(0.0005482688078317643) - rollout_corr/k3_kl:np.float64(0.0006431571834468741) - rollout_corr/rollout_ppl:np.float64(2.0423742262646556) - rollout_corr/rollout_log_ppl:np.float64(0.6906940678600222) - rollout_corr/log_ppl_diff:np.float64(0.0005482713459059596) - rollout_corr/log_ppl_abs_diff:np.float64(0.001258363830856979) - rollout_corr/log_ppl_diff_max:np.float64(0.005878806114196777) - rollout_corr/log_ppl_diff_min:np.float64(-0.005048781633377075) - rollout_corr/ppl_ratio:np.float64(1.000549679622054) - rollout_corr/chi2_token:np.float64(0.0014886993449181318) - rollout_corr/chi2_seq:np.float64(3.0285099220927805) - actor/pg_loss:np.float64(-0.000160741969011724) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00010068776555272052) - actor/ppo_kl:np.float64(7.83663859635908e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.23990549892187119) - perf/mfu/actor:np.float64(0.11424140158128934) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(106.15832138061523) - actor/lr:np.float64(1e-06) - training/global_step:95 - training/epoch:1 - critic/score/mean:0.7421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.7421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.014393975958228111 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.014393975958228111 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:644.0546875 - response_length/max:1596.0 - response_length/min:150.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:644.0546875 - response_length_non_aborted/max:1596.0 - response_length_non_aborted/min:150.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.1875 - prompt_length/max:646.0 - prompt_length/min:159.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.495019912719727e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.937621222808957) - timing_s/agent_loop/generate_sequences/max:np.float64(13.714711025357246) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.356954876297095) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(13.714711025357246) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:218 - timing_s/agent_loop/slowest/response_length:1596 - timing_s/gen:20.76876393146813 - timing_s/reward:0.07113218307495117 - timing_s/old_log_prob:3.2435854095965624 - timing_s/adv:0.1196875274181366 - timing_s/update_actor:13.214123142883182 - timing_s/step:37.502956656739116 - timing_s/stop_profile:0.00011238828301429749 - timing_per_token_ms/adv:0.0005153303168864115 - timing_per_token_ms/gen:0.12596443389335224 - timing_per_token_ms/update_actor:0.056895136974532974 - perf/total_num_tokens:232254 - perf/time_per_step:37.502956656739116 - perf/throughput:774.1189652251891
(TaskRunner pid=2224264) Training Progress: 95%|█████████▌| 95/100 [1:35:00<04:05, 49.08s/it]
(TaskRunner pid=2224264) step:96 - global_seqlen/min:27198 - global_seqlen/max:35708 - global_seqlen/minmax_diff:8510 - global_seqlen/balanced_min:32177 - global_seqlen/balanced_max:32213 - global_seqlen/mean:32201.875 - actor/entropy:0.7203529477119446 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.6764640808105469 - training/rollout_probs_diff_mean:0.004217594396322966 - training/rollout_probs_diff_std:0.008475503884255886 - training/rollout_actor_probs_pearson_corr:0.999649703502655 - rollout_corr/rollout_is_mean:1.0000306367874146 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:5.28119744558353e-06 - rollout_corr/rollout_is_max:1.7523001432418823 - rollout_corr/rollout_is_min:0.0011872303439304233 - rollout_corr/rollout_is_std:0.03390303626656532 - rollout_corr/rollout_is_eff_sample_size:0.9988520226061409 - rollout_corr/rollout_is_seq_mean:0.9999527931213379 - rollout_corr/rollout_is_seq_std:0.0014494654024019837 - rollout_corr/rollout_is_seq_max:1.0033957958221436 - rollout_corr/rollout_is_seq_min:0.9943961501121521 - rollout_corr/rollout_is_seq_max_deviation:0.0056038498878479 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.0765154217369854) - rollout_corr/training_log_ppl:np.float64(0.7100228805211373) - rollout_corr/kl:np.float64(0.0007996692028076069) - rollout_corr/k3_kl:np.float64(0.0006997747711352531) - rollout_corr/rollout_ppl:np.float64(2.074850105214864) - rollout_corr/rollout_log_ppl:np.float64(0.7092232063878328) - rollout_corr/log_ppl_diff:np.float64(0.0007996741333045065) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013392329565249383) - rollout_corr/log_ppl_diff_max:np.float64(0.006226241588592529) - rollout_corr/log_ppl_diff_min:np.float64(-0.002697765827178955) - rollout_corr/ppl_ratio:np.float64(1.0008012598846108) - rollout_corr/chi2_token:np.float64(0.0011850390583276749) - rollout_corr/chi2_seq:np.float64(0.8988333453889936) - actor/pg_loss:np.float64(-6.152375135570765e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024159732652151433) - actor/ppo_kl:np.float64(0.00012709150854451137) - actor/pg_clipfrac_lower:np.float64(4.822531082027126e-06) - actor/grad_norm:np.float64(0.21260051056742668) - perf/mfu/actor:np.float64(0.12350445771061563) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(106.11673736572266) - actor/lr:np.float64(1e-06) - training/global_step:96 - training/epoch:1 - critic/score/mean:0.640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.01730252243578434 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.01730252243578434 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:739.65234375 - response_length/max:1942.0 - response_length/min:151.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:739.65234375 - response_length_non_aborted/max:1942.0 - response_length_non_aborted/min:151.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.65625 - prompt_length/max:456.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010816939175128937 - timing_s/agent_loop/generate_sequences/min:np.float64(1.915278973057866) - timing_s/agent_loop/generate_sequences/max:np.float64(16.20021486841142) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.129882725828793) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.20021486841142) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:266 - timing_s/agent_loop/slowest/response_length:1903 - timing_s/gen:23.026053931564093 - timing_s/reward:0.07937229797244072 - timing_s/old_log_prob:3.2644342444837093 - timing_s/adv:0.1240159124135971 - timing_s/update_actor:13.31886836886406 - timing_s/step:39.908353347331285 - timing_s/stop_profile:0.00011824257671833038 - timing_per_token_ms/adv:0.00048140019957532403 - timing_per_token_ms/gen:0.12160513507488259 - timing_per_token_ms/update_actor:0.05170067103570856 - perf/total_num_tokens:257615 - perf/time_per_step:39.908353347331285 - perf/throughput:806.8956070359484
(TaskRunner pid=2224264) Training Progress: 96%|█████████▌| 96/100 [1:35:40<03:05, 46.34s/it]
(TaskRunner pid=2224264) step:97 - global_seqlen/min:31313 - global_seqlen/max:43694 - global_seqlen/minmax_diff:12381 - global_seqlen/balanced_min:35934 - global_seqlen/balanced_max:35960 - global_seqlen/mean:35950.125 - actor/entropy:0.7323668003082275 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2915082573890686 - training/rollout_probs_diff_mean:0.004103551618754864 - training/rollout_probs_diff_std:0.008156946860253811 - training/rollout_actor_probs_pearson_corr:0.9996232986450195 - rollout_corr/rollout_is_mean:0.9999871850013733 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.7118023758230265e-06 - rollout_corr/rollout_is_max:1.6210092306137085 - rollout_corr/rollout_is_min:0.4609464108943939 - rollout_corr/rollout_is_std:0.03369848057627678 - rollout_corr/rollout_is_eff_sample_size:0.9988657004043994 - rollout_corr/rollout_is_seq_mean:1.000003695487976 - rollout_corr/rollout_is_seq_std:0.0014436065685003996 - rollout_corr/rollout_is_seq_max:1.0053772926330566 - rollout_corr/rollout_is_seq_min:0.9946680665016174 - rollout_corr/rollout_is_seq_max_deviation:0.005377292633056641 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1260625598952174) - rollout_corr/training_log_ppl:np.float64(0.7336573095526546) - rollout_corr/kl:np.float64(0.0006526818591758854) - rollout_corr/k3_kl:np.float64(0.0006329746193500796) - rollout_corr/rollout_ppl:np.float64(2.124627975281328) - rollout_corr/rollout_log_ppl:np.float64(0.7330046314746141) - rollout_corr/log_ppl_diff:np.float64(0.0006526780780404806) - rollout_corr/log_ppl_abs_diff:np.float64(0.001235815929248929) - rollout_corr/log_ppl_diff_max:np.float64(0.00612562894821167) - rollout_corr/log_ppl_diff_min:np.float64(-0.00426250696182251) - rollout_corr/ppl_ratio:np.float64(1.0006539451424032) - rollout_corr/chi2_token:np.float64(0.0012377453967928886) - rollout_corr/chi2_seq:np.float64(0.6915090307593346) - actor/pg_loss:np.float64(-3.366696182638407e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00016491950168529002) - actor/ppo_kl:np.float64(6.479799488445792e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.20394239574670792) - perf/mfu/actor:np.float64(0.135031184027903) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(106.04956817626953) - actor/lr:np.float64(1e-06) - training/global_step:97 - training/epoch:1 - critic/score/mean:0.5859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.012476264499127865 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.012476264499127865 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:829.03515625 - response_length/max:2245.0 - response_length/min:215.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:829.03515625 - response_length_non_aborted/max:2245.0 - response_length_non_aborted/min:215.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:294.40625 - prompt_length/max:900.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010967254638671875 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4484062399715185) - timing_s/agent_loop/generate_sequences/max:np.float64(18.13408470340073) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.178321859493735) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(18.13408470340073) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:219 - timing_s/agent_loop/slowest/response_length:2245 - timing_s/gen:24.973692329600453 - timing_s/reward:0.08095383085310459 - timing_s/old_log_prob:3.3533995244652033 - timing_s/adv:0.12184758298099041 - timing_s/update_actor:13.665139958262444 - timing_s/step:42.28041913919151 - timing_s/stop_profile:4.5822933316230774e-05 - timing_per_token_ms/adv:0.0004236688432272155 - timing_per_token_ms/gen:0.11767110830832365 - timing_per_token_ms/update_actor:0.047514229638500716 - perf/total_num_tokens:287601 - perf/time_per_step:42.28041913919151 - perf/throughput:850.2783494564818
(TaskRunner pid=2224264) Training Progress: 97%|█████████▋| 97/100 [1:36:22<02:15, 45.13s/it]
(TaskRunner pid=2224264) step:98 - global_seqlen/min:25644 - global_seqlen/max:42510 - global_seqlen/minmax_diff:16866 - global_seqlen/balanced_min:34661 - global_seqlen/balanced_max:34705 - global_seqlen/mean:34687.75 - actor/entropy:0.7501354813575745 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.19508248567581177 - training/rollout_probs_diff_mean:0.004144626669585705 - training/rollout_probs_diff_std:0.008178996853530407 - training/rollout_actor_probs_pearson_corr:0.9996928572654724 - rollout_corr/rollout_is_mean:1.0000689029693604 - rollout_corr/rollout_is_ratio_fraction_high:4.974382136424538e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.2251296043395996 - rollout_corr/rollout_is_min:0.6142621636390686 - rollout_corr/rollout_is_std:0.03403989225625992 - rollout_corr/rollout_is_eff_sample_size:0.9988425078369757 - rollout_corr/rollout_is_seq_mean:1.000027060508728 - rollout_corr/rollout_is_seq_std:0.001423987210728228 - rollout_corr/rollout_is_seq_max:1.0042808055877686 - rollout_corr/rollout_is_seq_min:0.9920931458473206 - rollout_corr/rollout_is_seq_max_deviation:0.007906854152679443 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1749815698713064) - rollout_corr/training_log_ppl:np.float64(0.7591774334432557) - rollout_corr/kl:np.float64(0.0006937429601876488) - rollout_corr/k3_kl:np.float64(0.0006346612378251848) - rollout_corr/rollout_ppl:np.float64(2.17338493373245) - rollout_corr/rollout_log_ppl:np.float64(0.7584836900932714) - rollout_corr/log_ppl_diff:np.float64(0.0006937433499842882) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011937296949326992) - rollout_corr/log_ppl_diff_max:np.float64(0.004764825105667114) - rollout_corr/log_ppl_diff_min:np.float64(-0.003519713878631592) - rollout_corr/ppl_ratio:np.float64(1.000694976421073) - rollout_corr/chi2_token:np.float64(0.0011504706926643848) - rollout_corr/chi2_seq:np.float64(0.7677441604901105) - actor/pg_loss:np.float64(-0.00023154309019446373) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014778738523091306) - actor/ppo_kl:np.float64(0.00011894125558598034) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.22566131502389908) - perf/mfu/actor:np.float64(0.13134675089113737) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.8719253540039) - actor/lr:np.float64(1e-06) - training/global_step:98 - training/epoch:1 - critic/score/mean:0.56640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.56640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0074124508537352085 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0074124508537352085 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:785.2734375 - response_length/max:2121.0 - response_length/min:178.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:785.2734375 - response_length_non_aborted/max:2121.0 - response_length_non_aborted/min:178.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:298.71875 - prompt_length/max:631.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:3.526918590068817e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.268034750595689) - timing_s/agent_loop/generate_sequences/max:np.float64(17.475637735798955) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.994100491654535) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(17.475637735798955) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:229 - timing_s/agent_loop/slowest/response_length:2121 - timing_s/gen:24.36921063065529 - timing_s/reward:0.08015711978077888 - timing_s/old_log_prob:3.372231272980571 - timing_s/adv:0.11694180965423584 - timing_s/update_actor:13.581646379083395 - timing_s/step:41.60994468256831 - timing_s/stop_profile:0.00010863691568374634 - timing_per_token_ms/adv:0.0004214088895007454 - timing_per_token_ms/gen:0.12122176108369541 - timing_per_token_ms/update_actor:0.04894251709567281 - perf/total_num_tokens:277502 - perf/time_per_step:41.60994468256831 - perf/throughput:833.6408583242305
(TaskRunner pid=2224264) Training Progress: 98%|█████████▊| 98/100 [1:37:04<01:28, 44.09s/it]
(TaskRunner pid=2224264) step:99 - global_seqlen/min:31172 - global_seqlen/max:42762 - global_seqlen/minmax_diff:11590 - global_seqlen/balanced_min:36579 - global_seqlen/balanced_max:36620 - global_seqlen/mean:36605.125 - actor/entropy:0.7388753890991211 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2583175301551819 - training/rollout_probs_diff_mean:0.004119738936424255 - training/rollout_probs_diff_std:0.008252698928117752 - training/rollout_actor_probs_pearson_corr:0.9996559023857117 - rollout_corr/rollout_is_mean:1.0000630617141724 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.395273082191125e-06 - rollout_corr/rollout_is_max:1.7297682762145996 - rollout_corr/rollout_is_min:0.46209102869033813 - rollout_corr/rollout_is_std:0.03351224586367607 - rollout_corr/rollout_is_eff_sample_size:0.998878308098044 - rollout_corr/rollout_is_seq_mean:1.0000331401824951 - rollout_corr/rollout_is_seq_std:0.0012539547169581056 - rollout_corr/rollout_is_seq_max:1.0045143365859985 - rollout_corr/rollout_is_seq_min:0.9951211214065552 - rollout_corr/rollout_is_seq_max_deviation:0.004878878593444824 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.149727716576308) - rollout_corr/training_log_ppl:np.float64(0.7441099616116844) - rollout_corr/kl:np.float64(0.0006301436051412423) - rollout_corr/k3_kl:np.float64(0.0006160447873639896) - rollout_corr/rollout_ppl:np.float64(2.1482933391816914) - rollout_corr/rollout_log_ppl:np.float64(0.7434798121103086) - rollout_corr/log_ppl_diff:np.float64(0.000630149501375854) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011455799685791135) - rollout_corr/log_ppl_diff_max:np.float64(0.0055887699127197266) - rollout_corr/log_ppl_diff_min:np.float64(-0.00408819317817688) - rollout_corr/ppl_ratio:np.float64(1.0006313105113804) - rollout_corr/chi2_token:np.float64(0.0012098602019250393) - rollout_corr/chi2_seq:np.float64(0.841775347944349) - actor/pg_loss:np.float64(-8.723500650376081e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00012781737518707814) - actor/ppo_kl:np.float64(8.237922273224285e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.19401156529784203) - perf/mfu/actor:np.float64(0.13594204140089314) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.93074417114258) - actor/lr:np.float64(1e-06) - training/global_step:99 - training/epoch:1 - critic/score/mean:0.58984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006903177127242088 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006903177127242088 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:831.53515625 - response_length/max:1998.0 - response_length/min:192.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:831.53515625 - response_length_non_aborted/max:1998.0 - response_length_non_aborted/min:192.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.375 - prompt_length/max:836.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.847611904144287e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.493832629173994) - timing_s/agent_loop/generate_sequences/max:np.float64(17.390929045155644) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.606087130981905) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(17.390929045155644) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:493 - timing_s/agent_loop/slowest/response_length:1964 - timing_s/gen:24.117084549739957 - timing_s/reward:0.08298422396183014 - timing_s/old_log_prob:3.351647572591901 - timing_s/adv:0.12600908987224102 - timing_s/update_actor:14.029597079381347 - timing_s/step:41.793392254039645 - timing_s/stop_profile:0.00011212378740310669 - timing_per_token_ms/adv:0.0004302986599289069 - timing_per_token_ms/gen:0.11329329952478688 - timing_per_token_ms/update_actor:0.04790858206119138 - perf/total_num_tokens:292841 - perf/time_per_step:41.793392254039645 - perf/throughput:875.859149635355
(TaskRunner pid=2224264) Training Progress: 99%|█████████▉| 99/100 [1:37:46<00:43, 43.41s/it]
(TaskRunner pid=2224264) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 100}
(TaskRunner pid=2224264) validation generation end
(TaskRunner pid=2224264) [prompt] system
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2224264) <reasoning>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) ...
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) user
(TaskRunner pid=2224264) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2224264) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2224264) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2224264) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2224264) Please reason step by step.
(TaskRunner pid=2224264) assistant
(TaskRunner pid=2224264) <think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) </think>
(TaskRunner pid=2224264)
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) [response] <reasoning>
(TaskRunner pid=2224264) We are given a target product:
(TaskRunner pid=2224264) **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**
(TaskRunner pid=2224264) This is a chemical formula representing a molecule. Let's break it down:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - The main structure is: **Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264) - This can be interpreted as: a ring with a nitrogen (N) at position 1, and a substituent at position 1 (the "C..." part).
(TaskRunner pid=2224264) - The substituent is: **Cc2ccc(Cl)cc2C(F)(F)F**
(TaskRunner pid=2224264) This is a benzene ring (c2ccc...), with a chlorine (Cl) on one carbon, and a trifluoromethyl group (C(F)(F)F) on another carbon. The entire group is attached to a carbon (C) which is attached to the nitrogen (N) in the ring.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Now, we are to identify which of the four options is the **only correct reactant** that would be used in the synthesis of this product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) Let’s analyze each option:
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option A:**
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - This is a complex molecule.
(TaskRunner pid=2224264) - It contains a ring (C1) with two double bonds (c2ccccc2), and a carbonyl group (O=C).
(TaskRunner pid=2224264) - Then, **C(=O)N1** suggests a carbonyl group attached to a nitrogen (N1).
(TaskRunner pid=2224264) - Then, **c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1** matches the target product's ring and substituent.
(TaskRunner pid=2224264) - So, this option contains the full structure of the target product, possibly as a derivative or a starting point for synthesis.
(TaskRunner pid=2224264) ✅ **Possibly a correct reactant.**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option B:**
(TaskRunner pid=2224264) **O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - Contains a ring with Cl attached.
(TaskRunner pid=2224264) - The rest of the structure does not match the substituent **Cc2ccc(Cl)cc2C(F)(F)F**.
(TaskRunner pid=2224264) - **Does not match the target structure.**
(TaskRunner pid=2224264) ❌ **Not correct.**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option C:**
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - This has a ring with a substituent that includes a **C(F)(F)F** group.
(TaskRunner pid=2224264) - However, the **Oc2ccc(Cl)c...** is different from **Cc2ccc(Cl)cc2C(F)(F)F**.
(TaskRunner pid=2224264) - The structure is not matching the target.
(TaskRunner pid=2224264) ❌ **Not correct.**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) **Option D:**
(TaskRunner pid=2224264) **O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) - Contains a **C(F)(F)F** group.
(TaskRunner pid=2224264) - However, the **CC=C(Cc1ccccc1)** is not matching the **Cc2ccc(Cl)cc2C(F)(F)F** part.
(TaskRunner pid=2224264) - No Cl is present in the substituent.
(TaskRunner pid=2224264) ❌ **Not correct.**
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ---
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) ### Final Evaluation:
(TaskRunner pid=2224264) Only **Option A** contains the exact substituent **Cc2ccc(Cl)cc2C(F)(F)F**, which is part of the target molecule, and the rest of the structure is consistent with the ring and functional group of the product.
(TaskRunner pid=2224264)
(TaskRunner pid=2224264) So, **Option A** is the only correct reactant that could be used in the synthesis of the given product.
(TaskRunner pid=2224264) </reasoning>
(TaskRunner pid=2224264) <answer>
(TaskRunner pid=2224264) A
(TaskRunner pid=2224264) </answer>
(TaskRunner pid=2224264) [ground_truth] A
(TaskRunner pid=2224264) [score] 1.0
(TaskRunner pid=2224264) [acc] 1.0
(TaskRunner pid=2224264) [pred] A
(TaskRunner pid=2224264) [incorrect_format] 1
(TaskRunner pid=2224264) [feedback]
(TaskRunner pid=2224264) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_100
(TaskRunner pid=2224264) step:100 - global_seqlen/min:32181 - global_seqlen/max:40888 - global_seqlen/minmax_diff:8707 - global_seqlen/balanced_min:36112 - global_seqlen/balanced_max:36131 - global_seqlen/mean:36125.5 - actor/entropy:0.7533097267150879 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2046983242034912 - training/rollout_probs_diff_mean:0.0041552940383553505 - training/rollout_probs_diff_std:0.00816031452268362 - training/rollout_actor_probs_pearson_corr:0.9996163845062256 - rollout_corr/rollout_is_mean:0.999994695186615 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.690695732278982e-06 - rollout_corr/rollout_is_max:1.4722437858581543 - rollout_corr/rollout_is_min:0.08076818287372589 - rollout_corr/rollout_is_std:0.03385905548930168 - rollout_corr/rollout_is_eff_sample_size:0.9988548770722385 - rollout_corr/rollout_is_seq_mean:0.9999972581863403 - rollout_corr/rollout_is_seq_std:0.0013772835955023766 - rollout_corr/rollout_is_seq_max:1.0046179294586182 - rollout_corr/rollout_is_seq_min:0.9932000041007996 - rollout_corr/rollout_is_seq_max_deviation:0.0067999958992004395 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(2.1957929017953575) - rollout_corr/training_log_ppl:np.float64(0.7637257019523531) - rollout_corr/kl:np.float64(0.0006403183531915602) - rollout_corr/k3_kl:np.float64(0.0006129170444637566) - rollout_corr/rollout_ppl:np.float64(2.194364957511425) - rollout_corr/rollout_log_ppl:np.float64(0.7630853808368556) - rollout_corr/log_ppl_diff:np.float64(0.0006403211154974997) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010970584698952734) - rollout_corr/log_ppl_diff_max:np.float64(0.007486581802368164) - rollout_corr/log_ppl_diff_min:np.float64(-0.0030992627143859863) - rollout_corr/ppl_ratio:np.float64(1.000641372287646) - rollout_corr/chi2_token:np.float64(0.0011783752124756575) - rollout_corr/chi2_seq:np.float64(1.0062491311691701) - actor/pg_loss:np.float64(1.9038328900933266e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00016279506189675885) - actor/ppo_kl:np.float64(4.644049759150448e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.20593710616230965) - perf/mfu/actor:np.float64(0.13613751711674105) - perf/max_memory_allocated_gb:np.float64(132.26526165008545) - perf/max_memory_reserved_gb:np.float64(144.970703125) - perf/cpu_memory_used_gb:np.float64(105.93215942382812) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6901785714285714) - val-aux/sciknoweval/reward/std@16:np.float64(0.21497342883717385) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7771380952380953) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.17605374457438094) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6025857142857143) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.18719434569533389) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6898333333333334) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.21523251642703473) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8483380952380953) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.128790863116223) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5304333333333333) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.14423738321883778) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.7146857142857143) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.17575431812173817) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8984857142857143) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.08310585797508012) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.47553809523809526) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.10546043434417916) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7290142857142858) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.13381993498072356) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.9295428571428572) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.046578487700934236) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.4326952380952381) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.07212858795272402) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7341666666666667) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.10114866635456188) - val-aux/sciknoweval/score/mean@16:np.float64(0.6901785714285714) - val-aux/sciknoweval/score/std@16:np.float64(0.21497342883717385) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7771380952380953) - val-aux/sciknoweval/score/best@2/std:np.float64(0.17605374457438094) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6025857142857143) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.18719434569533389) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6898333333333334) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.21523251642703473) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8483380952380953) - val-aux/sciknoweval/score/best@4/std:np.float64(0.128790863116223) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5304333333333333) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.14423738321883778) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.7146857142857143) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.17575431812173817) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8984857142857143) - val-aux/sciknoweval/score/best@8/std:np.float64(0.08310585797508012) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.47553809523809526) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.10546043434417916) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7290142857142858) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.13381993498072356) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.9295428571428572) - val-aux/sciknoweval/score/best@16/std:np.float64(0.046578487700934236) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.4326952380952381) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.07212858795272402) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7341666666666667) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.10114866635456188) - val-core/sciknoweval/acc/mean@16:np.float64(0.6901785714285714) - val-aux/sciknoweval/acc/std@16:np.float64(0.21497342883717385) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7771380952380953) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.17605374457438094) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6025857142857143) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.18719434569533389) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6898333333333334) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.21523251642703473) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8483380952380953) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.128790863116223) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5304333333333333) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.14423738321883778) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.7146857142857143) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.17575431812173817) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8984857142857143) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.08310585797508012) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.47553809523809526) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.10546043434417916) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7290142857142858) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.13381993498072356) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.9295428571428572) - val-core/sciknoweval/acc/best@16/std:np.float64(0.046578487700934236) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.4326952380952381) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.07212858795272402) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7341666666666667) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.10114866635456188) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9645833333333333) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.08812713073175192) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9939428571428572) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.03432708478331659) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9356666666666668) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.11103403939640785) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9648952380952381) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.08777577522757704) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9996333333333334) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.004692755954297982) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.890790476190476) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.12713323596686327) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9825095238095238) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.06141628384478388) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.8332238095238094) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.12414222258147589) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9934761904761904) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.03401010083743055) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.7765571428571428) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.0965096914106855) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9985285714285715) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.011901839678247213) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:100 - training/epoch:1 - critic/score/mean:0.67578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.67578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0025611198507249355 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0025611198507249355 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:832.765625 - response_length/max:1661.0 - response_length/min:260.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:832.765625 - response_length_non_aborted/max:1661.0 - response_length_non_aborted/min:260.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:296.15625 - prompt_length/max:625.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00019051693379878998 - timing_s/agent_loop/generate_sequences/min:np.float64(3.3696037754416466) - timing_s/agent_loop/generate_sequences/max:np.float64(15.491487443447113) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.964753172069322) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.491487443447113) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:255 - timing_s/agent_loop/slowest/response_length:1661 - timing_s/gen:22.220765927806497 - timing_s/reward:0.08236124739050865 - timing_s/old_log_prob:3.2945325896143913 - timing_s/adv:0.11797017231583595 - timing_s/update_actor:13.748360374942422 - timing_s/step:39.5498822350055 - timing_s/testing:247.1615258976817 - timing_s/save_checkpoint:12.889757573604584 - timing_s/stop_profile:6.954558193683624e-05 - timing_per_token_ms/adv:0.00040819563852346666 - timing_per_token_ms/gen:0.1042308475514874 - timing_per_token_ms/update_actor:0.047571522798793175 - perf/total_num_tokens:289004 - perf/time_per_step:39.5498822350055 - perf/throughput:913.4161205674948
(TaskRunner pid=2224264) ("Final validation metrics: {'val-aux/sciknoweval/reward/mean@16': "
(TaskRunner pid=2224264) "np.float64(0.6901785714285714), 'val-aux/sciknoweval/reward/std@16': "
(TaskRunner pid=2224264) "np.float64(0.21497342883717385), 'val-aux/sciknoweval/reward/best@2/mean': "
(TaskRunner pid=2224264) "np.float64(0.7771380952380953), 'val-aux/sciknoweval/reward/best@2/std': "
(TaskRunner pid=2224264) "np.float64(0.17605374457438094), 'val-aux/sciknoweval/reward/worst@2/mean': "
(TaskRunner pid=2224264) "np.float64(0.6025857142857143), 'val-aux/sciknoweval/reward/worst@2/std': "
(TaskRunner pid=2224264) "np.float64(0.18719434569533389), 'val-aux/sciknoweval/reward/maj@2/mean': "
(TaskRunner pid=2224264) "np.float64(0.6898333333333334), 'val-aux/sciknoweval/reward/maj@2/std': "
(TaskRunner pid=2224264) "np.float64(0.21523251642703473), 'val-aux/sciknoweval/reward/best@4/mean': "
(TaskRunner pid=2224264) "np.float64(0.8483380952380953), 'val-aux/sciknoweval/reward/best@4/std': "
(TaskRunner pid=2224264) "np.float64(0.128790863116223), 'val-aux/sciknoweval/reward/worst@4/mean': "
(TaskRunner pid=2224264) "np.float64(0.5304333333333333), 'val-aux/sciknoweval/reward/worst@4/std': "
(TaskRunner pid=2224264) "np.float64(0.14423738321883778), 'val-aux/sciknoweval/reward/maj@4/mean': "
(TaskRunner pid=2224264) "np.float64(0.7146857142857143), 'val-aux/sciknoweval/reward/maj@4/std': "
(TaskRunner pid=2224264) "np.float64(0.17575431812173817), 'val-aux/sciknoweval/reward/best@8/mean': "
(TaskRunner pid=2224264) "np.float64(0.8984857142857143), 'val-aux/sciknoweval/reward/best@8/std': "
(TaskRunner pid=2224264) "np.float64(0.08310585797508012), 'val-aux/sciknoweval/reward/worst@8/mean': "
(TaskRunner pid=2224264) "np.float64(0.47553809523809526), 'val-aux/sciknoweval/reward/worst@8/std': "
(TaskRunner pid=2224264) "np.float64(0.10546043434417916), 'val-aux/sciknoweval/reward/maj@8/mean': "
(TaskRunner pid=2224264) "np.float64(0.7290142857142858), 'val-aux/sciknoweval/reward/maj@8/std': "
(TaskRunner pid=2224264) "np.float64(0.13381993498072356), 'val-aux/sciknoweval/reward/best@16/mean': "
(TaskRunner pid=2224264) "np.float64(0.9295428571428572), 'val-aux/sciknoweval/reward/best@16/std': "
(TaskRunner pid=2224264) 'np.float64(0.046578487700934236), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/reward/worst@16/mean': np.float64(0.4326952380952381), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/reward/worst@16/std': np.float64(0.07212858795272402), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/reward/maj@16/mean': np.float64(0.7341666666666667), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/reward/maj@16/std': np.float64(0.10114866635456188), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/mean@16': np.float64(0.6901785714285714), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/std@16': np.float64(0.21497342883717385), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/best@2/mean': np.float64(0.7771380952380953), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/best@2/std': np.float64(0.17605374457438094), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/worst@2/mean': np.float64(0.6025857142857143), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/worst@2/std': np.float64(0.18719434569533389), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/maj@2/mean': np.float64(0.6898333333333334), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/maj@2/std': np.float64(0.21523251642703473), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/best@4/mean': np.float64(0.8483380952380953), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/best@4/std': np.float64(0.128790863116223), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/worst@4/mean': np.float64(0.5304333333333333), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/worst@4/std': np.float64(0.14423738321883778), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/maj@4/mean': np.float64(0.7146857142857143), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/maj@4/std': np.float64(0.17575431812173817), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/best@8/mean': np.float64(0.8984857142857143), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/best@8/std': np.float64(0.08310585797508012), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/worst@8/mean': np.float64(0.47553809523809526), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/worst@8/std': np.float64(0.10546043434417916), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/maj@8/mean': np.float64(0.7290142857142858), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/maj@8/std': np.float64(0.13381993498072356), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/best@16/mean': np.float64(0.9295428571428572), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/best@16/std': np.float64(0.046578487700934236), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/worst@16/mean': np.float64(0.4326952380952381), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/worst@16/std': np.float64(0.07212858795272402), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/maj@16/mean': np.float64(0.7341666666666667), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/score/maj@16/std': np.float64(0.10114866635456188), "
(TaskRunner pid=2224264) "'val-core/sciknoweval/acc/mean@16': np.float64(0.6901785714285714), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/std@16': np.float64(0.21497342883717385), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/best@2/mean': np.float64(0.7771380952380953), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/best@2/std': np.float64(0.17605374457438094), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/worst@2/mean': np.float64(0.6025857142857143), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/worst@2/std': np.float64(0.18719434569533389), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/maj@2/mean': np.float64(0.6898333333333334), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/maj@2/std': np.float64(0.21523251642703473), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/best@4/mean': np.float64(0.8483380952380953), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/best@4/std': np.float64(0.128790863116223), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/worst@4/mean': np.float64(0.5304333333333333), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/worst@4/std': np.float64(0.14423738321883778), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/maj@4/mean': np.float64(0.7146857142857143), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/maj@4/std': np.float64(0.17575431812173817), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/best@8/mean': np.float64(0.8984857142857143), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/best@8/std': np.float64(0.08310585797508012), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/worst@8/mean': np.float64(0.47553809523809526), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/worst@8/std': np.float64(0.10546043434417916), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/maj@8/mean': np.float64(0.7290142857142858), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/maj@8/std': np.float64(0.13381993498072356), "
(TaskRunner pid=2224264) "'val-core/sciknoweval/acc/best@16/mean': np.float64(0.9295428571428572), "
(TaskRunner pid=2224264) "'val-core/sciknoweval/acc/best@16/std': np.float64(0.046578487700934236), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/worst@16/mean': np.float64(0.4326952380952381), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/acc/worst@16/std': np.float64(0.07212858795272402), "
(TaskRunner pid=2224264) "'val-core/sciknoweval/acc/maj@16/mean': np.float64(0.7341666666666667), "
(TaskRunner pid=2224264) "'val-core/sciknoweval/acc/maj@16/std': np.float64(0.10114866635456188), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/mean@16': "
(TaskRunner pid=2224264) 'np.float64(0.9645833333333333), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/std@16': "
(TaskRunner pid=2224264) 'np.float64(0.08812713073175192), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/best@2/mean': "
(TaskRunner pid=2224264) 'np.float64(0.9939428571428572), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/best@2/std': "
(TaskRunner pid=2224264) 'np.float64(0.03432708478331659), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/worst@2/mean': "
(TaskRunner pid=2224264) 'np.float64(0.9356666666666668), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/worst@2/std': "
(TaskRunner pid=2224264) 'np.float64(0.11103403939640785), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/maj@2/mean': "
(TaskRunner pid=2224264) 'np.float64(0.9648952380952381), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/maj@2/std': "
(TaskRunner pid=2224264) 'np.float64(0.08777577522757704), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/best@4/mean': "
(TaskRunner pid=2224264) 'np.float64(0.9996333333333334), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/best@4/std': "
(TaskRunner pid=2224264) 'np.float64(0.004692755954297982), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/worst@4/mean': "
(TaskRunner pid=2224264) 'np.float64(0.890790476190476), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/worst@4/std': "
(TaskRunner pid=2224264) 'np.float64(0.12713323596686327), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/maj@4/mean': "
(TaskRunner pid=2224264) 'np.float64(0.9825095238095238), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/maj@4/std': "
(TaskRunner pid=2224264) 'np.float64(0.06141628384478388), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/best@8/mean': np.float64(1.0), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/best@8/std': np.float64(0.0), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/worst@8/mean': "
(TaskRunner pid=2224264) 'np.float64(0.8332238095238094), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/worst@8/std': "
(TaskRunner pid=2224264) 'np.float64(0.12414222258147589), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/maj@8/mean': "
(TaskRunner pid=2224264) 'np.float64(0.9934761904761904), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/maj@8/std': "
(TaskRunner pid=2224264) 'np.float64(0.03401010083743055), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/best@16/mean': np.float64(1.0), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/best@16/std': np.float64(0.0), "
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/worst@16/mean': "
(TaskRunner pid=2224264) 'np.float64(0.7765571428571428), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/worst@16/std': "
(TaskRunner pid=2224264) 'np.float64(0.0965096914106855), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/maj@16/mean': "
(TaskRunner pid=2224264) 'np.float64(0.9985285714285715), '
(TaskRunner pid=2224264) "'val-aux/sciknoweval/incorrect_format/maj@16/std': "
(TaskRunner pid=2224264) "np.float64(0.011901839678247213), 'val-aux/num_turns/min': np.int32(2), "
(TaskRunner pid=2224264) "'val-aux/num_turns/max': np.int32(2), 'val-aux/num_turns/mean': "
(TaskRunner pid=2224264) 'np.float64(2.0)}')
(TaskRunner pid=2224264) Training Progress: 100%|██████████| 100/100 [1:42:46<00:00, 120.29s/it] Training Progress: 100%|██████████| 100/100 [1:42:46<00:00, 61.66s/it]
(TaskRunner pid=2224264) wandb: uploading output.log; uploading wandb-summary.json
(TaskRunner pid=2224264) wandb: uploading output.log
(TaskRunner pid=2224264) wandb: uploading history steps 98-99, summary, console lines 1082-1211
(TaskRunner pid=2224264) wandb:
(TaskRunner pid=2224264) wandb: Run history:
(TaskRunner pid=2224264) wandb: actor/entropy ▁▁▂▃▄▄▅▅▅▅▅▅▅▆▆▆▆▆▆▄▆▅▇▇▆▆█▆▆▆▆▆▆▇▆▅▅▆▅▆
(TaskRunner pid=2224264) wandb: actor/grad_norm █▇▇█▅▅▆▇▅▆▅▄▅▆▄▄▅▄▃▄▃▄▃▂▃▅▄▃▂▂▂▃▁▃▁▂▁▃▂▂
(TaskRunner pid=2224264) wandb: actor/kl_loss ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
(TaskRunner pid=2224264) wandb: actor/lr ▁▃▆█████████████████████████████████████
(TaskRunner pid=2224264) wandb: actor/pg_clipfrac ▇▃██▆▆▆▄▃▄▃▅▅▄▅▆▆▄▅▄▃▅▄▇▃▇▆▃▃▁▄▃▄▆▃▆▄▃▂▃
(TaskRunner pid=2224264) wandb: actor/pg_clipfrac_lower ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▅▁▁▁▁▁▁▁▁▁▅▁▁▁▁█▁▁▁▁
(TaskRunner pid=2224264) wandb: actor/pg_loss ▂▂▅▆▂▃▃█▂▄▃▁▄▆▅▄▇▆▄▅▃▆▄▃▄▂▅▄▄▄▅▅▅▆▃▄▅▄▄▄
(TaskRunner pid=2224264) wandb: actor/ppo_kl ▆▆▁▄▄▄▆▇▅▄▇▅▃▆▇▅▄▄█▆▃▃▄▅▅▄▇▅▁▄▅▆▇▄▄▃▂▇▄▅
(TaskRunner pid=2224264) wandb: critic/advantages/max ███████████████████████████████▁████████
(TaskRunner pid=2224264) wandb: critic/advantages/mean ▃▄▄▆▄▇▆▄▆▄▃▂▁▃▄▁▄▄▅▅▄▂▂▅▄▃▄█▆▇▇▆▇▃▅▄▃▁▇▆
(TaskRunner pid=2224264) wandb: +204 ...
(TaskRunner pid=2224264) wandb:
(TaskRunner pid=2224264) wandb: Run summary:
(TaskRunner pid=2224264) wandb: actor/entropy 0.75331
(TaskRunner pid=2224264) wandb: actor/grad_norm 0.20594
(TaskRunner pid=2224264) wandb: actor/kl_loss 0
(TaskRunner pid=2224264) wandb: actor/lr 0.0
(TaskRunner pid=2224264) wandb: actor/pg_clipfrac 0.00016
(TaskRunner pid=2224264) wandb: actor/pg_clipfrac_lower 0
(TaskRunner pid=2224264) wandb: actor/pg_loss 2e-05
(TaskRunner pid=2224264) wandb: actor/ppo_kl 5e-05
(TaskRunner pid=2224264) wandb: critic/advantages/max 0.875
(TaskRunner pid=2224264) wandb: critic/advantages/mean 0.00256
(TaskRunner pid=2224264) wandb: +204 ...
(TaskRunner pid=2224264) wandb:
(TaskRunner pid=2224264) wandb: 🚀 View run qwen3gen-chemistry-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8 at: https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root/runs/ffovp19e
(TaskRunner pid=2224264) wandb: ⭐️ View project at: https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root
(TaskRunner pid=2224264) wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
(TaskRunner pid=2224264) wandb: Find logs at: ./wandb/run-20260702_235814-ffovp19e/logs
(TaskRunner pid=2224264) Exception ignored in atexit callback: <function _start_and_connect_service.<locals>.teardown_atexit at 0x7797755e3880>
(TaskRunner pid=2224264) Traceback (most recent call last):
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_connection.py", line 65, in teardown_atexit
(TaskRunner pid=2224264) conn.teardown(hooks.exit_code)
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_connection.py", line 299, in teardown
(TaskRunner pid=2224264) self._asyncer.run(publish_teardown_and_close)
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 136, in run
(TaskRunner pid=2224264) return future.result()
(TaskRunner pid=2224264) ^^^^^^^^^^^^^^^
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(TaskRunner pid=2224264) return self.__get_result()
(TaskRunner pid=2224264) ^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(TaskRunner pid=2224264) raise self._exception
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 219, in _wrap
(TaskRunner pid=2224264) return await fn()
(TaskRunner pid=2224264) ^^^^^^^^^^
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_connection.py", line 297, in publish_teardown_and_close
(TaskRunner pid=2224264) await self._client.close()
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_client.py", line 69, in close
(TaskRunner pid=2224264) await self._writer.wait_closed()
(TaskRunner pid=2224264) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/asyncio/streams.py", line 364, in wait_closed
(TaskRunner pid=2224264) await self._protocol._get_close_waiter(self)
(TaskRunner pid=2224264) BrokenPipeError: [Errno 32] Broken pipe
main_ppo exit code: 0
[2026-07-03 01:41:07] Finished Qwen/Qwen3-8B chemistry grpo.
[2026-07-03 01:41:07] Starting Qwen/Qwen3-8B chemistry rlsd_tr.
Experiment: qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8
Dataset: chemistry
Method: rlsd_tr
Config: rlsd
Model: Qwen/Qwen3-8B
Train: /mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet
Val: /mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet
Ray tmp: /tmp/ray_q3g_chemistry_rlsd_tr_Qwen3_8B
2026-07-03 01:41:09,351 INFO scripts.py:1364 -- Did not find any active Ray processes.
Experiment: qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8
Config: rlsd
Task: datasets/sciknoweval/chemistry
Arguments: max_model_len=10240 data.train_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet] data.val_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet] data.train_batch_size=32 data.train_max_samples=3200 data.max_prompt_length=2048 data.max_response_length=8192 data.apply_chat_template_kwargs.enable_thinking=false trainer.group_name=QWEN3-RLSD-TR-GRPO-matched-generalization trainer.n_gpus_per_node=8 trainer.total_training_steps=100 trainer.val_before_train=False trainer.save_freq=10 trainer.test_freq=10 trainer.default_local_dir=/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8 custom_reward_function.path=/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py +ray_kwargs.ray_init._temp_dir=/tmp/ray_q3g_chemistry_rlsd_tr_Qwen3_8B +ray_kwargs.ray_init.include_dashboard=False +ray_kwargs.ray_init.runtime_env.env_vars.USER=root +ray_kwargs.ray_init.runtime_env.env_vars.TASK=datasets/sciknoweval/chemistry +ray_kwargs.ray_init.runtime_env.env_vars.EXPERIMENT=qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8 actor_rollout_ref.model.path=Qwen/Qwen3-8B actor_rollout_ref.actor.optim.lr=1e-6 actor_rollout_ref.actor.optim.lr_warmup_steps=10 actor_rollout_ref.actor.optim.weight_decay=0.01 actor_rollout_ref.actor.grad_clip=1.0 actor_rollout_ref.actor.ppo_mini_batch_size=8 actor_rollout_ref.actor.clip_ratio_high=0.28 actor_rollout_ref.actor.clip_ratio_low=0.2 actor_rollout_ref.actor.ppo_max_token_len_per_gpu=10240 actor_rollout_ref.rollout.n=8 actor_rollout_ref.rollout.temperature=1.0 actor_rollout_ref.rollout.top_p=1.0 actor_rollout_ref.rollout.max_model_len=10240 actor_rollout_ref.rollout.max_num_batched_tokens=10240 actor_rollout_ref.rollout.gpu_memory_utilization=0.8 actor_rollout_ref.rollout.calculate_log_probs=True actor_rollout_ref.rollout.val_kwargs.n=16 actor_rollout_ref.rollout.val_kwargs.temperature=0.6 actor_rollout_ref.rollout.val_kwargs.top_p=0.95 actor_rollout_ref.rollout.val_kwargs.do_sample=True algorithm.norm_adv_by_std_in_grpo=False algorithm.rollout_correction.rollout_is=token algorithm.rollout_correction.rollout_is_threshold=2.0 trainer.max_actor_ckpt_to_keep=100 actor_rollout_ref.model.use_fused_kernels=False actor_rollout_ref.actor.policy_loss.loss_mode=rlsd actor_rollout_ref.actor.self_distillation.token_reweight_lambda=0.5 actor_rollout_ref.actor.self_distillation.token_reweight_eps_w=0.2 actor_rollout_ref.actor.self_distillation.token_reweight_decay_steps=0 actor_rollout_ref.actor.self_distillation.teacher_regularization=trust-region actor_rollout_ref.actor.self_distillation.teacher_update_rate=0.1 actor_rollout_ref.actor.self_distillation.max_reprompt_len=10240
ray init kwargs: {'num_cpus': None, '_temp_dir': '/tmp/ray_q3g_chemistry_rlsd_tr_Qwen3_8B', 'include_dashboard': False, 'runtime_env': {'env_vars': {'TOKENIZERS_PARALLELISM': 'true', 'NCCL_DEBUG': 'WARN', 'VLLM_LOGGING_LEVEL': 'WARN', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': 'true', 'VLLM_ALLREDUCE_USE_SYMM_MEM': '0', 'CUDA_DEVICE_MAX_CONNECTIONS': '1', 'NCCL_CUMEM_ENABLE': '0', 'USER': 'root', 'TASK': 'datasets/sciknoweval/chemistry', 'EXPERIMENT': 'qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8'}, 'working_dir': None}}
2026-07-03 01:41:19,207 INFO worker.py:2007 -- Started a local Ray instance.
/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(TaskRunner pid=2255397) TaskRunner hostname: mole-workspace-rw, PID: 2255397
(TaskRunner pid=2255397) {'actor_rollout_ref': {'actor': {'_target_': 'verl.workers.config.FSDPActorConfig',
(TaskRunner pid=2255397) 'calculate_entropy': False,
(TaskRunner pid=2255397) 'calculate_sum_pi_squared': False,
(TaskRunner pid=2255397) 'checkpoint': {'_target_': 'verl.trainer.config.CheckpointConfig',
(TaskRunner pid=2255397) 'async_save': False,
(TaskRunner pid=2255397) 'load_contents': ['model',
(TaskRunner pid=2255397) 'optimizer',
(TaskRunner pid=2255397) 'extra'],
(TaskRunner pid=2255397) 'save_contents': ['model',
(TaskRunner pid=2255397) 'optimizer',
(TaskRunner pid=2255397) 'extra']},
(TaskRunner pid=2255397) 'clip_ratio': 0.2,
(TaskRunner pid=2255397) 'clip_ratio_c': 3.0,
(TaskRunner pid=2255397) 'clip_ratio_high': 0.28,
(TaskRunner pid=2255397) 'clip_ratio_low': 0.2,
(TaskRunner pid=2255397) 'data_loader_seed': 42,
(TaskRunner pid=2255397) 'entropy_checkpointing': False,
(TaskRunner pid=2255397) 'entropy_coeff': 0,
(TaskRunner pid=2255397) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2255397) 'freeze_vision_tower': False,
(TaskRunner pid=2255397) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2255397) 'dtype': 'bfloat16',
(TaskRunner pid=2255397) 'entropy_checkpointing': False,
(TaskRunner pid=2255397) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2255397) 'forward_only': False,
(TaskRunner pid=2255397) 'forward_prefetch': False,
(TaskRunner pid=2255397) 'fsdp_size': -1,
(TaskRunner pid=2255397) 'full_determinism': False,
(TaskRunner pid=2255397) 'model_dtype': 'fp32',
(TaskRunner pid=2255397) 'offload_policy': False,
(TaskRunner pid=2255397) 'optimizer_offload': False,
(TaskRunner pid=2255397) 'param_offload': False,
(TaskRunner pid=2255397) 'reshard_after_forward': True,
(TaskRunner pid=2255397) 'seed': 42,
(TaskRunner pid=2255397) 'strategy': 'fsdp',
(TaskRunner pid=2255397) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2255397) 'use_orig_params': False,
(TaskRunner pid=2255397) 'use_torch_compile': True,
(TaskRunner pid=2255397) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2255397) 'grad_clip': 1.0,
(TaskRunner pid=2255397) 'kl_loss_coef': 0.001,
(TaskRunner pid=2255397) 'kl_loss_type': 'low_var_kl',
(TaskRunner pid=2255397) 'loss_agg_mode': 'token-mean',
(TaskRunner pid=2255397) 'loss_scale_factor': None,
(TaskRunner pid=2255397) 'optim': {'_target_': 'verl.workers.config.FSDPOptimizerConfig',
(TaskRunner pid=2255397) 'betas': [0.9, 0.999],
(TaskRunner pid=2255397) 'clip_grad': 1.0,
(TaskRunner pid=2255397) 'lr': 1e-06,
(TaskRunner pid=2255397) 'lr_scheduler_type': 'constant',
(TaskRunner pid=2255397) 'lr_warmup_steps': 10,
(TaskRunner pid=2255397) 'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=2255397) 'min_lr_ratio': 0.0,
(TaskRunner pid=2255397) 'num_cycles': 0.5,
(TaskRunner pid=2255397) 'optimizer': 'AdamW',
(TaskRunner pid=2255397) 'optimizer_impl': 'torch.optim',
(TaskRunner pid=2255397) 'override_optimizer_config': None,
(TaskRunner pid=2255397) 'total_training_steps': -1,
(TaskRunner pid=2255397) 'warmup_style': None,
(TaskRunner pid=2255397) 'weight_decay': 0.01},
(TaskRunner pid=2255397) 'policy_loss': {'_target_': 'verl.workers.config.PolicyLossConfig',
(TaskRunner pid=2255397) 'clip_cov_lb': 1.0,
(TaskRunner pid=2255397) 'clip_cov_ratio': 0.0002,
(TaskRunner pid=2255397) 'clip_cov_ub': 5.0,
(TaskRunner pid=2255397) 'kl_cov_ratio': 0.0002,
(TaskRunner pid=2255397) 'loss_mode': 'rlsd',
(TaskRunner pid=2255397) 'ppo_kl_coef': 0.1},
(TaskRunner pid=2255397) 'ppo_epochs': 1,
(TaskRunner pid=2255397) 'ppo_max_token_len_per_gpu': 10240,
(TaskRunner pid=2255397) 'ppo_micro_batch_size': None,
(TaskRunner pid=2255397) 'ppo_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2255397) 'ppo_mini_batch_size': 8,
(TaskRunner pid=2255397) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2255397) 'all_ranks': False,
(TaskRunner pid=2255397) 'enable': False,
(TaskRunner pid=2255397) 'ranks': [],
(TaskRunner pid=2255397) 'save_path': 'outputs/profile',
(TaskRunner pid=2255397) 'tool': None,
(TaskRunner pid=2255397) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2255397) 'analysis': True,
(TaskRunner pid=2255397) 'contents': [],
(TaskRunner pid=2255397) 'discrete': False,
(TaskRunner pid=2255397) 'level': 'level0'},
(TaskRunner pid=2255397) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2255397) 'discrete': False},
(TaskRunner pid=2255397) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2255397) 'step_end': None,
(TaskRunner pid=2255397) 'step_start': 0},
(TaskRunner pid=2255397) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2255397) 'stack_depth': 32,
(TaskRunner pid=2255397) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2255397) 'rollout_n': 8,
(TaskRunner pid=2255397) 'router_replay': {'_target_': 'verl.workers.config.RouterReplayConfig',
(TaskRunner pid=2255397) 'mode': 'disabled',
(TaskRunner pid=2255397) 'record_file': None,
(TaskRunner pid=2255397) 'replay_file': None},
(TaskRunner pid=2255397) 'self_distillation': {'_target_': 'verl.workers.config.SelfDistillationConfig',
(TaskRunner pid=2255397) 'alpha': 0.5,
(TaskRunner pid=2255397) 'distillation_add_tail': True,
(TaskRunner pid=2255397) 'distillation_topk': 100,
(TaskRunner pid=2255397) 'dont_reprompt_on_self_success': True,
(TaskRunner pid=2255397) 'environment_feedback_only_without_solution': True,
(TaskRunner pid=2255397) 'evolving_teacher': {'_target_': 'verl.workers.config.EvolvingTeacherConfig',
(TaskRunner pid=2255397) 'enable': False,
(TaskRunner pid=2255397) 'loss_weight': 0.0,
(TaskRunner pid=2255397) 'mask': 'all'},
(TaskRunner pid=2255397) 'feedback_template': '\n'
(TaskRunner pid=2255397) 'The '
(TaskRunner pid=2255397) 'following '
(TaskRunner pid=2255397) 'is '
(TaskRunner pid=2255397) 'feedback '
(TaskRunner pid=2255397) 'from '
(TaskRunner pid=2255397) 'your '
(TaskRunner pid=2255397) 'unsuccessful '
(TaskRunner pid=2255397) 'earlier '
(TaskRunner pid=2255397) 'attempt:\n'
(TaskRunner pid=2255397) '\n'
(TaskRunner pid=2255397) '{feedback_raw}',
(TaskRunner pid=2255397) 'full_logit_distillation': True,
(TaskRunner pid=2255397) 'include_environment_feedback': True,
(TaskRunner pid=2255397) 'is_clip': 2,
(TaskRunner pid=2255397) 'max_reprompt_len': 10240,
(TaskRunner pid=2255397) 'remove_thinking_from_demonstration': False,
(TaskRunner pid=2255397) 'reprompt_template': '{prompt}{solution}{feedback}\n'
(TaskRunner pid=2255397) '\n'
(TaskRunner pid=2255397) 'Correctly '
(TaskRunner pid=2255397) 'solve '
(TaskRunner pid=2255397) 'the '
(TaskRunner pid=2255397) 'original '
(TaskRunner pid=2255397) 'question.',
(TaskRunner pid=2255397) 'reprompt_truncation': 'right',
(TaskRunner pid=2255397) 'solution_template': '\n'
(TaskRunner pid=2255397) 'Correct '
(TaskRunner pid=2255397) 'solution:\n'
(TaskRunner pid=2255397) '\n'
(TaskRunner pid=2255397) '{successful_previous_attempt}',
(TaskRunner pid=2255397) 'srpo_dynamic_weighting': False,
(TaskRunner pid=2255397) 'srpo_dynamic_weighting_temperature': 1.0,
(TaskRunner pid=2255397) 'success_reward_threshold': 0.5,
(TaskRunner pid=2255397) 'teacher_regularization': 'trust-region',
(TaskRunner pid=2255397) 'teacher_update_rate': 0.1,
(TaskRunner pid=2255397) 'token_reweight_decay_steps': 0,
(TaskRunner pid=2255397) 'token_reweight_eps_w': 0.2,
(TaskRunner pid=2255397) 'token_reweight_lambda': 0.5},
(TaskRunner pid=2255397) 'shuffle': False,
(TaskRunner pid=2255397) 'strategy': 'fsdp',
(TaskRunner pid=2255397) 'sum_pi_squared_checkpointing': False,
(TaskRunner pid=2255397) 'tau_neg': 1.05,
(TaskRunner pid=2255397) 'tau_pos': 1.0,
(TaskRunner pid=2255397) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2255397) 'use_dynamic_bsz': False,
(TaskRunner pid=2255397) 'use_fused_kernels': False,
(TaskRunner pid=2255397) 'use_kl_loss': False,
(TaskRunner pid=2255397) 'use_prefix_grouper': False,
(TaskRunner pid=2255397) 'use_remove_padding': True,
(TaskRunner pid=2255397) 'use_torch_compile': True},
(TaskRunner pid=2255397) 'hybrid_engine': True,
(TaskRunner pid=2255397) 'model': {'_target_': 'verl.workers.config.HFModelConfig',
(TaskRunner pid=2255397) 'custom_chat_template': None,
(TaskRunner pid=2255397) 'enable_activation_offload': False,
(TaskRunner pid=2255397) 'enable_gradient_checkpointing': True,
(TaskRunner pid=2255397) 'exclude_modules': None,
(TaskRunner pid=2255397) 'external_lib': None,
(TaskRunner pid=2255397) 'fused_kernel_options': {'impl_backend': 'torch'},
(TaskRunner pid=2255397) 'hf_config_path': None,
(TaskRunner pid=2255397) 'lora_adapter_path': None,
(TaskRunner pid=2255397) 'lora_alpha': 16,
(TaskRunner pid=2255397) 'lora_rank': 0,
(TaskRunner pid=2255397) 'override_config': {},
(TaskRunner pid=2255397) 'path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2255397) 'target_modules': 'all-linear',
(TaskRunner pid=2255397) 'tiled_mlp': {'enabled': False,
(TaskRunner pid=2255397) 'num_shards': 4},
(TaskRunner pid=2255397) 'tokenizer_path': None,
(TaskRunner pid=2255397) 'trust_remote_code': True,
(TaskRunner pid=2255397) 'use_fused_kernels': False,
(TaskRunner pid=2255397) 'use_liger': False,
(TaskRunner pid=2255397) 'use_remove_padding': True,
(TaskRunner pid=2255397) 'use_shm': False},
(TaskRunner pid=2255397) 'nccl_timeout': 600,
(TaskRunner pid=2255397) 'ref': {'_target_': 'verl.workers.config.FSDPActorConfig',
(TaskRunner pid=2255397) 'entropy_checkpointing': False,
(TaskRunner pid=2255397) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2255397) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2255397) 'dtype': 'bfloat16',
(TaskRunner pid=2255397) 'entropy_checkpointing': False,
(TaskRunner pid=2255397) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2255397) 'forward_only': True,
(TaskRunner pid=2255397) 'forward_prefetch': False,
(TaskRunner pid=2255397) 'fsdp_size': -1,
(TaskRunner pid=2255397) 'full_determinism': False,
(TaskRunner pid=2255397) 'model_dtype': 'fp32',
(TaskRunner pid=2255397) 'offload_policy': False,
(TaskRunner pid=2255397) 'optimizer_offload': False,
(TaskRunner pid=2255397) 'param_offload': False,
(TaskRunner pid=2255397) 'reshard_after_forward': True,
(TaskRunner pid=2255397) 'seed': 42,
(TaskRunner pid=2255397) 'strategy': 'fsdp',
(TaskRunner pid=2255397) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2255397) 'use_orig_params': False,
(TaskRunner pid=2255397) 'use_torch_compile': True,
(TaskRunner pid=2255397) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2255397) 'log_prob_max_token_len_per_gpu': 10240,
(TaskRunner pid=2255397) 'log_prob_micro_batch_size': None,
(TaskRunner pid=2255397) 'log_prob_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2255397) 'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=2255397) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2255397) 'all_ranks': False,
(TaskRunner pid=2255397) 'enable': False,
(TaskRunner pid=2255397) 'ranks': [],
(TaskRunner pid=2255397) 'save_path': 'outputs/profile',
(TaskRunner pid=2255397) 'tool': None,
(TaskRunner pid=2255397) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2255397) 'analysis': True,
(TaskRunner pid=2255397) 'contents': [],
(TaskRunner pid=2255397) 'discrete': False,
(TaskRunner pid=2255397) 'level': 'level0'},
(TaskRunner pid=2255397) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2255397) 'discrete': False},
(TaskRunner pid=2255397) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2255397) 'step_end': None,
(TaskRunner pid=2255397) 'step_start': 0},
(TaskRunner pid=2255397) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2255397) 'stack_depth': 32,
(TaskRunner pid=2255397) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2255397) 'rollout_n': 8,
(TaskRunner pid=2255397) 'router_replay': {'_target_': 'verl.workers.config.RouterReplayConfig',
(TaskRunner pid=2255397) 'mode': 'disabled',
(TaskRunner pid=2255397) 'record_file': None,
(TaskRunner pid=2255397) 'replay_file': None},
(TaskRunner pid=2255397) 'strategy': 'fsdp',
(TaskRunner pid=2255397) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2255397) 'use_torch_compile': True},
(TaskRunner pid=2255397) 'rollout': {'_target_': 'verl.workers.config.RolloutConfig',
(TaskRunner pid=2255397) 'agent': {'_target_': 'verl.workers.config.AgentLoopConfig',
(TaskRunner pid=2255397) 'agent_loop_config_path': None,
(TaskRunner pid=2255397) 'custom_async_server': {'_target_': 'verl.workers.config.CustomAsyncServerConfig',
(TaskRunner pid=2255397) 'name': None,
(TaskRunner pid=2255397) 'path': None},
(TaskRunner pid=2255397) 'default_agent_loop': 'single_turn_agent',
(TaskRunner pid=2255397) 'num_workers': 8},
(TaskRunner pid=2255397) 'calculate_log_probs': True,
(TaskRunner pid=2255397) 'cudagraph_capture_sizes': None,
(TaskRunner pid=2255397) 'data_parallel_size': 1,
(TaskRunner pid=2255397) 'disable_log_stats': True,
(TaskRunner pid=2255397) 'do_sample': True,
(TaskRunner pid=2255397) 'dtype': 'bfloat16',
(TaskRunner pid=2255397) 'enable_chunked_prefill': True,
(TaskRunner pid=2255397) 'enable_prefix_caching': True,
(TaskRunner pid=2255397) 'enable_rollout_routing_replay': False,
(TaskRunner pid=2255397) 'enforce_eager': False,
(TaskRunner pid=2255397) 'engine_kwargs': {'sglang': {}, 'vllm': {}},
(TaskRunner pid=2255397) 'expert_parallel_size': 1,
(TaskRunner pid=2255397) 'free_cache_engine': True,
(TaskRunner pid=2255397) 'gpu_memory_utilization': 0.8,
(TaskRunner pid=2255397) 'ignore_eos': False,
(TaskRunner pid=2255397) 'layered_summon': False,
(TaskRunner pid=2255397) 'load_format': 'dummy',
(TaskRunner pid=2255397) 'log_prob_max_token_len_per_gpu': 10240,
(TaskRunner pid=2255397) 'log_prob_micro_batch_size': None,
(TaskRunner pid=2255397) 'log_prob_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2255397) 'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=2255397) 'logprobs_mode': 'processed_logprobs',
(TaskRunner pid=2255397) 'max_model_len': 10240,
(TaskRunner pid=2255397) 'max_num_batched_tokens': 10240,
(TaskRunner pid=2255397) 'max_num_seqs': 1024,
(TaskRunner pid=2255397) 'mode': 'async',
(TaskRunner pid=2255397) 'multi_stage_wake_up': False,
(TaskRunner pid=2255397) 'multi_turn': {'_target_': 'verl.workers.config.MultiTurnConfig',
(TaskRunner pid=2255397) 'enable': False,
(TaskRunner pid=2255397) 'format': 'hermes',
(TaskRunner pid=2255397) 'interaction_config_path': None,
(TaskRunner pid=2255397) 'max_assistant_turns': None,
(TaskRunner pid=2255397) 'max_parallel_calls': 1,
(TaskRunner pid=2255397) 'max_tool_response_length': 256,
(TaskRunner pid=2255397) 'max_user_turns': None,
(TaskRunner pid=2255397) 'num_repeat_rollouts': None,
(TaskRunner pid=2255397) 'tokenization_sanity_check_mode': 'strict',
(TaskRunner pid=2255397) 'tool_config_path': None,
(TaskRunner pid=2255397) 'tool_response_truncate_side': 'middle',
(TaskRunner pid=2255397) 'use_inference_chat_template': False},
(TaskRunner pid=2255397) 'n': 8,
(TaskRunner pid=2255397) 'name': 'vllm',
(TaskRunner pid=2255397) 'over_sample_rate': 0,
(TaskRunner pid=2255397) 'pipeline_model_parallel_size': 1,
(TaskRunner pid=2255397) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2255397) 'all_ranks': False,
(TaskRunner pid=2255397) 'enable': False,
(TaskRunner pid=2255397) 'ranks': [],
(TaskRunner pid=2255397) 'save_path': 'outputs/profile',
(TaskRunner pid=2255397) 'tool': None,
(TaskRunner pid=2255397) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2255397) 'analysis': True,
(TaskRunner pid=2255397) 'contents': [],
(TaskRunner pid=2255397) 'discrete': False,
(TaskRunner pid=2255397) 'level': 'level0'},
(TaskRunner pid=2255397) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2255397) 'discrete': False},
(TaskRunner pid=2255397) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2255397) 'step_end': None,
(TaskRunner pid=2255397) 'step_start': 0},
(TaskRunner pid=2255397) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2255397) 'stack_depth': 32,
(TaskRunner pid=2255397) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2255397) 'prometheus': {'_target_': 'verl.workers.config.PrometheusConfig',
(TaskRunner pid=2255397) 'enable': False,
(TaskRunner pid=2255397) 'file': '/tmp/ray/session_latest/metrics/prometheus/prometheus.yml',
(TaskRunner pid=2255397) 'port': 9090,
(TaskRunner pid=2255397) 'served_model_name': 'Qwen/Qwen3-8B'},
(TaskRunner pid=2255397) 'prompt_length': 2048,
(TaskRunner pid=2255397) 'quantization': None,
(TaskRunner pid=2255397) 'quantization_config_file': None,
(TaskRunner pid=2255397) 'response_length': 8192,
(TaskRunner pid=2255397) 'scheduling_policy': 'fcfs',
(TaskRunner pid=2255397) 'skip_dump_dir': '/tmp/rollout_dump',
(TaskRunner pid=2255397) 'skip_rollout': False,
(TaskRunner pid=2255397) 'skip_tokenizer_init': True,
(TaskRunner pid=2255397) 'temperature': 1.0,
(TaskRunner pid=2255397) 'tensor_model_parallel_size': 2,
(TaskRunner pid=2255397) 'top_k': -1,
(TaskRunner pid=2255397) 'top_p': 1.0,
(TaskRunner pid=2255397) 'trace': {'_target_': 'verl.workers.config.TraceConfig',
(TaskRunner pid=2255397) 'backend': None,
(TaskRunner pid=2255397) 'max_samples_per_step_per_worker': None,
(TaskRunner pid=2255397) 'token2text': False},
(TaskRunner pid=2255397) 'update_weights_bucket_megabytes': 512,
(TaskRunner pid=2255397) 'val_kwargs': {'_target_': 'verl.workers.config.SamplingConfig',
(TaskRunner pid=2255397) 'do_sample': True,
(TaskRunner pid=2255397) 'n': 16,
(TaskRunner pid=2255397) 'temperature': 0.6,
(TaskRunner pid=2255397) 'top_k': -1,
(TaskRunner pid=2255397) 'top_p': 0.95}}},
(TaskRunner pid=2255397) 'algorithm': {'_target_': 'verl.trainer.config.AlgoConfig',
(TaskRunner pid=2255397) 'adv_estimator': 'grpo',
(TaskRunner pid=2255397) 'gamma': 1.0,
(TaskRunner pid=2255397) 'kl_ctrl': {'_target_': 'verl.trainer.config.KLControlConfig',
(TaskRunner pid=2255397) 'horizon': 10000,
(TaskRunner pid=2255397) 'kl_coef': 0.001,
(TaskRunner pid=2255397) 'target_kl': 0.1,
(TaskRunner pid=2255397) 'type': 'fixed'},
(TaskRunner pid=2255397) 'kl_penalty': 'kl',
(TaskRunner pid=2255397) 'lam': 1.0,
(TaskRunner pid=2255397) 'norm_adv_by_std_in_grpo': False,
(TaskRunner pid=2255397) 'pf_ppo': {'reweight_method': 'pow', 'weight_pow': 2.0},
(TaskRunner pid=2255397) 'rollout_correction': {'bypass_mode': False,
(TaskRunner pid=2255397) 'loss_type': 'ppo_clip',
(TaskRunner pid=2255397) 'rollout_is': 'token',
(TaskRunner pid=2255397) 'rollout_is_batch_normalize': False,
(TaskRunner pid=2255397) 'rollout_is_threshold': 2.0,
(TaskRunner pid=2255397) 'rollout_rs': None,
(TaskRunner pid=2255397) 'rollout_rs_threshold': None},
(TaskRunner pid=2255397) 'use_kl_in_reward': False,
(TaskRunner pid=2255397) 'use_pf_ppo': False},
(TaskRunner pid=2255397) 'critic': {'_target_': 'verl.workers.config.FSDPCriticConfig',
(TaskRunner pid=2255397) 'checkpoint': {'_target_': 'verl.trainer.config.CheckpointConfig',
(TaskRunner pid=2255397) 'async_save': False,
(TaskRunner pid=2255397) 'load_contents': ['model', 'optimizer', 'extra'],
(TaskRunner pid=2255397) 'save_contents': ['model', 'optimizer', 'extra']},
(TaskRunner pid=2255397) 'cliprange_value': 0.5,
(TaskRunner pid=2255397) 'data_loader_seed': 42,
(TaskRunner pid=2255397) 'enable': None,
(TaskRunner pid=2255397) 'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=2255397) 'forward_micro_batch_size': None,
(TaskRunner pid=2255397) 'forward_micro_batch_size_per_gpu': None,
(TaskRunner pid=2255397) 'grad_clip': 1.0,
(TaskRunner pid=2255397) 'loss_agg_mode': 'token-mean',
(TaskRunner pid=2255397) 'model': {'_target_': 'verl.workers.config.FSDPCriticModelCfg',
(TaskRunner pid=2255397) 'enable_activation_offload': False,
(TaskRunner pid=2255397) 'enable_gradient_checkpointing': True,
(TaskRunner pid=2255397) 'external_lib': None,
(TaskRunner pid=2255397) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2255397) 'dtype': 'bfloat16',
(TaskRunner pid=2255397) 'entropy_checkpointing': False,
(TaskRunner pid=2255397) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2255397) 'forward_only': False,
(TaskRunner pid=2255397) 'forward_prefetch': False,
(TaskRunner pid=2255397) 'fsdp_size': -1,
(TaskRunner pid=2255397) 'full_determinism': False,
(TaskRunner pid=2255397) 'model_dtype': 'fp32',
(TaskRunner pid=2255397) 'offload_policy': False,
(TaskRunner pid=2255397) 'optimizer_offload': False,
(TaskRunner pid=2255397) 'param_offload': False,
(TaskRunner pid=2255397) 'reshard_after_forward': True,
(TaskRunner pid=2255397) 'seed': 42,
(TaskRunner pid=2255397) 'strategy': 'fsdp',
(TaskRunner pid=2255397) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2255397) 'use_orig_params': False,
(TaskRunner pid=2255397) 'use_torch_compile': True,
(TaskRunner pid=2255397) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2255397) 'lora_alpha': 16,
(TaskRunner pid=2255397) 'lora_rank': 0,
(TaskRunner pid=2255397) 'override_config': {},
(TaskRunner pid=2255397) 'path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2255397) 'target_modules': 'all-linear',
(TaskRunner pid=2255397) 'tiled_mlp': {'enabled': False, 'num_shards': 4},
(TaskRunner pid=2255397) 'tokenizer_path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2255397) 'trust_remote_code': True,
(TaskRunner pid=2255397) 'use_remove_padding': False,
(TaskRunner pid=2255397) 'use_shm': False},
(TaskRunner pid=2255397) 'optim': {'_target_': 'verl.workers.config.FSDPOptimizerConfig',
(TaskRunner pid=2255397) 'betas': [0.9, 0.999],
(TaskRunner pid=2255397) 'clip_grad': 1.0,
(TaskRunner pid=2255397) 'lr': 1e-05,
(TaskRunner pid=2255397) 'lr_scheduler_type': 'constant',
(TaskRunner pid=2255397) 'lr_warmup_steps': -1,
(TaskRunner pid=2255397) 'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=2255397) 'min_lr_ratio': 0.0,
(TaskRunner pid=2255397) 'num_cycles': 0.5,
(TaskRunner pid=2255397) 'optimizer': 'AdamW',
(TaskRunner pid=2255397) 'optimizer_impl': 'torch.optim',
(TaskRunner pid=2255397) 'override_optimizer_config': None,
(TaskRunner pid=2255397) 'total_training_steps': -1,
(TaskRunner pid=2255397) 'warmup_style': None,
(TaskRunner pid=2255397) 'weight_decay': 0.01},
(TaskRunner pid=2255397) 'ppo_epochs': 1,
(TaskRunner pid=2255397) 'ppo_max_token_len_per_gpu': 32768,
(TaskRunner pid=2255397) 'ppo_micro_batch_size': None,
(TaskRunner pid=2255397) 'ppo_micro_batch_size_per_gpu': None,
(TaskRunner pid=2255397) 'ppo_mini_batch_size': 8,
(TaskRunner pid=2255397) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2255397) 'all_ranks': False,
(TaskRunner pid=2255397) 'enable': False,
(TaskRunner pid=2255397) 'ranks': [],
(TaskRunner pid=2255397) 'save_path': 'outputs/profile',
(TaskRunner pid=2255397) 'tool': None,
(TaskRunner pid=2255397) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2255397) 'analysis': True,
(TaskRunner pid=2255397) 'contents': [],
(TaskRunner pid=2255397) 'discrete': False,
(TaskRunner pid=2255397) 'level': 'level0'},
(TaskRunner pid=2255397) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2255397) 'discrete': False},
(TaskRunner pid=2255397) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2255397) 'step_end': None,
(TaskRunner pid=2255397) 'step_start': 0},
(TaskRunner pid=2255397) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2255397) 'stack_depth': 32,
(TaskRunner pid=2255397) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2255397) 'rollout_n': 8,
(TaskRunner pid=2255397) 'shuffle': False,
(TaskRunner pid=2255397) 'strategy': 'fsdp',
(TaskRunner pid=2255397) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2255397) 'use_dynamic_bsz': False},
(TaskRunner pid=2255397) 'custom_reward_function': {'name': 'compute_score',
(TaskRunner pid=2255397) 'path': '/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py'},
(TaskRunner pid=2255397) 'data': {'apply_chat_template_kwargs': {'enable_thinking': False},
(TaskRunner pid=2255397) 'custom_cls': {'name': None, 'path': None},
(TaskRunner pid=2255397) 'datagen': {'name': None, 'path': None},
(TaskRunner pid=2255397) 'dataloader_num_workers': 8,
(TaskRunner pid=2255397) 'filter_overlong_prompts': True,
(TaskRunner pid=2255397) 'filter_overlong_prompts_workers': 1,
(TaskRunner pid=2255397) 'image_key': 'images',
(TaskRunner pid=2255397) 'image_patch_size': 14,
(TaskRunner pid=2255397) 'max_prompt_length': 2048,
(TaskRunner pid=2255397) 'max_response_length': 8192,
(TaskRunner pid=2255397) 'prompt_key': 'prompt',
(TaskRunner pid=2255397) 'return_full_prompt': False,
(TaskRunner pid=2255397) 'return_multi_modal_inputs': True,
(TaskRunner pid=2255397) 'return_raw_chat': True,
(TaskRunner pid=2255397) 'return_raw_input_ids': False,
(TaskRunner pid=2255397) 'reward_fn_key': 'data_source',
(TaskRunner pid=2255397) 'sampler': {'class_name': None, 'class_path': None},
(TaskRunner pid=2255397) 'seed': None,
(TaskRunner pid=2255397) 'shuffle': True,
(TaskRunner pid=2255397) 'tokenizer': None,
(TaskRunner pid=2255397) 'tool_config_path': None,
(TaskRunner pid=2255397) 'train_batch_size': 32,
(TaskRunner pid=2255397) 'train_files': ['/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/train.parquet'],
(TaskRunner pid=2255397) 'train_max_samples': 3200,
(TaskRunner pid=2255397) 'truncation': 'error',
(TaskRunner pid=2255397) 'trust_remote_code': True,
(TaskRunner pid=2255397) 'use_shm': False,
(TaskRunner pid=2255397) 'val_batch_size': None,
(TaskRunner pid=2255397) 'val_files': ['/mnt/mole/SDPO/L2T/datasets/sciknoweval/chemistry/test.parquet'],
(TaskRunner pid=2255397) 'val_max_samples': -1,
(TaskRunner pid=2255397) 'validation_shuffle': False,
(TaskRunner pid=2255397) 'video_key': 'videos'},
(TaskRunner pid=2255397) 'global_profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2255397) 'global_tool_config': {'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2255397) 'controller_nsight_options': {'cuda-graph-trace': 'graph',
(TaskRunner pid=2255397) 'cuda-memory-usage': 'true',
(TaskRunner pid=2255397) 'trace': 'cuda,nvtx,cublas,ucx'},
(TaskRunner pid=2255397) 'discrete': False,
(TaskRunner pid=2255397) 'worker_nsight_options': {'capture-range': 'cudaProfilerApi',
(TaskRunner pid=2255397) 'capture-range-end': None,
(TaskRunner pid=2255397) 'cuda-graph-trace': 'graph',
(TaskRunner pid=2255397) 'cuda-memory-usage': 'true',
(TaskRunner pid=2255397) 'kill': 'none',
(TaskRunner pid=2255397) 'trace': 'cuda,nvtx,cublas,ucx'}},
(TaskRunner pid=2255397) 'torch_memory': {'context': 'all',
(TaskRunner pid=2255397) 'kw_args': {},
(TaskRunner pid=2255397) 'stack_depth': 32,
(TaskRunner pid=2255397) 'stacks': 'all',
(TaskRunner pid=2255397) 'trace_alloc_max_entries': 100000}},
(TaskRunner pid=2255397) 'profile_continuous_steps': False,
(TaskRunner pid=2255397) 'save_path': 'outputs/profile',
(TaskRunner pid=2255397) 'steps': None,
(TaskRunner pid=2255397) 'tool': None},
(TaskRunner pid=2255397) 'max_model_len': 10240,
(TaskRunner pid=2255397) 'ray_kwargs': {'ray_init': {'_temp_dir': '/tmp/ray_q3g_chemistry_rlsd_tr_Qwen3_8B',
(TaskRunner pid=2255397) 'include_dashboard': False,
(TaskRunner pid=2255397) 'num_cpus': None,
(TaskRunner pid=2255397) 'runtime_env': {'env_vars': {'EXPERIMENT': 'qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2255397) 'TASK': 'datasets/sciknoweval/chemistry',
(TaskRunner pid=2255397) 'USER': 'root'}}},
(TaskRunner pid=2255397) 'timeline_json_file': None},
(TaskRunner pid=2255397) 'reward_manager': {'_target_': 'verl.trainer.config.config.RewardManagerConfig',
(TaskRunner pid=2255397) 'module': {'_target_': 'verl.trainer.config.config.ModuleConfig',
(TaskRunner pid=2255397) 'name': 'custom_reward_manager',
(TaskRunner pid=2255397) 'path': None},
(TaskRunner pid=2255397) 'name': 'naive',
(TaskRunner pid=2255397) 'source': 'register'},
(TaskRunner pid=2255397) 'reward_model': {'enable': False,
(TaskRunner pid=2255397) 'enable_resource_pool': False,
(TaskRunner pid=2255397) 'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=2255397) 'launch_reward_fn_async': False,
(TaskRunner pid=2255397) 'max_length': None,
(TaskRunner pid=2255397) 'micro_batch_size': None,
(TaskRunner pid=2255397) 'micro_batch_size_per_gpu': None,
(TaskRunner pid=2255397) 'model': {'external_lib': None,
(TaskRunner pid=2255397) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2255397) 'forward_prefetch': False,
(TaskRunner pid=2255397) 'fsdp_size': -1,
(TaskRunner pid=2255397) 'param_offload': False,
(TaskRunner pid=2255397) 'reshard_after_forward': True,
(TaskRunner pid=2255397) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2255397) 'input_tokenizer': 'Qwen/Qwen3-8B',
(TaskRunner pid=2255397) 'override_config': {},
(TaskRunner pid=2255397) 'path': '~/models/FsfairX-LLaMA3-RM-v0.1',
(TaskRunner pid=2255397) 'trust_remote_code': False,
(TaskRunner pid=2255397) 'use_fused_kernels': False,
(TaskRunner pid=2255397) 'use_remove_padding': False,
(TaskRunner pid=2255397) 'use_shm': False},
(TaskRunner pid=2255397) 'n_gpus_per_node': 8,
(TaskRunner pid=2255397) 'nnodes': 0,
(TaskRunner pid=2255397) 'num_workers': 1,
(TaskRunner pid=2255397) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2255397) 'all_ranks': False,
(TaskRunner pid=2255397) 'enable': False,
(TaskRunner pid=2255397) 'ranks': [],
(TaskRunner pid=2255397) 'save_path': 'outputs/profile',
(TaskRunner pid=2255397) 'tool': None,
(TaskRunner pid=2255397) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2255397) 'analysis': True,
(TaskRunner pid=2255397) 'contents': [],
(TaskRunner pid=2255397) 'discrete': False,
(TaskRunner pid=2255397) 'level': 'level0'},
(TaskRunner pid=2255397) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2255397) 'discrete': False},
(TaskRunner pid=2255397) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2255397) 'step_end': None,
(TaskRunner pid=2255397) 'step_start': 0},
(TaskRunner pid=2255397) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2255397) 'stack_depth': 32,
(TaskRunner pid=2255397) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2255397) 'reward_loop_class_name': None,
(TaskRunner pid=2255397) 'reward_loop_module_path': None,
(TaskRunner pid=2255397) 'reward_loop_source': 'register',
(TaskRunner pid=2255397) 'reward_manager': 'naive',
(TaskRunner pid=2255397) 'rollout': {'_target_': 'verl.workers.config.RolloutConfig',
(TaskRunner pid=2255397) 'cudagraph_capture_sizes': None,
(TaskRunner pid=2255397) 'data_parallel_size': 1,
(TaskRunner pid=2255397) 'disable_log_stats': True,
(TaskRunner pid=2255397) 'dtype': 'bfloat16',
(TaskRunner pid=2255397) 'enable_chunked_prefill': True,
(TaskRunner pid=2255397) 'enable_prefix_caching': True,
(TaskRunner pid=2255397) 'enforce_eager': True,
(TaskRunner pid=2255397) 'engine_kwargs': {},
(TaskRunner pid=2255397) 'expert_parallel_size': 1,
(TaskRunner pid=2255397) 'free_cache_engine': True,
(TaskRunner pid=2255397) 'gpu_memory_utilization': 0.5,
(TaskRunner pid=2255397) 'limit_images': None,
(TaskRunner pid=2255397) 'load_format': 'auto',
(TaskRunner pid=2255397) 'max_model_len': None,
(TaskRunner pid=2255397) 'max_num_batched_tokens': 8192,
(TaskRunner pid=2255397) 'max_num_seqs': 1024,
(TaskRunner pid=2255397) 'name': '???',
(TaskRunner pid=2255397) 'prompt_length': 2048,
(TaskRunner pid=2255397) 'response_length': 2048,
(TaskRunner pid=2255397) 'skip_tokenizer_init': False,
(TaskRunner pid=2255397) 'tensor_model_parallel_size': 2},
(TaskRunner pid=2255397) 'sandbox_fusion': {'max_concurrent': 64,
(TaskRunner pid=2255397) 'memory_limit_mb': 1024,
(TaskRunner pid=2255397) 'url': None},
(TaskRunner pid=2255397) 'strategy': 'fsdp',
(TaskRunner pid=2255397) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2255397) 'use_dynamic_bsz': False,
(TaskRunner pid=2255397) 'use_reward_loop': False},
(TaskRunner pid=2255397) 'trainer': {'balance_batch': True,
(TaskRunner pid=2255397) 'critic_warmup': 0,
(TaskRunner pid=2255397) 'default_hdfs_dir': None,
(TaskRunner pid=2255397) 'default_local_dir': '/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2255397) 'del_local_ckpt_after_load': False,
(TaskRunner pid=2255397) 'device': 'cuda',
(TaskRunner pid=2255397) 'esi_redundant_time': 0,
(TaskRunner pid=2255397) 'experiment_name': 'qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2255397) 'group_name': 'QWEN3-RLSD-TR-GRPO-matched-generalization',
(TaskRunner pid=2255397) 'log_val_generations': 0,
(TaskRunner pid=2255397) 'logger': ['console', 'wandb'],
(TaskRunner pid=2255397) 'max_actor_ckpt_to_keep': 100,
(TaskRunner pid=2255397) 'max_critic_ckpt_to_keep': None,
(TaskRunner pid=2255397) 'n_gpus_per_node': 8,
(TaskRunner pid=2255397) 'nnodes': 1,
(TaskRunner pid=2255397) 'project_name': 'SDPO-root',
(TaskRunner pid=2255397) 'ray_wait_register_center_timeout': 300,
(TaskRunner pid=2255397) 'resume_from_path': None,
(TaskRunner pid=2255397) 'resume_mode': 'auto',
(TaskRunner pid=2255397) 'rollout_data_dir': None,
(TaskRunner pid=2255397) 'save_freq': 10,
(TaskRunner pid=2255397) 'test_freq': 10,
(TaskRunner pid=2255397) 'total_epochs': 30,
(TaskRunner pid=2255397) 'total_training_steps': 100,
(TaskRunner pid=2255397) 'use_legacy_worker_impl': 'auto',
(TaskRunner pid=2255397) 'val_before_train': False,
(TaskRunner pid=2255397) 'val_only': False,
(TaskRunner pid=2255397) 'validation_data_dir': None},
(TaskRunner pid=2255397) 'transfer_queue': {'enable': False},
(TaskRunner pid=2255397) 'vars': {'ckpt_dir': '/capstor/scratch/cscs/root/ttrl_runs/datasets/sciknoweval/chemistry',
(TaskRunner pid=2255397) 'dir': '/users/root/SDPO',
(TaskRunner pid=2255397) 'log_dir': '/users/root/output',
(TaskRunner pid=2255397) 'task': 'datasets/sciknoweval/chemistry'}}
(TaskRunner pid=2255397) [validate_config] All configuration checks passed successfully!
(TaskRunner pid=2255397) /mnt/mole/SDPO/L2T/verl/trainer/main_ppo.py:326: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=2255397) use_critic=need_critic(config),
(TaskRunner pid=2255397) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=2255397) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(TaskRunner pid=2255397) Using dataset class: RLHFDataset
(TaskRunner pid=2255397) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(TaskRunner pid=2255397) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(TaskRunner pid=2255397) dataset len: 1890
(TaskRunner pid=2255397) Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2255397) WARNING:2026-07-03 01:41:28,094:Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2255397) Filtering prompts longer than 2048 tokens (num_proc=1): 0%| | 0/1890 [00:00<?, ? examples/s]
(TaskRunner pid=2255397) Filtering prompts longer than 2048 tokens (num_proc=1): 53%|█████▎ | 1000/1890 [00:00<00:00, 1094.76 examples/s]
(TaskRunner pid=2255397) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 1890/1890 [00:01<00:00, 1587.58 examples/s]
(TaskRunner pid=2255397) filter dataset len: 1890
(TaskRunner pid=2255397) Using dataset class: RLHFDataset
(TaskRunner pid=2255397) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 1890/1890 [00:01<00:00, 1380.34 examples/s]
(TaskRunner pid=2255397) dataset len: 210
(TaskRunner pid=2255397) Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2255397) WARNING:2026-07-03 01:41:29,940:Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2255397) Filtering prompts longer than 2048 tokens (num_proc=1): 0%| | 0/210 [00:00<?, ? examples/s]
(TaskRunner pid=2255397) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 210/210 [00:00<00:00, 316.19 examples/s]
(TaskRunner pid=2255397) filter dataset len: 210
(TaskRunner pid=2255397) Size of train dataloader: 59, Size of val dataloader: 1
(TaskRunner pid=2255397) Total training steps: 100
(TaskRunner pid=2255397) colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
(TaskRunner pid=2255397) bind role actor_rollout_ref method chat_completion to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2255397) bind role actor_rollout_ref method generate to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2255397) bind role actor_rollout_ref method get_zeromq_address to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2255397) bind role actor_rollout_ref method sleep to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2255397) bind role actor_rollout_ref method wake_up to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2255397) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 210/210 [00:00<00:00, 276.01 examples/s]
(TaskRunner pid=2255397) /mnt/mole/SDPO/L2T/verl/trainer/ppo/ray_trainer.py:354: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=2255397) self.use_critic = need_critic(self.config)
(WorkerDict pid=2255793) [W703 01:41:37.908542315 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [::ffff:198.19.44.212]:41933 (errno: 97 - Address family not supported by protocol).
(WorkerDict pid=2255788) [Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
(WorkerDict pid=2255789) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(WorkerDict pid=2255789) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(WorkerDict pid=2255792) [W703 01:41:38.533011673 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [::ffff:198.19.44.212]:41933 (errno: 97 - Address family not supported by protocol). [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(WorkerDict pid=2255787) Model config after override: Qwen3Config {
(WorkerDict pid=2255787) "architectures": [
(WorkerDict pid=2255787) "Qwen3ForCausalLM"
(WorkerDict pid=2255787) ],
(WorkerDict pid=2255787) "attention_bias": false,
(WorkerDict pid=2255787) "attention_dropout": 0.0,
(WorkerDict pid=2255787) "dtype": "bfloat16",
(WorkerDict pid=2255787) "eos_token_id": 151645,
(WorkerDict pid=2255787) "head_dim": 128,
(WorkerDict pid=2255787) "hidden_act": "silu",
(WorkerDict pid=2255787) "hidden_size": 4096,
(WorkerDict pid=2255787) "initializer_range": 0.02,
(WorkerDict pid=2255787) "intermediate_size": 12288,
(WorkerDict pid=2255787) "layer_types": [
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention"
(WorkerDict pid=2255787) ],
(WorkerDict pid=2255787) "max_position_embeddings": 40960,
(WorkerDict pid=2255787) "max_window_layers": 36,
(WorkerDict pid=2255787) "model_type": "qwen3",
(WorkerDict pid=2255787) "num_attention_heads": 32,
(WorkerDict pid=2255787) "num_hidden_layers": 36,
(WorkerDict pid=2255787) "num_key_value_heads": 8,
(WorkerDict pid=2255787) "pad_token_id": 151643,
(WorkerDict pid=2255787) "rms_norm_eps": 1e-06,
(WorkerDict pid=2255787) "rope_scaling": null,
(WorkerDict pid=2255787) "rope_theta": 1000000,
(WorkerDict pid=2255787) "sliding_window": null,
(WorkerDict pid=2255787) "tie_word_embeddings": false,
(WorkerDict pid=2255787) "transformers_version": "4.57.1",
(WorkerDict pid=2255787) "use_cache": true,
(WorkerDict pid=2255787) "use_sliding_window": false,
(WorkerDict pid=2255787) "vocab_size": 151936
(WorkerDict pid=2255787) }
(WorkerDict pid=2255787)
(WorkerDict pid=2255794) `torch_dtype` is deprecated! Use `dtype` instead!
(WorkerDict pid=2255794) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
(WorkerDict pid=2255794) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
(WorkerDict pid=2255789) Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
(WorkerDict pid=2255787) Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.99s/it]
(WorkerDict pid=2255788) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(WorkerDict pid=2255788) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2255788) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(WorkerDict pid=2255788) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` [repeated 14x across cluster]
(WorkerDict pid=2255791) Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s] [repeated 7x across cluster]
(WorkerDict pid=2255787) Loading checkpoint shards: 60%|██████ | 3/5 [00:08<00:05, 2.97s/it] [repeated 16x across cluster]
(WorkerDict pid=2255787) Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.08s/it] Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.44s/it]
(WorkerDict pid=2255787) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=2255787) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch
(WorkerDict pid=2255792) [Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7 [repeated 7x across cluster]
(WorkerDict pid=2255787) Qwen3ForCausalLM contains 8.19B parameters
(WorkerDict pid=2255787) wrap_policy: functools.partial(<function _or_policy at 0x76064de9c540>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x76064de9c400>, transformer_layer_cls={<class 'transformers.models.qwen3.modeling_qwen3.Qwen3DecoderLayer'>})])
(WorkerDict pid=2255787) NCCL version 2.27.5+cuda12.9
(WorkerDict pid=2255788) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(WorkerDict pid=2255788) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(WorkerDict pid=2255790) Loading checkpoint shards: 80%|████████ | 4/5 [00:12<00:02, 2.90s/it] [repeated 15x across cluster]
(WorkerDict pid=2255790) Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.20s/it] Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.60s/it] [repeated 7x across cluster]
(WorkerDict pid=2255787) Total steps: 100, num_warmup_steps: 10
(WorkerDict pid=2255790) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 7x across cluster]
(WorkerDict pid=2255790) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch [repeated 7x across cluster]
(WorkerDict pid=2255787) Actor use_remove_padding=True
(WorkerDict pid=2255787) Actor use_fused_kernels=False
(WorkerDict pid=2255787) Actor use_prefix_grouper=False
(WorkerDict pid=2255794) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(WorkerDict pid=2255794) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(WorkerDict pid=2255788) [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
(WorkerDict pid=2255787) reference model: Qwen/Qwen3-8B
(WorkerDict pid=2255787) /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=2255787) warnings.warn(
(WorkerDict pid=2255787) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now. [repeated 7x across cluster]
(WorkerDict pid=2255787) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2255787) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 8x across cluster]
(WorkerDict pid=2255787) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 8x across cluster]
(WorkerDict pid=2255787) Model config after override: Qwen3Config {
(WorkerDict pid=2255787) "architectures": [
(WorkerDict pid=2255787) "Qwen3ForCausalLM"
(WorkerDict pid=2255787) ],
(WorkerDict pid=2255787) "attention_bias": false,
(WorkerDict pid=2255787) "attention_dropout": 0.0,
(WorkerDict pid=2255787) "dtype": "bfloat16",
(WorkerDict pid=2255787) "eos_token_id": 151645,
(WorkerDict pid=2255787) "head_dim": 128,
(WorkerDict pid=2255787) "hidden_act": "silu",
(WorkerDict pid=2255787) "hidden_size": 4096,
(WorkerDict pid=2255787) "initializer_range": 0.02,
(WorkerDict pid=2255787) "intermediate_size": 12288,
(WorkerDict pid=2255787) "layer_types": [
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention",
(WorkerDict pid=2255787) "full_attention"
(WorkerDict pid=2255787) ],
(WorkerDict pid=2255787) "max_position_embeddings": 40960,
(WorkerDict pid=2255787) "max_window_layers": 36,
(WorkerDict pid=2255787) "model_type": "qwen3",
(WorkerDict pid=2255787) "num_attention_heads": 32,
(WorkerDict pid=2255787) "num_hidden_layers": 36,
(WorkerDict pid=2255787) "num_key_value_heads": 8,
(WorkerDict pid=2255787) "pad_token_id": 151643,
(WorkerDict pid=2255787) "rms_norm_eps": 1e-06,
(WorkerDict pid=2255787) "rope_scaling": null,
(WorkerDict pid=2255787) "rope_theta": 1000000,
(WorkerDict pid=2255787) "sliding_window": null,
(WorkerDict pid=2255787) "tie_word_embeddings": false,
(WorkerDict pid=2255787) "transformers_version": "4.57.1",
(WorkerDict pid=2255787) "use_cache": true,
(WorkerDict pid=2255787) "use_sliding_window": false,
(WorkerDict pid=2255787) "vocab_size": 151936
(WorkerDict pid=2255787) }
(WorkerDict pid=2255787)
(WorkerDict pid=2255792) [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [repeated 23x across cluster]
(WorkerDict pid=2255787) Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
(WorkerDict pid=2255787) Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.96s/it]
(WorkerDict pid=2255791) /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . [repeated 7x across cluster]
(WorkerDict pid=2255791) warnings.warn( [repeated 7x across cluster]
(WorkerDict pid=2255791) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(WorkerDict pid=2255791) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2255793) Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s] [repeated 7x across cluster]
(WorkerDict pid=2255787) Loading checkpoint shards: 60%|██████ | 3/5 [00:08<00:05, 2.92s/it] [repeated 16x across cluster]
(WorkerDict pid=2255787) Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.05s/it] Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.41s/it]
(WorkerDict pid=2255787) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=2255787) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch
(WorkerDict pid=2255787) Qwen3ForCausalLM contains 8.19B parameters
(WorkerDict pid=2255787) wrap_policy: functools.partial(<function _or_policy at 0x76064de9c540>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x76064de9c400>, transformer_layer_cls={<class 'transformers.models.qwen3.modeling_qwen3.Qwen3DecoderLayer'>})])
(WorkerDict pid=2255787) Ref use_remove_padding=True
(WorkerDict pid=2255787) Ref use_fused_kernels=False
(WorkerDict pid=2255787) Ref use_prefix_grouper=False
(WorkerDict pid=2255792) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 7x across cluster]
(WorkerDict pid=2255792) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch [repeated 7x across cluster]
(TaskRunner pid=2255397) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=2255397) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(WorkerDict pid=2255792) Loading checkpoint shards: 80%|████████ | 4/5 [00:12<00:02, 2.93s/it] [repeated 15x across cluster]
(WorkerDict pid=2255792) Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.22s/it] Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.62s/it] [repeated 7x across cluster]
(vLLMHttpServer pid=2256706) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(vLLMHttpServer pid=2256706) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(vLLMHttpServer pid=2256707) ['serve',
(vLLMHttpServer pid=2256707) 'Qwen/Qwen3-8B',
(vLLMHttpServer pid=2256707) '--dtype',
(vLLMHttpServer pid=2256707) 'bfloat16',
(vLLMHttpServer pid=2256707) '--load_format',
(vLLMHttpServer pid=2256707) 'dummy',
(vLLMHttpServer pid=2256707) '--trust_remote_code',
(vLLMHttpServer pid=2256707) '--max_model_len',
(vLLMHttpServer pid=2256707) '40960',
(vLLMHttpServer pid=2256707) '--max_num_seqs',
(vLLMHttpServer pid=2256707) '1024',
(vLLMHttpServer pid=2256707) '--enable_chunked_prefill',
(vLLMHttpServer pid=2256707) '--max_num_batched_tokens',
(vLLMHttpServer pid=2256707) '10240',
(vLLMHttpServer pid=2256707) '--enable_prefix_caching',
(vLLMHttpServer pid=2256707) '--enable_sleep_mode',
(vLLMHttpServer pid=2256707) '--logprobs_mode',
(vLLMHttpServer pid=2256707) 'processed_logprobs',
(vLLMHttpServer pid=2256707) '--disable_custom_all_reduce',
(vLLMHttpServer pid=2256707) '--gpu_memory_utilization',
(vLLMHttpServer pid=2256707) '0.8',
(vLLMHttpServer pid=2256707) '--disable_log_stats',
(vLLMHttpServer pid=2256707) '--tensor_parallel_size',
(vLLMHttpServer pid=2256707) '2',
(vLLMHttpServer pid=2256707) '--seed',
(vLLMHttpServer pid=2256707) '0',
(vLLMHttpServer pid=2256707) '--override_generation_config',
(vLLMHttpServer pid=2256707) '{"temperature": 1.0, "top_k": -1, "top_p": 1.0, "repetition_penalty": 1.0, '
(vLLMHttpServer pid=2256707) '"max_new_tokens": 8192}',
(vLLMHttpServer pid=2256707) '--hf_overrides',
(vLLMHttpServer pid=2256707) '{}',
(vLLMHttpServer pid=2256707) '--scheduling_policy',
(vLLMHttpServer pid=2256707) 'fcfs']
(vLLMHttpServer pid=2256707) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(vLLMHttpServer pid=2256706) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(vLLMHttpServer pid=2256706) WARNING 07-03 01:42:55 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(WorkerDict pid=2255788) WARNING 07-03 01:43:03 [worker_base.py:301] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'.
(vLLMHttpServer pid=2256709) WARNING 07-03 01:42:55 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(WorkerDict pid=2255789) NCCL version 2.27.5+cuda12.9
(WorkerDict pid=2255793) 2026-07-03 01:43:17,091 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(vLLMHttpServer pid=2256709) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 3x across cluster]
(vLLMHttpServer pid=2256709) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 3x across cluster]
(vLLMHttpServer pid=2256709) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 3x across cluster]
(vLLMHttpServer pid=2256709) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 3x across cluster]
(WorkerDict pid=2255793) 2026-07-03 01:43:17,106 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 4%|▍ | 2/51 [00:00<00:03, 13.96it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 8%|▊ | 4/51 [00:00<00:02, 16.30it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 12%|█▏ | 6/51 [00:00<00:02, 16.43it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 8/51 [00:00<00:02, 16.93it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|█▉ | 10/51 [00:00<00:03, 13.32it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 24%|██▎ | 12/51 [00:00<00:03, 12.54it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 14/51 [00:00<00:02, 13.59it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 31%|███▏ | 16/51 [00:01<00:02, 14.21it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 35%|███▌ | 18/51 [00:01<00:02, 14.62it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 39%|███▉ | 20/51 [00:01<00:02, 15.08it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 43%|████▎ | 22/51 [00:01<00:01, 15.37it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 24/51 [00:01<00:01, 15.62it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 51%|█████ | 26/51 [00:01<00:01, 15.41it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 28/51 [00:01<00:01, 15.52it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 59%|█████▉ | 30/51 [00:02<00:01, 15.18it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 32/51 [00:02<00:01, 15.39it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 67%|██████▋ | 34/51 [00:02<00:01, 14.83it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████ | 36/51 [00:02<00:01, 14.44it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 75%|███████▍ | 38/51 [00:02<00:00, 14.61it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 78%|███████▊ | 40/51 [00:02<00:00, 14.18it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 42/51 [00:02<00:00, 14.27it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▋ | 44/51 [00:02<00:00, 14.40it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 90%|█████████ | 46/51 [00:03<00:00, 14.34it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 94%|█████████▍| 48/51 [00:03<00:00, 14.27it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 98%|█████████▊| 50/51 [00:03<00:00, 14.69it/s] Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 14.68it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 0%| | 0/51 [00:00<?, ?it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 6%|▌ | 3/51 [00:00<00:02, 21.25it/s]
(WorkerDict pid=2255787) 2026-07-03 01:43:18,119 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ... [repeated 7x across cluster]
(WorkerDict pid=2255787) 2026-07-03 01:43:18,149 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends [repeated 7x across cluster]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 12%|█▏ | 6/51 [00:00<00:02, 22.37it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 18%|█▊ | 9/51 [00:00<00:01, 22.78it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 24%|██▎ | 12/51 [00:00<00:01, 22.91it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 29%|██▉ | 15/51 [00:00<00:01, 22.90it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 35%|███▌ | 18/51 [00:00<00:01, 22.79it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 41%|████ | 21/51 [00:00<00:01, 22.34it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 47%|████▋ | 24/51 [00:01<00:01, 22.57it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 53%|█████▎ | 27/51 [00:01<00:01, 22.16it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 59%|█████▉ | 30/51 [00:01<00:00, 22.46it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 65%|██████▍ | 33/51 [00:01<00:00, 22.50it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 71%|███████ | 36/51 [00:01<00:00, 22.74it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 76%|███████▋ | 39/51 [00:01<00:00, 22.32it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 42/51 [00:01<00:00, 22.25it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 88%|████████▊ | 45/51 [00:02<00:00, 22.45it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 94%|█████████▍| 48/51 [00:02<00:00, 22.95it/s]
(WorkerDict pid=2255787) Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:02<00:00, 23.43it/s] Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:02<00:00, 22.69it/s]
(vLLMHttpServer pid=2256708) WARNING 07-03 01:43:25 [api_server.py:1358] LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development!
(WorkerDict pid=2255792) WARNING 07-03 01:43:04 [worker_base.py:301] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 7x across cluster]
(WorkerDict pid=2255793) NCCL version 2.27.5+cuda12.9 [repeated 2x across cluster]
(vLLMHttpServer pid=2256708) WARNING 07-03 01:43:25 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(TaskRunner pid=2255397) AgentLoopManager: ['198.19.44.212:37481', '198.19.44.212:36385', '198.19.44.212:42611', '198.19.44.212:38461']
(TaskRunner pid=2255397) wandb: Currently logged in as: seongryongjung (seongryongjung-chung-ang-university) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
(TaskRunner pid=2255397) wandb: setting up run zs3h1j56
(TaskRunner pid=2255397) wandb: Tracking run with wandb version 0.23.1
(TaskRunner pid=2255397) wandb: Run data is saved locally in /mnt/mole/SDPO/L2T/wandb/run-20260703_014331-zs3h1j56
(TaskRunner pid=2255397) wandb: Run `wandb offline` to turn off syncing.
(TaskRunner pid=2255397) wandb: Syncing run qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8
(TaskRunner pid=2255397) wandb: ⭐️ View project at https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root
(TaskRunner pid=2255397) wandb: 🚀 View run at https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root/runs/zs3h1j56
(TaskRunner pid=2255397) Checkpoint tracker file does not exist: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/latest_checkpointed_iteration.txt
(TaskRunner pid=2255397) Training from scratch
(vLLMHttpServer pid=2256707) WARNING 07-03 01:43:26 [api_server.py:1358] LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development! [repeated 3x across cluster]
(vLLMHttpServer pid=2256707) WARNING 07-03 01:43:26 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(TaskRunner pid=2255397) wandb: Detected [openai] in use.
(TaskRunner pid=2255397) wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
(TaskRunner pid=2255397) wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
(TaskRunner pid=2255397) Training Progress: 0%| | 0/100 [00:00<?, ?it/s]
(AgentLoopWorker pid=2257667) Using dataset class: RLHFDataset
(AgentLoopWorker pid=2257667) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(AgentLoopWorker pid=2257667) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(AgentLoopWorker pid=2257667) You're using a Qwen2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
(AgentLoopWorker pid=2257675) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(AgentLoopWorker pid=2257675) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(TaskRunner pid=2255397) step:1 - global_seqlen/min:18655 - global_seqlen/max:26853 - global_seqlen/minmax_diff:8198 - global_seqlen/balanced_min:23692 - global_seqlen/balanced_max:23701 - global_seqlen/mean:23696.5 - actor/entropy:0.5224554538726807 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.14097940921783447 - training/rollout_probs_diff_mean:0.0045183561742305756 - training/rollout_probs_diff_std:0.009310152381658554 - training/rollout_actor_probs_pearson_corr:0.9994592070579529 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.7421875 - self_distillation/correct_sample_fraction:0.46484375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7421875 - rollout_corr/rollout_is_mean:0.9999948143959045 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.428677908028476e-05 - rollout_corr/rollout_is_max:1.4933885335922241 - rollout_corr/rollout_is_min:0.4801149070262909 - rollout_corr/rollout_is_std:0.03428589925169945 - rollout_corr/rollout_is_eff_sample_size:0.9988258574271096 - rollout_corr/rollout_is_seq_mean:1.0000158548355103 - rollout_corr/rollout_is_seq_std:0.0017025339184328914 - rollout_corr/rollout_is_seq_max:1.0078747272491455 - rollout_corr/rollout_is_seq_min:0.9956079721450806 - rollout_corr/rollout_is_seq_max_deviation:0.007874727249145508 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7518747616559267) - rollout_corr/training_log_ppl:np.float64(0.5485204759170301) - rollout_corr/kl:np.float64(0.0006133353596755597) - rollout_corr/k3_kl:np.float64(0.0006291264688229603) - rollout_corr/rollout_ppl:np.float64(1.750815692357719) - rollout_corr/rollout_log_ppl:np.float64(0.5479071354493499) - rollout_corr/log_ppl_diff:np.float64(0.0006133404676802456) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013540681102313101) - rollout_corr/log_ppl_diff_max:np.float64(0.005063414573669434) - rollout_corr/log_ppl_diff_min:np.float64(-0.006822109222412109) - rollout_corr/ppl_ratio:np.float64(1.0006149462424219) - rollout_corr/chi2_token:np.float64(0.0012849036138504744) - rollout_corr/chi2_seq:np.float64(0.5767202903516591) - actor/pg_loss:np.float64(0.0015394691145047545) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0) - actor/ppo_kl:np.float64(0.0) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7421875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7421875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.46484375) - self_distillation/token_reweight_w_mean:np.float64(29057.488782484084) - self_distillation/token_reweight_w_std:np.float64(551978.9261493584) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0031458756420761) - self_distillation/token_reweight_w_clip_frac:np.float64(0.14994207414565608) - self_distillation/empty_target_batch:np.float64(0.2578125) - actor/grad_norm:np.float64(0.34795472025871277) - perf/mfu/actor:np.float64(0.039808731599246736) - perf/max_memory_allocated_gb:np.float64(130.98325157165527) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(208.84796524047852) - actor/lr:np.float64(0.0) - training/global_step:1 - training/epoch:0 - critic/score/mean:0.46484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004086250439286232 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004086250439286232 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:482.515625 - response_length/max:1194.0 - response_length/min:161.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:482.515625 - response_length_non_aborted/max:1194.0 - response_length_non_aborted/min:161.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:258.0 - prompt_length/max:491.0 - prompt_length/min:161.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00030501745641231537 - timing_s/agent_loop/generate_sequences/min:np.float64(14.297906694933772) - timing_s/agent_loop/generate_sequences/max:np.float64(25.201997511088848) - timing_s/agent_loop/generate_sequences/mean:np.float64(18.805905992172484) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(25.201997511088848) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:342 - timing_s/agent_loop/slowest/response_length:844 - timing_s/gen:36.206207644194365 - timing_s/reward:0.06375245191156864 - timing_s/old_log_prob:3.828576784580946 - timing_s/adv:0.6266427040100098 - timing_s/update_actor:30.915611987933517 - timing_s/step:71.7457935847342 - timing_s/stop_profile:8.898042142391205e-05 - timing_per_token_ms/update_actor:0.1630811089608883 - timing_per_token_ms/gen:0.2931107124461187 - timing_per_token_ms/adv:0.0033055657165088187 - perf/total_num_tokens:189572 - perf/time_per_step:71.7457935847342 - perf/throughput:330.28417160113554
(AgentLoopWorker pid=2257672) Using dataset class: RLHFDataset [repeated 7x across cluster]
(TaskRunner pid=2255397) Training Progress: 1%| | 1/100 [01:11<1:58:40, 71.93s/it]
(AgentLoopWorker pid=2257675) You're using a Qwen2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding. [repeated 7x across cluster]
(TaskRunner pid=2255397) step:2 - global_seqlen/min:18554 - global_seqlen/max:28889 - global_seqlen/minmax_diff:10335 - global_seqlen/balanced_min:23930 - global_seqlen/balanced_max:23950 - global_seqlen/mean:23945.25 - actor/entropy:0.5649150013923645 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.21355730295181274 - training/rollout_probs_diff_mean:0.004759751260280609 - training/rollout_probs_diff_std:0.009735357947647572 - training/rollout_actor_probs_pearson_corr:0.9994669556617737 - self_distillation/success_group_fraction:0.625 - self_distillation/success_sample_fraction:0.609375 - self_distillation/correct_sample_fraction:0.3671875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.609375 - rollout_corr/rollout_is_mean:0.999841034412384 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.8218586774310097e-05 - rollout_corr/rollout_is_max:1.8565502166748047 - rollout_corr/rollout_is_min:0.4281544089317322 - rollout_corr/rollout_is_std:0.03569299355149269 - rollout_corr/rollout_is_eff_sample_size:0.9987271556828731 - rollout_corr/rollout_is_seq_mean:0.9997751712799072 - rollout_corr/rollout_is_seq_std:0.002073427429422736 - rollout_corr/rollout_is_seq_max:1.0066989660263062 - rollout_corr/rollout_is_seq_min:0.993266224861145 - rollout_corr/rollout_is_seq_max_deviation:0.0067337751388549805 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8422705363482237) - rollout_corr/training_log_ppl:np.float64(0.5950889247469604) - rollout_corr/kl:np.float64(0.000921569124747279) - rollout_corr/k3_kl:np.float64(0.0006883058893549787) - rollout_corr/rollout_ppl:np.float64(1.8405294218100607) - rollout_corr/rollout_log_ppl:np.float64(0.5941673531197011) - rollout_corr/log_ppl_diff:np.float64(0.0009215716272592545) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017208073986694217) - rollout_corr/log_ppl_diff_max:np.float64(0.007348239421844482) - rollout_corr/log_ppl_diff_min:np.float64(-0.005036115646362305) - rollout_corr/ppl_ratio:np.float64(1.0009240037761629) - rollout_corr/chi2_token:np.float64(0.0009099591989070177) - rollout_corr/chi2_seq:np.float64(0.3998524572234601) - actor/pg_loss:np.float64(0.005818751524202526) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002621370244924037) - actor/ppo_kl:np.float64(9.864444990625998e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.609375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.609375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.3671875) - self_distillation/token_reweight_w_mean:np.float64(25264.271957140416) - self_distillation/token_reweight_w_std:np.float64(438644.3446098004) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0013297640252858) - self_distillation/token_reweight_w_clip_frac:np.float64(0.13903450994985178) - self_distillation/empty_target_batch:np.float64(0.390625) - actor/grad_norm:np.float64(0.3431096374988556) - perf/mfu/actor:np.float64(0.0564479877562713) - perf/max_memory_allocated_gb:np.float64(130.98325157165527) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(204.3056116104126) - actor/lr:np.float64(1e-07) - training/global_step:2 - training/epoch:0 - critic/score/mean:0.3671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.3671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0009724170668050647 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0009724170668050647 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:428.8203125 - response_length/max:1497.0 - response_length/min:127.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:428.8203125 - response_length_non_aborted/max:1497.0 - response_length_non_aborted/min:127.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:319.46875 - prompt_length/max:818.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.283795952796936e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.200064292177558) - timing_s/agent_loop/generate_sequences/max:np.float64(11.617594823241234) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.475670634448761) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.617594823241234) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:200 - timing_s/agent_loop/slowest/response_length:1497 - timing_s/gen:19.95695459470153 - timing_s/reward:0.06782706081867218 - timing_s/old_log_prob:3.249016724526882 - timing_s/adv:0.5516544189304113 - timing_s/update_actor:22.18794622644782 - timing_s/step:46.101478507742286 - timing_s/stop_profile:0.0001231711357831955 - timing_per_token_ms/update_actor:0.11582644901623401 - timing_per_token_ms/gen:0.18179375279838883 - timing_per_token_ms/adv:0.0028797695729341484 - perf/total_num_tokens:191562 - perf/time_per_step:46.101478507742286 - perf/throughput:519.4030815297742
(TaskRunner pid=2255397) Training Progress: 2%|▏ | 2/100 [01:58<1:32:44, 56.78s/it]
(TaskRunner pid=2255397) step:3 - global_seqlen/min:19604 - global_seqlen/max:28466 - global_seqlen/minmax_diff:8862 - global_seqlen/balanced_min:24149 - global_seqlen/balanced_max:24177 - global_seqlen/mean:24164.75 - actor/entropy:0.5242184400558472 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1616789698600769 - training/rollout_probs_diff_mean:0.004656430333852768 - training/rollout_probs_diff_std:0.009660309180617332 - training/rollout_actor_probs_pearson_corr:0.9994632601737976 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.765625 - self_distillation/correct_sample_fraction:0.4140625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.765625 - rollout_corr/rollout_is_mean:1.0001205205917358 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.629170224594418e-06 - rollout_corr/rollout_is_max:1.6356302499771118 - rollout_corr/rollout_is_min:0.49262791872024536 - rollout_corr/rollout_is_std:0.0352848656475544 - rollout_corr/rollout_is_eff_sample_size:0.9987568830709286 - rollout_corr/rollout_is_seq_mean:1.000150442123413 - rollout_corr/rollout_is_seq_std:0.0021288390271365643 - rollout_corr/rollout_is_seq_max:1.007064700126648 - rollout_corr/rollout_is_seq_min:0.9945412278175354 - rollout_corr/rollout_is_seq_max_deviation:0.007064700126647949 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7977697649039328) - rollout_corr/training_log_ppl:np.float64(0.5697851715376601) - rollout_corr/kl:np.float64(0.0006146689036183872) - rollout_corr/k3_kl:np.float64(0.0006752699637218029) - rollout_corr/rollout_ppl:np.float64(1.796580535825342) - rollout_corr/rollout_log_ppl:np.float64(0.5691705014905892) - rollout_corr/log_ppl_diff:np.float64(0.0006146700470708311) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015643324586562812) - rollout_corr/log_ppl_diff_max:np.float64(0.006385087966918945) - rollout_corr/log_ppl_diff_min:np.float64(-0.006580471992492676) - rollout_corr/ppl_ratio:np.float64(1.0006169071421027) - rollout_corr/chi2_token:np.float64(0.0014803188387304544) - rollout_corr/chi2_seq:np.float64(0.8378349014092237) - actor/pg_loss:np.float64(0.008528164820745587) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00031198100373330817) - actor/ppo_kl:np.float64(8.786334534249818e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.765625) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.765625) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.4140625) - self_distillation/token_reweight_w_mean:np.float64(19923.08156236657) - self_distillation/token_reweight_w_std:np.float64(356201.15468769113) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.001924671465531) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1814161114161834) - self_distillation/empty_target_batch:np.float64(0.234375) - actor/grad_norm:np.float64(0.4067366272211075) - perf/mfu/actor:np.float64(0.05678632217924573) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(202.01263046264648) - actor/lr:np.float64(2e-07) - training/global_step:3 - training/epoch:0 - critic/score/mean:0.4140625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4140625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.002736525610089302 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.002736525610089302 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:452.6796875 - response_length/max:1521.0 - response_length/min:145.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:452.6796875 - response_length_non_aborted/max:1521.0 - response_length_non_aborted/min:145.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.46875 - prompt_length/max:557.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001317691057920456 - timing_s/agent_loop/generate_sequences/min:np.float64(1.856243697926402) - timing_s/agent_loop/generate_sequences/max:np.float64(11.949713030830026) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.092739296589571) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.949713030830026) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:202 - timing_s/agent_loop/slowest/response_length:1521 - timing_s/gen:19.23995424248278 - timing_s/reward:0.06506127119064331 - timing_s/old_log_prob:3.2659819293767214 - timing_s/adv:0.6202606074512005 - timing_s/update_actor:22.100518580526114 - timing_s/step:45.38067802786827 - timing_s/stop_profile:0.00010930933058261871 - timing_per_token_ms/update_actor:0.11432209406535404 - timing_per_token_ms/gen:0.16602483684381877 - timing_per_token_ms/adv:0.0032084989884604664 - perf/total_num_tokens:193318 - perf/time_per_step:45.38067802786827 - perf/throughput:532.4898403933151
(TaskRunner pid=2255397) Training Progress: 3%|▎ | 3/100 [02:43<1:23:24, 51.60s/it]
(TaskRunner pid=2255397) step:4 - global_seqlen/min:23313 - global_seqlen/max:27346 - global_seqlen/minmax_diff:4033 - global_seqlen/balanced_min:24709 - global_seqlen/balanced_max:24717 - global_seqlen/mean:24714.5 - actor/entropy:0.54982590675354 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7950142621994019 - training/rollout_probs_diff_mean:0.004631490912288427 - training/rollout_probs_diff_std:0.009951530024409294 - training/rollout_actor_probs_pearson_corr:0.9994604587554932 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.79296875 - self_distillation/correct_sample_fraction:0.4453125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.79296875 - rollout_corr/rollout_is_mean:1.0000749826431274 - rollout_corr/rollout_is_ratio_fraction_high:8.3895429270342e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.67790858540684e-05 - rollout_corr/rollout_is_max:4.939357280731201 - rollout_corr/rollout_is_min:0.19150345027446747 - rollout_corr/rollout_is_std:0.0351443812251091 - rollout_corr/rollout_is_eff_sample_size:0.9987665151242391 - rollout_corr/rollout_is_seq_mean:1.0000817775726318 - rollout_corr/rollout_is_seq_std:0.0017939795507118106 - rollout_corr/rollout_is_seq_max:1.0054137706756592 - rollout_corr/rollout_is_seq_min:0.9938213229179382 - rollout_corr/rollout_is_seq_max_deviation:0.006178677082061768 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8050872194580734) - rollout_corr/training_log_ppl:np.float64(0.5758670450886711) - rollout_corr/kl:np.float64(0.0006997874668996928) - rollout_corr/k3_kl:np.float64(0.0006716491621432397) - rollout_corr/rollout_ppl:np.float64(1.803772633895278) - rollout_corr/rollout_log_ppl:np.float64(0.5751672576880082) - rollout_corr/log_ppl_diff:np.float64(0.000699787400662899) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015515540726482868) - rollout_corr/log_ppl_diff_max:np.float64(0.00667417049407959) - rollout_corr/log_ppl_diff_min:np.float64(-0.004105627536773682) - rollout_corr/ppl_ratio:np.float64(1.0007018391042948) - rollout_corr/chi2_token:np.float64(0.0013059712946414948) - rollout_corr/chi2_seq:np.float64(0.8464284441433847) - actor/pg_loss:np.float64(0.0077759657287970185) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014584792415917036) - actor/ppo_kl:np.float64(0.00010806482939074158) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.79296875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.79296875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.4453125) - self_distillation/token_reweight_w_mean:np.float64(18390.18527350994) - self_distillation/token_reweight_w_std:np.float64(278661.7640647436) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0027553322724998) - self_distillation/token_reweight_w_clip_frac:np.float64(0.18228401592932642) - self_distillation/empty_target_batch:np.float64(0.20703125) - actor/grad_norm:np.float64(0.41205956041812897) - perf/mfu/actor:np.float64(0.05901765112715433) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(199.91095733642578) - actor/lr:np.float64(3e-07) - training/global_step:4 - training/epoch:0 - critic/score/mean:0.4453125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4453125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0022473488934338093 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0022473488934338093 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:465.609375 - response_length/max:1140.0 - response_length/min:135.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:465.609375 - response_length_non_aborted/max:1140.0 - response_length_non_aborted/min:135.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:306.71875 - prompt_length/max:604.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001505669206380844 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3345985803753138) - timing_s/agent_loop/generate_sequences/max:np.float64(9.233254563063383) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.90943047656765) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.233254563063383) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:228 - timing_s/agent_loop/slowest/response_length:1011 - timing_s/gen:16.390039935708046 - timing_s/reward:0.06450367532670498 - timing_s/old_log_prob:3.0817583967000246 - timing_s/adv:0.5708468426018953 - timing_s/update_actor:21.8274494279176 - timing_s/step:42.01813451200724 - timing_s/stop_profile:0.00011336617171764374 - timing_per_token_ms/update_actor:0.11039799221063343 - timing_per_token_ms/gen:0.1375049492911511 - timing_per_token_ms/adv:0.002887206106748545 - perf/total_num_tokens:197716 - perf/time_per_step:42.01813451200724 - perf/throughput:588.1865124911127
(TaskRunner pid=2255397) Training Progress: 4%|▍ | 4/100 [03:25<1:16:32, 47.83s/it]
(TaskRunner pid=2255397) step:5 - global_seqlen/min:18324 - global_seqlen/max:26762 - global_seqlen/minmax_diff:8438 - global_seqlen/balanced_min:22550 - global_seqlen/balanced_max:22563 - global_seqlen/mean:22556.25 - actor/entropy:0.5743926167488098 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18463686108589172 - training/rollout_probs_diff_mean:0.004951741546392441 - training/rollout_probs_diff_std:0.009703434072434902 - training/rollout_actor_probs_pearson_corr:0.9994615912437439 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.8125 - self_distillation/correct_sample_fraction:0.42578125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8125 - rollout_corr/rollout_is_mean:1.000128984451294 - rollout_corr/rollout_is_ratio_fraction_high:9.058463547262363e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.8116927094524726e-05 - rollout_corr/rollout_is_max:2.5132858753204346 - rollout_corr/rollout_is_min:0.4844912588596344 - rollout_corr/rollout_is_std:0.03672673553228378 - rollout_corr/rollout_is_eff_sample_size:0.9986534393962608 - rollout_corr/rollout_is_seq_mean:1.0001883506774902 - rollout_corr/rollout_is_seq_std:0.0020744451321661472 - rollout_corr/rollout_is_seq_max:1.0107048749923706 - rollout_corr/rollout_is_seq_min:0.9943471550941467 - rollout_corr/rollout_is_seq_max_deviation:0.010704874992370605 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.8240970252081752) - rollout_corr/training_log_ppl:np.float64(0.5867100825998932) - rollout_corr/kl:np.float64(0.000413151877571849) - rollout_corr/k3_kl:np.float64(0.0007104324042757071) - rollout_corr/rollout_ppl:np.float64(1.8233414590358734) - rollout_corr/rollout_log_ppl:np.float64(0.5862969291047193) - rollout_corr/log_ppl_diff:np.float64(0.0004131534951739013) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015994933783076704) - rollout_corr/log_ppl_diff_max:np.float64(0.005790770053863525) - rollout_corr/log_ppl_diff_min:np.float64(-0.006320059299468994) - rollout_corr/ppl_ratio:np.float64(1.0004152327310294) - rollout_corr/chi2_token:np.float64(0.0020708865486085415) - rollout_corr/chi2_seq:np.float64(1.291173740522936) - actor/pg_loss:np.float64(0.00783018337097019) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00029261777604006056) - actor/ppo_kl:np.float64(-0.00011415763041355831) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.42578125) - self_distillation/token_reweight_w_mean:np.float64(7564.456899565645) - self_distillation/token_reweight_w_std:np.float64(147554.27462118154) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0038912589661777) - self_distillation/token_reweight_w_clip_frac:np.float64(0.19292002776637673) - self_distillation/empty_target_batch:np.float64(0.1875) - actor/grad_norm:np.float64(0.37297531962394714) - perf/mfu/actor:np.float64(0.05395284544548605) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(198.01452255249023) - actor/lr:np.float64(4e-07) - training/global_step:5 - training/epoch:0 - critic/score/mean:0.42578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.42578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0022725421003997326 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0022725421003997326 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:431.2265625 - response_length/max:1447.0 - response_length/min:132.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:431.2265625 - response_length_non_aborted/max:1447.0 - response_length_non_aborted/min:132.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.65625 - prompt_length/max:520.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00017325766384601593 - timing_s/agent_loop/generate_sequences/min:np.float64(1.665849743410945) - timing_s/agent_loop/generate_sequences/max:np.float64(11.205976894125342) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.922438242996577) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.205976894125342) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:275 - timing_s/agent_loop/slowest/response_length:1447 - timing_s/gen:18.029446551576257 - timing_s/reward:0.06849926523864269 - timing_s/old_log_prob:3.114251596853137 - timing_s/adv:0.5795028954744339 - timing_s/update_actor:21.41964470408857 - timing_s/step:43.29863533191383 - timing_s/stop_profile:0.00011399760842323303 - timing_per_token_ms/update_actor:0.11870127295144677 - timing_per_token_ms/gen:0.16331908030849734 - timing_per_token_ms/adv:0.0032114319505371787 - perf/total_num_tokens:180450 - perf/time_per_step:43.29863533191383 - perf/throughput:520.9459796386382
(TaskRunner pid=2255397) Training Progress: 5%|▌ | 5/100 [04:08<1:13:10, 46.22s/it]
(TaskRunner pid=2255397) step:6 - global_seqlen/min:19330 - global_seqlen/max:29022 - global_seqlen/minmax_diff:9692 - global_seqlen/balanced_min:23612 - global_seqlen/balanced_max:23621 - global_seqlen/mean:23617.625 - actor/entropy:0.5028008818626404 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5328631401062012 - training/rollout_probs_diff_mean:0.004628530703485012 - training/rollout_probs_diff_std:0.009826785884797573 - training/rollout_actor_probs_pearson_corr:0.9994211196899414 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.8515625 - self_distillation/correct_sample_fraction:0.421875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8515625 - rollout_corr/rollout_is_mean:0.9999843239784241 - rollout_corr/rollout_is_ratio_fraction_high:8.433054063061718e-06 - rollout_corr/rollout_is_ratio_fraction_low:8.433054063061718e-06 - rollout_corr/rollout_is_max:2.1156668663024902 - rollout_corr/rollout_is_min:0.4510452151298523 - rollout_corr/rollout_is_std:0.03478982672095299 - rollout_corr/rollout_is_eff_sample_size:0.998791012294431 - rollout_corr/rollout_is_seq_mean:0.9999687671661377 - rollout_corr/rollout_is_seq_std:0.0017612202791497111 - rollout_corr/rollout_is_seq_max:1.0042917728424072 - rollout_corr/rollout_is_seq_min:0.994244396686554 - rollout_corr/rollout_is_seq_max_deviation:0.005755603313446045 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.732960639987141) - rollout_corr/training_log_ppl:np.float64(0.5360075897770002) - rollout_corr/kl:np.float64(0.000639045071211708) - rollout_corr/k3_kl:np.float64(0.0006780325196018566) - rollout_corr/rollout_ppl:np.float64(1.7318218029104173) - rollout_corr/rollout_log_ppl:np.float64(0.5353685396839865) - rollout_corr/log_ppl_diff:np.float64(0.0006390500930137932) - rollout_corr/log_ppl_abs_diff:np.float64(0.001630233891773969) - rollout_corr/log_ppl_diff_max:np.float64(0.007409811019897461) - rollout_corr/log_ppl_diff_min:np.float64(-0.0069411396980285645) - rollout_corr/ppl_ratio:np.float64(1.0006413247901946) - rollout_corr/chi2_token:np.float64(0.0014372044242918491) - rollout_corr/chi2_seq:np.float64(1.4436022932641208) - actor/pg_loss:np.float64(0.00812386441975832) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004630846383406606) - actor/ppo_kl:np.float64(-4.196475500339858e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8515625) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8515625) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.421875) - self_distillation/token_reweight_w_mean:np.float64(19923.261127433972) - self_distillation/token_reweight_w_std:np.float64(319160.37308162905) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0048198685981333) - self_distillation/token_reweight_w_clip_frac:np.float64(0.18613416605512612) - self_distillation/empty_target_batch:np.float64(0.1484375) - actor/grad_norm:np.float64(0.3917096257209778) - perf/mfu/actor:np.float64(0.05549581290987645) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(196.47495651245117) - actor/lr:np.float64(5e-07) - training/global_step:6 - training/epoch:0 - critic/score/mean:0.421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.008403538726270199 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.008403538726270199 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:463.20703125 - response_length/max:1246.0 - response_length/min:123.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:463.20703125 - response_length_non_aborted/max:1246.0 - response_length_non_aborted/min:123.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:274.84375 - prompt_length/max:450.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010707974433898926 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5621601939201355) - timing_s/agent_loop/generate_sequences/max:np.float64(10.558630177751184) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.262586842865858) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.558630177751184) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:228 - timing_s/agent_loop/slowest/response_length:1246 - timing_s/gen:17.67251803725958 - timing_s/reward:0.07175176963210106 - timing_s/old_log_prob:3.0560552403330803 - timing_s/adv:0.6377361956983805 - timing_s/update_actor:21.91313385590911 - timing_s/step:43.43790529854596 - timing_s/stop_profile:0.00011335685849189758 - timing_per_token_ms/update_actor:0.11597871216892633 - timing_per_token_ms/gen:0.14903330244524485 - timing_per_token_ms/adv:0.003375319256796463 - perf/total_num_tokens:188941 - perf/time_per_step:43.43790529854596 - perf/throughput:543.7100347652026
(TaskRunner pid=2255397) Training Progress: 6%|▌ | 6/100 [04:52<1:10:57, 45.29s/it]
(TaskRunner pid=2255397) step:7 - global_seqlen/min:18757 - global_seqlen/max:24212 - global_seqlen/minmax_diff:5455 - global_seqlen/balanced_min:22489 - global_seqlen/balanced_max:22504 - global_seqlen/mean:22499.75 - actor/entropy:0.5316406488418579 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8049387335777283 - training/rollout_probs_diff_mean:0.004808515310287476 - training/rollout_probs_diff_std:0.0102939298376441 - training/rollout_actor_probs_pearson_corr:0.9993799924850464 - self_distillation/success_group_fraction:0.71875 - self_distillation/success_sample_fraction:0.69921875 - self_distillation/correct_sample_fraction:0.33203125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.69921875 - rollout_corr/rollout_is_mean:1.000180721282959 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.7163583076326177e-05 - rollout_corr/rollout_is_max:1.9837769269943237 - rollout_corr/rollout_is_min:0.12898790836334229 - rollout_corr/rollout_is_std:0.0356261320412159 - rollout_corr/rollout_is_eff_sample_size:0.9987328632041119 - rollout_corr/rollout_is_seq_mean:1.0002025365829468 - rollout_corr/rollout_is_seq_std:0.0019561529625207186 - rollout_corr/rollout_is_seq_max:1.0071183443069458 - rollout_corr/rollout_is_seq_min:0.9942708015441895 - rollout_corr/rollout_is_seq_max_deviation:0.007118344306945801 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7598580992780626) - rollout_corr/training_log_ppl:np.float64(0.5522571774781682) - rollout_corr/kl:np.float64(0.0006528583797624776) - rollout_corr/k3_kl:np.float64(0.0007534040726682179) - rollout_corr/rollout_ppl:np.float64(1.7586921667680144) - rollout_corr/rollout_log_ppl:np.float64(0.5516043193638325) - rollout_corr/log_ppl_diff:np.float64(0.000652858114335686) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017435748013667762) - rollout_corr/log_ppl_diff_max:np.float64(0.010204493999481201) - rollout_corr/log_ppl_diff_min:np.float64(-0.006472587585449219) - rollout_corr/ppl_ratio:np.float64(1.000655469717458) - rollout_corr/chi2_token:np.float64(0.001773336436599493) - rollout_corr/chi2_seq:np.float64(1.4482529927045107) - actor/pg_loss:np.float64(0.007577315904200077) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005699304592781118) - actor/ppo_kl:np.float64(0.00017036601963926756) - actor/pg_clipfrac_lower:np.float64(1.5439723938470706e-05) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.69921875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.69921875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.33203125) - self_distillation/token_reweight_w_mean:np.float64(40804.10983084305) - self_distillation/token_reweight_w_std:np.float64(550034.2616531486) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.002954869531095) - self_distillation/token_reweight_w_clip_frac:np.float64(0.17178265267284587) - self_distillation/empty_target_batch:np.float64(0.30078125) - actor/grad_norm:np.float64(0.42640380561351776) - perf/mfu/actor:np.float64(0.053623795084905154) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(196.01875686645508) - actor/lr:np.float64(6e-07) - training/global_step:7 - training/epoch:0 - critic/score/mean:0.33203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.33203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00599684938788414 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:0.00599684938788414 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:414.1171875 - response_length/max:1248.0 - response_length/min:138.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:414.1171875 - response_length_non_aborted/max:1248.0 - response_length_non_aborted/min:138.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:289.0 - prompt_length/max:662.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001475159078836441 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7632414139807224) - timing_s/agent_loop/generate_sequences/max:np.float64(10.236304745078087) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.697692676651059) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.236304745078087) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:190 - timing_s/agent_loop/slowest/response_length:1248 - timing_s/gen:17.044425999745727 - timing_s/reward:0.06204601190984249 - timing_s/old_log_prob:3.1439606230705976 - timing_s/adv:0.54639608040452 - timing_s/update_actor:21.744473489001393 - timing_s/step:42.631235618144274 - timing_s/stop_profile:0.00010830722749233246 - timing_per_token_ms/update_actor:0.1208039727608162 - timing_per_token_ms/gen:0.1607752372304198 - timing_per_token_ms/adv:0.0030355675085529843 - perf/total_num_tokens:179998 - perf/time_per_step:42.631235618144274 - perf/throughput:527.776163973626
(TaskRunner pid=2255397) Training Progress: 7%|▋ | 7/100 [05:35<1:08:52, 44.44s/it]
(TaskRunner pid=2255397) step:8 - global_seqlen/min:19293 - global_seqlen/max:28026 - global_seqlen/minmax_diff:8733 - global_seqlen/balanced_min:24653 - global_seqlen/balanced_max:24794 - global_seqlen/mean:24675.25 - actor/entropy:0.5080747604370117 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.408370703458786 - training/rollout_probs_diff_mean:0.0044975439086556435 - training/rollout_probs_diff_std:0.009530728682875633 - training/rollout_actor_probs_pearson_corr:0.9994667768478394 - self_distillation/success_group_fraction:0.90625 - self_distillation/success_sample_fraction:0.8828125 - self_distillation/correct_sample_fraction:0.46875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8828125 - rollout_corr/rollout_is_mean:1.0000174045562744 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.162332051142585e-06 - rollout_corr/rollout_is_max:1.8054882287979126 - rollout_corr/rollout_is_min:0.462515652179718 - rollout_corr/rollout_is_std:0.0343310683965683 - rollout_corr/rollout_is_eff_sample_size:0.9988226463404877 - rollout_corr/rollout_is_seq_mean:1.000030279159546 - rollout_corr/rollout_is_seq_std:0.0018812705529853702 - rollout_corr/rollout_is_seq_max:1.0089062452316284 - rollout_corr/rollout_is_seq_min:0.9933127760887146 - rollout_corr/rollout_is_seq_max_deviation:0.008906245231628418 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.747896637301892) - rollout_corr/training_log_ppl:np.float64(0.541930019389838) - rollout_corr/kl:np.float64(0.0006643355780084903) - rollout_corr/k3_kl:np.float64(0.0006548753977995148) - rollout_corr/rollout_ppl:np.float64(1.7467446983791888) - rollout_corr/rollout_log_ppl:np.float64(0.5412656838307157) - rollout_corr/log_ppl_diff:np.float64(0.0006643355591222644) - rollout_corr/log_ppl_abs_diff:np.float64(0.00149785284884274) - rollout_corr/log_ppl_diff_max:np.float64(0.007577002048492432) - rollout_corr/log_ppl_diff_min:np.float64(-0.0072296857833862305) - rollout_corr/ppl_ratio:np.float64(1.000666505889967) - rollout_corr/chi2_token:np.float64(0.0012856200337409973) - rollout_corr/chi2_seq:np.float64(0.7123665248509496) - actor/pg_loss:np.float64(0.007868309505283833) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004503007885432453) - actor/ppo_kl:np.float64(5.489206566622329e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8828125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8828125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.46875) - self_distillation/token_reweight_w_mean:np.float64(9891.629087405046) - self_distillation/token_reweight_w_std:np.float64(229237.51225678227) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0036344188265502) - self_distillation/token_reweight_w_clip_frac:np.float64(0.18639719724887982) - self_distillation/empty_target_batch:np.float64(0.1171875) - actor/grad_norm:np.float64(0.3860222175717354) - perf/mfu/actor:np.float64(0.05765871612685469) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(196.10034942626953) - actor/lr:np.float64(7e-07) - training/global_step:8 - training/epoch:0 - critic/score/mean:0.46875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.008611260913312435 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.008611260913312435 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:478.5703125 - response_length/max:1561.0 - response_length/min:170.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:478.5703125 - response_length_non_aborted/max:1561.0 - response_length_non_aborted/min:170.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:292.53125 - prompt_length/max:720.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014033541083335876 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1611996684223413) - timing_s/agent_loop/generate_sequences/max:np.float64(12.084481660276651) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.483152403154236) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.084481660276651) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:202 - timing_s/agent_loop/slowest/response_length:1561 - timing_s/gen:18.632152697071433 - timing_s/reward:0.06690721958875656 - timing_s/old_log_prob:3.0798434652388096 - timing_s/adv:0.581660982221365 - timing_s/update_actor:22.521120481193066 - timing_s/step:44.96886216290295 - timing_s/stop_profile:0.00011759623885154724 - timing_per_token_ms/update_actor:0.114087600334308 - timing_per_token_ms/gen:0.1520818249103893 - timing_per_token_ms/adv:0.0029465809982744096 - perf/total_num_tokens:197402 - perf/time_per_step:44.96886216290295 - perf/throughput:548.7185757694318
(TaskRunner pid=2255397) Training Progress: 8%|▊ | 8/100 [06:20<1:08:25, 44.62s/it]
(TaskRunner pid=2255397) step:9 - global_seqlen/min:19805 - global_seqlen/max:26584 - global_seqlen/minmax_diff:6779 - global_seqlen/balanced_min:23155 - global_seqlen/balanced_max:23161 - global_seqlen/mean:23157.5 - actor/entropy:0.5331449508666992 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5838292837142944 - training/rollout_probs_diff_mean:0.004796977620571852 - training/rollout_probs_diff_std:0.010206756182014942 - training/rollout_actor_probs_pearson_corr:0.9993701577186584 - self_distillation/success_group_fraction:0.6875 - self_distillation/success_sample_fraction:0.66796875 - self_distillation/correct_sample_fraction:0.359375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.66796875 - rollout_corr/rollout_is_mean:0.9999645352363586 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.796686956367921e-05 - rollout_corr/rollout_is_max:1.6195783615112305 - rollout_corr/rollout_is_min:0.3546355068683624 - rollout_corr/rollout_is_std:0.03580470383167267 - rollout_corr/rollout_is_eff_sample_size:0.9987194268518426 - rollout_corr/rollout_is_seq_mean:0.9999064207077026 - rollout_corr/rollout_is_seq_std:0.001988921547308564 - rollout_corr/rollout_is_seq_max:1.004913330078125 - rollout_corr/rollout_is_seq_min:0.9934803247451782 - rollout_corr/rollout_is_seq_max_deviation:0.006519675254821777 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7555229673162103) - rollout_corr/training_log_ppl:np.float64(0.5492386808618903) - rollout_corr/kl:np.float64(0.000775432549749766) - rollout_corr/k3_kl:np.float64(0.0008021395320838565) - rollout_corr/rollout_ppl:np.float64(1.7541374694555998) - rollout_corr/rollout_log_ppl:np.float64(0.5484632462612353) - rollout_corr/log_ppl_diff:np.float64(0.0007754346006549895) - rollout_corr/log_ppl_abs_diff:np.float64(0.001795541902538389) - rollout_corr/log_ppl_diff_max:np.float64(0.010334789752960205) - rollout_corr/log_ppl_diff_min:np.float64(-0.0038338303565979004) - rollout_corr/ppl_ratio:np.float64(1.000778182875365) - rollout_corr/chi2_token:np.float64(0.0016523946542292833) - rollout_corr/chi2_seq:np.float64(0.9519539077300578) - actor/pg_loss:np.float64(0.007556321215815842) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006206532375472307) - actor/ppo_kl:np.float64(-5.760855917635865e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.66796875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.66796875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.359375) - self_distillation/token_reweight_w_mean:np.float64(22852.849850326777) - self_distillation/token_reweight_w_std:np.float64(403812.0357487643) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0031312247738242) - self_distillation/token_reweight_w_clip_frac:np.float64(0.15560277574695647) - self_distillation/empty_target_batch:np.float64(0.33203125) - actor/grad_norm:np.float64(0.34761351346969604) - perf/mfu/actor:np.float64(0.05432608742997578) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(196.07957077026367) - actor/lr:np.float64(8e-07) - training/global_step:9 - training/epoch:0 - critic/score/mean:0.359375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.359375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.008080599829554558 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.008080599829554558 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:434.828125 - response_length/max:1105.0 - response_length/min:140.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:434.828125 - response_length_non_aborted/max:1105.0 - response_length_non_aborted/min:140.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:288.84375 - prompt_length/max:735.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010892748832702637 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7549341842532158) - timing_s/agent_loop/generate_sequences/max:np.float64(9.423352841287851) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.883239647970186) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.423352841287851) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:221 - timing_s/agent_loop/slowest/response_length:1105 - timing_s/gen:16.615697672590613 - timing_s/reward:0.06474532186985016 - timing_s/old_log_prob:3.0930375326424837 - timing_s/adv:0.585562577471137 - timing_s/update_actor:22.37637080810964 - timing_s/step:42.82246343977749 - timing_s/stop_profile:0.00014594756066799164 - timing_per_token_ms/update_actor:0.12078360578705409 - timing_per_token_ms/gen:0.14926603248940506 - timing_per_token_ms/adv:0.0031607609709118916 - perf/total_num_tokens:185260 - perf/time_per_step:42.82246343977749 - perf/throughput:540.7792578903612
(TaskRunner pid=2255397) Training Progress: 9%|▉ | 9/100 [07:02<1:06:50, 44.07s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 10}
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1", we need to examine the structure of the product and match it with the structure of the reactant provided in the options.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product has the following key features:
(TaskRunner pid=2255397) - A nitrogen-containing ring (likely a pyrrole or imidazole-like structure).
(TaskRunner pid=2255397) - A substituent "Cc2ccc(Cl)cc2C(F)(F)F", which is a benzene ring substituted with a chlorine atom and a trifluoromethyl group.
(TaskRunner pid=2255397) - The overall structure suggests that the reactant must have a similar core structure, with the necessary functional groups for the formation of the final product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Let's analyze the options:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option A**:
(TaskRunner pid=2255397) "O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"
(TaskRunner pid=2255397) This structure contains a nitrogen-containing ring with the correct substituent "Cc2ccc(Cl)cc2C(F)(F)F". It also has a carbonyl group and a nitrogen atom, which could be part of a condensation or coupling reaction. This matches the structure of the product closely.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option B**:
(TaskRunner pid=2255397) "O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1"
(TaskRunner pid=2255397) This option introduces a chlorine atom directly on the nitrogen, which is not present in the product. Also, the structure seems to have a different arrangement of substituents, making it unlikely to be the correct reactant.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option C**:
(TaskRunner pid=2255397) "O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1"
(TaskRunner pid=2255397) This structure has a hydroxyl group attached to the benzene ring, which is not present in the product. The presence of this hydroxyl group suggests a different synthetic pathway and thus is not the correct reactant.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option D**:
(TaskRunner pid=2255397) "O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F"
(TaskRunner pid=2255397) This option has a different arrangement of the substituents and introduces a double bond between the carbon atoms, which is not part of the product structure. It also lacks the necessary nitrogen-containing ring.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Conclusion**:
(TaskRunner pid=2255397) Only **Option A** contains the correct substituent "Cc2ccc(Cl)cc2C(F)(F)F" and the necessary nitrogen-containing ring structure. It is the only option that matches the structure of the product directly.
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_10
(TaskRunner pid=2255397) step:10 - global_seqlen/min:18995 - global_seqlen/max:25222 - global_seqlen/minmax_diff:6227 - global_seqlen/balanced_min:22810 - global_seqlen/balanced_max:22819 - global_seqlen/mean:22815.375 - actor/entropy:0.5086405277252197 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.48631101846694946 - training/rollout_probs_diff_mean:0.004814056679606438 - training/rollout_probs_diff_std:0.010112768970429897 - training/rollout_actor_probs_pearson_corr:0.9993922114372253 - self_distillation/success_group_fraction:0.6875 - self_distillation/success_sample_fraction:0.6796875 - self_distillation/correct_sample_fraction:0.37109375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.6796875 - rollout_corr/rollout_is_mean:1.0000672340393066 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.7611369912629016e-05 - rollout_corr/rollout_is_max:1.758644938468933 - rollout_corr/rollout_is_min:0.47177526354789734 - rollout_corr/rollout_is_std:0.0358346551656723 - rollout_corr/rollout_is_eff_sample_size:0.9987177621962007 - rollout_corr/rollout_is_seq_mean:1.0001494884490967 - rollout_corr/rollout_is_seq_std:0.001915071508847177 - rollout_corr/rollout_is_seq_max:1.0051805973052979 - rollout_corr/rollout_is_seq_min:0.9944745898246765 - rollout_corr/rollout_is_seq_max_deviation:0.005525410175323486 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7283513699658215) - rollout_corr/training_log_ppl:np.float64(0.5342370558064431) - rollout_corr/kl:np.float64(0.00048592705383754264) - rollout_corr/k3_kl:np.float64(0.0007897751748942028) - rollout_corr/rollout_ppl:np.float64(1.7275344631634653) - rollout_corr/rollout_log_ppl:np.float64(0.5337511280667968) - rollout_corr/log_ppl_diff:np.float64(0.00048592773964628577) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017402138910256326) - rollout_corr/log_ppl_diff_max:np.float64(0.008036613464355469) - rollout_corr/log_ppl_diff_min:np.float64(-0.007182866334915161) - rollout_corr/ppl_ratio:np.float64(1.0004885760135949) - rollout_corr/chi2_token:np.float64(0.002185102319344878) - rollout_corr/chi2_seq:np.float64(1.4425214361399412) - actor/pg_loss:np.float64(0.008299664361402392) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006863867054107686) - actor/ppo_kl:np.float64(-5.119195938618759e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.6796875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.6796875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.37109375) - self_distillation/token_reweight_w_mean:np.float64(42394.31746227876) - self_distillation/token_reweight_w_std:np.float64(702394.3469068523) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0012782558333129) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1625702439341694) - self_distillation/empty_target_batch:np.float64(0.3203125) - actor/grad_norm:np.float64(0.3786138668656349) - perf/mfu/actor:np.float64(0.054599702355521036) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(196.04931259155273) - actor/lr:np.float64(9e-07) - val-aux/sciknoweval/reward/mean@16:np.float64(0.4217261904761905) - val-aux/sciknoweval/reward/std@16:np.float64(0.28826355015906874) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.5430761904761905) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.26128001800402223) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.3013761904761905) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.2277461781455412) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.4231666666666667) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.2873520999211171) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.6505000000000001) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.2040037803005508) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.21330476190476189) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.15085658746075342) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.43689523809523806) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.24031934791752282) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7316095238095237) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.13822877715965945) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.15793333333333331) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.0882011251130811) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.444252380952381) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.19013145421249977) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.7838952380952381) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.07974930858404744) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.12557619047619048) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.04538785689784369) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.4482333333333334) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.14374275063423717) - val-aux/sciknoweval/score/mean@16:np.float64(0.4217261904761905) - val-aux/sciknoweval/score/std@16:np.float64(0.28826355015906874) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.5430761904761905) - val-aux/sciknoweval/score/best@2/std:np.float64(0.26128001800402223) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.3013761904761905) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.2277461781455412) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.4231666666666667) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.2873520999211171) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.6505000000000001) - val-aux/sciknoweval/score/best@4/std:np.float64(0.2040037803005508) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.21330476190476189) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.15085658746075342) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.43689523809523806) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.24031934791752282) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7316095238095237) - val-aux/sciknoweval/score/best@8/std:np.float64(0.13822877715965945) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.15793333333333331) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.0882011251130811) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.444252380952381) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.19013145421249977) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.7838952380952381) - val-aux/sciknoweval/score/best@16/std:np.float64(0.07974930858404744) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.12557619047619048) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.04538785689784369) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.4482333333333334) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.14374275063423717) - val-core/sciknoweval/acc/mean@16:np.float64(0.4217261904761905) - val-aux/sciknoweval/acc/std@16:np.float64(0.28826355015906874) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.5430761904761905) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.26128001800402223) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.3013761904761905) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.2277461781455412) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.4231666666666667) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.2873520999211171) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.6505000000000001) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.2040037803005508) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.21330476190476189) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.15085658746075342) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.43689523809523806) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.24031934791752282) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7316095238095237) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.13822877715965945) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.15793333333333331) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.0882011251130811) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.444252380952381) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.19013145421249977) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.7838952380952381) - val-core/sciknoweval/acc/best@16/std:np.float64(0.07974930858404744) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.12557619047619048) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.04538785689784369) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.4482333333333334) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.14374275063423717) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9946428571428572) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.018556639291927544) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9995666666666666) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.005201629481674079) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9896238095238097) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.024846621608394712) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9946904761904761) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.018505164380603217) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0001505093393264677) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9811428571428572) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.0310421950289628) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9980904761904762) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.011107450564340775) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9676523809523809) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.034830287112445846) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9995761904761905) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.004892275828419172) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9506190476190476) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.0319196096808831) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999571428571428) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0009723855004127487) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:10 - training/epoch:0 - critic/score/mean:0.37109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.37109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0003302131954114884 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0003302131954114884 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:443.60546875 - response_length/max:1164.0 - response_length/min:150.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:443.60546875 - response_length_non_aborted/max:1164.0 - response_length_non_aborted/min:150.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:269.375 - prompt_length/max:701.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001530386507511139 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8319511320441961) - timing_s/agent_loop/generate_sequences/max:np.float64(9.405133683234453) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.998352278737002) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.405133683234453) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:200 - timing_s/agent_loop/slowest/response_length:1164 - timing_s/gen:16.217690873891115 - timing_s/reward:0.0643529836088419 - timing_s/old_log_prob:3.16649679094553 - timing_s/adv:0.5727238319814205 - timing_s/update_actor:21.792541556060314 - timing_s/step:41.90203979611397 - timing_s/testing:145.202635621652 - timing_s/save_checkpoint:13.197123931720853 - timing_s/stop_profile:0.00014385022222995758 - timing_per_token_ms/update_actor:0.11939613942385516 - timing_per_token_ms/gen:0.14280787645528134 - timing_per_token_ms/adv:0.003137817327029583 - perf/total_num_tokens:182523 - perf/time_per_step:41.90203979611397 - perf/throughput:544.4931824563805
(TaskRunner pid=2255397) Training Progress: 10%|█ | 10/100 [10:23<2:18:30, 92.33s/it]
(TaskRunner pid=2255397) step:11 - global_seqlen/min:18238 - global_seqlen/max:25942 - global_seqlen/minmax_diff:7704 - global_seqlen/balanced_min:22325 - global_seqlen/balanced_max:22730 - global_seqlen/mean:22380.25 - actor/entropy:0.5434318780899048 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.38079291582107544 - training/rollout_probs_diff_mean:0.005151212681084871 - training/rollout_probs_diff_std:0.010387294925749302 - training/rollout_actor_probs_pearson_corr:0.9993837475776672 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.86328125 - self_distillation/correct_sample_fraction:0.48046875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.86328125 - rollout_corr/rollout_is_mean:1.0001193284988403 - rollout_corr/rollout_is_ratio_fraction_high:9.758572559803724e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:4.194515228271484 - rollout_corr/rollout_is_min:0.5521866679191589 - rollout_corr/rollout_is_std:0.03714950755238533 - rollout_corr/rollout_is_eff_sample_size:0.9986220538290654 - rollout_corr/rollout_is_seq_mean:1.0001423358917236 - rollout_corr/rollout_is_seq_std:0.0020691656973212957 - rollout_corr/rollout_is_seq_max:1.0071853399276733 - rollout_corr/rollout_is_seq_min:0.993625283241272 - rollout_corr/rollout_is_seq_max_deviation:0.00718533992767334 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7692446084693074) - rollout_corr/training_log_ppl:np.float64(0.5590797059703618) - rollout_corr/kl:np.float64(0.0006710044406990079) - rollout_corr/k3_kl:np.float64(0.0007906234051233696) - rollout_corr/rollout_ppl:np.float64(1.7680208389647305) - rollout_corr/rollout_log_ppl:np.float64(0.5584087027818896) - rollout_corr/log_ppl_diff:np.float64(0.0006710031884722412) - rollout_corr/log_ppl_abs_diff:np.float64(0.001697185740340501) - rollout_corr/log_ppl_diff_max:np.float64(0.006523817777633667) - rollout_corr/log_ppl_diff_min:np.float64(-0.005903065204620361) - rollout_corr/ppl_ratio:np.float64(1.0006734766066074) - rollout_corr/chi2_token:np.float64(0.0018631014972925186) - rollout_corr/chi2_seq:np.float64(1.0500562572851777) - actor/pg_loss:np.float64(0.010553930653259158) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004439412439296575) - actor/ppo_kl:np.float64(6.877072929745154e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.86328125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.86328125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.48046875) - self_distillation/token_reweight_w_mean:np.float64(85283.25226895092) - self_distillation/token_reweight_w_std:np.float64(1114820.7389926258) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0022825964260846) - self_distillation/token_reweight_w_clip_frac:np.float64(0.21137651632307097) - self_distillation/empty_target_batch:np.float64(0.13671875) - actor/grad_norm:np.float64(0.4761762246489525) - perf/mfu/actor:np.float64(0.05262875043012334) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.41241455078125) - actor/lr:np.float64(1e-06) - training/global_step:11 - training/epoch:0 - critic/score/mean:0.48046875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.48046875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.003749731695279479 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.003749731695279479 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:400.2890625 - response_length/max:1690.0 - response_length/min:156.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:400.2890625 - response_length_non_aborted/max:1690.0 - response_length_non_aborted/min:156.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.09375 - prompt_length/max:620.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012122094631195068 - timing_s/agent_loop/generate_sequences/min:np.float64(2.067027226090431) - timing_s/agent_loop/generate_sequences/max:np.float64(12.363675134256482) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.645478486927459) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.363675134256482) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:299 - timing_s/agent_loop/slowest/response_length:1690 - timing_s/gen:18.65842711366713 - timing_s/reward:0.055716294795274734 - timing_s/old_log_prob:3.1651459094136953 - timing_s/adv:0.6307941991835833 - timing_s/update_actor:22.146489353850484 - timing_s/step:44.69863370805979 - timing_s/stop_profile:0.00010724924504756927 - timing_per_token_ms/update_actor:0.1236943809488862 - timing_per_token_ms/gen:0.18207962130557145 - timing_per_token_ms/adv:0.0035231632755642994 - perf/total_num_tokens:179042 - perf/time_per_step:44.69863370805979 - perf/throughput:500.69203784107003
(TaskRunner pid=2255397) Training Progress: 11%|█ | 11/100 [11:08<1:55:21, 77.77s/it]
(TaskRunner pid=2255397) step:12 - global_seqlen/min:19921 - global_seqlen/max:27357 - global_seqlen/minmax_diff:7436 - global_seqlen/balanced_min:23213 - global_seqlen/balanced_max:23297 - global_seqlen/mean:23233.875 - actor/entropy:0.5037309527397156 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.16815555095672607 - training/rollout_probs_diff_mean:0.004780740477144718 - training/rollout_probs_diff_std:0.009941385127604008 - training/rollout_actor_probs_pearson_corr:0.9994201064109802 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.85546875 - self_distillation/correct_sample_fraction:0.46484375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.85546875 - rollout_corr/rollout_is_mean:0.999710738658905 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.6984993635560386e-05 - rollout_corr/rollout_is_max:1.6297073364257812 - rollout_corr/rollout_is_min:0.3698853552341461 - rollout_corr/rollout_is_std:0.035432368516922 - rollout_corr/rollout_is_eff_sample_size:0.9987455864550007 - rollout_corr/rollout_is_seq_mean:0.9997212886810303 - rollout_corr/rollout_is_seq_std:0.0021238892804831266 - rollout_corr/rollout_is_seq_max:1.0065120458602905 - rollout_corr/rollout_is_seq_min:0.9934496879577637 - rollout_corr/rollout_is_seq_max_deviation:0.006550312042236328 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7103436021134257) - rollout_corr/training_log_ppl:np.float64(0.5244237306469586) - rollout_corr/kl:np.float64(0.0010640324547948055) - rollout_corr/k3_kl:np.float64(0.0007240620033144296) - rollout_corr/rollout_ppl:np.float64(1.7084603938274086) - rollout_corr/rollout_log_ppl:np.float64(0.5233596969628707) - rollout_corr/log_ppl_diff:np.float64(0.0010640336840879172) - rollout_corr/log_ppl_abs_diff:np.float64(0.001916373468702659) - rollout_corr/log_ppl_diff_max:np.float64(0.007551968097686768) - rollout_corr/log_ppl_diff_min:np.float64(-0.006095826625823975) - rollout_corr/ppl_ratio:np.float64(1.001067083561793) - rollout_corr/chi2_token:np.float64(0.0007682361174374819) - rollout_corr/chi2_seq:np.float64(0.6293782293796539) - actor/pg_loss:np.float64(0.007974700187332928) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00048827441059984267) - actor/ppo_kl:np.float64(0.00010889819034787251) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.85546875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.85546875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.46484375) - self_distillation/token_reweight_w_mean:np.float64(5987.44374671299) - self_distillation/token_reweight_w_std:np.float64(110518.26587392716) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0042279572226107) - self_distillation/token_reweight_w_clip_frac:np.float64(0.17869807308306918) - self_distillation/empty_target_batch:np.float64(0.14453125) - actor/grad_norm:np.float64(0.39314158260822296) - perf/mfu/actor:np.float64(0.05490032892199064) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.42404556274414) - actor/lr:np.float64(1e-06) - training/global_step:12 - training/epoch:0 - critic/score/mean:0.46484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0070593878626823425 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0070593878626823425 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:459.96484375 - response_length/max:1546.0 - response_length/min:137.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:459.96484375 - response_length_non_aborted/max:1546.0 - response_length_non_aborted/min:137.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.09375 - prompt_length/max:665.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00016945600509643555 - timing_s/agent_loop/generate_sequences/min:np.float64(1.736042970791459) - timing_s/agent_loop/generate_sequences/max:np.float64(12.078167820349336) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.221707455886644) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.078167820349336) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:264 - timing_s/agent_loop/slowest/response_length:1546 - timing_s/gen:18.70925861224532 - timing_s/reward:0.06456440314650536 - timing_s/old_log_prob:3.0842963717877865 - timing_s/adv:0.5920961294323206 - timing_s/update_actor:22.019973430782557 - timing_s/step:44.561775021255016 - timing_s/stop_profile:0.0001148320734500885 - timing_per_token_ms/update_actor:0.11846911799464444 - timing_per_token_ms/gen:0.15888832037303566 - timing_per_token_ms/adv:0.0031855218373620445 - perf/total_num_tokens:185871 - perf/time_per_step:44.561775021255016 - perf/throughput:521.3857614270961
(TaskRunner pid=2255397) Training Progress: 12%|█▏ | 12/100 [11:52<1:39:16, 67.68s/it]
(TaskRunner pid=2255397) step:13 - global_seqlen/min:18814 - global_seqlen/max:23875 - global_seqlen/minmax_diff:5061 - global_seqlen/balanced_min:21556 - global_seqlen/balanced_max:21562 - global_seqlen/mean:21560.5 - actor/entropy:0.540065348148346 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2304113507270813 - training/rollout_probs_diff_mean:0.005129465367645025 - training/rollout_probs_diff_std:0.010314878076314926 - training/rollout_actor_probs_pearson_corr:0.9993942975997925 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.796875 - self_distillation/correct_sample_fraction:0.375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.796875 - rollout_corr/rollout_is_mean:0.9999523758888245 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.84678445092868e-06 - rollout_corr/rollout_is_max:1.643043875694275 - rollout_corr/rollout_is_min:0.3293769061565399 - rollout_corr/rollout_is_std:0.03673160448670387 - rollout_corr/rollout_is_eff_sample_size:0.9986523694012616 - rollout_corr/rollout_is_seq_mean:0.9999930262565613 - rollout_corr/rollout_is_seq_std:0.0019651646725833416 - rollout_corr/rollout_is_seq_max:1.0069950819015503 - rollout_corr/rollout_is_seq_min:0.9950668215751648 - rollout_corr/rollout_is_seq_max_deviation:0.006995081901550293 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.75340425549075) - rollout_corr/training_log_ppl:np.float64(0.5529010602040216) - rollout_corr/kl:np.float64(0.0008567688251606143) - rollout_corr/k3_kl:np.float64(0.0008690143870353495) - rollout_corr/rollout_ppl:np.float64(1.7518776911310852) - rollout_corr/rollout_log_ppl:np.float64(0.5520442884298973) - rollout_corr/log_ppl_diff:np.float64(0.0008567717741243541) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018868366605602205) - rollout_corr/log_ppl_diff_max:np.float64(0.00990372896194458) - rollout_corr/log_ppl_diff_min:np.float64(-0.005962073802947998) - rollout_corr/ppl_ratio:np.float64(1.0008597834967077) - rollout_corr/chi2_token:np.float64(0.0017287565860897303) - rollout_corr/chi2_seq:np.float64(0.7434897748753428) - actor/pg_loss:np.float64(0.009388480219058692) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.001301017089645029) - actor/ppo_kl:np.float64(0.0001421315267471357) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.796875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.796875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.375) - self_distillation/token_reweight_w_mean:np.float64(1917.8817092673853) - self_distillation/token_reweight_w_std:np.float64(43886.33348266815) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0036602555774152) - self_distillation/token_reweight_w_clip_frac:np.float64(0.2082537584064994) - self_distillation/empty_target_batch:np.float64(0.203125) - actor/grad_norm:np.float64(0.43746187537908554) - perf/mfu/actor:np.float64(0.051655857866616195) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.21465253829956) - actor/lr:np.float64(1e-06) - training/global_step:13 - training/epoch:0 - critic/score/mean:0.375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.003404525574296713 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.003404525574296713 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:396.703125 - response_length/max:972.0 - response_length/min:141.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:396.703125 - response_length_non_aborted/max:972.0 - response_length_non_aborted/min:141.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.0625 - prompt_length/max:612.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013491325080394745 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8798387572169304) - timing_s/agent_loop/generate_sequences/max:np.float64(8.628671053797007) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.614353348057193) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.628671053797007) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:232 - timing_s/agent_loop/slowest/response_length:972 - timing_s/gen:15.247638115659356 - timing_s/reward:0.06261933967471123 - timing_s/old_log_prob:3.0381316114217043 - timing_s/adv:0.5441188924014568 - timing_s/update_actor:21.71716057509184 - timing_s/step:40.69797484949231 - timing_s/stop_profile:0.00012857653200626373 - timing_per_token_ms/update_actor:0.12590826149145334 - timing_per_token_ms/gen:0.15014019965003894 - timing_per_token_ms/adv:0.0031546050207639947 - perf/total_num_tokens:172484 - perf/time_per_step:40.69797484949231 - perf/throughput:529.7683749556143
(TaskRunner pid=2255397) Training Progress: 13%|█▎ | 13/100 [12:33<1:26:18, 59.52s/it]
(TaskRunner pid=2255397) step:14 - global_seqlen/min:17920 - global_seqlen/max:21796 - global_seqlen/minmax_diff:3876 - global_seqlen/balanced_min:20032 - global_seqlen/balanced_max:20152 - global_seqlen/mean:20054.25 - actor/entropy:0.5020557641983032 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18802747130393982 - training/rollout_probs_diff_mean:0.005197192542254925 - training/rollout_probs_diff_std:0.01069364883005619 - training/rollout_actor_probs_pearson_corr:0.9992953538894653 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.76953125 - self_distillation/correct_sample_fraction:0.40625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.76953125 - rollout_corr/rollout_is_mean:1.000118374824524 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6423780918121338 - rollout_corr/rollout_is_min:0.6023645997047424 - rollout_corr/rollout_is_std:0.03736548125743866 - rollout_corr/rollout_is_eff_sample_size:0.9986062429080049 - rollout_corr/rollout_is_seq_mean:1.0001424551010132 - rollout_corr/rollout_is_seq_std:0.002069722395390272 - rollout_corr/rollout_is_seq_max:1.0075104236602783 - rollout_corr/rollout_is_seq_min:0.9939577579498291 - rollout_corr/rollout_is_seq_max_deviation:0.00751042366027832 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7107822382822633) - rollout_corr/training_log_ppl:np.float64(0.5240909514250234) - rollout_corr/kl:np.float64(0.0007750868314850834) - rollout_corr/k3_kl:np.float64(0.0011061159723340097) - rollout_corr/rollout_ppl:np.float64(1.7093983073718846) - rollout_corr/rollout_log_ppl:np.float64(0.5233158656046726) - rollout_corr/log_ppl_diff:np.float64(0.000775085820350796) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019063884974457324) - rollout_corr/log_ppl_diff_max:np.float64(0.009017229080200195) - rollout_corr/log_ppl_diff_min:np.float64(-0.007934510707855225) - rollout_corr/ppl_ratio:np.float64(1.0007783074397594) - rollout_corr/chi2_token:np.float64(0.0027928389608860016) - rollout_corr/chi2_seq:np.float64(0.6327834515832365) - actor/pg_loss:np.float64(0.009141277987509966) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0023054060643517005) - actor/ppo_kl:np.float64(0.00017033189165172047) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.76953125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.76953125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.40625) - self_distillation/token_reweight_w_mean:np.float64(51790.3643477282) - self_distillation/token_reweight_w_std:np.float64(710581.8369147222) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0032089522574097) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1833768331562169) - self_distillation/empty_target_batch:np.float64(0.23046875) - actor/grad_norm:np.float64(0.5500108152627945) - perf/mfu/actor:np.float64(0.04745480566246875) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.28086853027344) - actor/lr:np.float64(1e-06) - training/global_step:14 - training/epoch:0 - critic/score/mean:0.40625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.40625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:1.6309666534652933e-05 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:1.6309666534652933e-05 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:359.2578125 - response_length/max:1234.0 - response_length/min:131.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:359.2578125 - response_length_non_aborted/max:1234.0 - response_length_non_aborted/min:131.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:267.4375 - prompt_length/max:666.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013267621397972107 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5782812405377626) - timing_s/agent_loop/generate_sequences/max:np.float64(9.458859227597713) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.9894911859955755) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.458859227597713) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:274 - timing_s/agent_loop/slowest/response_length:1234 - timing_s/gen:16.56269684806466 - timing_s/reward:0.05258593708276749 - timing_s/old_log_prob:2.994526704773307 - timing_s/adv:0.5849139746278524 - timing_s/update_actor:21.962995065376163 - timing_s/step:42.20455348305404 - timing_s/stop_profile:0.0001186169683933258 - timing_per_token_ms/update_actor:0.13689738500178367 - timing_per_token_ms/gen:0.18008803792611353 - timing_per_token_ms/adv:0.003645823046410689 - perf/total_num_tokens:160434 - perf/time_per_step:42.20455348305404 - perf/throughput:475.1679225335763
(TaskRunner pid=2255397) Training Progress: 14%|█▍ | 14/100 [13:15<1:17:50, 54.31s/it]
(TaskRunner pid=2255397) step:15 - global_seqlen/min:17292 - global_seqlen/max:23462 - global_seqlen/minmax_diff:6170 - global_seqlen/balanced_min:20153 - global_seqlen/balanced_max:20223 - global_seqlen/mean:20173.0 - actor/entropy:0.4989277422428131 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23447632789611816 - training/rollout_probs_diff_mean:0.005335042718797922 - training/rollout_probs_diff_std:0.011061806231737137 - training/rollout_actor_probs_pearson_corr:0.99927818775177 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.76953125 - self_distillation/correct_sample_fraction:0.46875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.76953125 - rollout_corr/rollout_is_mean:0.9997453093528748 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.5897420644760132 - rollout_corr/rollout_is_min:0.5000474452972412 - rollout_corr/rollout_is_std:0.03860742598772049 - rollout_corr/rollout_is_eff_sample_size:0.9985110313307147 - rollout_corr/rollout_is_seq_mean:0.9997358918190002 - rollout_corr/rollout_is_seq_std:0.0024659933988004923 - rollout_corr/rollout_is_seq_max:1.0084636211395264 - rollout_corr/rollout_is_seq_min:0.9930576682090759 - rollout_corr/rollout_is_seq_max_deviation:0.008463621139526367 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7213433790020645) - rollout_corr/training_log_ppl:np.float64(0.530678162467666) - rollout_corr/kl:np.float64(0.0010814671836847367) - rollout_corr/k3_kl:np.float64(0.000888961919315534) - rollout_corr/rollout_ppl:np.float64(1.7193831540644169) - rollout_corr/rollout_log_ppl:np.float64(0.5295966898556799) - rollout_corr/log_ppl_diff:np.float64(0.0010814726119861007) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019996468909084797) - rollout_corr/log_ppl_diff_max:np.float64(0.012681245803833008) - rollout_corr/log_ppl_diff_min:np.float64(-0.0078064799308776855) - rollout_corr/ppl_ratio:np.float64(1.0010852627456188) - rollout_corr/chi2_token:np.float64(0.0014137444086372852) - rollout_corr/chi2_seq:np.float64(0.5360429051797837) - actor/pg_loss:np.float64(0.007955617504194379) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006377134768627002) - actor/ppo_kl:np.float64(1.769121835337728e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.76953125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.76953125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.46875) - self_distillation/token_reweight_w_mean:np.float64(8754.255326425191) - self_distillation/token_reweight_w_std:np.float64(117017.50059508975) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.001879719318822) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1550164521613624) - self_distillation/empty_target_batch:np.float64(0.23046875) - actor/grad_norm:np.float64(0.4198375418782234) - perf/mfu/actor:np.float64(0.04879610399473919) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.12841796875) - actor/lr:np.float64(1e-06) - training/global_step:15 - training/epoch:0 - critic/score/mean:0.46875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00847491342574358 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00847491342574358 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:343.5 - response_length/max:1036.0 - response_length/min:108.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:343.5 - response_length_non_aborted/max:1036.0 - response_length_non_aborted/min:108.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.90625 - prompt_length/max:494.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012453272938728333 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3633983470499516) - timing_s/agent_loop/generate_sequences/max:np.float64(8.056044036522508) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.874184662854532) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.056044036522508) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:353 - timing_s/agent_loop/slowest/response_length:1036 - timing_s/gen:14.75450462475419 - timing_s/reward:0.052955711260437965 - timing_s/old_log_prob:3.0404567383229733 - timing_s/adv:0.516296835616231 - timing_s/update_actor:21.39274295605719 - timing_s/step:39.795078329741955 - timing_s/stop_profile:3.5120174288749695e-05 - timing_per_token_ms/update_actor:0.13255801663149502 - timing_per_token_ms/gen:0.16778685208281238 - timing_per_token_ms/adv:0.003199182295743264 - perf/total_num_tokens:161384 - perf/time_per_step:39.795078329741955 - perf/throughput:506.9219824835261
(TaskRunner pid=2255397) Training Progress: 15%|█▌ | 15/100 [13:55<1:10:45, 49.94s/it]
(TaskRunner pid=2255397) step:16 - global_seqlen/min:17365 - global_seqlen/max:24430 - global_seqlen/minmax_diff:7065 - global_seqlen/balanced_min:20809 - global_seqlen/balanced_max:20815 - global_seqlen/mean:20814.0 - actor/entropy:0.4525703191757202 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.44003361463546753 - training/rollout_probs_diff_mean:0.0051152328960597515 - training/rollout_probs_diff_std:0.011277093552052975 - training/rollout_actor_probs_pearson_corr:0.9991934299468994 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.82421875 - self_distillation/correct_sample_fraction:0.47265625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.82421875 - rollout_corr/rollout_is_mean:1.000049352645874 - rollout_corr/rollout_is_ratio_fraction_high:1.0565463526290841e-05 - rollout_corr/rollout_is_ratio_fraction_low:6.339278479572386e-05 - rollout_corr/rollout_is_max:2.1190927028656006 - rollout_corr/rollout_is_min:0.16761082410812378 - rollout_corr/rollout_is_std:0.03827483579516411 - rollout_corr/rollout_is_eff_sample_size:0.9985372988785958 - rollout_corr/rollout_is_seq_mean:1.0000265836715698 - rollout_corr/rollout_is_seq_std:0.0022894139401614666 - rollout_corr/rollout_is_seq_max:1.0063250064849854 - rollout_corr/rollout_is_seq_min:0.992612361907959 - rollout_corr/rollout_is_seq_max_deviation:0.007387638092041016 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6002246476709843) - rollout_corr/training_log_ppl:np.float64(0.46000494839972816) - rollout_corr/kl:np.float64(0.0006955834787873982) - rollout_corr/k3_kl:np.float64(0.0009704830327450509) - rollout_corr/rollout_ppl:np.float64(1.5991095416247845) - rollout_corr/rollout_log_ppl:np.float64(0.45930936612421647) - rollout_corr/log_ppl_diff:np.float64(0.0006955822755116969) - rollout_corr/log_ppl_abs_diff:np.float64(0.0021134040143806487) - rollout_corr/log_ppl_diff_max:np.float64(0.007766246795654297) - rollout_corr/log_ppl_diff_min:np.float64(-0.005490928888320923) - rollout_corr/ppl_ratio:np.float64(1.0006991014815867) - rollout_corr/chi2_token:np.float64(0.0025813004467636347) - rollout_corr/chi2_seq:np.float64(3.1832106430083513) - actor/pg_loss:np.float64(0.009350149193778634) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0011533425140441977) - actor/ppo_kl:np.float64(-8.012323441874969e-05) - actor/pg_clipfrac_lower:np.float64(1.187310044770129e-05) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.82421875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.82421875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.47265625) - self_distillation/token_reweight_w_mean:np.float64(35795.58769591758) - self_distillation/token_reweight_w_std:np.float64(482687.72877264104) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0030966859776527) - self_distillation/token_reweight_w_clip_frac:np.float64(0.18662144461995922) - self_distillation/empty_target_batch:np.float64(0.17578125) - actor/grad_norm:np.float64(0.49581966549158096) - perf/mfu/actor:np.float64(0.05050208271241423) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.1108741760254) - actor/lr:np.float64(1e-06) - training/global_step:16 - training/epoch:0 - critic/score/mean:0.47265625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.47265625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.01058791484683752 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.01058791484683752 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:369.71875 - response_length/max:1008.0 - response_length/min:123.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:369.71875 - response_length_non_aborted/max:1008.0 - response_length_non_aborted/min:123.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.71875 - prompt_length/max:859.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:3.693439066410065e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3586726989597082) - timing_s/agent_loop/generate_sequences/max:np.float64(7.759539771825075) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.9693014876902453) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.759539771825075) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:226 - timing_s/agent_loop/slowest/response_length:1008 - timing_s/gen:14.533444579690695 - timing_s/reward:0.06137754209339619 - timing_s/old_log_prob:3.1735431142151356 - timing_s/adv:0.5521691609174013 - timing_s/update_actor:21.5248114336282 - timing_s/step:39.9321037363261 - timing_s/stop_profile:0.00011518038809299469 - timing_per_token_ms/update_actor:0.1292688300760798 - timing_per_token_ms/gen:0.15355257987163695 - timing_per_token_ms/adv:0.0033160922991580266 - perf/total_num_tokens:166512 - perf/time_per_step:39.9321037363261 - perf/throughput:521.2347472959601
(TaskRunner pid=2255397) Training Progress: 16%|█▌ | 16/100 [14:35<1:05:43, 46.94s/it]
(TaskRunner pid=2255397) step:17 - global_seqlen/min:22601 - global_seqlen/max:25800 - global_seqlen/minmax_diff:3199 - global_seqlen/balanced_min:23884 - global_seqlen/balanced_max:24138 - global_seqlen/mean:23919.75 - actor/entropy:0.5084906220436096 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23104846477508545 - training/rollout_probs_diff_mean:0.004854131024330854 - training/rollout_probs_diff_std:0.010198368690907955 - training/rollout_actor_probs_pearson_corr:0.9993425011634827 - self_distillation/success_group_fraction:0.6875 - self_distillation/success_sample_fraction:0.6796875 - self_distillation/correct_sample_fraction:0.41015625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.6796875 - rollout_corr/rollout_is_mean:1.0001150369644165 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.8287475288379937e-05 - rollout_corr/rollout_is_max:1.7099158763885498 - rollout_corr/rollout_is_min:0.4197293221950531 - rollout_corr/rollout_is_std:0.036216843873262405 - rollout_corr/rollout_is_eff_sample_size:0.9986904150765131 - rollout_corr/rollout_is_seq_mean:1.0001122951507568 - rollout_corr/rollout_is_seq_std:0.0019326574401929975 - rollout_corr/rollout_is_seq_max:1.0060887336730957 - rollout_corr/rollout_is_seq_min:0.9933558106422424 - rollout_corr/rollout_is_seq_max_deviation:0.006644189357757568 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7412785273045301) - rollout_corr/training_log_ppl:np.float64(0.5394562017754652) - rollout_corr/kl:np.float64(0.00046429189457519726) - rollout_corr/k3_kl:np.float64(0.0007774811874128318) - rollout_corr/rollout_ppl:np.float64(1.7404227014631033) - rollout_corr/rollout_log_ppl:np.float64(0.538991907320451) - rollout_corr/log_ppl_diff:np.float64(0.0004642944550141692) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017179182032123208) - rollout_corr/log_ppl_diff_max:np.float64(0.007203280925750732) - rollout_corr/log_ppl_diff_min:np.float64(-0.006280422210693359) - rollout_corr/ppl_ratio:np.float64(1.0004668079782277) - rollout_corr/chi2_token:np.float64(0.002199894981458783) - rollout_corr/chi2_seq:np.float64(2.1964850607328117) - actor/pg_loss:np.float64(0.006313061923719943) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00043986083028357825) - actor/ppo_kl:np.float64(-0.00013203885243839864) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.6796875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.6796875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.41015625) - self_distillation/token_reweight_w_mean:np.float64(1803.160887064878) - self_distillation/token_reweight_w_std:np.float64(34584.773948686314) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0014728792011738) - self_distillation/token_reweight_w_clip_frac:np.float64(0.12900599383283406) - self_distillation/empty_target_batch:np.float64(0.3203125) - actor/grad_norm:np.float64(0.38481806963682175) - perf/mfu/actor:np.float64(0.05734825677685727) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.12204360961914) - actor/lr:np.float64(1e-06) - training/global_step:17 - training/epoch:0 - critic/score/mean:0.41015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.41015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005190751980990171 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005190751980990171 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:414.2734375 - response_length/max:1415.0 - response_length/min:154.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:414.2734375 - response_length_non_aborted/max:1415.0 - response_length_non_aborted/min:154.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:333.21875 - prompt_length/max:617.0 - prompt_length/min:189.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010232068598270416 - timing_s/agent_loop/generate_sequences/min:np.float64(1.907282380387187) - timing_s/agent_loop/generate_sequences/max:np.float64(11.15155959688127) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.6968326987480395) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(11.15155959688127) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:297 - timing_s/agent_loop/slowest/response_length:1415 - timing_s/gen:17.835143700242043 - timing_s/reward:0.06522902101278305 - timing_s/old_log_prob:3.041203547269106 - timing_s/adv:0.5475675780326128 - timing_s/update_actor:21.574721600860357 - timing_s/step:43.15138044394553 - timing_s/stop_profile:4.641711711883545e-05 - timing_per_token_ms/update_actor:0.11274533388131333 - timing_per_token_ms/gen:0.16817040093011149 - timing_per_token_ms/adv:0.002861482551200435 - perf/total_num_tokens:191358 - perf/time_per_step:43.15138044394553 - perf/throughput:554.3217796026761
(TaskRunner pid=2255397) Training Progress: 17%|█▋ | 17/100 [15:18<1:03:22, 45.81s/it]
(TaskRunner pid=2255397) step:18 - global_seqlen/min:17607 - global_seqlen/max:24855 - global_seqlen/minmax_diff:7248 - global_seqlen/balanced_min:20730 - global_seqlen/balanced_max:20735 - global_seqlen/mean:20733.125 - actor/entropy:0.4945189952850342 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.30840808153152466 - training/rollout_probs_diff_mean:0.005118089262396097 - training/rollout_probs_diff_std:0.010652771219611168 - training/rollout_actor_probs_pearson_corr:0.9993004202842712 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.80078125 - self_distillation/correct_sample_fraction:0.49609375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.80078125 - rollout_corr/rollout_is_mean:0.9998429417610168 - rollout_corr/rollout_is_ratio_fraction_high:2.0853969544987194e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.032266139984131 - rollout_corr/rollout_is_min:0.5130106210708618 - rollout_corr/rollout_is_std:0.03713827580213547 - rollout_corr/rollout_is_eff_sample_size:0.9986221727100675 - rollout_corr/rollout_is_seq_mean:0.9997930526733398 - rollout_corr/rollout_is_seq_std:0.0022385723423212767 - rollout_corr/rollout_is_seq_max:1.007550835609436 - rollout_corr/rollout_is_seq_min:0.9926072359085083 - rollout_corr/rollout_is_seq_max_deviation:0.007550835609436035 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6632476961240172) - rollout_corr/training_log_ppl:np.float64(0.49761353904614225) - rollout_corr/kl:np.float64(0.0008254678116763614) - rollout_corr/k3_kl:np.float64(0.0008354976650934987) - rollout_corr/rollout_ppl:np.float64(1.6618767492473125) - rollout_corr/rollout_log_ppl:np.float64(0.49678807152668014) - rollout_corr/log_ppl_diff:np.float64(0.0008254675194621086) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019702696008607745) - rollout_corr/log_ppl_diff_max:np.float64(0.008582055568695068) - rollout_corr/log_ppl_diff_min:np.float64(-0.006475538015365601) - rollout_corr/ppl_ratio:np.float64(1.0008287394884974) - rollout_corr/chi2_token:np.float64(0.0016966825351119041) - rollout_corr/chi2_seq:np.float64(0.7993498002178967) - actor/pg_loss:np.float64(0.006876209634356201) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009630748777453846) - actor/ppo_kl:np.float64(-0.00010461421089758005) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.80078125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.80078125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.49609375) - self_distillation/token_reweight_w_mean:np.float64(6797.09068774269) - self_distillation/token_reweight_w_std:np.float64(117938.16381233273) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0013849884271622) - self_distillation/token_reweight_w_clip_frac:np.float64(0.17171310502453707) - self_distillation/empty_target_batch:np.float64(0.19921875) - actor/grad_norm:np.float64(0.4298411011695862) - perf/mfu/actor:np.float64(0.05005069497056926) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.3214340209961) - actor/lr:np.float64(1e-06) - training/global_step:18 - training/epoch:0 - critic/score/mean:0.49609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.49609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005354256834834814 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005354256834834814 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:374.62890625 - response_length/max:907.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:374.62890625 - response_length_non_aborted/max:907.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.28125 - prompt_length/max:540.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001042727380990982 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7595505379140377) - timing_s/agent_loop/generate_sequences/max:np.float64(7.9033639673143625) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.420700289687375) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.9033639673143625) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:245 - timing_s/agent_loop/slowest/response_length:907 - timing_s/gen:14.91362101584673 - timing_s/reward:0.05069681629538536 - timing_s/old_log_prob:3.109211662784219 - timing_s/adv:0.5708821713924408 - timing_s/update_actor:21.388794276863337 - timing_s/step:40.0685121640563 - timing_s/stop_profile:0.00011024437844753265 - timing_per_token_ms/update_actor:0.12895302973420153 - timing_per_token_ms/gen:0.15550410318384578 - timing_per_token_ms/adv:0.0034418483187679183 - perf/total_num_tokens:165865 - perf/time_per_step:40.0685121640563 - perf/throughput:517.4418484796842
(TaskRunner pid=2255397) Training Progress: 18%|█▊ | 18/100 [15:58<1:00:16, 44.11s/it]
(TaskRunner pid=2255397) step:19 - global_seqlen/min:16539 - global_seqlen/max:26420 - global_seqlen/minmax_diff:9881 - global_seqlen/balanced_min:21542 - global_seqlen/balanced_max:21548 - global_seqlen/mean:21545.625 - actor/entropy:0.48952940106391907 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.21504220366477966 - training/rollout_probs_diff_mean:0.0051950011402368546 - training/rollout_probs_diff_std:0.010906816460192204 - training/rollout_actor_probs_pearson_corr:0.9992679953575134 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.7421875 - self_distillation/correct_sample_fraction:0.5 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7421875 - rollout_corr/rollout_is_mean:1.0002551078796387 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.993241310119629 - rollout_corr/rollout_is_min:0.508678138256073 - rollout_corr/rollout_is_std:0.03832463547587395 - rollout_corr/rollout_is_eff_sample_size:0.9985340896470543 - rollout_corr/rollout_is_seq_mean:1.000322699546814 - rollout_corr/rollout_is_seq_std:0.0022897543385624886 - rollout_corr/rollout_is_seq_max:1.007610559463501 - rollout_corr/rollout_is_seq_min:0.9937204718589783 - rollout_corr/rollout_is_seq_max_deviation:0.0076105594635009766 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6905396389774978) - rollout_corr/training_log_ppl:np.float64(0.5122466792818159) - rollout_corr/kl:np.float64(0.0005300049446929078) - rollout_corr/k3_kl:np.float64(0.0008489201253496503) - rollout_corr/rollout_ppl:np.float64(1.6896098447032273) - rollout_corr/rollout_log_ppl:np.float64(0.511716675653588) - rollout_corr/log_ppl_diff:np.float64(0.0005300036282278597) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018085126648657024) - rollout_corr/log_ppl_diff_max:np.float64(0.011858522891998291) - rollout_corr/log_ppl_diff_min:np.float64(-0.007662832736968994) - rollout_corr/ppl_ratio:np.float64(1.0005329437553883) - rollout_corr/chi2_token:np.float64(0.00233305711299181) - rollout_corr/chi2_seq:np.float64(0.9715974144637585) - actor/pg_loss:np.float64(0.006131203263066709) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005582695239354507) - actor/ppo_kl:np.float64(7.291653415109067e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7421875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7421875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5) - self_distillation/token_reweight_w_mean:np.float64(19979.935181184206) - self_distillation/token_reweight_w_std:np.float64(310836.0626574742) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0012645116075873) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1186036104336381) - self_distillation/empty_target_batch:np.float64(0.2578125) - actor/grad_norm:np.float64(0.3470497876405716) - perf/mfu/actor:np.float64(0.051614476906136436) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.35358047485352) - actor/lr:np.float64(1e-06) - training/global_step:19 - training/epoch:0 - critic/score/mean:0.5 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004195125307887793 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004195125307887793 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:371.17578125 - response_length/max:894.0 - response_length/min:152.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:371.17578125 - response_length_non_aborted/max:894.0 - response_length_non_aborted/min:152.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.125 - prompt_length/max:836.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014657527208328247 - timing_s/agent_loop/generate_sequences/min:np.float64(1.919897235929966) - timing_s/agent_loop/generate_sequences/max:np.float64(7.8232190404087305) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.320013291013311) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.8232190404087305) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:213 - timing_s/agent_loop/slowest/response_length:894 - timing_s/gen:14.59238414466381 - timing_s/reward:0.06268099136650562 - timing_s/old_log_prob:2.9667536336928606 - timing_s/adv:0.558667978271842 - timing_s/update_actor:21.68359724804759 - timing_s/step:39.951241597533226 - timing_s/stop_profile:0.00011624395847320557 - timing_per_token_ms/update_actor:0.12580046557043245 - timing_per_token_ms/gen:0.153570096554065 - timing_per_token_ms/adv:0.0032411915311800076 - perf/total_num_tokens:172365 - perf/time_per_step:39.951241597533226 - perf/throughput:539.2980077327641
(TaskRunner pid=2255397) Training Progress: 19%|█▉ | 19/100 [16:38<57:52, 42.87s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 20}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**, we must identify the structure of the product and match it with the most appropriate reactant that would lead to its formation through a plausible synthetic pathway.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product has the following key structural features:
(TaskRunner pid=2255397) - A **nitrogen-containing ring** (likely a pyrrole or imidazole-like structure).
(TaskRunner pid=2255397) - A **chlorinated aromatic ring** (e.g., **Cl-phenyl**).
(TaskRunner pid=2255397) - A **trifluoromethyl group** (C(F)(F)F).
(TaskRunner pid=2255397) - A **nitrogen atom** bonded to a carbon chain that includes the **Cl-phenyl** and **trifluoromethyl** groups.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Now, let’s examine the options:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option A**:
(TaskRunner pid=2255397) `O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1`
(TaskRunner pid=2255397) This structure contains a **nitrogen ring** with the same **Cl-phenyl** and **trifluoromethyl** substituents as the product. The **carbonyl groups** and **nitrogen ring** suggest a possible **amide** or **nitrile** structure that could be a **precursor** to the product. It is the **most structurally similar** to the product and would likely be the **correct reactant** in a synthesis involving ring formation or substitution.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option B**:
(TaskRunner pid=2255397) `O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1`
(TaskRunner pid=2255397) This structure contains **two chlorines** and a **nitrogen ring**, but it lacks the **trifluoromethyl group** and the **correct arrangement of substituents** as in the product. It is **not a good match**.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option C**:
(TaskRunner pid=2255397) `O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1`
(TaskRunner pid=2255397) This structure includes a **trifluoromethyl group** and **Cl-phenyl**, but the **nitrogen ring** is **not directly bonded to the Cl-phenyl** or **trifluoromethyl**, and the **structure is more complex**. It might be a **byproduct** or **intermediate**, but not the **direct reactant**.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option D**:
(TaskRunner pid=2255397) `O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F`
(TaskRunner pid=2255397) This structure contains a **trifluoromethyl group**, but it **lacks the nitrogen ring** and the **Cl-phenyl** substituents. It is **structurally different** from the product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Conclusion**:
(TaskRunner pid=2255397) Only **Option A** contains the **correct substituents** (Cl-phenyl, trifluoromethyl, and a nitrogen ring), and the **structure is most consistent** with the product. It is the **only plausible reactant** that would lead to the synthesis of the given product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_20
(TaskRunner pid=2255397) step:20 - global_seqlen/min:18853 - global_seqlen/max:23232 - global_seqlen/minmax_diff:4379 - global_seqlen/balanced_min:20703 - global_seqlen/balanced_max:20714 - global_seqlen/mean:20709.5 - actor/entropy:0.4852030277252197 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3635062277317047 - training/rollout_probs_diff_mean:0.0052037229761481285 - training/rollout_probs_diff_std:0.010991442948579788 - training/rollout_actor_probs_pearson_corr:0.9992546439170837 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83203125 - self_distillation/correct_sample_fraction:0.640625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83203125 - rollout_corr/rollout_is_mean:1.0001426935195923 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.1487816411536187e-05 - rollout_corr/rollout_is_max:1.6168099641799927 - rollout_corr/rollout_is_min:0.4058627188205719 - rollout_corr/rollout_is_std:0.03768158331513405 - rollout_corr/rollout_is_eff_sample_size:0.9985827057725277 - rollout_corr/rollout_is_seq_mean:1.0002155303955078 - rollout_corr/rollout_is_seq_std:0.002177149523049593 - rollout_corr/rollout_is_seq_max:1.0087448358535767 - rollout_corr/rollout_is_seq_min:0.9949045181274414 - rollout_corr/rollout_is_seq_max_deviation:0.00874483585357666 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.65907796099782) - rollout_corr/training_log_ppl:np.float64(0.49622454700875096) - rollout_corr/kl:np.float64(0.0007081711968908166) - rollout_corr/k3_kl:np.float64(0.000836590931498904) - rollout_corr/rollout_ppl:np.float64(1.6578862462192774) - rollout_corr/rollout_log_ppl:np.float64(0.49551637563854456) - rollout_corr/log_ppl_diff:np.float64(0.0007081713702064008) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018356915388721973) - rollout_corr/log_ppl_diff_max:np.float64(0.007012069225311279) - rollout_corr/log_ppl_diff_min:np.float64(-0.006520330905914307) - rollout_corr/ppl_ratio:np.float64(1.0007108724676073) - rollout_corr/chi2_token:np.float64(0.001943168230354786) - rollout_corr/chi2_seq:np.float64(0.7969548560213298) - actor/pg_loss:np.float64(0.006117190117947757) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006000254938953731) - actor/ppo_kl:np.float64(0.0001683622528645401) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83203125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83203125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.640625) - self_distillation/token_reweight_w_mean:np.float64(23049.971880125348) - self_distillation/token_reweight_w_std:np.float64(317665.6890155331) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9982667530421168) - self_distillation/token_reweight_w_clip_frac:np.float64(0.15381890590651892) - self_distillation/empty_target_batch:np.float64(0.16796875) - actor/grad_norm:np.float64(0.3626445010304451) - perf/mfu/actor:np.float64(0.05021794625693002) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.33641052246094) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.5196428571428572) - val-aux/sciknoweval/reward/std@16:np.float64(0.2570318204515784) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.6278380952380953) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.21727863221866373) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.41209523809523807) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.2203367172556475) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.5193428571428571) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.2582634523002606) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.713442857142857) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.15828990121163083) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.3252571428571428) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.15921732171540784) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.5385380952380953) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.2095760136213612) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.7737142857142857) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.1018065135469096) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.2657142857142857) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.10472175061646756) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.5479142857142857) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.15900279342137813) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8120428571428572) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.05885508241318426) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.22566190476190473) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.06395190241306174) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.5533238095238096) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.11781703809110204) - val-aux/sciknoweval/score/mean@16:np.float64(0.5196428571428572) - val-aux/sciknoweval/score/std@16:np.float64(0.2570318204515784) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.6278380952380953) - val-aux/sciknoweval/score/best@2/std:np.float64(0.21727863221866373) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.41209523809523807) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.2203367172556475) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.5193428571428571) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.2582634523002606) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.713442857142857) - val-aux/sciknoweval/score/best@4/std:np.float64(0.15828990121163083) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.3252571428571428) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.15921732171540784) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.5385380952380953) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.2095760136213612) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.7737142857142857) - val-aux/sciknoweval/score/best@8/std:np.float64(0.1018065135469096) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.2657142857142857) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.10472175061646756) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.5479142857142857) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.15900279342137813) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8120428571428572) - val-aux/sciknoweval/score/best@16/std:np.float64(0.05885508241318426) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.22566190476190473) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.06395190241306174) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.5533238095238096) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.11781703809110204) - val-core/sciknoweval/acc/mean@16:np.float64(0.5196428571428572) - val-aux/sciknoweval/acc/std@16:np.float64(0.2570318204515784) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.6278380952380953) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.21727863221866373) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.41209523809523807) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.2203367172556475) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.5193428571428571) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.2582634523002606) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.713442857142857) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.15828990121163083) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.3252571428571428) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.15921732171540784) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.5385380952380953) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.2095760136213612) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.7737142857142857) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.1018065135469096) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.2657142857142857) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.10472175061646756) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.5479142857142857) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.15900279342137813) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8120428571428572) - val-core/sciknoweval/acc/best@16/std:np.float64(0.05885508241318426) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.22566190476190473) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.06395190241306174) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.5533238095238096) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.11781703809110204) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9979166666666667) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.005738829557423338) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9996761904761906) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0021556297891449894) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9960809523809524) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.007408678245820718) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9977904761904762) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0058674039295688075) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9999904761904761) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0002127457895589398) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9931095238095238) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.008738507306529836) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9991428571428571) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0038031567601600064) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9892238095238096) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.008745323711299243) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9997809523809524) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0019148844274678545) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9851285714285715) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.006912686317569252) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999857142857144) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.00036325512888540755) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:20 - training/epoch:0 - critic/score/mean:0.640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004073284100741148 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004073284100741148 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:363.578125 - response_length/max:1073.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:363.578125 - response_length_non_aborted/max:1073.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:283.59375 - prompt_length/max:392.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001299697905778885 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8756434936076403) - timing_s/agent_loop/generate_sequences/max:np.float64(8.891039799898863) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.303868516799412) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.891039799898863) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:270 - timing_s/agent_loop/slowest/response_length:1073 - timing_s/gen:15.654097167775035 - timing_s/reward:0.05749098025262356 - timing_s/old_log_prob:3.016568712890148 - timing_s/adv:0.5642522405833006 - timing_s/update_actor:21.53763866983354 - timing_s/step:40.86700990051031 - timing_s/testing:131.59580136090517 - timing_s/save_checkpoint:13.167504344135523 - timing_s/stop_profile:0.00014436431229114532 - timing_per_token_ms/update_actor:0.12999854336073746 - timing_per_token_ms/gen:0.16818618298782753 - timing_per_token_ms/adv:0.003405757264681068 - perf/total_num_tokens:165676 - perf/time_per_step:40.86700990051031 - perf/throughput:506.7534926195175
(TaskRunner pid=2255397) Training Progress: 20%|██ | 20/100 [19:44<1:54:20, 85.76s/it]
(TaskRunner pid=2255397) step:21 - global_seqlen/min:17376 - global_seqlen/max:26208 - global_seqlen/minmax_diff:8832 - global_seqlen/balanced_min:22075 - global_seqlen/balanced_max:22088 - global_seqlen/mean:22085.25 - actor/entropy:0.4656538665294647 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4102373421192169 - training/rollout_probs_diff_mean:0.004885641857981682 - training/rollout_probs_diff_std:0.010877853259444237 - training/rollout_actor_probs_pearson_corr:0.9992596507072449 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.84765625 - self_distillation/correct_sample_fraction:0.43359375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.84765625 - rollout_corr/rollout_is_mean:0.9998842477798462 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.879465500882361e-05 - rollout_corr/rollout_is_max:1.9200012683868408 - rollout_corr/rollout_is_min:0.25445759296417236 - rollout_corr/rollout_is_std:0.03666176274418831 - rollout_corr/rollout_is_eff_sample_size:0.9986574816202841 - rollout_corr/rollout_is_seq_mean:0.9998617172241211 - rollout_corr/rollout_is_seq_std:0.0022321075666695833 - rollout_corr/rollout_is_seq_max:1.0074141025543213 - rollout_corr/rollout_is_seq_min:0.9920339584350586 - rollout_corr/rollout_is_seq_max_deviation:0.007966041564941406 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6367876464501023) - rollout_corr/training_log_ppl:np.float64(0.47835954203037545) - rollout_corr/kl:np.float64(0.0009155909243130367) - rollout_corr/k3_kl:np.float64(0.0008391680780164279) - rollout_corr/rollout_ppl:np.float64(1.6352251395583153) - rollout_corr/rollout_log_ppl:np.float64(0.4774439476314001) - rollout_corr/log_ppl_diff:np.float64(0.0009155943989753723) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018019898561760783) - rollout_corr/log_ppl_diff_max:np.float64(0.008913397789001465) - rollout_corr/log_ppl_diff_min:np.float64(-0.006456807255744934) - rollout_corr/ppl_ratio:np.float64(1.0009184170048684) - rollout_corr/chi2_token:np.float64(0.001522636041045189) - rollout_corr/chi2_seq:np.float64(0.3302574837580323) - actor/pg_loss:np.float64(0.00977602880448103) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008731294096833153) - actor/ppo_kl:np.float64(2.3401729779659597e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.84765625) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.84765625) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.43359375) - self_distillation/token_reweight_w_mean:np.float64(19834.05608875933) - self_distillation/token_reweight_w_std:np.float64(292799.855937861) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0034907914232463) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1998413290712051) - self_distillation/empty_target_batch:np.float64(0.15234375) - actor/grad_norm:np.float64(0.506427951157093) - perf/mfu/actor:np.float64(0.05293222627939937) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(195.2050437927246) - actor/lr:np.float64(1e-06) - training/global_step:21 - training/epoch:0 - critic/score/mean:0.43359375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.43359375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00826286617666483 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00826286617666483 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:406.9765625 - response_length/max:1132.0 - response_length/min:133.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:406.9765625 - response_length_non_aborted/max:1132.0 - response_length_non_aborted/min:133.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:283.1875 - prompt_length/max:669.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.725825369358063e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8232946544885635) - timing_s/agent_loop/generate_sequences/max:np.float64(9.148927487432957) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.626729334348056) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.148927487432957) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:234 - timing_s/agent_loop/slowest/response_length:1132 - timing_s/gen:15.357284164056182 - timing_s/reward:0.06359587237238884 - timing_s/old_log_prob:3.120852567255497 - timing_s/adv:0.6262220405042171 - timing_s/update_actor:21.799812253564596 - timing_s/step:41.07217570580542 - timing_s/stop_profile:7.509812712669373e-05 - timing_per_token_ms/update_actor:0.12338445485994383 - timing_per_token_ms/gen:0.14740257005793658 - timing_per_token_ms/adv:0.0035443454370236763 - perf/total_num_tokens:176682 - perf/time_per_step:41.07217570580542 - perf/throughput:537.7180444053837
(TaskRunner pid=2255397) Training Progress: 21%|██ | 21/100 [20:25<1:35:16, 72.35s/it]
(TaskRunner pid=2255397) step:22 - global_seqlen/min:15780 - global_seqlen/max:20714 - global_seqlen/minmax_diff:4934 - global_seqlen/balanced_min:18793 - global_seqlen/balanced_max:18799 - global_seqlen/mean:18797.25 - actor/entropy:0.4407426416873932 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23966830968856812 - training/rollout_probs_diff_mean:0.0051012844778597355 - training/rollout_probs_diff_std:0.011158647015690804 - training/rollout_actor_probs_pearson_corr:0.9991984963417053 - self_distillation/success_group_fraction:0.6875 - self_distillation/success_sample_fraction:0.6796875 - self_distillation/correct_sample_fraction:0.4375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.6796875 - rollout_corr/rollout_is_mean:0.9999550580978394 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.586863398551941 - rollout_corr/rollout_is_min:0.6212320327758789 - rollout_corr/rollout_is_std:0.03741012141108513 - rollout_corr/rollout_is_eff_sample_size:0.9986023199750391 - rollout_corr/rollout_is_seq_mean:0.9999154806137085 - rollout_corr/rollout_is_seq_std:0.0023052047472447157 - rollout_corr/rollout_is_seq_max:1.0075262784957886 - rollout_corr/rollout_is_seq_min:0.9921414256095886 - rollout_corr/rollout_is_seq_max_deviation:0.007858574390411377 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6121524260379374) - rollout_corr/training_log_ppl:np.float64(0.46336599756614305) - rollout_corr/kl:np.float64(0.0012106716192334943) - rollout_corr/k3_kl:np.float64(0.001044533465915265) - rollout_corr/rollout_ppl:np.float64(1.6101201642304659) - rollout_corr/rollout_log_ppl:np.float64(0.4621553277829662) - rollout_corr/log_ppl_diff:np.float64(0.0012106697831768543) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022202231630217284) - rollout_corr/log_ppl_diff_max:np.float64(0.011315524578094482) - rollout_corr/log_ppl_diff_min:np.float64(-0.007631242275238037) - rollout_corr/ppl_ratio:np.float64(1.0012149058748037) - rollout_corr/chi2_token:np.float64(0.001704103546217084) - rollout_corr/chi2_seq:np.float64(0.4757801443338394) - actor/pg_loss:np.float64(0.006979008438065648) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0011709280256582133) - actor/ppo_kl:np.float64(0.00036738665134805615) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.6796875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.6796875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.4375) - self_distillation/token_reweight_w_mean:np.float64(53386.58389568445) - self_distillation/token_reweight_w_std:np.float64(762098.1771182105) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0011509482283145) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1306488252885174) - self_distillation/empty_target_batch:np.float64(0.3203125) - actor/grad_norm:np.float64(0.4595351442694664) - perf/mfu/actor:np.float64(0.0457439323763552) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(195.23744583129883) - actor/lr:np.float64(1e-06) - training/global_step:22 - training/epoch:0 - critic/score/mean:0.4375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.4375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0024807543959468603 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0024807543959468603 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:331.8515625 - response_length/max:891.0 - response_length/min:134.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:331.8515625 - response_length_non_aborted/max:891.0 - response_length_non_aborted/min:134.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:255.5625 - prompt_length/max:592.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.7911500334739685e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6936665810644627) - timing_s/agent_loop/generate_sequences/max:np.float64(7.335243610665202) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.780559276121494) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.335243610665202) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:197 - timing_s/agent_loop/slowest/response_length:891 - timing_s/gen:14.717649631202221 - timing_s/reward:0.05497334711253643 - timing_s/old_log_prob:2.9507923033088446 - timing_s/adv:0.4809166081249714 - timing_s/update_actor:21.40921910852194 - timing_s/step:39.70510539226234 - timing_s/stop_profile:0.00011640414595603943 - timing_per_token_ms/update_actor:0.1423693566114853 - timing_per_token_ms/gen:0.1732425739953648 - timing_per_token_ms/adv:0.003198051630723719 - perf/total_num_tokens:150378 - perf/time_per_step:39.70510539226234 - perf/throughput:473.42148608584665
(TaskRunner pid=2255397) Training Progress: 22%|██▏ | 22/100 [21:05<1:21:20, 62.57s/it]
(TaskRunner pid=2255397) step:23 - global_seqlen/min:15655 - global_seqlen/max:22837 - global_seqlen/minmax_diff:7182 - global_seqlen/balanced_min:19794 - global_seqlen/balanced_max:19804 - global_seqlen/mean:19802.0 - actor/entropy:0.43892350792884827 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.22363799810409546 - training/rollout_probs_diff_mean:0.005129741504788399 - training/rollout_probs_diff_std:0.011249431408941746 - training/rollout_actor_probs_pearson_corr:0.9991777539253235 - self_distillation/success_group_fraction:0.9375 - self_distillation/success_sample_fraction:0.9296875 - self_distillation/correct_sample_fraction:0.69921875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.9296875 - rollout_corr/rollout_is_mean:0.9999939203262329 - rollout_corr/rollout_is_ratio_fraction_high:1.1452129911049269e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.1670188903808594 - rollout_corr/rollout_is_min:0.5688050389289856 - rollout_corr/rollout_is_std:0.03843645006418228 - rollout_corr/rollout_is_eff_sample_size:0.9985248186495871 - rollout_corr/rollout_is_seq_mean:0.9999755024909973 - rollout_corr/rollout_is_seq_std:0.0022403132170438766 - rollout_corr/rollout_is_seq_max:1.008310079574585 - rollout_corr/rollout_is_seq_min:0.9935483336448669 - rollout_corr/rollout_is_seq_max_deviation:0.008310079574584961 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.585592514835298) - rollout_corr/training_log_ppl:np.float64(0.45206161157693714) - rollout_corr/kl:np.float64(0.0010271025659691624) - rollout_corr/k3_kl:np.float64(0.0009754648472721783) - rollout_corr/rollout_ppl:np.float64(1.583907154854387) - rollout_corr/rollout_log_ppl:np.float64(0.45103450742317364) - rollout_corr/log_ppl_diff:np.float64(0.0010271041537635028) - rollout_corr/log_ppl_abs_diff:np.float64(0.0021158340969122946) - rollout_corr/log_ppl_diff_max:np.float64(0.008697479963302612) - rollout_corr/log_ppl_diff_min:np.float64(-0.007354050874710083) - rollout_corr/ppl_ratio:np.float64(1.0010308369528502) - rollout_corr/chi2_token:np.float64(0.0018195987213402987) - rollout_corr/chi2_seq:np.float64(1.1621388338971883) - actor/pg_loss:np.float64(0.007446382311172783) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000959160078309651) - actor/ppo_kl:np.float64(0.00021888582077522756) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.9296875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.9296875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.69921875) - self_distillation/token_reweight_w_mean:np.float64(6609.447738413466) - self_distillation/token_reweight_w_std:np.float64(122546.46291852725) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9963779090903699) - self_distillation/token_reweight_w_clip_frac:np.float64(0.17003195008146577) - self_distillation/empty_target_batch:np.float64(0.0703125) - actor/grad_norm:np.float64(0.43443264812231064) - perf/mfu/actor:np.float64(0.04804480134893495) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(195.1064910888672) - actor/lr:np.float64(1e-06) - training/global_step:23 - training/epoch:0 - critic/score/mean:0.69921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.69921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0033225491642951965 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0033225491642951965 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:341.09375 - response_length/max:846.0 - response_length/min:146.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:341.09375 - response_length_non_aborted/max:846.0 - response_length_non_aborted/min:146.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.71875 - prompt_length/max:766.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010677054524421692 - timing_s/agent_loop/generate_sequences/min:np.float64(1.892102275043726) - timing_s/agent_loop/generate_sequences/max:np.float64(7.085802031680942) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.9653804742338252) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.085802031680942) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:224 - timing_s/agent_loop/slowest/response_length:846 - timing_s/gen:13.813347652554512 - timing_s/reward:0.060129137709736824 - timing_s/old_log_prob:2.9367834143340588 - timing_s/adv:0.5313698258250952 - timing_s/update_actor:21.404147198423743 - timing_s/step:38.83369122073054 - timing_s/stop_profile:0.00010784715414047241 - timing_per_token_ms/update_actor:0.1351135440765058 - timing_per_token_ms/gen:0.15819225438106405 - timing_per_token_ms/adv:0.0033542686712522423 - perf/total_num_tokens:158416 - perf/time_per_step:38.83369122073054 - perf/throughput:509.91804738430636
(TaskRunner pid=2255397) Training Progress: 23%|██▎ | 23/100 [21:44<1:11:10, 55.46s/it]
(TaskRunner pid=2255397) step:24 - global_seqlen/min:18708 - global_seqlen/max:22807 - global_seqlen/minmax_diff:4099 - global_seqlen/balanced_min:20352 - global_seqlen/balanced_max:20355 - global_seqlen/mean:20353.75 - actor/entropy:0.48541581630706787 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.16695591807365417 - training/rollout_probs_diff_mean:0.005245619919151068 - training/rollout_probs_diff_std:0.01105139683932066 - training/rollout_actor_probs_pearson_corr:0.9992536306381226 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.82421875 - self_distillation/correct_sample_fraction:0.58984375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.82421875 - rollout_corr/rollout_is_mean:0.9997487664222717 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6379640102386475 - rollout_corr/rollout_is_min:0.55048668384552 - rollout_corr/rollout_is_std:0.038326967507600784 - rollout_corr/rollout_is_eff_sample_size:0.998532425609191 - rollout_corr/rollout_is_seq_mean:0.9996849894523621 - rollout_corr/rollout_is_seq_std:0.0021940250881016254 - rollout_corr/rollout_is_seq_max:1.0061367750167847 - rollout_corr/rollout_is_seq_min:0.9898834228515625 - rollout_corr/rollout_is_seq_max_deviation:0.0101165771484375 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6475044381804764) - rollout_corr/training_log_ppl:np.float64(0.4905120711773634) - rollout_corr/kl:np.float64(0.0011423102059779922) - rollout_corr/k3_kl:np.float64(0.0009413222521743592) - rollout_corr/rollout_ppl:np.float64(1.6456106547266245) - rollout_corr/rollout_log_ppl:np.float64(0.48936975869582966) - rollout_corr/log_ppl_diff:np.float64(0.001142312481533736) - rollout_corr/log_ppl_abs_diff:np.float64(0.002205758646596223) - rollout_corr/log_ppl_diff_max:np.float64(0.011655747890472412) - rollout_corr/log_ppl_diff_min:np.float64(-0.006167978048324585) - rollout_corr/ppl_ratio:np.float64(1.0011465374846011) - rollout_corr/chi2_token:np.float64(0.001514944015070796) - rollout_corr/chi2_seq:np.float64(0.9692259337753057) - actor/pg_loss:np.float64(0.006828168756328523) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008421741476922762) - actor/ppo_kl:np.float64(5.722692924337025e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.82421875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.82421875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.58984375) - self_distillation/token_reweight_w_mean:np.float64(16169.235121513018) - self_distillation/token_reweight_w_std:np.float64(204586.64157430374) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.000086387153715) - self_distillation/token_reweight_w_clip_frac:np.float64(0.14809982472797856) - self_distillation/empty_target_batch:np.float64(0.17578125) - actor/grad_norm:np.float64(0.38066764548420906) - perf/mfu/actor:np.float64(0.04935736118807353) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.85000133514404) - actor/lr:np.float64(1e-06) - training/global_step:24 - training/epoch:0 - critic/score/mean:0.58984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-8.236870780820027e-05 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-8.236870780820027e-05 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:355.6796875 - response_length/max:641.0 - response_length/min:162.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:355.6796875 - response_length_non_aborted/max:641.0 - response_length_non_aborted/min:162.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.375 - prompt_length/max:848.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011152960360050201 - timing_s/agent_loop/generate_sequences/min:np.float64(2.046153662726283) - timing_s/agent_loop/generate_sequences/max:np.float64(6.436841234564781) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.186256590895937) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.436841234564781) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:234 - timing_s/agent_loop/slowest/response_length:641 - timing_s/gen:13.05754235945642 - timing_s/reward:0.05888623557984829 - timing_s/old_log_prob:2.9728885516524315 - timing_s/adv:0.5305459387600422 - timing_s/update_actor:21.35632175952196 - timing_s/step:38.063012946397066 - timing_s/stop_profile:0.00010835938155651093 - timing_per_token_ms/update_actor:0.13115716857779255 - timing_per_token_ms/gen:0.14340437937329958 - timing_per_token_ms/adv:0.0032582812673342884 - perf/total_num_tokens:162830 - perf/time_per_step:38.063012946397066 - perf/throughput:534.7382780407726
(TaskRunner pid=2255397) Training Progress: 24%|██▍ | 24/100 [22:22<1:03:39, 50.25s/it]
(TaskRunner pid=2255397) step:25 - global_seqlen/min:16381 - global_seqlen/max:25188 - global_seqlen/minmax_diff:8807 - global_seqlen/balanced_min:21088 - global_seqlen/balanced_max:21097 - global_seqlen/mean:21093.125 - actor/entropy:0.4773712158203125 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.26223766803741455 - training/rollout_probs_diff_mean:0.005165540147572756 - training/rollout_probs_diff_std:0.01117149367928505 - training/rollout_actor_probs_pearson_corr:0.9992371797561646 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.86328125 - self_distillation/correct_sample_fraction:0.5546875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.86328125 - rollout_corr/rollout_is_mean:0.9999747276306152 - rollout_corr/rollout_is_ratio_fraction_high:1.0959985047520604e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.094183921813965 - rollout_corr/rollout_is_min:0.5062851309776306 - rollout_corr/rollout_is_std:0.038087502121925354 - rollout_corr/rollout_is_eff_sample_size:0.9985513246511216 - rollout_corr/rollout_is_seq_mean:1.0000536441802979 - rollout_corr/rollout_is_seq_std:0.0022864702623337507 - rollout_corr/rollout_is_seq_max:1.0067065954208374 - rollout_corr/rollout_is_seq_min:0.9937045574188232 - rollout_corr/rollout_is_seq_max_deviation:0.006706595420837402 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.669403042178601) - rollout_corr/training_log_ppl:np.float64(0.4993552331288811) - rollout_corr/kl:np.float64(0.0006479631488460758) - rollout_corr/k3_kl:np.float64(0.0008453664596572708) - rollout_corr/rollout_ppl:np.float64(1.668324178084731) - rollout_corr/rollout_log_ppl:np.float64(0.4987072692310903) - rollout_corr/log_ppl_diff:np.float64(0.0006479638977907598) - rollout_corr/log_ppl_abs_diff:np.float64(0.001830984663683921) - rollout_corr/log_ppl_diff_max:np.float64(0.008455544710159302) - rollout_corr/log_ppl_diff_min:np.float64(-0.007198035717010498) - rollout_corr/ppl_ratio:np.float64(1.0006508629303426) - rollout_corr/chi2_token:np.float64(0.002105830702930689) - rollout_corr/chi2_seq:np.float64(0.6466955123469234) - actor/pg_loss:np.float64(0.007964683580212295) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000689070585394802) - actor/ppo_kl:np.float64(-6.814886182837654e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.86328125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.86328125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5546875) - self_distillation/token_reweight_w_mean:np.float64(26990.30881792493) - self_distillation/token_reweight_w_std:np.float64(421488.5579388592) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.002008835086599) - self_distillation/token_reweight_w_clip_frac:np.float64(0.15447656423202716) - self_distillation/empty_target_batch:np.float64(0.13671875) - actor/grad_norm:np.float64(0.4783755913376808) - perf/mfu/actor:np.float64(0.0494590320538676) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.90256881713867) - actor/lr:np.float64(1e-06) - training/global_step:25 - training/epoch:0 - critic/score/mean:0.5546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.02120209112763405 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.02120209112763405 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:356.41015625 - response_length/max:956.0 - response_length/min:139.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:356.41015625 - response_length_non_aborted/max:956.0 - response_length_non_aborted/min:139.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.75 - prompt_length/max:646.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.000149555504322052 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6570597048848867) - timing_s/agent_loop/generate_sequences/max:np.float64(7.838871484622359) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.9465300647134427) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.838871484622359) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:202 - timing_s/agent_loop/slowest/response_length:956 - timing_s/gen:14.78430269099772 - timing_s/reward:0.06230742111802101 - timing_s/old_log_prob:3.0198799911886454 - timing_s/adv:0.5167359504848719 - timing_s/update_actor:22.208827931433916 - timing_s/step:40.68130003660917 - timing_s/stop_profile:0.0001026783138513565 - timing_per_token_ms/update_actor:0.13161176883127745 - timing_per_token_ms/gen:0.16203573712473252 - timing_per_token_ms/adv:0.003062229698568087 - perf/total_num_tokens:168745 - perf/time_per_step:40.68130003660917 - perf/throughput:518.496827314227
(TaskRunner pid=2255397) Training Progress: 25%|██▌ | 25/100 [23:03<59:14, 47.40s/it]
(TaskRunner pid=2255397) step:26 - global_seqlen/min:18171 - global_seqlen/max:23062 - global_seqlen/minmax_diff:4891 - global_seqlen/balanced_min:20385 - global_seqlen/balanced_max:20394 - global_seqlen/mean:20391.25 - actor/entropy:0.5068786144256592 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.17874199151992798 - training/rollout_probs_diff_mean:0.005472649820148945 - training/rollout_probs_diff_std:0.011403916403651237 - training/rollout_actor_probs_pearson_corr:0.9991965889930725 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.74609375 - self_distillation/correct_sample_fraction:0.6328125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.74609375 - rollout_corr/rollout_is_mean:1.0000735521316528 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.292263525305316e-05 - rollout_corr/rollout_is_max:1.8139365911483765 - rollout_corr/rollout_is_min:0.43659546971321106 - rollout_corr/rollout_is_std:0.039478376507759094 - rollout_corr/rollout_is_eff_sample_size:0.9984440018610561 - rollout_corr/rollout_is_seq_mean:1.0000500679016113 - rollout_corr/rollout_is_seq_std:0.0023560679983347654 - rollout_corr/rollout_is_seq_max:1.0076462030410767 - rollout_corr/rollout_is_seq_min:0.9908225536346436 - rollout_corr/rollout_is_seq_max_deviation:0.009177446365356445 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6881171250715852) - rollout_corr/training_log_ppl:np.float64(0.510973722091876) - rollout_corr/kl:np.float64(0.0007455622756609159) - rollout_corr/k3_kl:np.float64(0.0008864333188967066) - rollout_corr/rollout_ppl:np.float64(1.6867859680205584) - rollout_corr/rollout_log_ppl:np.float64(0.5102281617000699) - rollout_corr/log_ppl_diff:np.float64(0.000745560391806066) - rollout_corr/log_ppl_abs_diff:np.float64(0.002050930866971612) - rollout_corr/log_ppl_diff_max:np.float64(0.00910806655883789) - rollout_corr/log_ppl_diff_min:np.float64(-0.006127357482910156) - rollout_corr/ppl_ratio:np.float64(1.0007490692660213) - rollout_corr/chi2_token:np.float64(0.0020302317570894957) - rollout_corr/chi2_seq:np.float64(1.0315312510356307) - actor/pg_loss:np.float64(0.005001060431823134) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00048566531631877297) - actor/ppo_kl:np.float64(-2.1595971896104516e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.74609375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.74609375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6328125) - self_distillation/token_reweight_w_mean:np.float64(13624.06982974615) - self_distillation/token_reweight_w_std:np.float64(206433.80913565995) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9992130598984659) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09108548518270254) - self_distillation/empty_target_batch:np.float64(0.25390625) - actor/grad_norm:np.float64(0.3169581815600395) - perf/mfu/actor:np.float64(0.04962560549786612) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.88824081420898) - actor/lr:np.float64(1e-06) - training/global_step:26 - training/epoch:0 - critic/score/mean:0.6328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0027206302620470524 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0027206302620470524 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:340.8203125 - response_length/max:592.0 - response_length/min:156.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:340.8203125 - response_length_non_aborted/max:592.0 - response_length_non_aborted/min:156.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:296.40625 - prompt_length/max:709.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00016283243894577026 - timing_s/agent_loop/generate_sequences/min:np.float64(2.038805991411209) - timing_s/agent_loop/generate_sequences/max:np.float64(5.906553823500872) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.0381240394635824) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.906553823500872) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:331 - timing_s/agent_loop/slowest/response_length:592 - timing_s/gen:12.749464539811015 - timing_s/reward:0.061460068449378014 - timing_s/old_log_prob:3.0010161008685827 - timing_s/adv:0.5166078116744757 - timing_s/update_actor:21.353253232315183 - timing_s/step:37.770273169502616 - timing_s/stop_profile:0.00010509975254535675 - timing_per_token_ms/update_actor:0.13089715706684965 - timing_per_token_ms/gen:0.1461256680780632 - timing_per_token_ms/adv:0.0031668473712650994 - perf/total_num_tokens:163130 - perf/time_per_step:37.770273169502616 - perf/throughput:539.8756294001282
(TaskRunner pid=2255397) Training Progress: 26%|██▌ | 26/100 [23:40<54:55, 44.53s/it]
(TaskRunner pid=2255397) step:27 - global_seqlen/min:17480 - global_seqlen/max:23456 - global_seqlen/minmax_diff:5976 - global_seqlen/balanced_min:20067 - global_seqlen/balanced_max:20076 - global_seqlen/mean:20073.875 - actor/entropy:0.4675213694572449 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23105892539024353 - training/rollout_probs_diff_mean:0.005088143516331911 - training/rollout_probs_diff_std:0.010994603857398033 - training/rollout_actor_probs_pearson_corr:0.9992254972457886 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.85546875 - self_distillation/correct_sample_fraction:0.51171875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.85546875 - rollout_corr/rollout_is_mean:0.9997977018356323 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6167669296264648 - rollout_corr/rollout_is_min:0.5378821492195129 - rollout_corr/rollout_is_std:0.03790003061294556 - rollout_corr/rollout_is_eff_sample_size:0.9985651130827546 - rollout_corr/rollout_is_seq_mean:0.9997107982635498 - rollout_corr/rollout_is_seq_std:0.002149628708139062 - rollout_corr/rollout_is_seq_max:1.007951021194458 - rollout_corr/rollout_is_seq_min:0.9923518896102905 - rollout_corr/rollout_is_seq_max_deviation:0.007951021194458008 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6248419084586203) - rollout_corr/training_log_ppl:np.float64(0.47598731669131666) - rollout_corr/kl:np.float64(0.0010493483772009427) - rollout_corr/k3_kl:np.float64(0.0008248598063573809) - rollout_corr/rollout_ppl:np.float64(1.623152432963252) - rollout_corr/rollout_log_ppl:np.float64(0.47493796655908227) - rollout_corr/log_ppl_diff:np.float64(0.0010493501322343946) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018501168815419078) - rollout_corr/log_ppl_diff_max:np.float64(0.007712453603744507) - rollout_corr/log_ppl_diff_min:np.float64(-0.004121840000152588) - rollout_corr/ppl_ratio:np.float64(1.0010521123185754) - rollout_corr/chi2_token:np.float64(0.0011714110150933266) - rollout_corr/chi2_seq:np.float64(0.32961429841816425) - actor/pg_loss:np.float64(0.008322581415995955) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008014981349333539) - actor/ppo_kl:np.float64(4.5028834745330926e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.85546875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.85546875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.51171875) - self_distillation/token_reweight_w_mean:np.float64(30577.34729726822) - self_distillation/token_reweight_w_std:np.float64(521484.4462188269) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0020933467894793) - self_distillation/token_reweight_w_clip_frac:np.float64(0.16031731336261146) - self_distillation/empty_target_batch:np.float64(0.14453125) - actor/grad_norm:np.float64(0.42759493738412857) - perf/mfu/actor:np.float64(0.048367516590069666) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.80517578125) - actor/lr:np.float64(1e-06) - training/global_step:27 - training/epoch:0 - critic/score/mean:0.51171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.51171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0014940870460122824 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0014940870460122824 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:369.62109375 - response_length/max:943.0 - response_length/min:156.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:369.62109375 - response_length_non_aborted/max:943.0 - response_length_non_aborted/min:156.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:257.6875 - prompt_length/max:380.0 - prompt_length/min:160.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013620592653751373 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9484890308231115) - timing_s/agent_loop/generate_sequences/max:np.float64(7.838706968352199) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.320610681475955) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.838706968352199) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:213 - timing_s/agent_loop/slowest/response_length:943 - timing_s/gen:14.69153231009841 - timing_s/reward:0.06106424517929554 - timing_s/old_log_prob:3.0161088667809963 - timing_s/adv:0.5223662443459034 - timing_s/update_actor:21.658704286441207 - timing_s/step:40.0367049369961 - timing_s/stop_profile:0.00010894984006881714 - timing_per_token_ms/update_actor:0.13486873041727873 - timing_per_token_ms/gen:0.15526386090166672 - timing_per_token_ms/adv:0.003252774092856408 - perf/total_num_tokens:160591 - perf/time_per_step:40.0367049369961 - perf/throughput:501.38679073588406
(TaskRunner pid=2255397) Training Progress: 27%|██▋ | 27/100 [24:21<52:33, 43.20s/it]
(TaskRunner pid=2255397) step:28 - global_seqlen/min:18201 - global_seqlen/max:23560 - global_seqlen/minmax_diff:5359 - global_seqlen/balanced_min:20410 - global_seqlen/balanced_max:20497 - global_seqlen/mean:20425.125 - actor/entropy:0.4737909138202667 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2294958233833313 - training/rollout_probs_diff_mean:0.0052576144225895405 - training/rollout_probs_diff_std:0.011254720389842987 - training/rollout_actor_probs_pearson_corr:0.999194860458374 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83984375 - self_distillation/correct_sample_fraction:0.59765625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83984375 - rollout_corr/rollout_is_mean:1.0001050233840942 - rollout_corr/rollout_is_ratio_fraction_high:1.0676346391846891e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:3.8465442657470703 - rollout_corr/rollout_is_min:0.6126700639724731 - rollout_corr/rollout_is_std:0.03862826153635979 - rollout_corr/rollout_is_eff_sample_size:0.9985103182038877 - rollout_corr/rollout_is_seq_mean:1.0001041889190674 - rollout_corr/rollout_is_seq_std:0.002421688986942172 - rollout_corr/rollout_is_seq_max:1.00888991355896 - rollout_corr/rollout_is_seq_min:0.9913090467453003 - rollout_corr/rollout_is_seq_max_deviation:0.008889913558959961 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6370238508097827) - rollout_corr/training_log_ppl:np.float64(0.4820473712461535) - rollout_corr/kl:np.float64(0.0006596640439546775) - rollout_corr/k3_kl:np.float64(0.000909588395757055) - rollout_corr/rollout_ppl:np.float64(1.6358301592990756) - rollout_corr/rollout_log_ppl:np.float64(0.48138770982041024) - rollout_corr/log_ppl_diff:np.float64(0.000659661425743252) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018744217813946307) - rollout_corr/log_ppl_diff_max:np.float64(0.007977485656738281) - rollout_corr/log_ppl_diff_min:np.float64(-0.0074624717235565186) - rollout_corr/ppl_ratio:np.float64(1.0006625556852669) - rollout_corr/chi2_token:np.float64(0.0023598361294716597) - rollout_corr/chi2_seq:np.float64(0.9705478034447879) - actor/pg_loss:np.float64(0.006433643866330385) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006121892383816885) - actor/ppo_kl:np.float64(-2.0443672603676077e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83984375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83984375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.59765625) - self_distillation/token_reweight_w_mean:np.float64(22090.04282637965) - self_distillation/token_reweight_w_std:np.float64(363460.48439942) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0007038170006126) - self_distillation/token_reweight_w_clip_frac:np.float64(0.12272839152137749) - self_distillation/empty_target_batch:np.float64(0.16015625) - actor/grad_norm:np.float64(0.4047693833708763) - perf/mfu/actor:np.float64(0.0494901405269842) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.81372833251953) - actor/lr:np.float64(1e-06) - training/global_step:28 - training/epoch:0 - critic/score/mean:0.59765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004289222415536642 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004289222415536642 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:365.87890625 - response_length/max:1022.0 - response_length/min:163.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:365.87890625 - response_length_non_aborted/max:1022.0 - response_length_non_aborted/min:163.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:272.40625 - prompt_length/max:443.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001571252942085266 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0796666648238897) - timing_s/agent_loop/generate_sequences/max:np.float64(8.435203347355127) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.275154207913147) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.435203347355127) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:330 - timing_s/agent_loop/slowest/response_length:1022 - timing_s/gen:15.208884691819549 - timing_s/reward:0.060688696801662445 - timing_s/old_log_prob:3.0850410480052233 - timing_s/adv:0.5393318682909012 - timing_s/update_actor:21.212657935917377 - timing_s/step:40.19403674826026 - timing_s/stop_profile:0.00011958740651607513 - timing_per_token_ms/update_actor:0.1298196335145891 - timing_per_token_ms/gen:0.1623753236728719 - timing_per_token_ms/adv:0.003300664428558584 - perf/total_num_tokens:163401 - perf/time_per_step:40.19403674826026 - perf/throughput:508.1630672710193
(TaskRunner pid=2255397) Training Progress: 28%|██▊ | 28/100 [25:01<50:46, 42.31s/it]
(TaskRunner pid=2255397) step:29 - global_seqlen/min:17728 - global_seqlen/max:24689 - global_seqlen/minmax_diff:6961 - global_seqlen/balanced_min:21875 - global_seqlen/balanced_max:21882 - global_seqlen/mean:21878.875 - actor/entropy:0.45051109790802 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.25539880990982056 - training/rollout_probs_diff_mean:0.004729867912828922 - training/rollout_probs_diff_std:0.010498874820768833 - training/rollout_actor_probs_pearson_corr:0.9993234872817993 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.734375 - self_distillation/correct_sample_fraction:0.48046875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.734375 - rollout_corr/rollout_is_mean:0.9999346733093262 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.9196990251657553e-05 - rollout_corr/rollout_is_max:1.6618187427520752 - rollout_corr/rollout_is_min:0.48319488763809204 - rollout_corr/rollout_is_std:0.03575139120221138 - rollout_corr/rollout_is_eff_sample_size:0.998723350704955 - rollout_corr/rollout_is_seq_mean:0.9999228119850159 - rollout_corr/rollout_is_seq_std:0.00180347659625113 - rollout_corr/rollout_is_seq_max:1.0053354501724243 - rollout_corr/rollout_is_seq_min:0.9929618835449219 - rollout_corr/rollout_is_seq_max_deviation:0.007038116455078125 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5977162146009505) - rollout_corr/training_log_ppl:np.float64(0.4593473710701801) - rollout_corr/kl:np.float64(0.0006978295002575408) - rollout_corr/k3_kl:np.float64(0.0007481996697720206) - rollout_corr/rollout_ppl:np.float64(1.5965832527726889) - rollout_corr/rollout_log_ppl:np.float64(0.45864954427815974) - rollout_corr/log_ppl_diff:np.float64(0.0006978267920203507) - rollout_corr/log_ppl_abs_diff:np.float64(0.001622283656615764) - rollout_corr/log_ppl_diff_max:np.float64(0.006626904010772705) - rollout_corr/log_ppl_diff_min:np.float64(-0.00486445426940918) - rollout_corr/ppl_ratio:np.float64(1.0007000237237662) - rollout_corr/chi2_token:np.float64(0.001576458802446723) - rollout_corr/chi2_seq:np.float64(0.6520844814367592) - actor/pg_loss:np.float64(0.0060147366020828485) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005360728382584057) - actor/ppo_kl:np.float64(-6.641209190716424e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.734375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.734375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.48046875) - self_distillation/token_reweight_w_mean:np.float64(26070.862267469056) - self_distillation/token_reweight_w_std:np.float64(478879.7805042105) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0025979080237448) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1094071113911923) - self_distillation/empty_target_batch:np.float64(0.265625) - actor/grad_norm:np.float64(0.3129398711025715) - perf/mfu/actor:np.float64(0.052564897191348076) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.9274024963379) - actor/lr:np.float64(1e-06) - training/global_step:29 - training/epoch:0 - critic/score/mean:0.48046875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.48046875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.01068192534148693 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.01068192534148693 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:406.96484375 - response_length/max:856.0 - response_length/min:160.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:406.96484375 - response_length_non_aborted/max:856.0 - response_length_non_aborted/min:160.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:276.75 - prompt_length/max:492.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011670216917991638 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0912411771714687) - timing_s/agent_loop/generate_sequences/max:np.float64(7.972104227170348) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.765111505068489) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.972104227170348) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:299 - timing_s/agent_loop/slowest/response_length:856 - timing_s/gen:14.771529672667384 - timing_s/reward:0.06256661377847195 - timing_s/old_log_prob:3.17423870973289 - timing_s/adv:0.5342923663556576 - timing_s/update_actor:21.539450408890843 - timing_s/step:40.168734323233366 - timing_s/stop_profile:0.00010610558092594147 - timing_per_token_ms/update_actor:0.12306077442790617 - timing_per_token_ms/gen:0.1417844530553678 - timing_per_token_ms/adv:0.003052558497384221 - perf/total_num_tokens:175031 - perf/time_per_step:40.168734323233366 - perf/throughput:544.6742440013945
(TaskRunner pid=2255397) Training Progress: 29%|██▉ | 29/100 [25:41<49:19, 41.68s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 30}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1", we need to analyze the structure of the product and identify the key functional groups and substituents that must be present in the reactant.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product has the following key features:
(TaskRunner pid=2255397) - A nitrogen-containing ring (likely a pyrrole or similar heterocycle).
(TaskRunner pid=2255397) - A benzene ring with a chlorine substituent.
(TaskRunner pid=2255397) - A trifluoromethyl group (C(F)(F)F) attached to the benzene ring.
(TaskRunner pid=2255397) - A substituent group (likely a nitrogen-containing chain) that connects to the ring.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Now, let's examine each option:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option A:**
(TaskRunner pid=2255397) O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) This option contains the trifluoromethyl group (C(F)(F)F) and the chlorinated benzene ring. It also contains a nitrogen-containing chain and a ring structure similar to the product. This option closely matches the structure of the product and is a strong candidate.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option B:**
(TaskRunner pid=2255397) O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) This option has a chlorine substituent on the ring, but it lacks the trifluoromethyl group and the correct nitrogen-containing chain. It also has a different ring structure than the product. Therefore, it is not the correct reactant.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option C:**
(TaskRunner pid=2255397) O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) This option contains the trifluoromethyl group and the chlorinated benzene ring. However, the structure is more complex and includes an oxygen-containing side chain that is not present in the product. It introduces an unnecessary oxygen group, making it incompatible with the product structure.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option D:**
(TaskRunner pid=2255397) O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) This option contains the trifluoromethyl group and a benzene ring. However, it lacks the correct nitrogen-containing chain and the chlorinated benzene ring. The structure is not aligned with the product, so it is not the correct reactant.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Conclusion:**
(TaskRunner pid=2255397) Only **Option A** contains all the necessary functional groups and substituents present in the product, making it the only correct reactant for the synthesis of the given product.
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_30
(TaskRunner pid=2255397) step:30 - global_seqlen/min:18031 - global_seqlen/max:24561 - global_seqlen/minmax_diff:6530 - global_seqlen/balanced_min:20880 - global_seqlen/balanced_max:20888 - global_seqlen/mean:20884.625 - actor/entropy:0.4658919870853424 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3171748220920563 - training/rollout_probs_diff_mean:0.005262547638267279 - training/rollout_probs_diff_std:0.01141402218490839 - training/rollout_actor_probs_pearson_corr:0.9991620779037476 - self_distillation/success_group_fraction:0.6875 - self_distillation/success_sample_fraction:0.66796875 - self_distillation/correct_sample_fraction:0.44921875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.66796875 - rollout_corr/rollout_is_mean:0.9999656677246094 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.412619844311848e-05 - rollout_corr/rollout_is_max:1.9128998517990112 - rollout_corr/rollout_is_min:0.2724512219429016 - rollout_corr/rollout_is_std:0.03873920068144798 - rollout_corr/rollout_is_eff_sample_size:0.9985014042044956 - rollout_corr/rollout_is_seq_mean:1.0000505447387695 - rollout_corr/rollout_is_seq_std:0.002504864940419793 - rollout_corr/rollout_is_seq_max:1.0067648887634277 - rollout_corr/rollout_is_seq_min:0.9914875626564026 - rollout_corr/rollout_is_seq_max_deviation:0.008512437343597412 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6606428353115916) - rollout_corr/training_log_ppl:np.float64(0.49546970275696367) - rollout_corr/kl:np.float64(0.0007864780202773147) - rollout_corr/k3_kl:np.float64(0.000911510776063551) - rollout_corr/rollout_ppl:np.float64(1.659348614513874) - rollout_corr/rollout_log_ppl:np.float64(0.4946832267160062) - rollout_corr/log_ppl_diff:np.float64(0.0007864760409574956) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019896051089745015) - rollout_corr/log_ppl_diff_max:np.float64(0.009873658418655396) - rollout_corr/log_ppl_diff_min:np.float64(-0.007493495941162109) - rollout_corr/ppl_ratio:np.float64(1.0007900123018771) - rollout_corr/chi2_token:np.float64(0.002100849524140358) - rollout_corr/chi2_seq:np.float64(0.6980828407686204) - actor/pg_loss:np.float64(0.0044937689090147614) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004526598731899867) - actor/ppo_kl:np.float64(2.1002829232230624e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.66796875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.66796875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.44921875) - self_distillation/token_reweight_w_mean:np.float64(16658.25212058681) - self_distillation/token_reweight_w_std:np.float64(262596.2304097906) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.002752567641437) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09863085992401466) - self_distillation/empty_target_batch:np.float64(0.33203125) - actor/grad_norm:np.float64(0.3747030645608902) - perf/mfu/actor:np.float64(0.050935440136856504) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(194.77979278564453) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.606845238095238) - val-aux/sciknoweval/reward/std@16:np.float64(0.2138942013523591) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.6934285714285715) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.16904326809342193) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.521195238095238) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.1926339687939083) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6076809523809524) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.21426439691535565) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7574380952380952) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.11875313696867805) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.4448476190476191) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.15604155437402734) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6224000000000001) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.16648090775602947) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8030238095238095) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.07678824143757296) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.38081904761904767) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.11615050215883854) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6278476190476191) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.11644569510751888) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8329428571428572) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.04605969898285256) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.3325333333333333) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.07403573137391646) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6297142857142858) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.079464901369361) - val-aux/sciknoweval/score/mean@16:np.float64(0.606845238095238) - val-aux/sciknoweval/score/std@16:np.float64(0.2138942013523591) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.6934285714285715) - val-aux/sciknoweval/score/best@2/std:np.float64(0.16904326809342193) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.521195238095238) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.1926339687939083) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6076809523809524) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.21426439691535565) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7574380952380952) - val-aux/sciknoweval/score/best@4/std:np.float64(0.11875313696867805) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.4448476190476191) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.15604155437402734) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6224000000000001) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.16648090775602947) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8030238095238095) - val-aux/sciknoweval/score/best@8/std:np.float64(0.07678824143757296) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.38081904761904767) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.11615050215883854) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6278476190476191) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.11644569510751888) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8329428571428572) - val-aux/sciknoweval/score/best@16/std:np.float64(0.04605969898285256) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.3325333333333333) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.07403573137391646) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6297142857142858) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.079464901369361) - val-core/sciknoweval/acc/mean@16:np.float64(0.606845238095238) - val-aux/sciknoweval/acc/std@16:np.float64(0.2138942013523591) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.6934285714285715) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.16904326809342193) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.521195238095238) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.1926339687939083) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6076809523809524) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.21426439691535565) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7574380952380952) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.11875313696867805) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.4448476190476191) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.15604155437402734) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6224000000000001) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.16648090775602947) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8030238095238095) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.07678824143757296) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.38081904761904767) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.11615050215883854) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6278476190476191) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.11644569510751888) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8329428571428572) - val-core/sciknoweval/acc/best@16/std:np.float64(0.04605969898285256) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.3325333333333333) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.07403573137391646) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6297142857142858) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.079464901369361) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:30 - training/epoch:0 - critic/score/mean:0.44921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.44921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0019352398812770844 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:0.0019352398812770844 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:343.39453125 - response_length/max:941.0 - response_length/min:138.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:343.39453125 - response_length_non_aborted/max:941.0 - response_length_non_aborted/min:138.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:309.25 - prompt_length/max:516.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015097111463546753 - timing_s/agent_loop/generate_sequences/min:np.float64(1.869465732946992) - timing_s/agent_loop/generate_sequences/max:np.float64(7.8817349802702665) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.971169372940494) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.8817349802702665) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:235 - timing_s/agent_loop/slowest/response_length:941 - timing_s/gen:15.309523962438107 - timing_s/reward:0.06142178550362587 - timing_s/old_log_prob:3.0037817489355803 - timing_s/adv:0.5514516998082399 - timing_s/update_actor:21.344387443736196 - timing_s/step:40.35734134726226 - timing_s/testing:78.26906821876764 - timing_s/save_checkpoint:13.220651166513562 - timing_s/stop_profile:0.00014671683311462402 - timing_per_token_ms/update_actor:0.12775179973147827 - timing_per_token_ms/gen:0.17415195216005308 - timing_per_token_ms/adv:0.0033005841606459294 - perf/total_num_tokens:167077 - perf/time_per_step:40.35734134726226 - perf/throughput:517.4925875392621
(TaskRunner pid=2255397) Training Progress: 30%|███ | 30/100 [27:53<1:20:12, 68.76s/it]
(TaskRunner pid=2255397) step:31 - global_seqlen/min:19249 - global_seqlen/max:25029 - global_seqlen/minmax_diff:5780 - global_seqlen/balanced_min:21607 - global_seqlen/balanced_max:21619 - global_seqlen/mean:21615.375 - actor/entropy:0.4569137990474701 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.303112655878067 - training/rollout_probs_diff_mean:0.004967319313436747 - training/rollout_probs_diff_std:0.010901066474616528 - training/rollout_actor_probs_pearson_corr:0.9992502331733704 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83203125 - self_distillation/correct_sample_fraction:0.6015625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83203125 - rollout_corr/rollout_is_mean:1.000017762184143 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.816590428352356 - rollout_corr/rollout_is_min:0.5354147553443909 - rollout_corr/rollout_is_std:0.03712382912635803 - rollout_corr/rollout_is_eff_sample_size:0.9986238370470686 - rollout_corr/rollout_is_seq_mean:1.0000417232513428 - rollout_corr/rollout_is_seq_std:0.002106655156239867 - rollout_corr/rollout_is_seq_max:1.0081474781036377 - rollout_corr/rollout_is_seq_min:0.9948439002037048 - rollout_corr/rollout_is_seq_max_deviation:0.008147478103637695 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6235803924500942) - rollout_corr/training_log_ppl:np.float64(0.4749105938244611) - rollout_corr/kl:np.float64(0.0008353461136003659) - rollout_corr/k3_kl:np.float64(0.00080985213742224) - rollout_corr/rollout_ppl:np.float64(1.6222249972634017) - rollout_corr/rollout_log_ppl:np.float64(0.4740752446523402) - rollout_corr/log_ppl_diff:np.float64(0.0008353491721209139) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017292899719905108) - rollout_corr/log_ppl_diff_max:np.float64(0.006912916898727417) - rollout_corr/log_ppl_diff_min:np.float64(-0.0048788487911224365) - rollout_corr/ppl_ratio:np.float64(1.0008378000929952) - rollout_corr/chi2_token:np.float64(0.0016067922115325928) - rollout_corr/chi2_seq:np.float64(0.6984529327601194) - actor/pg_loss:np.float64(0.005961017799563706) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00028992991656195954) - actor/ppo_kl:np.float64(0.0001396356336407223) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83203125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83203125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6015625) - self_distillation/token_reweight_w_mean:np.float64(9105.852561026346) - self_distillation/token_reweight_w_std:np.float64(160757.80174179428) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9988879463635385) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1225835490913596) - self_distillation/empty_target_batch:np.float64(0.16796875) - actor/grad_norm:np.float64(0.33824220299720764) - perf/mfu/actor:np.float64(0.05203885094538403) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(196.26508331298828) - actor/lr:np.float64(1e-06) - training/global_step:31 - training/epoch:0 - critic/score/mean:0.6015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.001895678462460637 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.001895678462460637 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:395.63671875 - response_length/max:1088.0 - response_length/min:120.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:395.63671875 - response_length_non_aborted/max:1088.0 - response_length_non_aborted/min:120.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.84375 - prompt_length/max:635.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.687268614768982e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6187728364020586) - timing_s/agent_loop/generate_sequences/max:np.float64(9.020029896870255) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.584768897177128) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.020029896870255) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:194 - timing_s/agent_loop/slowest/response_length:1088 - timing_s/gen:15.492790538817644 - timing_s/reward:0.05757281742990017 - timing_s/old_log_prob:3.137273196130991 - timing_s/adv:0.5795937422662973 - timing_s/update_actor:21.805233089253306 - timing_s/step:41.175627982243896 - timing_s/stop_profile:0.00012617185711860657 - timing_per_token_ms/update_actor:0.12609793427857086 - timing_per_token_ms/gen:0.15296535982166448 - timing_per_token_ms/adv:0.00335174466245842 - perf/total_num_tokens:172923 - perf/time_per_step:41.175627982243896 - perf/throughput:524.9555637456499
(TaskRunner pid=2255397) Training Progress: 31%|███ | 31/100 [28:34<1:09:34, 60.50s/it]
(TaskRunner pid=2255397) step:32 - global_seqlen/min:17172 - global_seqlen/max:25152 - global_seqlen/minmax_diff:7980 - global_seqlen/balanced_min:20878 - global_seqlen/balanced_max:20886 - global_seqlen/mean:20883.0 - actor/entropy:0.4830181300640106 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3013932406902313 - training/rollout_probs_diff_mean:0.00495893694460392 - training/rollout_probs_diff_std:0.010702594183385372 - training/rollout_actor_probs_pearson_corr:0.9992987513542175 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.8125 - self_distillation/correct_sample_fraction:0.55078125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8125 - rollout_corr/rollout_is_mean:1.0001959800720215 - rollout_corr/rollout_is_ratio_fraction_high:1.0022450624092016e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.0067350962781347e-05 - rollout_corr/rollout_is_max:2.244924545288086 - rollout_corr/rollout_is_min:0.39147886633872986 - rollout_corr/rollout_is_std:0.037390995770692825 - rollout_corr/rollout_is_eff_sample_size:0.998604459752836 - rollout_corr/rollout_is_seq_mean:1.0002524852752686 - rollout_corr/rollout_is_seq_std:0.0021919296123087406 - rollout_corr/rollout_is_seq_max:1.0080853700637817 - rollout_corr/rollout_is_seq_min:0.9943554401397705 - rollout_corr/rollout_is_seq_max_deviation:0.008085370063781738 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6529409568756819) - rollout_corr/training_log_ppl:np.float64(0.4933903770870529) - rollout_corr/kl:np.float64(0.0006465979992356097) - rollout_corr/k3_kl:np.float64(0.0008415120931317688) - rollout_corr/rollout_ppl:np.float64(1.651906920131296) - rollout_corr/rollout_log_ppl:np.float64(0.49274377603433095) - rollout_corr/log_ppl_diff:np.float64(0.0006466010527219623) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017735535220708698) - rollout_corr/log_ppl_diff_max:np.float64(0.007363229990005493) - rollout_corr/log_ppl_diff_min:np.float64(-0.010225236415863037) - rollout_corr/ppl_ratio:np.float64(1.000649468274787) - rollout_corr/chi2_token:np.float64(0.0021671864669770002) - rollout_corr/chi2_seq:np.float64(1.2125774417072535) - actor/pg_loss:np.float64(0.007539922837167978) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005248926790955011) - actor/ppo_kl:np.float64(0.00015378757747441796) - actor/pg_clipfrac_lower:np.float64(1.2849506674683653e-05) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.55078125) - self_distillation/token_reweight_w_mean:np.float64(28962.190404822817) - self_distillation/token_reweight_w_std:np.float64(594865.4367179495) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9994088341481984) - self_distillation/token_reweight_w_clip_frac:np.float64(0.15152001177193597) - self_distillation/empty_target_batch:np.float64(0.1875) - actor/grad_norm:np.float64(0.4039623737335205) - perf/mfu/actor:np.float64(0.050354520500870484) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(195.79799795150757) - actor/lr:np.float64(1e-06) - training/global_step:32 - training/epoch:0 - critic/score/mean:0.55078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005901970434933901 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:0.005901970434933901 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:389.75 - response_length/max:1026.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:389.75 - response_length_non_aborted/max:1026.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.84375 - prompt_length/max:404.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001097116619348526 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7604525182396173) - timing_s/agent_loop/generate_sequences/max:np.float64(8.474037919193506) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.519642143648525) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.474037919193506) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:286 - timing_s/agent_loop/slowest/response_length:1026 - timing_s/gen:15.50579597428441 - timing_s/reward:0.05374091491103172 - timing_s/old_log_prob:3.0371324717998505 - timing_s/adv:0.5603868458420038 - timing_s/update_actor:21.465026825666428 - timing_s/step:40.65800975076854 - timing_s/stop_profile:0.0001172088086605072 - timing_per_token_ms/update_actor:0.12848385544262336 - timing_per_token_ms/gen:0.1554060693381616 - timing_per_token_ms/adv:0.0033543243657640416 - perf/total_num_tokens:167064 - perf/time_per_step:40.65800975076854 - perf/throughput:513.6257315105114
(TaskRunner pid=2255397) Training Progress: 32%|███▏ | 32/100 [29:15<1:01:50, 54.56s/it]
(TaskRunner pid=2255397) step:33 - global_seqlen/min:16273 - global_seqlen/max:24765 - global_seqlen/minmax_diff:8492 - global_seqlen/balanced_min:20358 - global_seqlen/balanced_max:20368 - global_seqlen/mean:20365.75 - actor/entropy:0.5308820605278015 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2607361674308777 - training/rollout_probs_diff_mean:0.005311037413775921 - training/rollout_probs_diff_std:0.010893904604017735 - training/rollout_actor_probs_pearson_corr:0.9993165731430054 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.80078125 - self_distillation/correct_sample_fraction:0.60546875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.80078125 - rollout_corr/rollout_is_mean:1.0000280141830444 - rollout_corr/rollout_is_ratio_fraction_high:2.2798258214606903e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.1399129107303452e-05 - rollout_corr/rollout_is_max:2.4295754432678223 - rollout_corr/rollout_is_min:0.4068605899810791 - rollout_corr/rollout_is_std:0.038521647453308105 - rollout_corr/rollout_is_eff_sample_size:0.9985184003675729 - rollout_corr/rollout_is_seq_mean:1.0000193119049072 - rollout_corr/rollout_is_seq_std:0.0023478458169847727 - rollout_corr/rollout_is_seq_max:1.0072718858718872 - rollout_corr/rollout_is_seq_min:0.9926888346672058 - rollout_corr/rollout_is_seq_max_deviation:0.0073111653327941895 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7451868052594364) - rollout_corr/training_log_ppl:np.float64(0.5472010709927417) - rollout_corr/kl:np.float64(0.000750623744014689) - rollout_corr/k3_kl:np.float64(0.0009170059764755933) - rollout_corr/rollout_ppl:np.float64(1.7438560076989233) - rollout_corr/rollout_log_ppl:np.float64(0.5464504445553757) - rollout_corr/log_ppl_diff:np.float64(0.0007506264373660088) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020804237574338913) - rollout_corr/log_ppl_diff_max:np.float64(0.011185050010681152) - rollout_corr/log_ppl_diff_min:np.float64(-0.008495688438415527) - rollout_corr/ppl_ratio:np.float64(1.0007542204111814) - rollout_corr/chi2_token:np.float64(0.0021686023101210594) - rollout_corr/chi2_seq:np.float64(1.0599001871887594) - actor/pg_loss:np.float64(0.0046818595146760345) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000626342507530353) - actor/ppo_kl:np.float64(-7.711236621688045e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.80078125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.80078125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.60546875) - self_distillation/token_reweight_w_mean:np.float64(406.9383929811884) - self_distillation/token_reweight_w_std:np.float64(6888.8986393570085) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0006252727471292) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10766920002060942) - self_distillation/empty_target_batch:np.float64(0.19921875) - actor/grad_norm:np.float64(0.35059650987386703) - perf/mfu/actor:np.float64(0.04906215016751242) - perf/max_memory_allocated_gb:np.float64(130.98335647583008) - perf/max_memory_reserved_gb:np.float64(140.86328125) - perf/cpu_memory_used_gb:np.float64(195.48283004760742) - actor/lr:np.float64(1e-06) - training/global_step:33 - training/epoch:0 - critic/score/mean:0.60546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.60546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0038529057055711746 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0038529057055711746 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:342.6796875 - response_length/max:820.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:342.6796875 - response_length_non_aborted/max:820.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:293.75 - prompt_length/max:474.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011102668941020966 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7211053427308798) - timing_s/agent_loop/generate_sequences/max:np.float64(6.794723700731993) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.86705601822905) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.794723700731993) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:378 - timing_s/agent_loop/slowest/response_length:820 - timing_s/gen:13.564623160287738 - timing_s/reward:0.06091204658150673 - timing_s/old_log_prob:2.9387052599340677 - timing_s/adv:0.5545471422374249 - timing_s/update_actor:21.675176206976175 - timing_s/step:38.87970031797886 - timing_s/stop_profile:3.417767584323883e-05 - timing_per_token_ms/update_actor:0.1330369382847193 - timing_per_token_ms/gen:0.15462489068563182 - timing_per_token_ms/adv:0.0034036749336350544 - perf/total_num_tokens:162926 - perf/time_per_step:38.87970031797886 - perf/throughput:523.8144798812251
(TaskRunner pid=2255397) Training Progress: 33%|███▎ | 33/100 [29:54<55:40, 49.86s/it]
(TaskRunner pid=2255397) step:34 - global_seqlen/min:19407 - global_seqlen/max:28954 - global_seqlen/minmax_diff:9547 - global_seqlen/balanced_min:20764 - global_seqlen/balanced_max:28104 - global_seqlen/mean:21689.625 - actor/entropy:0.4711659848690033 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.26104798913002014 - training/rollout_probs_diff_mean:0.004729302600026131 - training/rollout_probs_diff_std:0.010431492701172829 - training/rollout_actor_probs_pearson_corr:0.9993188977241516 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.85546875 - self_distillation/correct_sample_fraction:0.59765625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.85546875 - rollout_corr/rollout_is_mean:0.9998348355293274 - rollout_corr/rollout_is_ratio_fraction_high:1.0172008842346258e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.372540235519409 - rollout_corr/rollout_is_min:0.5585512518882751 - rollout_corr/rollout_is_std:0.03565623611211777 - rollout_corr/rollout_is_eff_sample_size:0.9987297716220098 - rollout_corr/rollout_is_seq_mean:0.9998084902763367 - rollout_corr/rollout_is_seq_std:0.0021642311476171017 - rollout_corr/rollout_is_seq_max:1.0065914392471313 - rollout_corr/rollout_is_seq_min:0.9937243461608887 - rollout_corr/rollout_is_seq_max_deviation:0.006591439247131348 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.732406364288181) - rollout_corr/training_log_ppl:np.float64(0.53354305683024) - rollout_corr/kl:np.float64(0.0010742717632254706) - rollout_corr/k3_kl:np.float64(0.0007844316083662761) - rollout_corr/rollout_ppl:np.float64(1.7304212134331465) - rollout_corr/rollout_log_ppl:np.float64(0.5324687818174425) - rollout_corr/log_ppl_diff:np.float64(0.0010742750127974432) - rollout_corr/log_ppl_abs_diff:np.float64(0.002004527148528723) - rollout_corr/log_ppl_diff_max:np.float64(0.010212719440460205) - rollout_corr/log_ppl_diff_min:np.float64(-0.0042626261711120605) - rollout_corr/ppl_ratio:np.float64(1.0010776109993458) - rollout_corr/chi2_token:np.float64(0.000989705789834261) - rollout_corr/chi2_seq:np.float64(0.5540260763373226) - actor/pg_loss:np.float64(0.005981583381071687) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004929028409605962) - actor/ppo_kl:np.float64(0.00015171869483765477) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.85546875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.85546875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.59765625) - self_distillation/token_reweight_w_mean:np.float64(11402.142899290891) - self_distillation/token_reweight_w_std:np.float64(223975.93910462921) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0007735511753708) - self_distillation/token_reweight_w_clip_frac:np.float64(0.13933006773004308) - self_distillation/empty_target_batch:np.float64(0.14453125) - actor/grad_norm:np.float64(0.3575190082192421) - perf/mfu/actor:np.float64(0.04880687442998337) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.40957641601562) - actor/lr:np.float64(1e-06) - training/global_step:34 - training/epoch:0 - critic/score/mean:0.59765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.05439735949039459 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.05439735949039459 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:384.01953125 - response_length/max:8192.0 - response_length/min:160.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:384.01953125 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:160.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:293.78125 - prompt_length/max:621.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010099261999130249 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5427038352936506) - timing_s/agent_loop/generate_sequences/max:np.float64(57.38408918492496) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.160430314703262) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(57.38408918492496) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:438 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:64.30811233632267 - timing_s/reward:0.06254505552351475 - timing_s/old_log_prob:3.2206109445542097 - timing_s/adv:0.5401895362883806 - timing_s/update_actor:23.241615699604154 - timing_s/step:91.45890662260354 - timing_s/stop_profile:2.9904767870903015e-05 - timing_per_token_ms/update_actor:0.1339443149639756 - timing_per_token_ms/gen:0.6541426760146342 - timing_per_token_ms/adv:0.003113179321267545 - perf/total_num_tokens:173517 - perf/time_per_step:91.45890662260354 - perf/throughput:237.15158863094845
(TaskRunner pid=2255397) Training Progress: 34%|███▍ | 34/100 [31:25<1:08:35, 62.35s/it]
(TaskRunner pid=2255397) step:35 - global_seqlen/min:19463 - global_seqlen/max:21625 - global_seqlen/minmax_diff:2162 - global_seqlen/balanced_min:20560 - global_seqlen/balanced_max:20574 - global_seqlen/mean:20569.125 - actor/entropy:0.4701376259326935 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.27064740657806396 - training/rollout_probs_diff_mean:0.00488562835380435 - training/rollout_probs_diff_std:0.010647308081388474 - training/rollout_actor_probs_pearson_corr:0.999289870262146 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.75 - self_distillation/correct_sample_fraction:0.5078125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.75 - rollout_corr/rollout_is_mean:0.9999282956123352 - rollout_corr/rollout_is_ratio_fraction_high:1.0713864867284428e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.1437816619873047 - rollout_corr/rollout_is_min:0.5562102794647217 - rollout_corr/rollout_is_std:0.036710504442453384 - rollout_corr/rollout_is_eff_sample_size:0.9986539149503298 - rollout_corr/rollout_is_seq_mean:0.9999589323997498 - rollout_corr/rollout_is_seq_std:0.0022408063523471355 - rollout_corr/rollout_is_seq_max:1.0093075037002563 - rollout_corr/rollout_is_seq_min:0.9939238429069519 - rollout_corr/rollout_is_seq_max_deviation:0.009307503700256348 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.643273327499628) - rollout_corr/training_log_ppl:np.float64(0.48507256753509864) - rollout_corr/kl:np.float64(0.0008028178892658389) - rollout_corr/k3_kl:np.float64(0.0007829796426221947) - rollout_corr/rollout_ppl:np.float64(1.641896395944059) - rollout_corr/rollout_log_ppl:np.float64(0.4842697436106391) - rollout_corr/log_ppl_diff:np.float64(0.000802823924459517) - rollout_corr/log_ppl_abs_diff:np.float64(0.002036678488366306) - rollout_corr/log_ppl_diff_max:np.float64(0.008617639541625977) - rollout_corr/log_ppl_diff_min:np.float64(-0.010091245174407959) - rollout_corr/ppl_ratio:np.float64(1.0008062841370702) - rollout_corr/chi2_token:np.float64(0.0015425898600369692) - rollout_corr/chi2_seq:np.float64(1.5650525961536914) - actor/pg_loss:np.float64(0.0024570953100919724) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004893736586382147) - actor/ppo_kl:np.float64(4.371891989851662e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.75) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.75) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5078125) - self_distillation/token_reweight_w_mean:np.float64(28164.23297266336) - self_distillation/token_reweight_w_std:np.float64(438345.5305433142) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0024336164351553) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09937885942053981) - self_distillation/empty_target_batch:np.float64(0.25) - actor/grad_norm:np.float64(0.2858409211039543) - perf/mfu/actor:np.float64(0.04990645011261699) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.3169288635254) - actor/lr:np.float64(1e-06) - training/global_step:35 - training/epoch:0 - critic/score/mean:0.5078125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5078125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004351168405264616 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004351168405264616 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:364.59765625 - response_length/max:846.0 - response_length/min:119.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:364.59765625 - response_length_non_aborted/max:846.0 - response_length_non_aborted/min:119.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.1875 - prompt_length/max:514.0 - prompt_length/min:181.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.146092295646667e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5682779010385275) - timing_s/agent_loop/generate_sequences/max:np.float64(7.532088892534375) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.356410696163948) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.532088892534375) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:216 - timing_s/agent_loop/slowest/response_length:846 - timing_s/gen:14.197907466441393 - timing_s/reward:0.05897785350680351 - timing_s/old_log_prob:3.0003139451146126 - timing_s/adv:0.5386089980602264 - timing_s/update_actor:21.315383506938815 - timing_s/step:39.194119876250625 - timing_s/stop_profile:0.00012629106640815735 - timing_per_token_ms/update_actor:0.12953506473257137 - timing_per_token_ms/gen:0.15211446121518146 - timing_per_token_ms/adv:0.003273164257474652 - perf/total_num_tokens:164553 - perf/time_per_step:39.194119876250625 - perf/throughput:524.8012983821
(TaskRunner pid=2255397) Training Progress: 35%|███▌ | 35/100 [32:05<1:00:02, 55.42s/it]
(TaskRunner pid=2255397) step:36 - global_seqlen/min:15946 - global_seqlen/max:23368 - global_seqlen/minmax_diff:7422 - global_seqlen/balanced_min:21144 - global_seqlen/balanced_max:21148 - global_seqlen/mean:21146.25 - actor/entropy:0.4903087913990021 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18462920188903809 - training/rollout_probs_diff_mean:0.005082453601062298 - training/rollout_probs_diff_std:0.010714489035308361 - training/rollout_actor_probs_pearson_corr:0.9993000030517578 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.828125 - self_distillation/correct_sample_fraction:0.5625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.828125 - rollout_corr/rollout_is_mean:1.0000641345977783 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6886519193649292 - rollout_corr/rollout_is_min:0.6301243901252747 - rollout_corr/rollout_is_std:0.037293627858161926 - rollout_corr/rollout_is_eff_sample_size:0.9986113546547832 - rollout_corr/rollout_is_seq_mean:1.000097632408142 - rollout_corr/rollout_is_seq_std:0.002074406947940588 - rollout_corr/rollout_is_seq_max:1.0062254667282104 - rollout_corr/rollout_is_seq_min:0.9938981533050537 - rollout_corr/rollout_is_seq_max_deviation:0.006225466728210449 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6528221243061125) - rollout_corr/training_log_ppl:np.float64(0.4915998738142662) - rollout_corr/kl:np.float64(0.0008901074607035042) - rollout_corr/k3_kl:np.float64(0.000780510345350649) - rollout_corr/rollout_ppl:np.float64(1.6513647134415805) - rollout_corr/rollout_log_ppl:np.float64(0.49070976028451696) - rollout_corr/log_ppl_diff:np.float64(0.0008901135297492146) - rollout_corr/log_ppl_abs_diff:np.float64(0.001717470120638609) - rollout_corr/log_ppl_diff_max:np.float64(0.006238460540771484) - rollout_corr/log_ppl_diff_min:np.float64(-0.005200505256652832) - rollout_corr/ppl_ratio:np.float64(1.0008924398571253) - rollout_corr/chi2_token:np.float64(0.0013305631000548601) - rollout_corr/chi2_seq:np.float64(0.19096036697737873) - actor/pg_loss:np.float64(0.0061755290953442454) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007742183429400029) - actor/ppo_kl:np.float64(0.0002702638142206837) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.828125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.828125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5625) - self_distillation/token_reweight_w_mean:np.float64(5297.740719528403) - self_distillation/token_reweight_w_std:np.float64(105322.54200576013) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0011047541629523) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1423137153033167) - self_distillation/empty_target_batch:np.float64(0.171875) - actor/grad_norm:np.float64(0.3517342582345009) - perf/mfu/actor:np.float64(0.051460493101783154) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.31007385253906) - actor/lr:np.float64(1e-06) - training/global_step:36 - training/epoch:0 - critic/score/mean:0.5625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0027716271579265594 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0027716271579265594 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:382.8203125 - response_length/max:718.0 - response_length/min:171.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:382.8203125 - response_length_non_aborted/max:718.0 - response_length_non_aborted/min:171.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.0 - prompt_length/max:704.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014107488095760345 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2529861833900213) - timing_s/agent_loop/generate_sequences/max:np.float64(7.170260231941938) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.674387803606805) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.170260231941938) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:344 - timing_s/agent_loop/slowest/response_length:682 - timing_s/gen:14.726460101082921 - timing_s/reward:0.061883002519607544 - timing_s/old_log_prob:2.9912548549473286 - timing_s/adv:0.5396729540079832 - timing_s/update_actor:21.34972732514143 - timing_s/step:39.758028803393245 - timing_s/stop_profile:0.0001045670360326767 - timing_per_token_ms/update_actor:0.12620279792600006 - timing_per_token_ms/gen:0.15026693435932859 - timing_per_token_ms/adv:0.0031901220902523096 - perf/total_num_tokens:169170 - perf/time_per_step:39.758028803393245 - perf/throughput:531.8737029084103
(TaskRunner pid=2255397) Training Progress: 36%|███▌ | 36/100 [32:44<54:07, 50.74s/it]
(TaskRunner pid=2255397) step:37 - global_seqlen/min:18794 - global_seqlen/max:24933 - global_seqlen/minmax_diff:6139 - global_seqlen/balanced_min:22097 - global_seqlen/balanced_max:22101 - global_seqlen/mean:22100.0 - actor/entropy:0.4772471785545349 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8962392210960388 - training/rollout_probs_diff_mean:0.004988708533346653 - training/rollout_probs_diff_std:0.011156635358929634 - training/rollout_actor_probs_pearson_corr:0.9992755651473999 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.7734375 - self_distillation/correct_sample_fraction:0.54296875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7734375 - rollout_corr/rollout_is_mean:0.9999436140060425 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.1014806154416874e-05 - rollout_corr/rollout_is_max:1.5039196014404297 - rollout_corr/rollout_is_min:0.0692535936832428 - rollout_corr/rollout_is_std:0.03678025305271149 - rollout_corr/rollout_is_eff_sample_size:0.9986488027678234 - rollout_corr/rollout_is_seq_mean:0.9999807476997375 - rollout_corr/rollout_is_seq_std:0.002165940823033452 - rollout_corr/rollout_is_seq_max:1.0075201988220215 - rollout_corr/rollout_is_seq_min:0.9936956763267517 - rollout_corr/rollout_is_seq_max_deviation:0.007520198822021484 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6701486743986607) - rollout_corr/training_log_ppl:np.float64(0.5012833332875744) - rollout_corr/kl:np.float64(0.0008904281232275935) - rollout_corr/k3_kl:np.float64(0.0009389091193838794) - rollout_corr/rollout_ppl:np.float64(1.6685718595981598) - rollout_corr/rollout_log_ppl:np.float64(0.5003929063677788) - rollout_corr/log_ppl_diff:np.float64(0.0008904269197955728) - rollout_corr/log_ppl_abs_diff:np.float64(0.001979760592803359) - rollout_corr/log_ppl_diff_max:np.float64(0.02197730541229248) - rollout_corr/log_ppl_diff_min:np.float64(-0.00723642110824585) - rollout_corr/ppl_ratio:np.float64(1.0008947688620538) - rollout_corr/chi2_token:np.float64(0.0017904054839164019) - rollout_corr/chi2_seq:np.float64(0.7441312682349235) - actor/pg_loss:np.float64(0.0051974470261484385) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006028023444741848) - actor/ppo_kl:np.float64(0.00011284644474807148) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7734375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7734375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.54296875) - self_distillation/token_reweight_w_mean:np.float64(17486.880942083197) - self_distillation/token_reweight_w_std:np.float64(248698.93032729038) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0008525024168193) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10978465108200908) - self_distillation/empty_target_batch:np.float64(0.2265625) - actor/grad_norm:np.float64(0.33661027252674103) - perf/mfu/actor:np.float64(0.053537470413758295) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.20091247558594) - actor/lr:np.float64(1e-06) - training/global_step:37 - training/epoch:0 - critic/score/mean:0.54296875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.54296875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00370756140910089 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00370756140910089 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:377.84375 - response_length/max:858.0 - response_length/min:146.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:377.84375 - response_length_non_aborted/max:858.0 - response_length_non_aborted/min:146.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.78125 - prompt_length/max:1009.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010385364294052124 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9452894497662783) - timing_s/agent_loop/generate_sequences/max:np.float64(7.737249631434679) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.423385596266598) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.737249631434679) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:258 - timing_s/agent_loop/slowest/response_length:858 - timing_s/gen:14.69729588367045 - timing_s/reward:0.05707615241408348 - timing_s/old_log_prob:3.1259428039193153 - timing_s/adv:0.5428317990154028 - timing_s/update_actor:21.357854707166553 - timing_s/step:39.88826394826174 - timing_s/stop_profile:0.00012905150651931763 - timing_per_token_ms/update_actor:0.1208023456287701 - timing_per_token_ms/gen:0.1519445856801593 - timing_per_token_ms/adv:0.00307031560529074 - perf/total_num_tokens:176800 - perf/time_per_step:39.88826394826174 - perf/throughput:554.0476774989622
(TaskRunner pid=2255397) Training Progress: 37%|███▋ | 37/100 [33:24<49:52, 47.50s/it]
(TaskRunner pid=2255397) step:38 - global_seqlen/min:17401 - global_seqlen/max:23183 - global_seqlen/minmax_diff:5782 - global_seqlen/balanced_min:20614 - global_seqlen/balanced_max:20619 - global_seqlen/mean:20617.125 - actor/entropy:0.4591502249240875 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.21379658579826355 - training/rollout_probs_diff_mean:0.005224332213401794 - training/rollout_probs_diff_std:0.011259016580879688 - training/rollout_actor_probs_pearson_corr:0.9992035627365112 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.7734375 - self_distillation/correct_sample_fraction:0.6484375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7734375 - rollout_corr/rollout_is_mean:1.0001171827316284 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.0498313460848294e-05 - rollout_corr/rollout_is_max:1.9823302030563354 - rollout_corr/rollout_is_min:0.3327290415763855 - rollout_corr/rollout_is_std:0.03824835270643234 - rollout_corr/rollout_is_eff_sample_size:0.9985396761004448 - rollout_corr/rollout_is_seq_mean:1.0001174211502075 - rollout_corr/rollout_is_seq_std:0.002267003059387207 - rollout_corr/rollout_is_seq_max:1.007875919342041 - rollout_corr/rollout_is_seq_min:0.9941402077674866 - rollout_corr/rollout_is_seq_max_deviation:0.007875919342041016 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.597476608119905) - rollout_corr/training_log_ppl:np.float64(0.4596842197352089) - rollout_corr/kl:np.float64(0.0007120158846873892) - rollout_corr/k3_kl:np.float64(0.0008242313452342387) - rollout_corr/rollout_ppl:np.float64(1.5963355526328087) - rollout_corr/rollout_log_ppl:np.float64(0.45897220366168767) - rollout_corr/log_ppl_diff:np.float64(0.0007120160735212266) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018952106474898756) - rollout_corr/log_ppl_diff_max:np.float64(0.006997823715209961) - rollout_corr/log_ppl_diff_min:np.float64(-0.006651520729064941) - rollout_corr/ppl_ratio:np.float64(1.0007150201126933) - rollout_corr/chi2_token:np.float64(0.001860827673226595) - rollout_corr/chi2_seq:np.float64(0.6607308345846832) - actor/pg_loss:np.float64(0.003047003992833197) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00027509686378834886) - actor/ppo_kl:np.float64(6.150551095629453e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7734375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7734375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6484375) - self_distillation/token_reweight_w_mean:np.float64(24324.67800112092) - self_distillation/token_reweight_w_std:np.float64(319731.90878068085) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0001283828169107) - self_distillation/token_reweight_w_clip_frac:np.float64(0.07227780664106831) - self_distillation/empty_target_batch:np.float64(0.2265625) - actor/grad_norm:np.float64(0.25013007409870625) - perf/mfu/actor:np.float64(0.05036454480977212) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.1811408996582) - actor/lr:np.float64(1e-06) - training/global_step:38 - training/epoch:0 - critic/score/mean:0.6484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0011094711953774095 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0011094711953774095 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:381.12890625 - response_length/max:764.0 - response_length/min:145.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:381.12890625 - response_length_non_aborted/max:764.0 - response_length_non_aborted/min:145.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.15625 - prompt_length/max:500.0 - prompt_length/min:164.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010602176189422607 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6879636067897081) - timing_s/agent_loop/generate_sequences/max:np.float64(6.8860987443476915) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.44739578160079) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.8860987443476915) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:335 - timing_s/agent_loop/slowest/response_length:743 - timing_s/gen:14.023487392812967 - timing_s/reward:0.054636452347040176 - timing_s/old_log_prob:2.9677660800516605 - timing_s/adv:0.5456443149596453 - timing_s/update_actor:21.413069240748882 - timing_s/step:39.11168333515525 - timing_s/stop_profile:0.00010557286441326141 - timing_per_token_ms/update_actor:0.1298257470473507 - timing_per_token_ms/gen:0.14372892407232796 - timing_per_token_ms/adv:0.003308198372467338 - perf/total_num_tokens:164937 - perf/time_per_step:39.11168333515525 - perf/throughput:527.1346882037278
(TaskRunner pid=2255397) Training Progress: 38%|███▊ | 38/100 [34:04<46:30, 45.01s/it]
(TaskRunner pid=2255397) step:39 - global_seqlen/min:16098 - global_seqlen/max:24192 - global_seqlen/minmax_diff:8094 - global_seqlen/balanced_min:21469 - global_seqlen/balanced_max:21484 - global_seqlen/mean:21479.5 - actor/entropy:0.4425225853919983 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.17422929406166077 - training/rollout_probs_diff_mean:0.004969022236764431 - training/rollout_probs_diff_std:0.010954358614981174 - training/rollout_actor_probs_pearson_corr:0.9992186427116394 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.80078125 - self_distillation/correct_sample_fraction:0.6015625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.80078125 - rollout_corr/rollout_is_mean:1.0003507137298584 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6815071105957031 - rollout_corr/rollout_is_min:0.5040706396102905 - rollout_corr/rollout_is_std:0.03746902570128441 - rollout_corr/rollout_is_eff_sample_size:0.9985991103255378 - rollout_corr/rollout_is_seq_mean:1.0004005432128906 - rollout_corr/rollout_is_seq_std:0.0023794230073690414 - rollout_corr/rollout_is_seq_max:1.012694001197815 - rollout_corr/rollout_is_seq_min:0.9931012392044067 - rollout_corr/rollout_is_seq_max_deviation:0.012694001197814941 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6068592290394008) - rollout_corr/training_log_ppl:np.float64(0.4637760063924361) - rollout_corr/kl:np.float64(0.0005125066249966892) - rollout_corr/k3_kl:np.float64(0.0008192024238837803) - rollout_corr/rollout_ppl:np.float64(1.6059974883683026) - rollout_corr/rollout_log_ppl:np.float64(0.46326349792070687) - rollout_corr/log_ppl_diff:np.float64(0.0005125084717292339) - rollout_corr/log_ppl_abs_diff:np.float64(0.001737583806971088) - rollout_corr/log_ppl_diff_max:np.float64(0.006271004676818848) - rollout_corr/log_ppl_diff_min:np.float64(-0.01115313172340393) - rollout_corr/ppl_ratio:np.float64(1.0005151890218258) - rollout_corr/chi2_token:np.float64(0.0022532844450324774) - rollout_corr/chi2_seq:np.float64(1.0010677098762244) - actor/pg_loss:np.float64(0.004201453877612948) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00037899658491369337) - actor/ppo_kl:np.float64(0.000158508868715046) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.80078125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.80078125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6015625) - self_distillation/token_reweight_w_mean:np.float64(21191.4891625538) - self_distillation/token_reweight_w_std:np.float64(305819.6095345903) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0013274641241878) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08719093064428307) - self_distillation/empty_target_batch:np.float64(0.19921875) - actor/grad_norm:np.float64(0.2567609027028084) - perf/mfu/actor:np.float64(0.05189741811293595) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.25640106201172) - actor/lr:np.float64(1e-06) - training/global_step:39 - training/epoch:0 - critic/score/mean:0.6015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00024856385425664485 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00024856385425664485 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:388.953125 - response_length/max:1077.0 - response_length/min:166.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:388.953125 - response_length_non_aborted/max:1077.0 - response_length_non_aborted/min:166.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:282.28125 - prompt_length/max:452.0 - prompt_length/min:181.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001035984605550766 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4913760963827372) - timing_s/agent_loop/generate_sequences/max:np.float64(8.839037172496319) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.348892316258571) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.839037172496319) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:296 - timing_s/agent_loop/slowest/response_length:1077 - timing_s/gen:15.795821461826563 - timing_s/reward:0.061745528131723404 - timing_s/old_log_prob:3.0005341172218323 - timing_s/adv:0.5373612400144339 - timing_s/update_actor:21.613999467343092 - timing_s/step:41.096161579713225 - timing_s/stop_profile:2.9422342777252197e-05 - timing_per_token_ms/update_actor:0.12578271996172566 - timing_per_token_ms/gen:0.15863718175618208 - timing_per_token_ms/adv:0.0031271749808796405 - perf/total_num_tokens:171836 - perf/time_per_step:41.096161579713225 - perf/throughput:522.6643845639145
(TaskRunner pid=2255397) Training Progress: 39%|███▉ | 39/100 [34:45<44:34, 43.84s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 40}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1", we must examine the structure of the product and compare it to the structures of the given options.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product contains a nitrogen-containing ring with a substituent group that includes a chlorinated benzene ring and a trifluoromethyl group. Specifically, the product has a structure where a nitrogen atom is part of a ring that is connected to a carbon chain leading to a chlorinated benzene ring with a trifluoromethyl group.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Now, let's analyze the options:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) - **Option A**: Contains a structure with a nitrogen ring and a chlorinated benzene ring, along with a trifluoromethyl group. It closely resembles the structure of the target product, especially in terms of the nitrogen-containing ring and the substituent groups.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) - **Option B**: Contains a chlorine atom directly attached to the nitrogen ring, which is not present in the target product. This makes it an incorrect match.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) - **Option C**: Features a hydroxyl group and a different substituent pattern, which does not match the target structure.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) - **Option D**: Includes a double bond and a fluorinated carbon chain, but not the specific chlorinated benzene ring and nitrogen ring arrangement of the target product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Only **Option A** contains the correct structural elements that match the target product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_40
(TaskRunner pid=2255397) step:40 - global_seqlen/min:18616 - global_seqlen/max:26413 - global_seqlen/minmax_diff:7797 - global_seqlen/balanced_min:22648 - global_seqlen/balanced_max:22652 - global_seqlen/mean:22651.25 - actor/entropy:0.46682530641555786 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4102363586425781 - training/rollout_probs_diff_mean:0.005161741282790899 - training/rollout_probs_diff_std:0.011252960190176964 - training/rollout_actor_probs_pearson_corr:0.9991944432258606 - self_distillation/success_group_fraction:0.65625 - self_distillation/success_sample_fraction:0.6484375 - self_distillation/correct_sample_fraction:0.55859375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.6484375 - rollout_corr/rollout_is_mean:1.0001730918884277 - rollout_corr/rollout_is_ratio_fraction_high:9.88513511401834e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.977027022803668e-05 - rollout_corr/rollout_is_max:2.2787070274353027 - rollout_corr/rollout_is_min:0.2646176815032959 - rollout_corr/rollout_is_std:0.03861745819449425 - rollout_corr/rollout_is_eff_sample_size:0.9985113878945102 - rollout_corr/rollout_is_seq_mean:1.0001778602600098 - rollout_corr/rollout_is_seq_std:0.0021594883874058723 - rollout_corr/rollout_is_seq_max:1.0066732168197632 - rollout_corr/rollout_is_seq_min:0.9935801029205322 - rollout_corr/rollout_is_seq_max_deviation:0.006673216819763184 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6297845151275396) - rollout_corr/training_log_ppl:np.float64(0.4774156858911738) - rollout_corr/kl:np.float64(0.0007412694044468537) - rollout_corr/k3_kl:np.float64(0.0009004596647628205) - rollout_corr/rollout_ppl:np.float64(1.628512756433338) - rollout_corr/rollout_log_ppl:np.float64(0.47667441627709195) - rollout_corr/log_ppl_diff:np.float64(0.0007412696140818298) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018867639009840786) - rollout_corr/log_ppl_diff_max:np.float64(0.00960838794708252) - rollout_corr/log_ppl_diff_min:np.float64(-0.005440711975097656) - rollout_corr/ppl_ratio:np.float64(1.0007440785411745) - rollout_corr/chi2_token:np.float64(0.0020869714207947254) - rollout_corr/chi2_seq:np.float64(0.9351563951931894) - actor/pg_loss:np.float64(0.003354526706971228) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000359561878212844) - actor/ppo_kl:np.float64(0.00013469158743095733) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.6484375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.6484375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.55859375) - self_distillation/token_reweight_w_mean:np.float64(9349.600620643701) - self_distillation/token_reweight_w_std:np.float64(148328.15592557297) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9980465415865183) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08430611382937059) - self_distillation/empty_target_batch:np.float64(0.3515625) - actor/grad_norm:np.float64(0.28926024958491325) - perf/mfu/actor:np.float64(0.054041057160877155) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.16180038452148) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6455357142857143) - val-aux/sciknoweval/reward/std@16:np.float64(0.19709042049186706) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7234142857142857) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.15652759126874422) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5656380952380953) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.17688873171951364) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6442142857142857) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.19706989234623676) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7823238095238095) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.11006778616480029) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.4971333333333333) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.14506188764023242) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.657395238095238) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.15434237064022824) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8243714285714285) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.07581488544967824) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.43891904761904754) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.10956011972160636) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6651571428571428) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.10880726297391681) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8535380952380953) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.04777075262896029) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.3930904761904762) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.0728366598950776) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6670523809523811) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.07675932385019639) - val-aux/sciknoweval/score/mean@16:np.float64(0.6455357142857143) - val-aux/sciknoweval/score/std@16:np.float64(0.19709042049186706) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7234142857142857) - val-aux/sciknoweval/score/best@2/std:np.float64(0.15652759126874422) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5656380952380953) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.17688873171951364) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6442142857142857) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.19706989234623676) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7823238095238095) - val-aux/sciknoweval/score/best@4/std:np.float64(0.11006778616480029) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.4971333333333333) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.14506188764023242) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.657395238095238) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.15434237064022824) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8243714285714285) - val-aux/sciknoweval/score/best@8/std:np.float64(0.07581488544967824) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.43891904761904754) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.10956011972160636) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6651571428571428) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.10880726297391681) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8535380952380953) - val-aux/sciknoweval/score/best@16/std:np.float64(0.04777075262896029) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.3930904761904762) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.0728366598950776) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6670523809523811) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.07675932385019639) - val-core/sciknoweval/acc/mean@16:np.float64(0.6455357142857143) - val-aux/sciknoweval/acc/std@16:np.float64(0.19709042049186706) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7234142857142857) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.15652759126874422) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5656380952380953) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.17688873171951364) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6442142857142857) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.19706989234623676) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7823238095238095) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.11006778616480029) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.4971333333333333) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.14506188764023242) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.657395238095238) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.15434237064022824) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8243714285714285) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.07581488544967824) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.43891904761904754) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.10956011972160636) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6651571428571428) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.10880726297391681) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8535380952380953) - val-core/sciknoweval/acc/best@16/std:np.float64(0.04777075262896029) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.3930904761904762) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.0728366598950776) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6670523809523811) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.07675932385019639) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9970238095238095) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.008247511677748322) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.999504761904762) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0031118665720636365) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9944428571428572) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.010692005762772086) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.996947619047619) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.00845281646011912) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9999857142857144) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0002604287412166727) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9903238095238096) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.012632535914270398) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9988238095238094) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.005212831716796075) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9847809523809524) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.01316242814531538) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9997285714285713) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.002375569550118018) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9783666666666666) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.010799831142838577) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999714285714286) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0004863840055089889) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:40 - training/epoch:0 - critic/score/mean:0.55859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.55859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0015742076793685555 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0015742076793685555 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:395.1640625 - response_length/max:913.0 - response_length/min:159.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:395.1640625 - response_length_non_aborted/max:913.0 - response_length_non_aborted/min:159.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.6875 - prompt_length/max:501.0 - prompt_length/min:159.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.6823952794075012e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.958938129246235) - timing_s/agent_loop/generate_sequences/max:np.float64(8.071589579805732) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.574361211241921) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.071589579805732) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:272 - timing_s/agent_loop/slowest/response_length:913 - timing_s/gen:15.109674572944641 - timing_s/reward:0.06512491777539253 - timing_s/old_log_prob:3.012589380145073 - timing_s/adv:0.5300268922001123 - timing_s/update_actor:21.85186207294464 - timing_s/step:40.66220090165734 - timing_s/testing:138.401902012527 - timing_s/save_checkpoint:13.304814949631691 - timing_s/stop_profile:3.7167221307754517e-05 - timing_per_token_ms/update_actor:0.12058861030265791 - timing_per_token_ms/gen:0.1493611689462905 - timing_per_token_ms/adv:0.0029249318039849476 - perf/total_num_tokens:181210 - perf/time_per_step:40.66220090165734 - perf/throughput:557.0591236510458
(TaskRunner pid=2255397) Training Progress: 40%|████ | 40/100 [37:57<1:28:24, 88.41s/it]
(TaskRunner pid=2255397) step:41 - global_seqlen/min:18283 - global_seqlen/max:24700 - global_seqlen/minmax_diff:6417 - global_seqlen/balanced_min:21101 - global_seqlen/balanced_max:21109 - global_seqlen/mean:21105.5 - actor/entropy:0.42351460456848145 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9826107025146484 - training/rollout_probs_diff_mean:0.005102348513901234 - training/rollout_probs_diff_std:0.01174638606607914 - training/rollout_actor_probs_pearson_corr:0.9991093873977661 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.8046875 - self_distillation/correct_sample_fraction:0.62890625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8046875 - rollout_corr/rollout_is_mean:0.999843180179596 - rollout_corr/rollout_is_ratio_fraction_high:9.873228009382728e-06 - rollout_corr/rollout_is_ratio_fraction_low:9.873228009382728e-06 - rollout_corr/rollout_is_max:2.636786937713623 - rollout_corr/rollout_is_min:0.013885053806006908 - rollout_corr/rollout_is_std:0.03809376060962677 - rollout_corr/rollout_is_eff_sample_size:0.9985506114667393 - rollout_corr/rollout_is_seq_mean:0.9998304843902588 - rollout_corr/rollout_is_seq_std:0.001992590492591262 - rollout_corr/rollout_is_seq_max:1.0068778991699219 - rollout_corr/rollout_is_seq_min:0.9926929473876953 - rollout_corr/rollout_is_seq_max_deviation:0.0073070526123046875 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5631140493787825) - rollout_corr/training_log_ppl:np.float64(0.436920084990561) - rollout_corr/kl:np.float64(0.0011501897124975846) - rollout_corr/k3_kl:np.float64(0.0009579621572584074) - rollout_corr/rollout_ppl:np.float64(1.5612433003261685) - rollout_corr/rollout_log_ppl:np.float64(0.4357698948588222) - rollout_corr/log_ppl_diff:np.float64(0.001150190131738782) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019631539471447468) - rollout_corr/log_ppl_diff_max:np.float64(0.010622143745422363) - rollout_corr/log_ppl_diff_min:np.float64(-0.00589299201965332) - rollout_corr/ppl_ratio:np.float64(1.001153603894636) - rollout_corr/chi2_token:np.float64(0.0014891347382217646) - rollout_corr/chi2_seq:np.float64(0.3713759642560035) - actor/pg_loss:np.float64(0.007653480512090027) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008902258696252829) - actor/ppo_kl:np.float64(0.0001822680885723571) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8046875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8046875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.62890625) - self_distillation/token_reweight_w_mean:np.float64(1181.0103185500484) - self_distillation/token_reweight_w_std:np.float64(19672.96590518701) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9963315855711699) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1375392735353671) - self_distillation/empty_target_batch:np.float64(0.1953125) - actor/grad_norm:np.float64(0.34661057591438293) - perf/mfu/actor:np.float64(0.05125395817020568) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.22666931152344) - actor/lr:np.float64(1e-06) - training/global_step:41 - training/epoch:0 - critic/score/mean:0.62890625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62890625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004591051023453474 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004591051023453474 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:395.640625 - response_length/max:930.0 - response_length/min:123.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:395.640625 - response_length_non_aborted/max:930.0 - response_length_non_aborted/min:123.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.90625 - prompt_length/max:631.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.7123838663101196e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.571682434529066) - timing_s/agent_loop/generate_sequences/max:np.float64(8.058521373197436) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.402195893351745) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.058521373197436) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:354 - timing_s/agent_loop/slowest/response_length:930 - timing_s/gen:14.408957185223699 - timing_s/reward:0.055850500240921974 - timing_s/old_log_prob:3.1053357888013124 - timing_s/adv:0.5568513758480549 - timing_s/update_actor:21.503483591601253 - timing_s/step:39.73492740653455 - timing_s/stop_profile:3.6053359508514404e-05 - timing_per_token_ms/update_actor:0.12735710828694685 - timing_per_token_ms/gen:0.1422629160106601 - timing_per_token_ms/adv:0.003298022884130054 - perf/total_num_tokens:168844 - perf/time_per_step:39.73492740653455 - perf/throughput:531.157381617089
(TaskRunner pid=2255397) Training Progress: 41%|████ | 41/100 [38:37<1:12:34, 73.81s/it]
(TaskRunner pid=2255397) step:42 - global_seqlen/min:15988 - global_seqlen/max:22821 - global_seqlen/minmax_diff:6833 - global_seqlen/balanced_min:19797 - global_seqlen/balanced_max:19801 - global_seqlen/mean:19798.875 - actor/entropy:0.39622214436531067 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.6605160236358643 - training/rollout_probs_diff_mean:0.005419079214334488 - training/rollout_probs_diff_std:0.012383614666759968 - training/rollout_actor_probs_pearson_corr:0.998907744884491 - self_distillation/success_group_fraction:0.96875 - self_distillation/success_sample_fraction:0.9609375 - self_distillation/correct_sample_fraction:0.6171875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.9609375 - rollout_corr/rollout_is_mean:0.9999876022338867 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.1576137442025356e-05 - rollout_corr/rollout_is_max:1.7245110273361206 - rollout_corr/rollout_is_min:0.2930092215538025 - rollout_corr/rollout_is_std:0.03913719579577446 - rollout_corr/rollout_is_eff_sample_size:0.9984703847272242 - rollout_corr/rollout_is_seq_mean:0.9999865293502808 - rollout_corr/rollout_is_seq_std:0.002267190022394061 - rollout_corr/rollout_is_seq_max:1.0080516338348389 - rollout_corr/rollout_is_seq_min:0.9936971664428711 - rollout_corr/rollout_is_seq_max_deviation:0.008051633834838867 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5080496412701905) - rollout_corr/training_log_ppl:np.float64(0.4042311883531511) - rollout_corr/kl:np.float64(0.0007036857559299392) - rollout_corr/k3_kl:np.float64(0.0009411578781737262) - rollout_corr/rollout_ppl:np.float64(1.5069640525616705) - rollout_corr/rollout_log_ppl:np.float64(0.40352750167949125) - rollout_corr/log_ppl_diff:np.float64(0.0007036866736598313) - rollout_corr/log_ppl_abs_diff:np.float64(0.001983315742108971) - rollout_corr/log_ppl_diff_max:np.float64(0.0076701343059539795) - rollout_corr/log_ppl_diff_min:np.float64(-0.006415665149688721) - rollout_corr/ppl_ratio:np.float64(1.0007067790720612) - rollout_corr/chi2_token:np.float64(0.002371277194470167) - rollout_corr/chi2_seq:np.float64(0.9433175600133836) - actor/pg_loss:np.float64(0.010519832605496049) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0009025154786286294) - actor/ppo_kl:np.float64(-0.00013643282175568316) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.9609375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.9609375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6171875) - self_distillation/token_reweight_w_mean:np.float64(12954.285722455475) - self_distillation/token_reweight_w_std:np.float64(227959.35449937586) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0025726119056344) - self_distillation/token_reweight_w_clip_frac:np.float64(0.14800434719654731) - self_distillation/empty_target_batch:np.float64(0.0390625) - actor/grad_norm:np.float64(0.45579010248184204) - perf/mfu/actor:np.float64(0.04795526011784892) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.25069046020508) - actor/lr:np.float64(1e-06) - training/global_step:42 - training/epoch:0 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0013215383514761925 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0013215383514761925 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:362.08984375 - response_length/max:864.0 - response_length/min:150.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:362.08984375 - response_length_non_aborted/max:864.0 - response_length_non_aborted/min:150.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:256.625 - prompt_length/max:557.0 - prompt_length/min:175.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.4523348808288574e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8027989882975817) - timing_s/agent_loop/generate_sequences/max:np.float64(7.439005896449089) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.104638532247918) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.439005896449089) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:199 - timing_s/agent_loop/slowest/response_length:864 - timing_s/gen:14.218583673238754 - timing_s/reward:0.05999644473195076 - timing_s/old_log_prob:2.961333120241761 - timing_s/adv:0.5369714200496674 - timing_s/update_actor:21.455332016572356 - timing_s/step:39.31888073682785 - timing_s/stop_profile:0.00011654756963253021 - timing_per_token_ms/update_actor:0.1354580248661373 - timing_per_token_ms/gen:0.15339105316617677 - timing_per_token_ms/adv:0.00339016370911016 - perf/total_num_tokens:158391 - perf/time_per_step:39.31888073682785 - perf/throughput:503.54625129131597
(TaskRunner pid=2255397) Training Progress: 42%|████▏ | 42/100 [39:16<1:01:21, 63.48s/it]
(TaskRunner pid=2255397) step:43 - global_seqlen/min:18065 - global_seqlen/max:23096 - global_seqlen/minmax_diff:5031 - global_seqlen/balanced_min:20756 - global_seqlen/balanced_max:20762 - global_seqlen/mean:20760.875 - actor/entropy:0.42775002121925354 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3424738645553589 - training/rollout_probs_diff_mean:0.005239918828010559 - training/rollout_probs_diff_std:0.01176416501402855 - training/rollout_actor_probs_pearson_corr:0.999097466468811 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.79296875 - self_distillation/correct_sample_fraction:0.61328125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.79296875 - rollout_corr/rollout_is_mean:0.9999403357505798 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1294968317088205e-05 - rollout_corr/rollout_is_max:1.6902360916137695 - rollout_corr/rollout_is_min:0.39213547110557556 - rollout_corr/rollout_is_std:0.038980014622211456 - rollout_corr/rollout_is_eff_sample_size:0.9984827447477223 - rollout_corr/rollout_is_seq_mean:0.9998909831047058 - rollout_corr/rollout_is_seq_std:0.002513354877009988 - rollout_corr/rollout_is_seq_max:1.0098495483398438 - rollout_corr/rollout_is_seq_min:0.9915899038314819 - rollout_corr/rollout_is_seq_max_deviation:0.00984954833984375 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5651745153591037) - rollout_corr/training_log_ppl:np.float64(0.4374735167075414) - rollout_corr/kl:np.float64(0.0007840867953743391) - rollout_corr/k3_kl:np.float64(0.0008964727625198066) - rollout_corr/rollout_ppl:np.float64(1.5639647683128715) - rollout_corr/rollout_log_ppl:np.float64(0.4366894279664848) - rollout_corr/log_ppl_diff:np.float64(0.0007840887410566211) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020498010562732816) - rollout_corr/log_ppl_diff_max:np.float64(0.009105801582336426) - rollout_corr/log_ppl_diff_min:np.float64(-0.00906282663345337) - rollout_corr/ppl_ratio:np.float64(1.0007877822499722) - rollout_corr/chi2_token:np.float64(0.00206923414953053) - rollout_corr/chi2_seq:np.float64(1.3589389722328633) - actor/pg_loss:np.float64(0.004318759543821216) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004535638540801301) - actor/ppo_kl:np.float64(-0.00012510620860073374) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.79296875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.79296875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.61328125) - self_distillation/token_reweight_w_mean:np.float64(39451.46745206439) - self_distillation/token_reweight_w_std:np.float64(461544.07824653096) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9995274562388659) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09244409290840849) - self_distillation/empty_target_batch:np.float64(0.20703125) - actor/grad_norm:np.float64(0.32581663131713867) - perf/mfu/actor:np.float64(0.05044233796581919) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.0129623413086) - actor/lr:np.float64(1e-06) - training/global_step:43 - training/epoch:0 - critic/score/mean:0.61328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.61328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0011210255324840546 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0011210255324840546 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:345.83984375 - response_length/max:801.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:345.83984375 - response_length_non_aborted/max:801.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.9375 - prompt_length/max:813.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001546517014503479 - timing_s/agent_loop/generate_sequences/min:np.float64(1.693193232640624) - timing_s/agent_loop/generate_sequences/max:np.float64(6.869511658325791) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.046683865577506) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.869511658325791) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:315 - timing_s/agent_loop/slowest/response_length:801 - timing_s/gen:14.301691135391593 - timing_s/reward:0.06055835448205471 - timing_s/old_log_prob:2.9482695516198874 - timing_s/adv:0.5146379265934229 - timing_s/update_actor:21.42802003212273 - timing_s/step:39.34179303236306 - timing_s/stop_profile:0.00010741874575614929 - timing_per_token_ms/update_actor:0.12901684076491676 - timing_per_token_ms/gen:0.16153714503181332 - timing_per_token_ms/adv:0.0030986045060325184 - perf/total_num_tokens:166087 - perf/time_per_step:39.34179303236306 - perf/throughput:527.7053585972008
(TaskRunner pid=2255397) Training Progress: 43%|████▎ | 43/100 [39:56<53:26, 56.25s/it]
(TaskRunner pid=2255397) step:44 - global_seqlen/min:19213 - global_seqlen/max:22622 - global_seqlen/minmax_diff:3409 - global_seqlen/balanced_min:20879 - global_seqlen/balanced_max:20890 - global_seqlen/mean:20886.75 - actor/entropy:0.43929049372673035 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.22547024488449097 - training/rollout_probs_diff_mean:0.0053957682102918625 - training/rollout_probs_diff_std:0.011915630660951138 - training/rollout_actor_probs_pearson_corr:0.9990841150283813 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.80859375 - self_distillation/correct_sample_fraction:0.66796875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.80859375 - rollout_corr/rollout_is_mean:1.000036358833313 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.9659661054611206 - rollout_corr/rollout_is_min:0.5844541192054749 - rollout_corr/rollout_is_std:0.039155468344688416 - rollout_corr/rollout_is_eff_sample_size:0.9984691962798411 - rollout_corr/rollout_is_seq_mean:1.0000959634780884 - rollout_corr/rollout_is_seq_std:0.0024589113891124725 - rollout_corr/rollout_is_seq_max:1.0087720155715942 - rollout_corr/rollout_is_seq_min:0.9921355843544006 - rollout_corr/rollout_is_seq_max_deviation:0.008772015571594238 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6047232481651008) - rollout_corr/training_log_ppl:np.float64(0.46260377258295193) - rollout_corr/kl:np.float64(0.0009372506496678756) - rollout_corr/k3_kl:np.float64(0.000976888309367041) - rollout_corr/rollout_ppl:np.float64(1.6032404340803623) - rollout_corr/rollout_log_ppl:np.float64(0.4616665196372196) - rollout_corr/log_ppl_diff:np.float64(0.0009372529457323253) - rollout_corr/log_ppl_abs_diff:np.float64(0.002075349271763116) - rollout_corr/log_ppl_diff_max:np.float64(0.013183057308197021) - rollout_corr/log_ppl_diff_min:np.float64(-0.006263077259063721) - rollout_corr/ppl_ratio:np.float64(1.000940943369642) - rollout_corr/chi2_token:np.float64(0.0020958164241164923) - rollout_corr/chi2_seq:np.float64(0.34359696647152305) - actor/pg_loss:np.float64(0.005748242023400962) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005829084911965765) - actor/ppo_kl:np.float64(0.0002053698056716513) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.80859375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.80859375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.66796875) - self_distillation/token_reweight_w_mean:np.float64(14803.191471951548) - self_distillation/token_reweight_w_std:np.float64(246306.37441055127) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9985546870157123) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10855769729823805) - self_distillation/empty_target_batch:np.float64(0.19140625) - actor/grad_norm:np.float64(0.36420872062444687) - perf/mfu/actor:np.float64(0.05008791245123715) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.9407615661621) - actor/lr:np.float64(1e-06) - training/global_step:44 - training/epoch:0 - critic/score/mean:0.66796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.66796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00279244058765471 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00279244058765471 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:341.1484375 - response_length/max:1060.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:341.1484375 - response_length_non_aborted/max:1060.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:311.5625 - prompt_length/max:701.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014213845133781433 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7569324560463428) - timing_s/agent_loop/generate_sequences/max:np.float64(8.538507629185915) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.876661294561927) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.538507629185915) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:198 - timing_s/agent_loop/slowest/response_length:1060 - timing_s/gen:16.138035461306572 - timing_s/reward:0.06066634878516197 - timing_s/old_log_prob:3.082423932850361 - timing_s/adv:0.5534890294075012 - timing_s/update_actor:21.732121851295233 - timing_s/step:41.653873067349195 - timing_s/stop_profile:7.493607699871063e-05 - timing_per_token_ms/update_actor:0.13005925916726652 - timing_per_token_ms/gen:0.18478525501301407 - timing_per_token_ms/adv:0.00331244107752224 - perf/total_num_tokens:167094 - perf/time_per_step:41.653873067349195 - perf/throughput:501.4359640993934
(TaskRunner pid=2255397) Training Progress: 44%|████▍ | 44/100 [40:37<48:25, 51.89s/it]
(TaskRunner pid=2255397) step:45 - global_seqlen/min:18835 - global_seqlen/max:26317 - global_seqlen/minmax_diff:7482 - global_seqlen/balanced_min:21440 - global_seqlen/balanced_max:21451 - global_seqlen/mean:21447.875 - actor/entropy:0.4101407527923584 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.28218671679496765 - training/rollout_probs_diff_mean:0.004841715097427368 - training/rollout_probs_diff_std:0.01116796862334013 - training/rollout_actor_probs_pearson_corr:0.9992083311080933 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.73046875 - self_distillation/correct_sample_fraction:0.5390625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.73046875 - rollout_corr/rollout_is_mean:0.9999707937240601 - rollout_corr/rollout_is_ratio_fraction_high:1.0986233974108472e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.0986233974108472e-05 - rollout_corr/rollout_is_max:2.0019893646240234 - rollout_corr/rollout_is_min:0.25255540013313293 - rollout_corr/rollout_is_std:0.036969371140003204 - rollout_corr/rollout_is_eff_sample_size:0.998635012024894 - rollout_corr/rollout_is_seq_mean:1.0000436305999756 - rollout_corr/rollout_is_seq_std:0.002479569287970662 - rollout_corr/rollout_is_seq_max:1.0116991996765137 - rollout_corr/rollout_is_seq_min:0.9924298524856567 - rollout_corr/rollout_is_seq_max_deviation:0.011699199676513672 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5578296538442373) - rollout_corr/training_log_ppl:np.float64(0.432097623241134) - rollout_corr/kl:np.float64(0.0007768278248150295) - rollout_corr/k3_kl:np.float64(0.0008250089271086836) - rollout_corr/rollout_ppl:np.float64(1.5565427541732788) - rollout_corr/rollout_log_ppl:np.float64(0.4313207929953933) - rollout_corr/log_ppl_diff:np.float64(0.0007768302457407117) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019471754785627127) - rollout_corr/log_ppl_diff_max:np.float64(0.011698126792907715) - rollout_corr/log_ppl_diff_min:np.float64(-0.010318875312805176) - rollout_corr/ppl_ratio:np.float64(1.0007803542539477) - rollout_corr/chi2_token:np.float64(0.0017588012851774693) - rollout_corr/chi2_seq:np.float64(0.6141417354810983) - actor/pg_loss:np.float64(0.0026712483959272504) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00039884855823402177) - actor/ppo_kl:np.float64(6.892802709934642e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.73046875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.73046875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5390625) - self_distillation/token_reweight_w_mean:np.float64(38907.24203802785) - self_distillation/token_reweight_w_std:np.float64(568287.8935087718) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0006591160781682) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08028327141073532) - self_distillation/empty_target_batch:np.float64(0.26953125) - actor/grad_norm:np.float64(0.28873957321047783) - perf/mfu/actor:np.float64(0.05157141247510403) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.14898681640625) - actor/lr:np.float64(1e-06) - training/global_step:45 - training/epoch:0 - critic/score/mean:0.5390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005405227188020945 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005405227188020945 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:355.55859375 - response_length/max:782.0 - response_length/min:125.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:355.55859375 - response_length_non_aborted/max:782.0 - response_length_non_aborted/min:125.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:314.6875 - prompt_length/max:845.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:4.7983601689338684e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.374290145933628) - timing_s/agent_loop/generate_sequences/max:np.float64(7.021162023767829) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.9449067628447665) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.021162023767829) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:229 - timing_s/agent_loop/slowest/response_length:782 - timing_s/gen:13.897942123934627 - timing_s/reward:0.06329947151243687 - timing_s/old_log_prob:3.025046695023775 - timing_s/adv:0.56037213280797 - timing_s/update_actor:21.67712887376547 - timing_s/step:39.31348724476993 - timing_s/stop_profile:0.00011695176362991333 - timing_per_token_ms/update_actor:0.1263361106506208 - timing_per_token_ms/gen:0.15268604774545583 - timing_per_token_ms/adv:0.0032658954139277788 - perf/total_num_tokens:171583 - perf/time_per_step:39.31348724476993 - perf/throughput:545.5602263534461
(TaskRunner pid=2255397) Training Progress: 45%|████▌ | 45/100 [41:17<44:07, 48.13s/it]
(TaskRunner pid=2255397) step:46 - global_seqlen/min:17315 - global_seqlen/max:21947 - global_seqlen/minmax_diff:4632 - global_seqlen/balanced_min:19767 - global_seqlen/balanced_max:19776 - global_seqlen/mean:19773.25 - actor/entropy:0.4300471842288971 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5463374257087708 - training/rollout_probs_diff_mean:0.0050852419808506966 - training/rollout_probs_diff_std:0.011606025509536266 - training/rollout_actor_probs_pearson_corr:0.9991208910942078 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.8203125 - self_distillation/correct_sample_fraction:0.5546875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8203125 - rollout_corr/rollout_is_mean:0.9999197125434875 - rollout_corr/rollout_is_ratio_fraction_high:1.1397570233384613e-05 - rollout_corr/rollout_is_ratio_fraction_low:2.2795140466769226e-05 - rollout_corr/rollout_is_max:2.4014859199523926 - rollout_corr/rollout_is_min:0.26859816908836365 - rollout_corr/rollout_is_std:0.03753577917814255 - rollout_corr/rollout_is_eff_sample_size:0.9985928099624436 - rollout_corr/rollout_is_seq_mean:1.0000156164169312 - rollout_corr/rollout_is_seq_std:0.002315930090844631 - rollout_corr/rollout_is_seq_max:1.0097026824951172 - rollout_corr/rollout_is_seq_min:0.9926007986068726 - rollout_corr/rollout_is_seq_max_deviation:0.009702682495117188 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5882981927134097) - rollout_corr/training_log_ppl:np.float64(0.44962009048322216) - rollout_corr/kl:np.float64(0.0011486389046666545) - rollout_corr/k3_kl:np.float64(0.0009466409361493788) - rollout_corr/rollout_ppl:np.float64(1.5864842752926052) - rollout_corr/rollout_log_ppl:np.float64(0.4484714485006407) - rollout_corr/log_ppl_diff:np.float64(0.0011486419825814664) - rollout_corr/log_ppl_abs_diff:np.float64(0.0022174433688633144) - rollout_corr/log_ppl_diff_max:np.float64(0.015058279037475586) - rollout_corr/log_ppl_diff_min:np.float64(-0.00858527421951294) - rollout_corr/ppl_ratio:np.float64(1.0011532127391547) - rollout_corr/chi2_token:np.float64(0.0015598325990140438) - rollout_corr/chi2_seq:np.float64(0.6058120087254792) - actor/pg_loss:np.float64(0.005358678288757801) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006527249729515461) - actor/ppo_kl:np.float64(0.0004008172310321356) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8203125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8203125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5546875) - self_distillation/token_reweight_w_mean:np.float64(13853.629367629066) - self_distillation/token_reweight_w_std:np.float64(223926.18253548152) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.000649391906336) - self_distillation/token_reweight_w_clip_frac:np.float64(0.12465728094684891) - self_distillation/empty_target_batch:np.float64(0.1796875) - actor/grad_norm:np.float64(0.37411294877529144) - perf/mfu/actor:np.float64(0.04759140321566382) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.96075439453125) - actor/lr:np.float64(1e-06) - training/global_step:46 - training/epoch:0 - critic/score/mean:0.5546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004386639688163996 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004386639688163996 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:342.7265625 - response_length/max:874.0 - response_length/min:124.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:342.7265625 - response_length_non_aborted/max:874.0 - response_length_non_aborted/min:124.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:275.1875 - prompt_length/max:595.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012195669114589691 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6501337420195341) - timing_s/agent_loop/generate_sequences/max:np.float64(7.341919541358948) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.8824371568043716) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.341919541358948) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:177 - timing_s/agent_loop/slowest/response_length:874 - timing_s/gen:14.224360808730125 - timing_s/reward:0.07079944014549255 - timing_s/old_log_prob:2.932241167873144 - timing_s/adv:0.537244874984026 - timing_s/update_actor:21.595252191647887 - timing_s/step:39.45111795514822 - timing_s/stop_profile:0.00012188218533992767 - timing_per_token_ms/update_actor:0.13651810015834454 - timing_per_token_ms/gen:0.16212314856424953 - timing_per_token_ms/adv:0.0033962858595831865 - perf/total_num_tokens:158186 - perf/time_per_step:39.45111795514822 - perf/throughput:501.2088636494436
(TaskRunner pid=2255397) Training Progress: 46%|████▌ | 46/100 [41:56<40:59, 45.54s/it]
(TaskRunner pid=2255397) step:47 - global_seqlen/min:16338 - global_seqlen/max:23481 - global_seqlen/minmax_diff:7143 - global_seqlen/balanced_min:20960 - global_seqlen/balanced_max:20969 - global_seqlen/mean:20966.0 - actor/entropy:0.4562165141105652 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24135512113571167 - training/rollout_probs_diff_mean:0.004941638559103012 - training/rollout_probs_diff_std:0.01093968003988266 - training/rollout_actor_probs_pearson_corr:0.9992673993110657 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.8671875 - self_distillation/correct_sample_fraction:0.64453125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8671875 - rollout_corr/rollout_is_mean:1.0000667572021484 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.9514927864074707 - rollout_corr/rollout_is_min:0.5686209797859192 - rollout_corr/rollout_is_std:0.03698226809501648 - rollout_corr/rollout_is_eff_sample_size:0.998634417604882 - rollout_corr/rollout_is_seq_mean:1.0001049041748047 - rollout_corr/rollout_is_seq_std:0.0021275379694998264 - rollout_corr/rollout_is_seq_max:1.0070573091506958 - rollout_corr/rollout_is_seq_min:0.9932472705841064 - rollout_corr/rollout_is_seq_max_deviation:0.007057309150695801 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6164640178903937) - rollout_corr/training_log_ppl:np.float64(0.46599303945549764) - rollout_corr/kl:np.float64(0.0011402188001117253) - rollout_corr/k3_kl:np.float64(0.0008741923521711215) - rollout_corr/rollout_ppl:np.float64(1.6145951971411705) - rollout_corr/rollout_log_ppl:np.float64(0.46485281962668523) - rollout_corr/log_ppl_diff:np.float64(0.0011402198288124055) - rollout_corr/log_ppl_abs_diff:np.float64(0.001965065923286602) - rollout_corr/log_ppl_diff_max:np.float64(0.008194833993911743) - rollout_corr/log_ppl_diff_min:np.float64(-0.005479037761688232) - rollout_corr/ppl_ratio:np.float64(1.0011436399072409) - rollout_corr/chi2_token:np.float64(0.0012568566016852856) - rollout_corr/chi2_seq:np.float64(0.40265263244509697) - actor/pg_loss:np.float64(0.006021094392053783) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005619370990643802) - actor/ppo_kl:np.float64(0.0005166811963022155) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8671875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8671875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.64453125) - self_distillation/token_reweight_w_mean:np.float64(12667.028565393994) - self_distillation/token_reweight_w_std:np.float64(223391.28332038334) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0007588975131512) - self_distillation/token_reweight_w_clip_frac:np.float64(0.12597810916486196) - self_distillation/empty_target_batch:np.float64(0.1328125) - actor/grad_norm:np.float64(0.35431264340877533) - perf/mfu/actor:np.float64(0.050087304585172576) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.9956817626953) - actor/lr:np.float64(1e-06) - training/global_step:47 - training/epoch:0 - critic/score/mean:0.64453125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.64453125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005755655933171511 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005755655933171511 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:361.90625 - response_length/max:784.0 - response_length/min:139.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:361.90625 - response_length_non_aborted/max:784.0 - response_length_non_aborted/min:139.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:293.28125 - prompt_length/max:950.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.334273636341095e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7290426678955555) - timing_s/agent_loop/generate_sequences/max:np.float64(7.266604153439403) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.170146645323257) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.266604153439403) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:279 - timing_s/agent_loop/slowest/response_length:784 - timing_s/gen:14.346795970574021 - timing_s/reward:0.06195123679935932 - timing_s/old_log_prob:2.8864920753985643 - timing_s/adv:0.5422863606363535 - timing_s/update_actor:21.55750539712608 - timing_s/step:39.47956304065883 - timing_s/stop_profile:2.954714000225067e-05 - timing_per_token_ms/update_actor:0.12852657515218735 - timing_per_token_ms/gen:0.1548527326070074 - timing_per_token_ms/adv:0.0032331295945599633 - perf/total_num_tokens:167728 - perf/time_per_step:39.47956304065883 - perf/throughput:531.0595757710828
(TaskRunner pid=2255397) Training Progress: 47%|████▋ | 47/100 [42:36<38:37, 43.73s/it]
(TaskRunner pid=2255397) step:48 - global_seqlen/min:16138 - global_seqlen/max:20539 - global_seqlen/minmax_diff:4401 - global_seqlen/balanced_min:19360 - global_seqlen/balanced_max:19367 - global_seqlen/mean:19365.625 - actor/entropy:0.45748016238212585 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.423759400844574 - training/rollout_probs_diff_mean:0.00517674908041954 - training/rollout_probs_diff_std:0.011389809660613537 - training/rollout_actor_probs_pearson_corr:0.9992020130157471 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.80859375 - self_distillation/correct_sample_fraction:0.671875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.80859375 - rollout_corr/rollout_is_mean:1.0000624656677246 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.4522043279139325e-05 - rollout_corr/rollout_is_max:1.9825339317321777 - rollout_corr/rollout_is_min:0.29720374941825867 - rollout_corr/rollout_is_std:0.038425594568252563 - rollout_corr/rollout_is_eff_sample_size:0.9985258883712781 - rollout_corr/rollout_is_seq_mean:1.0000547170639038 - rollout_corr/rollout_is_seq_std:0.0022232853807508945 - rollout_corr/rollout_is_seq_max:1.0069233179092407 - rollout_corr/rollout_is_seq_min:0.9934024810791016 - rollout_corr/rollout_is_seq_max_deviation:0.006923317909240723 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6189399659633636) - rollout_corr/training_log_ppl:np.float64(0.47103061771485955) - rollout_corr/kl:np.float64(0.0008533685957203829) - rollout_corr/k3_kl:np.float64(0.0008472814761262271) - rollout_corr/rollout_ppl:np.float64(1.6175275961868465) - rollout_corr/rollout_log_ppl:np.float64(0.47017724718898535) - rollout_corr/log_ppl_diff:np.float64(0.0008533705258741975) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018438339466229081) - rollout_corr/log_ppl_diff_max:np.float64(0.009492874145507812) - rollout_corr/log_ppl_diff_min:np.float64(-0.004758328199386597) - rollout_corr/ppl_ratio:np.float64(1.000856165541336) - rollout_corr/chi2_token:np.float64(0.0016905255615711212) - rollout_corr/chi2_seq:np.float64(0.5319139340426773) - actor/pg_loss:np.float64(0.004102788749150932) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002375331782786816) - actor/ppo_kl:np.float64(0.00012016280814464153) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.80859375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.80859375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.671875) - self_distillation/token_reweight_w_mean:np.float64(1381.5336034777574) - self_distillation/token_reweight_w_std:np.float64(20333.09212728584) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9975523005705327) - self_distillation/token_reweight_w_clip_frac:np.float64(0.11302358307875693) - self_distillation/empty_target_batch:np.float64(0.19140625) - actor/grad_norm:np.float64(0.2960100993514061) - perf/mfu/actor:np.float64(0.04606089601890587) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.1104965209961) - actor/lr:np.float64(1e-06) - training/global_step:48 - training/epoch:0 - critic/score/mean:0.671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0013794433325529099 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0013794433325529099 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:339.45703125 - response_length/max:832.0 - response_length/min:151.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:339.45703125 - response_length_non_aborted/max:832.0 - response_length_non_aborted/min:151.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.71875 - prompt_length/max:563.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.479366958141327e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9788021426647902) - timing_s/agent_loop/generate_sequences/max:np.float64(7.109138231724501) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.9673785015329486) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.109138231724501) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:311 - timing_s/agent_loop/slowest/response_length:832 - timing_s/gen:13.836422380059958 - timing_s/reward:0.06020788662135601 - timing_s/old_log_prob:2.906719511374831 - timing_s/adv:0.5168796796351671 - timing_s/update_actor:21.836700728163123 - timing_s/step:39.24484575167298 - timing_s/stop_profile:0.00012063421308994293 - timing_per_token_ms/update_actor:0.1409501418632443 - timing_per_token_ms/gen:0.1592205196725004 - timing_per_token_ms/adv:0.0033363219598848935 - perf/total_num_tokens:154925 - perf/time_per_step:39.24484575167298 - perf/throughput:493.4565196800259
(TaskRunner pid=2255397) Training Progress: 48%|████▊ | 48/100 [43:15<36:44, 42.40s/it]
(TaskRunner pid=2255397) step:49 - global_seqlen/min:19946 - global_seqlen/max:22692 - global_seqlen/minmax_diff:2746 - global_seqlen/balanced_min:21730 - global_seqlen/balanced_max:21733 - global_seqlen/mean:21731.75 - actor/entropy:0.4979252517223358 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.43991324305534363 - training/rollout_probs_diff_mean:0.004984842147678137 - training/rollout_probs_diff_std:0.01066806074231863 - training/rollout_actor_probs_pearson_corr:0.9993179440498352 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.86328125 - self_distillation/correct_sample_fraction:0.640625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.86328125 - rollout_corr/rollout_is_mean:0.9998939037322998 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.758192391018383e-06 - rollout_corr/rollout_is_max:1.9085274934768677 - rollout_corr/rollout_is_min:0.12017349898815155 - rollout_corr/rollout_is_std:0.03726325184106827 - rollout_corr/rollout_is_eff_sample_size:0.9986130189557251 - rollout_corr/rollout_is_seq_mean:0.9998945593833923 - rollout_corr/rollout_is_seq_std:0.001821837155148387 - rollout_corr/rollout_is_seq_max:1.005088448524475 - rollout_corr/rollout_is_seq_min:0.9940054416656494 - rollout_corr/rollout_is_seq_max_deviation:0.005994558334350586 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6754766842350364) - rollout_corr/training_log_ppl:np.float64(0.5048167807399295) - rollout_corr/kl:np.float64(0.0009508746554258707) - rollout_corr/k3_kl:np.float64(0.0007735003869697721) - rollout_corr/rollout_ppl:np.float64(1.6738910162821412) - rollout_corr/rollout_log_ppl:np.float64(0.503865905397106) - rollout_corr/log_ppl_diff:np.float64(0.000950875342823565) - rollout_corr/log_ppl_abs_diff:np.float64(0.001617251429706812) - rollout_corr/log_ppl_diff_max:np.float64(0.008522868156433105) - rollout_corr/log_ppl_diff_min:np.float64(-0.00493285059928894) - rollout_corr/ppl_ratio:np.float64(1.0009531062096357) - rollout_corr/chi2_token:np.float64(0.0011934449430555105) - rollout_corr/chi2_seq:np.float64(0.1550909043289721) - actor/pg_loss:np.float64(0.005771090276539326) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002795115901790268) - actor/ppo_kl:np.float64(0.00010383528175317291) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.86328125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.86328125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.640625) - self_distillation/token_reweight_w_mean:np.float64(4803.391529331682) - self_distillation/token_reweight_w_std:np.float64(100370.09618178767) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.001389476004988) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10803735570516437) - self_distillation/empty_target_batch:np.float64(0.13671875) - actor/grad_norm:np.float64(0.3376929610967636) - perf/mfu/actor:np.float64(0.05224311765341433) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.75490951538086) - actor/lr:np.float64(1e-06) - training/global_step:49 - training/epoch:0 - critic/score/mean:0.640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0023309881798923016 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:0.0023309881798923016 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:400.3046875 - response_length/max:826.0 - response_length/min:122.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:400.3046875 - response_length_non_aborted/max:826.0 - response_length_non_aborted/min:122.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.8125 - prompt_length/max:418.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013683363795280457 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6328133698552847) - timing_s/agent_loop/generate_sequences/max:np.float64(7.4781682174652815) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.834140434708388) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.4781682174652815) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:191 - timing_s/agent_loop/slowest/response_length:826 - timing_s/gen:14.471364917233586 - timing_s/reward:0.060647664591670036 - timing_s/old_log_prob:2.9566055797040462 - timing_s/adv:0.5396076701581478 - timing_s/update_actor:21.56921719573438 - timing_s/step:39.68868784233928 - timing_s/stop_profile:2.8738752007484436e-05 - timing_per_token_ms/update_actor:0.12406511898336754 - timing_per_token_ms/gen:0.14121435739606147 - timing_per_token_ms/adv:0.0031037978427769725 - perf/total_num_tokens:173854 - perf/time_per_step:39.68868784233928 - perf/throughput:547.5552652767952
(TaskRunner pid=2255397) Training Progress: 49%|████▉ | 49/100 [43:55<35:21, 41.59s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 50}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1", we need to examine the structure of the product and compare it with the structures of the given options.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product has a nitrogen-containing ring with a substituent group: "Cc2ccc(Cl)cc2C(F)(F)F". This suggests that the molecule contains a chlorinated aromatic ring attached to a trifluoromethyl group, which is connected to a nitrogen-containing ring.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Now, let's evaluate the options:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) - **Option A**: Contains a structure with a nitrogen ring and a chlorinated aromatic ring connected to a trifluoromethyl group. The structure is very similar to the product, with the correct substituents and connectivity.
(TaskRunner pid=2255397) - **Option B**: Contains a chlorinated nitrogen ring but lacks the trifluoromethyl group and the correct aromatic ring structure.
(TaskRunner pid=2255397) - **Option C**: Contains a hydroxyl group and a different substituent, not matching the product.
(TaskRunner pid=2255397) - **Option D**: Contains a different ring structure and lacks the trifluoromethyl group and the correct aromatic ring.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Only **Option A** matches the structure and substituents of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1".
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_50
(TaskRunner pid=2255397) step:50 - global_seqlen/min:19045 - global_seqlen/max:30156 - global_seqlen/minmax_diff:11111 - global_seqlen/balanced_min:21698 - global_seqlen/balanced_max:21699 - global_seqlen/mean:21698.5 - actor/entropy:0.39573362469673157 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24491804838180542 - training/rollout_probs_diff_mean:0.004589053802192211 - training/rollout_probs_diff_std:0.01072707585990429 - training/rollout_actor_probs_pearson_corr:0.9992213845252991 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83203125 - self_distillation/correct_sample_fraction:0.609375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83203125 - rollout_corr/rollout_is_mean:1.0000442266464233 - rollout_corr/rollout_is_ratio_fraction_high:9.806613888940774e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.004254102706909 - rollout_corr/rollout_is_min:0.6065316796302795 - rollout_corr/rollout_is_std:0.03603039309382439 - rollout_corr/rollout_is_eff_sample_size:0.9987034939469002 - rollout_corr/rollout_is_seq_mean:1.000090479850769 - rollout_corr/rollout_is_seq_std:0.0021530245430767536 - rollout_corr/rollout_is_seq_max:1.0064139366149902 - rollout_corr/rollout_is_seq_min:0.991096556186676 - rollout_corr/rollout_is_seq_max_deviation:0.008903443813323975 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.524656287394464) - rollout_corr/training_log_ppl:np.float64(0.4108033778902609) - rollout_corr/kl:np.float64(0.0007370334822254243) - rollout_corr/k3_kl:np.float64(0.0008599899813646061) - rollout_corr/rollout_ppl:np.float64(1.5234609101898968) - rollout_corr/rollout_log_ppl:np.float64(0.41006634262157604) - rollout_corr/log_ppl_diff:np.float64(0.0007370352686848491) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018268449639435858) - rollout_corr/log_ppl_diff_max:np.float64(0.00987979769706726) - rollout_corr/log_ppl_diff_min:np.float64(-0.00624692440032959) - rollout_corr/ppl_ratio:np.float64(1.000740017509088) - rollout_corr/chi2_token:np.float64(0.0019624873530119658) - rollout_corr/chi2_seq:np.float64(0.6737326441798359) - actor/pg_loss:np.float64(0.00585430464707315) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0007581320141980541) - actor/ppo_kl:np.float64(0.00011956099752552518) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83203125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83203125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.609375) - self_distillation/token_reweight_w_mean:np.float64(4577.494679181604) - self_distillation/token_reweight_w_std:np.float64(75540.59034832189) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9987221618648618) - self_distillation/token_reweight_w_clip_frac:np.float64(0.11705390218412504) - self_distillation/empty_target_batch:np.float64(0.16796875) - actor/grad_norm:np.float64(0.3779703937470913) - perf/mfu/actor:np.float64(0.0522593230068327) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(195.6965789794922) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.66875) - val-aux/sciknoweval/reward/std@16:np.float64(0.17551919057486814) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7372904761904763) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.13660825245257055) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6003428571428571) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.15818906699634022) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6687047619047619) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.17558003827147262) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7877285714285714) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.09825969433764362) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5388809523809523) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.13247872360110166) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6810333333333333) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.13288868774672) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8258047619047618) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.07020796516216546) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.4855857142857143) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.10698787291527515) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6865809523809525) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.09029984972979069) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8547666666666667) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.04666779709351366) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.4396285714285714) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.07582314271837051) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6883333333333334) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.06437108221973292) - val-aux/sciknoweval/score/mean@16:np.float64(0.66875) - val-aux/sciknoweval/score/std@16:np.float64(0.17551919057486814) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7372904761904763) - val-aux/sciknoweval/score/best@2/std:np.float64(0.13660825245257055) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6003428571428571) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.15818906699634022) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6687047619047619) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.17558003827147262) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7877285714285714) - val-aux/sciknoweval/score/best@4/std:np.float64(0.09825969433764362) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5388809523809523) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.13247872360110166) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6810333333333333) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.13288868774672) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8258047619047618) - val-aux/sciknoweval/score/best@8/std:np.float64(0.07020796516216546) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.4855857142857143) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.10698787291527515) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6865809523809525) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.09029984972979069) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8547666666666667) - val-aux/sciknoweval/score/best@16/std:np.float64(0.04666779709351366) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.4396285714285714) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.07582314271837051) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6883333333333334) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.06437108221973292) - val-core/sciknoweval/acc/mean@16:np.float64(0.66875) - val-aux/sciknoweval/acc/std@16:np.float64(0.17551919057486814) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7372904761904763) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.13660825245257055) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6003428571428571) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.15818906699634022) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6687047619047619) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.17558003827147262) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7877285714285714) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.09825969433764362) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5388809523809523) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.13247872360110166) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6810333333333333) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.13288868774672) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8258047619047618) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.07020796516216546) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.4855857142857143) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.10698787291527515) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6865809523809525) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.09029984972979069) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8547666666666667) - val-core/sciknoweval/acc/best@16/std:np.float64(0.04666779709351366) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.4396285714285714) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.07582314271837051) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6883333333333334) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.06437108221973292) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9985119047619048) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.005763368074713418) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9998666666666667) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.001758076041122951) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9971714285714286) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.007695791728287569) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9986190476190475) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.00554804164831311) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9947428571428573) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.00986918416859868) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9995904761904763) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.003032413337024054) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9906380952380953) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.011629673038340291) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9999619047619047) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0006991297950679286) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9847428571428571) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.011415170996457474) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.00015050933932646775) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:50 - training/epoch:0 - critic/score/mean:0.609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0015592515701428056 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0015592515701428056 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:398.328125 - response_length/max:872.0 - response_length/min:157.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:398.328125 - response_length_non_aborted/max:872.0 - response_length_non_aborted/min:157.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.75 - prompt_length/max:900.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.404837429523468e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0187409687787294) - timing_s/agent_loop/generate_sequences/max:np.float64(7.631963180378079) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.57371993160632) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.631963180378079) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:345 - timing_s/agent_loop/slowest/response_length:872 - timing_s/gen:14.39490077085793 - timing_s/reward:0.06266830302774906 - timing_s/old_log_prob:3.103602586314082 - timing_s/adv:0.5463018678128719 - timing_s/update_actor:21.542298316955566 - timing_s/step:39.73873019963503 - timing_s/testing:133.93844243697822 - timing_s/save_checkpoint:13.662320464849472 - timing_s/stop_profile:0.0001660715788602829 - timing_per_token_ms/update_actor:0.12410015851876607 - timing_per_token_ms/gen:0.14116522938510503 - timing_per_token_ms/adv:0.003147117702910754 - perf/total_num_tokens:173588 - perf/time_per_step:39.73873019963503 - perf/throughput:546.0290223415162
(TaskRunner pid=2255397) Training Progress: 50%|█████ | 50/100 [47:02<1:11:07, 85.35s/it]
(TaskRunner pid=2255397) step:51 - global_seqlen/min:19988 - global_seqlen/max:24236 - global_seqlen/minmax_diff:4248 - global_seqlen/balanced_min:21993 - global_seqlen/balanced_max:22000 - global_seqlen/mean:21997.625 - actor/entropy:0.47643905878067017 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.6292527914047241 - training/rollout_probs_diff_mean:0.004974652547389269 - training/rollout_probs_diff_std:0.010949385352432728 - training/rollout_actor_probs_pearson_corr:0.9992679953575134 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83203125 - self_distillation/correct_sample_fraction:0.5546875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83203125 - rollout_corr/rollout_is_mean:1.000044584274292 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.0419596947031096e-05 - rollout_corr/rollout_is_max:1.6338132619857788 - rollout_corr/rollout_is_min:0.19046327471733093 - rollout_corr/rollout_is_std:0.03751512989401817 - rollout_corr/rollout_is_eff_sample_size:0.998594711950472 - rollout_corr/rollout_is_seq_mean:0.9999472498893738 - rollout_corr/rollout_is_seq_std:0.002286979230120778 - rollout_corr/rollout_is_seq_max:1.0055614709854126 - rollout_corr/rollout_is_seq_min:0.9931483864784241 - rollout_corr/rollout_is_seq_max_deviation:0.006851613521575928 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6695182812400162) - rollout_corr/training_log_ppl:np.float64(0.49964700039708987) - rollout_corr/kl:np.float64(0.0010526214147859037) - rollout_corr/k3_kl:np.float64(0.0008514240415706809) - rollout_corr/rollout_ppl:np.float64(1.6677322825416923) - rollout_corr/rollout_log_ppl:np.float64(0.49859437631675974) - rollout_corr/log_ppl_diff:np.float64(0.0010526240803301334) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019851572578772902) - rollout_corr/log_ppl_diff_max:np.float64(0.008082211017608643) - rollout_corr/log_ppl_diff_min:np.float64(-0.008058786392211914) - rollout_corr/ppl_ratio:np.float64(1.0010559679940343) - rollout_corr/chi2_token:np.float64(0.0013342886231839657) - rollout_corr/chi2_seq:np.float64(1.1409833203069866) - actor/pg_loss:np.float64(0.005735161481425166) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006951400860089052) - actor/ppo_kl:np.float64(0.00024209431599508946) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83203125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83203125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5546875) - self_distillation/token_reweight_w_mean:np.float64(7385.579081017524) - self_distillation/token_reweight_w_std:np.float64(136267.17127367936) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0020049593877047) - self_distillation/token_reweight_w_clip_frac:np.float64(0.11803711034008302) - self_distillation/empty_target_batch:np.float64(0.16796875) - actor/grad_norm:np.float64(0.3889169320464134) - perf/mfu/actor:np.float64(0.053154140926217126) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.41073989868164) - actor/lr:np.float64(1e-06) - training/global_step:51 - training/epoch:0 - critic/score/mean:0.5546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0006512248073704541 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0006512248073704541 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:374.89453125 - response_length/max:928.0 - response_length/min:141.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:374.89453125 - response_length_non_aborted/max:928.0 - response_length_non_aborted/min:141.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.53125 - prompt_length/max:608.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013144873082637787 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7525868192315102) - timing_s/agent_loop/generate_sequences/max:np.float64(7.869243094697595) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.208094254077878) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.869243094697595) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:226 - timing_s/agent_loop/slowest/response_length:928 - timing_s/gen:14.724404215812683 - timing_s/reward:0.059697989374399185 - timing_s/old_log_prob:3.061056010425091 - timing_s/adv:0.5633108653128147 - timing_s/update_actor:21.545322271063924 - timing_s/step:40.06024991162121 - timing_s/stop_profile:0.00010966137051582336 - timing_per_token_ms/update_actor:0.12242982066850355 - timing_per_token_ms/gen:0.15342236062030656 - timing_per_token_ms/adv:0.00320097547640265 - perf/total_num_tokens:175981 - perf/time_per_step:40.06024991162121 - perf/throughput:549.1135239677732
(TaskRunner pid=2255397) Training Progress: 51%|█████ | 51/100 [47:42<58:37, 71.78s/it]
(TaskRunner pid=2255397) step:52 - global_seqlen/min:16418 - global_seqlen/max:23760 - global_seqlen/minmax_diff:7342 - global_seqlen/balanced_min:20391 - global_seqlen/balanced_max:20400 - global_seqlen/mean:20398.0 - actor/entropy:0.4327685832977295 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4141288995742798 - training/rollout_probs_diff_mean:0.004852033220231533 - training/rollout_probs_diff_std:0.011078495532274246 - training/rollout_actor_probs_pearson_corr:0.9992164969444275 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.859375 - self_distillation/correct_sample_fraction:0.61328125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.859375 - rollout_corr/rollout_is_mean:0.999881386756897 - rollout_corr/rollout_is_ratio_fraction_high:1.0429704161651898e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.107421636581421 - rollout_corr/rollout_is_min:0.5139061212539673 - rollout_corr/rollout_is_std:0.03718959912657738 - rollout_corr/rollout_is_eff_sample_size:0.9986184874121644 - rollout_corr/rollout_is_seq_mean:0.9998266696929932 - rollout_corr/rollout_is_seq_std:0.0023117384407669306 - rollout_corr/rollout_is_seq_max:1.010014533996582 - rollout_corr/rollout_is_seq_min:0.9893986582756042 - rollout_corr/rollout_is_seq_max_deviation:0.010601341724395752 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5747977024875581) - rollout_corr/training_log_ppl:np.float64(0.44207731005735695) - rollout_corr/kl:np.float64(0.0007742779266513011) - rollout_corr/k3_kl:np.float64(0.0008074032141394127) - rollout_corr/rollout_ppl:np.float64(1.5735519244335592) - rollout_corr/rollout_log_ppl:np.float64(0.44130302700796165) - rollout_corr/log_ppl_diff:np.float64(0.0007742830493953079) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017560763808432966) - rollout_corr/log_ppl_diff_max:np.float64(0.011528313159942627) - rollout_corr/log_ppl_diff_min:np.float64(-0.008214741945266724) - rollout_corr/ppl_ratio:np.float64(1.0007772489916533) - rollout_corr/chi2_token:np.float64(0.0016987936105579138) - rollout_corr/chi2_seq:np.float64(0.6159643346909434) - actor/pg_loss:np.float64(0.004326079739257693) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003785648150369525) - actor/ppo_kl:np.float64(-0.00013583274305517534) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.859375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.859375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.61328125) - self_distillation/token_reweight_w_mean:np.float64(23087.692809279542) - self_distillation/token_reweight_w_std:np.float64(411971.67904203804) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0013474761508405) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10722107632318512) - self_distillation/empty_target_batch:np.float64(0.140625) - actor/grad_norm:np.float64(0.31624583154916763) - perf/mfu/actor:np.float64(0.04994189212775807) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.2727928161621) - actor/lr:np.float64(1e-06) - training/global_step:52 - training/epoch:0 - critic/score/mean:0.61328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.61328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00683667091652751 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00683667091652751 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:374.53125 - response_length/max:748.0 - response_length/min:127.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:374.53125 - response_length_non_aborted/max:748.0 - response_length_non_aborted/min:127.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.90625 - prompt_length/max:453.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.608851253986359e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5058961473405361) - timing_s/agent_loop/generate_sequences/max:np.float64(6.993606846779585) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.290860692432034) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.993606846779585) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:260 - timing_s/agent_loop/slowest/response_length:748 - timing_s/gen:13.7207720130682 - timing_s/reward:0.060612039640545845 - timing_s/old_log_prob:2.9474493078887463 - timing_s/adv:0.5020784568041563 - timing_s/update_actor:21.135599818080664 - timing_s/step:38.45533044077456 - timing_s/stop_profile:0.0001184120774269104 - timing_per_token_ms/update_actor:0.1295200498705796 - timing_per_token_ms/gen:0.14310358795440342 - timing_per_token_ms/adv:0.0030767627757877997 - perf/total_num_tokens:163184 - perf/time_per_step:38.45533044077456 - perf/throughput:530.4336165155352
(TaskRunner pid=2255397) Training Progress: 52%|█████▏ | 52/100 [48:21<49:26, 61.79s/it]
(TaskRunner pid=2255397) step:53 - global_seqlen/min:19241 - global_seqlen/max:23446 - global_seqlen/minmax_diff:4205 - global_seqlen/balanced_min:21257 - global_seqlen/balanced_max:21274 - global_seqlen/mean:21261.625 - actor/entropy:0.4501931071281433 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2294929027557373 - training/rollout_probs_diff_mean:0.0048794192261993885 - training/rollout_probs_diff_std:0.010888302698731422 - training/rollout_actor_probs_pearson_corr:0.9992465376853943 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.73828125 - self_distillation/correct_sample_fraction:0.62109375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.73828125 - rollout_corr/rollout_is_mean:0.9998956918716431 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.1179934265092015e-05 - rollout_corr/rollout_is_max:1.6338140964508057 - rollout_corr/rollout_is_min:0.37676042318344116 - rollout_corr/rollout_is_std:0.036510247737169266 - rollout_corr/rollout_is_eff_sample_size:0.9986683006749928 - rollout_corr/rollout_is_seq_mean:0.9998424053192139 - rollout_corr/rollout_is_seq_std:0.0020514947827905416 - rollout_corr/rollout_is_seq_max:1.0057796239852905 - rollout_corr/rollout_is_seq_min:0.9921916127204895 - rollout_corr/rollout_is_seq_max_deviation:0.007808387279510498 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6163378930650651) - rollout_corr/training_log_ppl:np.float64(0.4686294782150071) - rollout_corr/kl:np.float64(0.000762243592463463) - rollout_corr/k3_kl:np.float64(0.000810159646334796) - rollout_corr/rollout_ppl:np.float64(1.6150716617703438) - rollout_corr/rollout_log_ppl:np.float64(0.46786723379045725) - rollout_corr/log_ppl_diff:np.float64(0.0007622444245498627) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019291590724606067) - rollout_corr/log_ppl_diff_max:np.float64(0.008153051137924194) - rollout_corr/log_ppl_diff_min:np.float64(-0.005449235439300537) - rollout_corr/ppl_ratio:np.float64(1.000765364151448) - rollout_corr/chi2_token:np.float64(0.001715675462037325) - rollout_corr/chi2_seq:np.float64(0.6910448384005576) - actor/pg_loss:np.float64(0.0023863104870542884) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002726298084780865) - actor/ppo_kl:np.float64(-0.00011610987491650349) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.73828125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.73828125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.62109375) - self_distillation/token_reweight_w_mean:np.float64(1507.4308137190528) - self_distillation/token_reweight_w_std:np.float64(33027.9989340055) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9995625410228968) - self_distillation/token_reweight_w_clip_frac:np.float64(0.06903543486259878) - self_distillation/empty_target_batch:np.float64(0.26171875) - actor/grad_norm:np.float64(0.21907289698719978) - perf/mfu/actor:np.float64(0.051939957351732395) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.24185180664062) - actor/lr:np.float64(1e-06) - training/global_step:53 - training/epoch:0 - critic/score/mean:0.62109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.003284213598817587 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.003284213598817587 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:368.86328125 - response_length/max:1113.0 - response_length/min:100.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:368.86328125 - response_length_non_aborted/max:1113.0 - response_length_non_aborted/min:100.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:295.5625 - prompt_length/max:532.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.354894816875458e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2522868532687426) - timing_s/agent_loop/generate_sequences/max:np.float64(8.869224239140749) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.229014496719174) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.869224239140749) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:227 - timing_s/agent_loop/slowest/response_length:1113 - timing_s/gen:15.673873921856284 - timing_s/reward:0.061943380162119865 - timing_s/old_log_prob:2.946569699794054 - timing_s/adv:0.5094821080565453 - timing_s/update_actor:21.332264576107264 - timing_s/step:40.6122511010617 - timing_s/stop_profile:0.00011801905930042267 - timing_per_token_ms/update_actor:0.1254152997248991 - timing_per_token_ms/gen:0.16598580861659326 - timing_per_token_ms/adv:0.0029953149633232716 - perf/total_num_tokens:170093 - perf/time_per_step:40.6122511010617 - perf/throughput:523.5273697853249
(TaskRunner pid=2255397) Training Progress: 53%|█████▎ | 53/100 [49:01<43:26, 55.45s/it]
(TaskRunner pid=2255397) step:54 - global_seqlen/min:16771 - global_seqlen/max:23434 - global_seqlen/minmax_diff:6663 - global_seqlen/balanced_min:20121 - global_seqlen/balanced_max:20134 - global_seqlen/mean:20129.375 - actor/entropy:0.39255428314208984 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2144627422094345 - training/rollout_probs_diff_mean:0.004672019276767969 - training/rollout_probs_diff_std:0.011115854606032372 - training/rollout_actor_probs_pearson_corr:0.9991840720176697 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.8046875 - self_distillation/correct_sample_fraction:0.5546875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8046875 - rollout_corr/rollout_is_mean:0.9999427795410156 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.969622254371643 - rollout_corr/rollout_is_min:0.5503029227256775 - rollout_corr/rollout_is_std:0.03688544034957886 - rollout_corr/rollout_is_eff_sample_size:0.998641194034968 - rollout_corr/rollout_is_seq_mean:0.9999832510948181 - rollout_corr/rollout_is_seq_std:0.002377620432525873 - rollout_corr/rollout_is_seq_max:1.0109039545059204 - rollout_corr/rollout_is_seq_min:0.9921522736549377 - rollout_corr/rollout_is_seq_max_deviation:0.01090395450592041 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5304169207811356) - rollout_corr/training_log_ppl:np.float64(0.4121860411542002) - rollout_corr/kl:np.float64(0.0008754907883350427) - rollout_corr/k3_kl:np.float64(0.0008680712330715323) - rollout_corr/rollout_ppl:np.float64(1.5290372776798904) - rollout_corr/rollout_log_ppl:np.float64(0.411310548952315) - rollout_corr/log_ppl_diff:np.float64(0.0008754922018852085) - rollout_corr/log_ppl_abs_diff:np.float64(0.002070396934868768) - rollout_corr/log_ppl_diff_max:np.float64(0.016439110040664673) - rollout_corr/log_ppl_diff_min:np.float64(-0.00867973268032074) - rollout_corr/ppl_ratio:np.float64(1.0008794867899269) - rollout_corr/chi2_token:np.float64(0.0017218261491507292) - rollout_corr/chi2_seq:np.float64(0.5381895105820149) - actor/pg_loss:np.float64(0.006413624971173704) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005543413315081125) - actor/ppo_kl:np.float64(9.862682850325655e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8046875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8046875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5546875) - self_distillation/token_reweight_w_mean:np.float64(21560.225262402324) - self_distillation/token_reweight_w_std:np.float64(340430.1742267248) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0004913297016174) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10845586232608184) - self_distillation/empty_target_batch:np.float64(0.1953125) - actor/grad_norm:np.float64(0.3543579652905464) - perf/mfu/actor:np.float64(0.04868076222396018) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.25546646118164) - actor/lr:np.float64(1e-06) - training/global_step:54 - training/epoch:0 - critic/score/mean:0.5546875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5546875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.011497573927044868 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.011497573927044868 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:358.26171875 - response_length/max:1148.0 - response_length/min:105.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:358.26171875 - response_length_non_aborted/max:1148.0 - response_length_non_aborted/min:105.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:270.78125 - prompt_length/max:597.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010633282363414764 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4096278883516788) - timing_s/agent_loop/generate_sequences/max:np.float64(9.100342744961381) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.021014133017161) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.100342744961381) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:219 - timing_s/agent_loop/slowest/response_length:1148 - timing_s/gen:15.916911805048585 - timing_s/reward:0.059211257845163345 - timing_s/old_log_prob:3.0376540310680866 - timing_s/adv:0.5192954838275909 - timing_s/update_actor:21.510440975427628 - timing_s/step:41.131219344213605 - timing_s/stop_profile:3.115646541118622e-05 - timing_per_token_ms/update_actor:0.13357618514874176 - timing_per_token_ms/gen:0.17354753099327902 - timing_per_token_ms/adv:0.003224736758019008 - perf/total_num_tokens:161035 - perf/time_per_step:41.131219344213605 - perf/throughput:489.3940739160661
(TaskRunner pid=2255397) Training Progress: 54%|█████▍ | 54/100 [49:42<39:13, 51.16s/it]
(TaskRunner pid=2255397) step:55 - global_seqlen/min:19138 - global_seqlen/max:21905 - global_seqlen/minmax_diff:2767 - global_seqlen/balanced_min:20379 - global_seqlen/balanced_max:20392 - global_seqlen/mean:20387.375 - actor/entropy:0.46607786417007446 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.44003328680992126 - training/rollout_probs_diff_mean:0.005262596998363733 - training/rollout_probs_diff_std:0.011458019725978374 - training/rollout_actor_probs_pearson_corr:0.9992002844810486 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.8046875 - self_distillation/correct_sample_fraction:0.640625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8046875 - rollout_corr/rollout_is_mean:1.0000497102737427 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.523318446241319e-05 - rollout_corr/rollout_is_max:1.6244064569473267 - rollout_corr/rollout_is_min:0.396142840385437 - rollout_corr/rollout_is_std:0.03780318796634674 - rollout_corr/rollout_is_eff_sample_size:0.99857307726417 - rollout_corr/rollout_is_seq_mean:0.9999815225601196 - rollout_corr/rollout_is_seq_std:0.002393631497398019 - rollout_corr/rollout_is_seq_max:1.0126123428344727 - rollout_corr/rollout_is_seq_min:0.9941840171813965 - rollout_corr/rollout_is_seq_max_deviation:0.012612342834472656 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.630201779305935) - rollout_corr/training_log_ppl:np.float64(0.47879417467629537) - rollout_corr/kl:np.float64(0.0010309481362873463) - rollout_corr/k3_kl:np.float64(0.0008061554526079817) - rollout_corr/rollout_ppl:np.float64(1.6284840498119593) - rollout_corr/rollout_log_ppl:np.float64(0.4777632247423753) - rollout_corr/log_ppl_diff:np.float64(0.0010309499339200556) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020579289994202554) - rollout_corr/log_ppl_diff_max:np.float64(0.007941722869873047) - rollout_corr/log_ppl_diff_min:np.float64(-0.0057179927825927734) - rollout_corr/ppl_ratio:np.float64(1.0010342469904572) - rollout_corr/chi2_token:np.float64(0.00116553227417171) - rollout_corr/chi2_seq:np.float64(0.674807739444077) - actor/pg_loss:np.float64(0.004637886886484921) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00022305224865704076) - actor/ppo_kl:np.float64(0.0002447084712073888) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8046875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8046875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.640625) - self_distillation/token_reweight_w_mean:np.float64(5661.525033625308) - self_distillation/token_reweight_w_std:np.float64(108029.05970195204) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0004851918201894) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08942344659590162) - self_distillation/empty_target_batch:np.float64(0.1953125) - actor/grad_norm:np.float64(0.34092599898576736) - perf/mfu/actor:np.float64(0.04916186108901947) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.24573516845703) - actor/lr:np.float64(1e-06) - training/global_step:55 - training/epoch:0 - critic/score/mean:0.640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0027775494381785393 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0027775494381785393 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:332.60546875 - response_length/max:929.0 - response_length/min:101.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:332.60546875 - response_length_non_aborted/max:929.0 - response_length_non_aborted/min:101.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:304.5 - prompt_length/max:625.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.5229528546333313e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2314282320439816) - timing_s/agent_loop/generate_sequences/max:np.float64(7.571410246193409) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.8092300324497046) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.571410246193409) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:337 - timing_s/agent_loop/slowest/response_length:895 - timing_s/gen:14.387957438826561 - timing_s/reward:0.05958708934485912 - timing_s/old_log_prob:2.9241322185844183 - timing_s/adv:0.5914540458470583 - timing_s/update_actor:21.573941450566053 - timing_s/step:39.624077497050166 - timing_s/stop_profile:0.0001155603677034378 - timing_per_token_ms/update_actor:0.13227513013915507 - timing_per_token_ms/gen:0.16897785522480605 - timing_per_token_ms/adv:0.003626349921502022 - perf/total_num_tokens:163099 - perf/time_per_step:39.624077497050166 - perf/throughput:514.5198648856305
(TaskRunner pid=2255397) Training Progress: 55%|█████▌ | 55/100 [50:22<35:47, 47.71s/it]
(TaskRunner pid=2255397) step:56 - global_seqlen/min:16165 - global_seqlen/max:23611 - global_seqlen/minmax_diff:7446 - global_seqlen/balanced_min:19603 - global_seqlen/balanced_max:19611 - global_seqlen/mean:19608.875 - actor/entropy:0.42714759707450867 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2066904902458191 - training/rollout_probs_diff_mean:0.004840976092964411 - training/rollout_probs_diff_std:0.011059464886784554 - training/rollout_actor_probs_pearson_corr:0.9992184638977051 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.87109375 - self_distillation/correct_sample_fraction:0.6484375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.87109375 - rollout_corr/rollout_is_mean:1.0001850128173828 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.7041244506835938 - rollout_corr/rollout_is_min:0.5909921526908875 - rollout_corr/rollout_is_std:0.03720882534980774 - rollout_corr/rollout_is_eff_sample_size:0.9986178930118242 - rollout_corr/rollout_is_seq_mean:1.0002026557922363 - rollout_corr/rollout_is_seq_std:0.0023563196882605553 - rollout_corr/rollout_is_seq_max:1.0108153820037842 - rollout_corr/rollout_is_seq_min:0.9901830554008484 - rollout_corr/rollout_is_seq_max_deviation:0.01081538200378418 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5864489898085594) - rollout_corr/training_log_ppl:np.float64(0.4489033415738959) - rollout_corr/kl:np.float64(0.0005826077157036202) - rollout_corr/k3_kl:np.float64(0.0008374885150033151) - rollout_corr/rollout_ppl:np.float64(1.5854855361394584) - rollout_corr/rollout_log_ppl:np.float64(0.4483207310840953) - rollout_corr/log_ppl_diff:np.float64(0.0005826104898005724) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019630271708592772) - rollout_corr/log_ppl_diff_max:np.float64(0.011552989482879639) - rollout_corr/log_ppl_diff_min:np.float64(-0.00930735468864441) - rollout_corr/ppl_ratio:np.float64(1.0005861730314791) - rollout_corr/chi2_token:np.float64(0.0022083548828959465) - rollout_corr/chi2_seq:np.float64(1.1513643299695104) - actor/pg_loss:np.float64(0.0053109838627278805) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004679971348195977) - actor/ppo_kl:np.float64(4.2931282393965375e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.87109375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.87109375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6484375) - self_distillation/token_reweight_w_mean:np.float64(133.22875495348126) - self_distillation/token_reweight_w_std:np.float64(2217.535897645983) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0020651405211538) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08932297839783132) - self_distillation/empty_target_batch:np.float64(0.12890625) - actor/grad_norm:np.float64(0.3138376846909523) - perf/mfu/actor:np.float64(0.04696252700818672) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.17254257202148) - actor/lr:np.float64(1e-06) - training/global_step:56 - training/epoch:0 - critic/score/mean:0.6484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004433032590895891 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:-0.004433032590895891 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:347.62109375 - response_length/max:909.0 - response_length/min:143.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:347.62109375 - response_length_non_aborted/max:909.0 - response_length_non_aborted/min:143.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.15625 - prompt_length/max:458.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.741447865962982e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4314610064029694) - timing_s/agent_loop/generate_sequences/max:np.float64(7.775194456800818) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.5458272197138285) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.775194456800818) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:230 - timing_s/agent_loop/slowest/response_length:909 - timing_s/gen:14.528320418670774 - timing_s/reward:0.0595672782510519 - timing_s/old_log_prob:2.9559899885207415 - timing_s/adv:0.5271639805287123 - timing_s/update_actor:21.64877647906542 - timing_s/step:39.80573454126716 - timing_s/stop_profile:2.855248749256134e-05 - timing_per_token_ms/update_actor:0.13800368760998158 - timing_per_token_ms/gen:0.16325606430617448 - timing_per_token_ms/adv:0.00336049352989853 - perf/total_num_tokens:156871 - perf/time_per_step:39.80573454126716 - perf/throughput:492.61432368924653
(TaskRunner pid=2255397) Training Progress: 56%|█████▌ | 56/100 [51:02<33:15, 45.35s/it]
(TaskRunner pid=2255397) step:57 - global_seqlen/min:17187 - global_seqlen/max:22734 - global_seqlen/minmax_diff:5547 - global_seqlen/balanced_min:19426 - global_seqlen/balanced_max:19436 - global_seqlen/mean:19433.875 - actor/entropy:0.419066846370697 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.45647868514060974 - training/rollout_probs_diff_mean:0.005257493816316128 - training/rollout_probs_diff_std:0.012042389251291752 - training/rollout_actor_probs_pearson_corr:0.9990561008453369 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83203125 - self_distillation/correct_sample_fraction:0.66796875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83203125 - rollout_corr/rollout_is_mean:0.9998766779899597 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.874779187957756e-05 - rollout_corr/rollout_is_max:1.9517626762390137 - rollout_corr/rollout_is_min:0.412737101316452 - rollout_corr/rollout_is_std:0.039478376507759094 - rollout_corr/rollout_is_eff_sample_size:0.9984436453453855 - rollout_corr/rollout_is_seq_mean:0.9998682141304016 - rollout_corr/rollout_is_seq_std:0.002758133690804243 - rollout_corr/rollout_is_seq_max:1.010321855545044 - rollout_corr/rollout_is_seq_min:0.9932553768157959 - rollout_corr/rollout_is_seq_max_deviation:0.010321855545043945 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5485623660497367) - rollout_corr/training_log_ppl:np.float64(0.4282529861666262) - rollout_corr/kl:np.float64(0.0008480568455153659) - rollout_corr/k3_kl:np.float64(0.0008880625406391118) - rollout_corr/rollout_ppl:np.float64(1.5472203940153122) - rollout_corr/rollout_log_ppl:np.float64(0.42740493212477304) - rollout_corr/log_ppl_diff:np.float64(0.0008480540418531746) - rollout_corr/log_ppl_abs_diff:np.float64(0.002164474077289924) - rollout_corr/log_ppl_diff_max:np.float64(0.007681757211685181) - rollout_corr/log_ppl_diff_min:np.float64(-0.008211076259613037) - rollout_corr/ppl_ratio:np.float64(1.0008519866969436) - rollout_corr/chi2_token:np.float64(0.0018828820902854204) - rollout_corr/chi2_seq:np.float64(1.0425616323482245) - actor/pg_loss:np.float64(0.002906394307501614) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00029778121393064794) - actor/ppo_kl:np.float64(-0.00014163734993388744) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83203125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83203125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.66796875) - self_distillation/token_reweight_w_mean:np.float64(3090.494025749387) - self_distillation/token_reweight_w_std:np.float64(77747.68742029642) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9994963144417852) - self_distillation/token_reweight_w_clip_frac:np.float64(0.06993347342358902) - self_distillation/empty_target_batch:np.float64(0.16796875) - actor/grad_norm:np.float64(0.22349200583994389) - perf/mfu/actor:np.float64(0.04662792309159809) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.20368194580078) - actor/lr:np.float64(1e-06) - training/global_step:57 - training/epoch:0 - critic/score/mean:0.66796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.66796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0013542745728045702 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0013542745728045702 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:320.52734375 - response_length/max:1046.0 - response_length/min:113.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:320.52734375 - response_length_non_aborted/max:1046.0 - response_length_non_aborted/min:113.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.78125 - prompt_length/max:660.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.3910775780677795e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5050442274659872) - timing_s/agent_loop/generate_sequences/max:np.float64(8.29741439782083) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.6069312687322963) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.29741439782083) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:242 - timing_s/agent_loop/slowest/response_length:1046 - timing_s/gen:15.101491201668978 - timing_s/reward:0.05853153392672539 - timing_s/old_log_prob:2.997370980679989 - timing_s/adv:0.5432093199342489 - timing_s/update_actor:21.456135919317603 - timing_s/step:40.24460173584521 - timing_s/stop_profile:0.00010164454579353333 - timing_per_token_ms/update_actor:0.13800731917410708 - timing_per_token_ms/gen:0.18404108465869207 - timing_per_token_ms/adv:0.003493959130218812 - perf/total_num_tokens:155471 - perf/time_per_step:40.24460173584521 - perf/throughput:482.89395749419396
(TaskRunner pid=2255397) Training Progress: 57%|█████▋ | 57/100 [51:42<31:24, 43.83s/it]
(TaskRunner pid=2255397) step:58 - global_seqlen/min:17591 - global_seqlen/max:22208 - global_seqlen/minmax_diff:4617 - global_seqlen/balanced_min:19936 - global_seqlen/balanced_max:19980 - global_seqlen/mean:19944.0 - actor/entropy:0.39805784821510315 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5443886518478394 - training/rollout_probs_diff_mean:0.0047958018258214 - training/rollout_probs_diff_std:0.011325978673994541 - training/rollout_actor_probs_pearson_corr:0.9991068243980408 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.8671875 - self_distillation/correct_sample_fraction:0.703125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8671875 - rollout_corr/rollout_is_mean:0.9999314546585083 - rollout_corr/rollout_is_ratio_fraction_high:1.1276500117674004e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.1276500117674004e-05 - rollout_corr/rollout_is_max:2.245145320892334 - rollout_corr/rollout_is_min:0.3792571425437927 - rollout_corr/rollout_is_std:0.037710048258304596 - rollout_corr/rollout_is_eff_sample_size:0.9985797339908784 - rollout_corr/rollout_is_seq_mean:0.9998851418495178 - rollout_corr/rollout_is_seq_std:0.002318696118891239 - rollout_corr/rollout_is_seq_max:1.007555365562439 - rollout_corr/rollout_is_seq_min:0.9894842505455017 - rollout_corr/rollout_is_seq_max_deviation:0.010515749454498291 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5350726060569286) - rollout_corr/training_log_ppl:np.float64(0.4149510355782695) - rollout_corr/kl:np.float64(0.0008157491595808253) - rollout_corr/k3_kl:np.float64(0.0008106242991630097) - rollout_corr/rollout_ppl:np.float64(1.533764130435884) - rollout_corr/rollout_log_ppl:np.float64(0.41413528152043) - rollout_corr/log_ppl_diff:np.float64(0.0008157540578395128) - rollout_corr/log_ppl_abs_diff:np.float64(0.00186522479634732) - rollout_corr/log_ppl_diff_max:np.float64(0.012087315320968628) - rollout_corr/log_ppl_diff_min:np.float64(-0.005733311176300049) - rollout_corr/ppl_ratio:np.float64(1.0008189915679395) - rollout_corr/chi2_token:np.float64(0.0016144579276442528) - rollout_corr/chi2_seq:np.float64(0.5739311659708619) - actor/pg_loss:np.float64(0.0032648365013301373) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00039790487971913535) - actor/ppo_kl:np.float64(-6.927822899172043e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8671875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8671875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.703125) - self_distillation/token_reweight_w_mean:np.float64(20.40596836898476) - self_distillation/token_reweight_w_std:np.float64(398.28047909680754) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0010924299713224) - self_distillation/token_reweight_w_clip_frac:np.float64(0.060014044371200725) - self_distillation/empty_target_batch:np.float64(0.1328125) - actor/grad_norm:np.float64(0.22483210638165474) - perf/mfu/actor:np.float64(0.048109296920814684) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.0834503173828) - actor/lr:np.float64(1e-06) - training/global_step:58 - training/epoch:0 - critic/score/mean:0.703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00522806728258729 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00522806728258729 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:346.40625 - response_length/max:1083.0 - response_length/min:134.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:346.40625 - response_length_non_aborted/max:1083.0 - response_length_non_aborted/min:134.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:276.84375 - prompt_length/max:598.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012578070163726807 - timing_s/agent_loop/generate_sequences/min:np.float64(1.711790930479765) - timing_s/agent_loop/generate_sequences/max:np.float64(8.412942063063383) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.874052923943964) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.412942063063383) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:312 - timing_s/agent_loop/slowest/response_length:1083 - timing_s/gen:14.847420612350106 - timing_s/reward:0.05969010666012764 - timing_s/old_log_prob:2.983126612380147 - timing_s/adv:0.5337398238480091 - timing_s/update_actor:21.578375332057476 - timing_s/step:40.09062473848462 - timing_s/stop_profile:0.00010734237730503082 - timing_per_token_ms/update_actor:0.1352435277029274 - timing_per_token_ms/gen:0.16742693518662727 - timing_per_token_ms/adv:0.003345240572653487 - perf/total_num_tokens:159552 - perf/time_per_step:40.09062473848462 - perf/throughput:497.4729161767076
(TaskRunner pid=2255397) Training Progress: 58%|█████▊ | 58/100 [52:22<29:54, 42.72s/it]
(TaskRunner pid=2255397) step:59 - global_seqlen/min:16516 - global_seqlen/max:23158 - global_seqlen/minmax_diff:6642 - global_seqlen/balanced_min:19285 - global_seqlen/balanced_max:19288 - global_seqlen/mean:19286.5 - actor/entropy:0.39392659068107605 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23105931282043457 - training/rollout_probs_diff_mean:0.004885942675173283 - training/rollout_probs_diff_std:0.01138182170689106 - training/rollout_actor_probs_pearson_corr:0.9991214275360107 - self_distillation/success_group_fraction:0.65625 - self_distillation/success_sample_fraction:0.65234375 - self_distillation/correct_sample_fraction:0.5390625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.65234375 - rollout_corr/rollout_is_mean:0.9998358488082886 - rollout_corr/rollout_is_ratio_fraction_high:1.1690437531797215e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.1690437531797215e-05 - rollout_corr/rollout_is_max:3.892078161239624 - rollout_corr/rollout_is_min:0.4614630937576294 - rollout_corr/rollout_is_std:0.03779766708612442 - rollout_corr/rollout_is_eff_sample_size:0.9985728395255719 - rollout_corr/rollout_is_seq_mean:0.999951183795929 - rollout_corr/rollout_is_seq_std:0.002800993388518691 - rollout_corr/rollout_is_seq_max:1.0143204927444458 - rollout_corr/rollout_is_seq_min:0.9928278923034668 - rollout_corr/rollout_is_seq_max_deviation:0.0143204927444458 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5359033830463886) - rollout_corr/training_log_ppl:np.float64(0.4180586190195754) - rollout_corr/kl:np.float64(0.0011108192077600165) - rollout_corr/k3_kl:np.float64(0.0008730394356462057) - rollout_corr/rollout_ppl:np.float64(1.5341403023339808) - rollout_corr/rollout_log_ppl:np.float64(0.41694779935642146) - rollout_corr/log_ppl_diff:np.float64(0.0011108196631539613) - rollout_corr/log_ppl_abs_diff:np.float64(0.002291953394887969) - rollout_corr/log_ppl_diff_max:np.float64(0.008277744054794312) - rollout_corr/log_ppl_diff_min:np.float64(-0.007644861936569214) - rollout_corr/ppl_ratio:np.float64(1.0011151337530464) - rollout_corr/chi2_token:np.float64(0.0013092008884996176) - rollout_corr/chi2_seq:np.float64(0.4192327775526792) - actor/pg_loss:np.float64(0.0025457896990701556) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003396774759494292) - actor/ppo_kl:np.float64(0.0002558892238049282) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.65234375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.65234375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5390625) - self_distillation/token_reweight_w_mean:np.float64(9165.766619424336) - self_distillation/token_reweight_w_std:np.float64(181822.19756039116) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9998559767846018) - self_distillation/token_reweight_w_clip_frac:np.float64(0.057435350317973644) - self_distillation/empty_target_batch:np.float64(0.34765625) - actor/grad_norm:np.float64(0.21796750277280807) - perf/mfu/actor:np.float64(0.0472734036832577) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(198.940327167511) - actor/lr:np.float64(1e-06) - training/global_step:59 - training/epoch:0 - critic/score/mean:0.5390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0026581131387501955 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0026581131387501955 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:334.140625 - response_length/max:859.0 - response_length/min:124.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:334.140625 - response_length_non_aborted/max:859.0 - response_length_non_aborted/min:124.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:268.5625 - prompt_length/max:615.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001378711313009262 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6116437073796988) - timing_s/agent_loop/generate_sequences/max:np.float64(7.250096688047051) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.7876410359021975) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.250096688047051) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:217 - timing_s/agent_loop/slowest/response_length:844 - timing_s/gen:14.891584478318691 - timing_s/reward:0.05919893458485603 - timing_s/old_log_prob:2.9019656274467707 - timing_s/adv:0.4909003674983978 - timing_s/update_actor:21.302226703613997 - timing_s/step:39.73218045942485 - timing_s/stop_profile:0.00011939182877540588 - timing_per_token_ms/update_actor:0.1380643630493739 - timing_per_token_ms/gen:0.17408913348513785 - timing_per_token_ms/adv:0.0031816320191480948 - perf/total_num_tokens:154292 - perf/time_per_step:39.73218045942485 - perf/throughput:485.41257431606823
(TaskRunner pid=2255397) Training Progress: 59%|█████▉ | 59/100 [53:02<28:35, 41.85s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 60}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1", we need to analyze the structure of the product and compare it with the structures of the given options.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product contains a nitrogen-containing ring with a substituent group: a chlorine-substituted benzene ring connected to a trifluoromethyl group (C(F)(F)F). This suggests that the correct reactant must contain a similar structure, specifically a nitrogen-containing ring with the same substituents.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Option A: "O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" contains a nitrogen-containing ring with the exact substituents: a chlorine-substituted benzene ring and a trifluoromethyl group. This matches the structure of the product exactly.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Option B: Contains a nitrogen but not the correct substituents.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Option C: Contains a hydroxyl group and a different substituent, not matching the product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Option D: Contains a different structure with a carbon chain and a fluorinated group, not matching the product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Therefore, the only correct reactant is Option A.
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_60
(TaskRunner pid=2255397) step:60 - global_seqlen/min:15969 - global_seqlen/max:20753 - global_seqlen/minmax_diff:4784 - global_seqlen/balanced_min:19004 - global_seqlen/balanced_max:19008 - global_seqlen/mean:19006.625 - actor/entropy:0.40371620655059814 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8045270442962646 - training/rollout_probs_diff_mean:0.005058089271187782 - training/rollout_probs_diff_std:0.01212889701128006 - training/rollout_actor_probs_pearson_corr:0.9990420341491699 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.859375 - self_distillation/correct_sample_fraction:0.59375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.859375 - rollout_corr/rollout_is_mean:0.9997034668922424 - rollout_corr/rollout_is_ratio_fraction_high:1.1908589840459172e-05 - rollout_corr/rollout_is_ratio_fraction_low:4.763435936183669e-05 - rollout_corr/rollout_is_max:2.487139940261841 - rollout_corr/rollout_is_min:0.0640438050031662 - rollout_corr/rollout_is_std:0.03801152855157852 - rollout_corr/rollout_is_eff_sample_size:0.9985564358356739 - rollout_corr/rollout_is_seq_mean:0.9996296167373657 - rollout_corr/rollout_is_seq_std:0.0025690472684800625 - rollout_corr/rollout_is_seq_max:1.0076467990875244 - rollout_corr/rollout_is_seq_min:0.9901750683784485 - rollout_corr/rollout_is_seq_max_deviation:0.009824931621551514 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.534271144773811) - rollout_corr/training_log_ppl:np.float64(0.41600528117851354) - rollout_corr/kl:np.float64(0.001141798264093552) - rollout_corr/k3_kl:np.float64(0.000918725037195145) - rollout_corr/rollout_ppl:np.float64(1.5324818124063313) - rollout_corr/rollout_log_ppl:np.float64(0.4148634799348656) - rollout_corr/log_ppl_diff:np.float64(0.001141801243647933) - rollout_corr/log_ppl_abs_diff:np.float64(0.0023173787049017847) - rollout_corr/log_ppl_diff_max:np.float64(0.009830474853515625) - rollout_corr/log_ppl_diff_min:np.float64(-0.006625443696975708) - rollout_corr/ppl_ratio:np.float64(1.0011462338734418) - rollout_corr/chi2_token:np.float64(0.0013863001950085163) - rollout_corr/chi2_seq:np.float64(0.6778646670281887) - actor/pg_loss:np.float64(0.005434742895886302) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0011875206650984182) - actor/ppo_kl:np.float64(-4.705816970229648e-05) - actor/pg_clipfrac_lower:np.float64(2.3182492441264912e-05) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.859375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.859375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.59375) - self_distillation/token_reweight_w_mean:np.float64(18888.49168556533) - self_distillation/token_reweight_w_std:np.float64(311593.95773308317) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0030431225895882) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10766072731348686) - self_distillation/empty_target_batch:np.float64(0.140625) - actor/grad_norm:np.float64(0.35879671573638916) - perf/mfu/actor:np.float64(0.045344773814958786) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.29048919677734) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6875) - val-aux/sciknoweval/reward/std@16:np.float64(0.16029546205916068) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7511857142857143) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.13365319867771744) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6229333333333333) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.13860952641542026) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6874666666666667) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.16050258419670824) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8023714285714285) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.10001117146576141) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5698666666666666) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.10827654197635113) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.694952380952381) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.12671450210804772) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8414666666666667) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.07147207072829928) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.5278238095238095) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.07866088591113561) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7011380952380954) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.09304988511758287) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8708) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.0457160969007615) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.49634285714285714) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.05141404847998308) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7056047619047618) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.06605128200501377) - val-aux/sciknoweval/score/mean@16:np.float64(0.6875) - val-aux/sciknoweval/score/std@16:np.float64(0.16029546205916068) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7511857142857143) - val-aux/sciknoweval/score/best@2/std:np.float64(0.13365319867771744) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6229333333333333) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.13860952641542026) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6874666666666667) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.16050258419670824) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8023714285714285) - val-aux/sciknoweval/score/best@4/std:np.float64(0.10001117146576141) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5698666666666666) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.10827654197635113) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.694952380952381) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.12671450210804772) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8414666666666667) - val-aux/sciknoweval/score/best@8/std:np.float64(0.07147207072829928) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.5278238095238095) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.07866088591113561) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7011380952380954) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.09304988511758287) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8708) - val-aux/sciknoweval/score/best@16/std:np.float64(0.0457160969007615) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.49634285714285714) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.05141404847998308) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7056047619047618) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.06605128200501377) - val-core/sciknoweval/acc/mean@16:np.float64(0.6875) - val-aux/sciknoweval/acc/std@16:np.float64(0.16029546205916068) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7511857142857143) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.13365319867771744) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6229333333333333) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.13860952641542026) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6874666666666667) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.16050258419670824) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8023714285714285) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.10001117146576141) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5698666666666666) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.10827654197635113) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.694952380952381) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.12671450210804772) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8414666666666667) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.07147207072829928) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.5278238095238095) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.07866088591113561) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7011380952380954) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.09304988511758287) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8708) - val-core/sciknoweval/acc/best@16/std:np.float64(0.0457160969007615) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.49634285714285714) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.05141404847998308) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7056047619047618) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.06605128200501377) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9982142857142857) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.006185546430642514) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9998571428571429) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0017362747371820737) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9967380952380952) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.008056772590456738) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9983857142857143) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.005874140160911555) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.00015050933932646775) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9938904761904761) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.01020614644674161) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9993380952380951) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0038135892390515993) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9893857142857142) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.011570601968058545) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9998333333333334) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.00189074176643801) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9839095238095237) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.01075169136568833) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999857142857144) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.00036325512888540755) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:60 - training/epoch:1 - critic/score/mean:0.59375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0029056959319859743 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0029056959319859743 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:328.01953125 - response_length/max:944.0 - response_length/min:125.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:328.01953125 - response_length_non_aborted/max:944.0 - response_length_non_aborted/min:125.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.9375 - prompt_length/max:451.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00029645487666130066 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6444534286856651) - timing_s/agent_loop/generate_sequences/max:np.float64(7.757375167682767) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.735986279869394) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.757375167682767) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:187 - timing_s/agent_loop/slowest/response_length:944 - timing_s/gen:14.574015535414219 - timing_s/reward:0.05933819338679314 - timing_s/old_log_prob:3.192529769614339 - timing_s/adv:0.5636057835072279 - timing_s/update_actor:21.57127202488482 - timing_s/step:40.06707218103111 - timing_s/testing:128.52822875231504 - timing_s/save_checkpoint:13.383056795224547 - timing_s/stop_profile:0.00014889799058437347 - timing_per_token_ms/update_actor:0.14186679660963494 - timing_per_token_ms/gen:0.17355597079316232 - timing_per_token_ms/adv:0.0037066403392713586 - perf/total_num_tokens:152053 - perf/time_per_step:40.06707218103111 - perf/throughput:474.3701989035844
(TaskRunner pid=2255397) Training Progress: 60%|██████ | 60/100 [56:05<56:00, 84.02s/it]
(TaskRunner pid=2255397) step:61 - global_seqlen/min:15598 - global_seqlen/max:23498 - global_seqlen/minmax_diff:7900 - global_seqlen/balanced_min:19661 - global_seqlen/balanced_max:19666 - global_seqlen/mean:19664.0 - actor/entropy:0.375679075717926 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4102344214916229 - training/rollout_probs_diff_mean:0.00489032082259655 - training/rollout_probs_diff_std:0.011647011153399944 - training/rollout_actor_probs_pearson_corr:0.9990482330322266 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83984375 - self_distillation/correct_sample_fraction:0.66015625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83984375 - rollout_corr/rollout_is_mean:1.00008225440979 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1082543096563313e-05 - rollout_corr/rollout_is_max:1.639464259147644 - rollout_corr/rollout_is_min:0.3959808051586151 - rollout_corr/rollout_is_std:0.03841473534703255 - rollout_corr/rollout_is_eff_sample_size:0.9985268392369275 - rollout_corr/rollout_is_seq_mean:1.000138759613037 - rollout_corr/rollout_is_seq_std:0.002528289332985878 - rollout_corr/rollout_is_seq_max:1.007035493850708 - rollout_corr/rollout_is_seq_min:0.993572473526001 - rollout_corr/rollout_is_seq_max_deviation:0.007035493850708008 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4903184147551656) - rollout_corr/training_log_ppl:np.float64(0.3903155774460174) - rollout_corr/kl:np.float64(0.0006912411112880434) - rollout_corr/k3_kl:np.float64(0.0009254292584728319) - rollout_corr/rollout_ppl:np.float64(1.4893131870776415) - rollout_corr/rollout_log_ppl:np.float64(0.38962433370761573) - rollout_corr/log_ppl_diff:np.float64(0.0006912437384016812) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020783296204172075) - rollout_corr/log_ppl_diff_max:np.float64(0.008693814277648926) - rollout_corr/log_ppl_diff_min:np.float64(-0.006119638681411743) - rollout_corr/ppl_ratio:np.float64(1.0006948560476303) - rollout_corr/chi2_token:np.float64(0.002362349070608616) - rollout_corr/chi2_seq:np.float64(0.9650198624003679) - actor/pg_loss:np.float64(0.005624711047858) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006064838689781027) - actor/ppo_kl:np.float64(1.730111737785478e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83984375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83984375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.66015625) - self_distillation/token_reweight_w_mean:np.float64(5193.730784527026) - self_distillation/token_reweight_w_std:np.float64(104429.90018021135) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9993646144866943) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08162092015845701) - self_distillation/empty_target_batch:np.float64(0.16015625) - actor/grad_norm:np.float64(0.354709729552269) - perf/mfu/actor:np.float64(0.04738440538771457) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.02718353271484) - actor/lr:np.float64(1e-06) - training/global_step:61 - training/epoch:1 - critic/score/mean:0.66015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.66015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004777961410582066 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004777961410582066 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:352.46875 - response_length/max:838.0 - response_length/min:127.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:352.46875 - response_length_non_aborted/max:838.0 - response_length_non_aborted/min:127.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.03125 - prompt_length/max:462.0 - prompt_length/min:177.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.24973326921463e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6630962695926428) - timing_s/agent_loop/generate_sequences/max:np.float64(7.389603292569518) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.984407690099033) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.389603292569518) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:202 - timing_s/agent_loop/slowest/response_length:830 - timing_s/gen:13.70044944435358 - timing_s/reward:0.06281237117946148 - timing_s/old_log_prob:3.0475483629852533 - timing_s/adv:0.5286510195583105 - timing_s/update_actor:21.522440258413553 - timing_s/step:38.972098929807544 - timing_s/stop_profile:0.0001547839492559433 - timing_per_token_ms/update_actor:0.13681372214715695 - timing_per_token_ms/gen:0.15183581705330237 - timing_per_token_ms/adv:0.0033605257040677794 - perf/total_num_tokens:157312 - perf/time_per_step:38.972098929807544 - perf/throughput:504.5661008768538
(TaskRunner pid=2255397) Training Progress: 61%|██████ | 61/100 [56:44<45:50, 70.52s/it]
(TaskRunner pid=2255397) step:62 - global_seqlen/min:17175 - global_seqlen/max:22516 - global_seqlen/minmax_diff:5341 - global_seqlen/balanced_min:19524 - global_seqlen/balanced_max:19535 - global_seqlen/mean:19531.875 - actor/entropy:0.4294959306716919 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.20105724036693573 - training/rollout_probs_diff_mean:0.005539817735552788 - training/rollout_probs_diff_std:0.012213723734021187 - training/rollout_actor_probs_pearson_corr:0.9990189075469971 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.80078125 - self_distillation/correct_sample_fraction:0.62890625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.80078125 - rollout_corr/rollout_is_mean:1.0000041723251343 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.9759001731872559 - rollout_corr/rollout_is_min:0.5684848427772522 - rollout_corr/rollout_is_std:0.04096618667244911 - rollout_corr/rollout_is_eff_sample_size:0.9983247021636537 - rollout_corr/rollout_is_seq_mean:0.9999959468841553 - rollout_corr/rollout_is_seq_std:0.0028718190733343363 - rollout_corr/rollout_is_seq_max:1.008870244026184 - rollout_corr/rollout_is_seq_min:0.987141489982605 - rollout_corr/rollout_is_seq_max_deviation:0.01285851001739502 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.584152389317751) - rollout_corr/training_log_ppl:np.float64(0.4482707906863652) - rollout_corr/kl:np.float64(0.001287838817653153) - rollout_corr/k3_kl:np.float64(0.0013142074097913792) - rollout_corr/rollout_ppl:np.float64(1.5821097679436207) - rollout_corr/rollout_log_ppl:np.float64(0.4469829489826225) - rollout_corr/log_ppl_diff:np.float64(0.0012878417037427425) - rollout_corr/log_ppl_abs_diff:np.float64(0.0026649965438991785) - rollout_corr/log_ppl_diff_max:np.float64(0.014283061027526855) - rollout_corr/log_ppl_diff_min:np.float64(-0.010016441345214844) - rollout_corr/ppl_ratio:np.float64(1.0012943353503942) - rollout_corr/chi2_token:np.float64(0.002664147410541773) - rollout_corr/chi2_seq:np.float64(0.6478703722823411) - actor/pg_loss:np.float64(0.005159451276995242) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.001237628416674852) - actor/ppo_kl:np.float64(0.0003923675134416271) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.80078125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.80078125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.62890625) - self_distillation/token_reweight_w_mean:np.float64(9608.619164069882) - self_distillation/token_reweight_w_std:np.float64(147490.825859931) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.000067490618676) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10762355217593722) - self_distillation/empty_target_batch:np.float64(0.19921875) - actor/grad_norm:np.float64(0.41205330193042755) - perf/mfu/actor:np.float64(0.04771627973118822) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.9746551513672) - actor/lr:np.float64(1e-06) - training/global_step:62 - training/epoch:1 - critic/score/mean:0.62890625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62890625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.012106545269489288 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.012106545269489288 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:311.12109375 - response_length/max:1015.0 - response_length/min:111.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:311.12109375 - response_length_non_aborted/max:1015.0 - response_length_non_aborted/min:111.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.25 - prompt_length/max:662.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013557448983192444 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3771552108228207) - timing_s/agent_loop/generate_sequences/max:np.float64(7.987416280433536) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.6242045445760596) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.987416280433536) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:198 - timing_s/agent_loop/slowest/response_length:1015 - timing_s/gen:14.992265343666077 - timing_s/reward:0.06058591604232788 - timing_s/old_log_prob:2.887193411588669 - timing_s/adv:0.5111278612166643 - timing_s/update_actor:21.240903863683343 - timing_s/step:39.78005450963974 - timing_s/stop_profile:0.00012206099927425385 - timing_per_token_ms/update_actor:0.13593743472966205 - timing_per_token_ms/gen:0.18823389887461017 - timing_per_token_ms/adv:0.0032711136361502947 - perf/total_num_tokens:156255 - perf/time_per_step:39.78005450963974 - perf/throughput:490.99668768093113
(TaskRunner pid=2255397) Training Progress: 62%|██████▏ | 62/100 [57:23<38:49, 61.31s/it]
(TaskRunner pid=2255397) step:63 - global_seqlen/min:15986 - global_seqlen/max:20551 - global_seqlen/minmax_diff:4565 - global_seqlen/balanced_min:18758 - global_seqlen/balanced_max:18761 - global_seqlen/mean:18759.625 - actor/entropy:0.35515502095222473 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24133962392807007 - training/rollout_probs_diff_mean:0.004971614107489586 - training/rollout_probs_diff_std:0.012004957534372807 - training/rollout_actor_probs_pearson_corr:0.9989368915557861 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.8125 - self_distillation/correct_sample_fraction:0.53125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8125 - rollout_corr/rollout_is_mean:1.0001026391983032 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.3019985999562778e-05 - rollout_corr/rollout_is_max:1.8590831756591797 - rollout_corr/rollout_is_min:0.4984112083911896 - rollout_corr/rollout_is_std:0.03878994286060333 - rollout_corr/rollout_is_eff_sample_size:0.998497719797901 - rollout_corr/rollout_is_seq_mean:1.000045895576477 - rollout_corr/rollout_is_seq_std:0.002819458954036236 - rollout_corr/rollout_is_seq_max:1.0079857110977173 - rollout_corr/rollout_is_seq_min:0.9885396957397461 - rollout_corr/rollout_is_seq_max_deviation:0.011460304260253906 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4885337855666876) - rollout_corr/training_log_ppl:np.float64(0.38630677238688804) - rollout_corr/kl:np.float64(0.0008763675811707117) - rollout_corr/k3_kl:np.float64(0.0011061657141908654) - rollout_corr/rollout_ppl:np.float64(1.4872424700297415) - rollout_corr/rollout_log_ppl:np.float64(0.3854304051492363) - rollout_corr/log_ppl_diff:np.float64(0.0008763672376517206) - rollout_corr/log_ppl_abs_diff:np.float64(0.0025414098927285522) - rollout_corr/log_ppl_diff_max:np.float64(0.016067981719970703) - rollout_corr/log_ppl_diff_min:np.float64(-0.010530859231948853) - rollout_corr/ppl_ratio:np.float64(1.0008826348930597) - rollout_corr/chi2_token:np.float64(0.002648768713697791) - rollout_corr/chi2_seq:np.float64(1.621614676201716) - actor/pg_loss:np.float64(0.004201351199299097) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006837096566414402) - actor/ppo_kl:np.float64(9.11460165120559e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.53125) - self_distillation/token_reweight_w_mean:np.float64(328.2442679554224) - self_distillation/token_reweight_w_std:np.float64(5757.5124940559035) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0031547287944704) - self_distillation/token_reweight_w_clip_frac:np.float64(0.0877059873600956) - self_distillation/empty_target_batch:np.float64(0.1875) - actor/grad_norm:np.float64(0.3450646325945854) - perf/mfu/actor:np.float64(0.04625351205637495) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.83617782592773) - actor/lr:np.float64(1e-06) - training/global_step:63 - training/epoch:1 - critic/score/mean:0.53125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.53125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.001894407905638218 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.001894407905638218 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:300.01953125 - response_length/max:723.0 - response_length/min:105.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:300.01953125 - response_length_non_aborted/max:723.0 - response_length_non_aborted/min:105.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.21875 - prompt_length/max:532.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013262778520584106 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4198575951159) - timing_s/agent_loop/generate_sequences/max:np.float64(6.652159977704287) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.4228165260647074) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.652159977704287) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:226 - timing_s/agent_loop/slowest/response_length:723 - timing_s/gen:13.547540098428726 - timing_s/reward:0.06083318777382374 - timing_s/old_log_prob:2.9801845103502274 - timing_s/adv:0.5016210991889238 - timing_s/update_actor:21.087883727625012 - timing_s/step:38.26442988216877 - timing_s/stop_profile:0.00010959990322589874 - timing_per_token_ms/update_actor:0.1405137611201251 - timing_per_token_ms/gen:0.17638877805388617 - timing_per_token_ms/adv:0.003342424883152807 - perf/total_num_tokens:150077 - perf/time_per_step:38.26442988216877 - perf/throughput:490.26275989916127
(TaskRunner pid=2255397) Training Progress: 63%|██████▎ | 63/100 [58:02<33:33, 54.41s/it]
(TaskRunner pid=2255397) step:64 - global_seqlen/min:15100 - global_seqlen/max:20599 - global_seqlen/minmax_diff:5499 - global_seqlen/balanced_min:18082 - global_seqlen/balanced_max:18090 - global_seqlen/mean:18088.125 - actor/entropy:0.3758540451526642 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24836724996566772 - training/rollout_probs_diff_mean:0.005264882929623127 - training/rollout_probs_diff_std:0.012344393879175186 - training/rollout_actor_probs_pearson_corr:0.9989309310913086 - self_distillation/success_group_fraction:0.90625 - self_distillation/success_sample_fraction:0.8984375 - self_distillation/correct_sample_fraction:0.765625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8984375 - rollout_corr/rollout_is_mean:1.0000416040420532 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.8669313905993477e-05 - rollout_corr/rollout_is_max:1.78631591796875 - rollout_corr/rollout_is_min:0.3966720402240753 - rollout_corr/rollout_is_std:0.04027950391173363 - rollout_corr/rollout_is_eff_sample_size:0.9983804272587137 - rollout_corr/rollout_is_seq_mean:1.0000438690185547 - rollout_corr/rollout_is_seq_std:0.0027673090808093548 - rollout_corr/rollout_is_seq_max:1.0073356628417969 - rollout_corr/rollout_is_seq_min:0.9913079738616943 - rollout_corr/rollout_is_seq_max_deviation:0.008692026138305664 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.488935824483633) - rollout_corr/training_log_ppl:np.float64(0.3868299189489335) - rollout_corr/kl:np.float64(0.0007672079092806428) - rollout_corr/k3_kl:np.float64(0.0009736951585637144) - rollout_corr/rollout_ppl:np.float64(1.4878136315383017) - rollout_corr/rollout_log_ppl:np.float64(0.38606271165190265) - rollout_corr/log_ppl_diff:np.float64(0.0007672072970308363) - rollout_corr/log_ppl_abs_diff:np.float64(0.0023564034490846097) - rollout_corr/log_ppl_diff_max:np.float64(0.011275023221969604) - rollout_corr/log_ppl_diff_min:np.float64(-0.008923828601837158) - rollout_corr/ppl_ratio:np.float64(1.000771946972236) - rollout_corr/chi2_token:np.float64(0.002396456664428115) - rollout_corr/chi2_seq:np.float64(0.5857332549057901) - actor/pg_loss:np.float64(0.0037766702007502317) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00045624542644873145) - actor/ppo_kl:np.float64(-8.097473682511236e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8984375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8984375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.765625) - self_distillation/token_reweight_w_mean:np.float64(34646.944526499836) - self_distillation/token_reweight_w_std:np.float64(444681.0361109813) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9998654320370406) - self_distillation/token_reweight_w_clip_frac:np.float64(0.06305540914763696) - self_distillation/empty_target_batch:np.float64(0.1015625) - actor/grad_norm:np.float64(0.2998654544353485) - perf/mfu/actor:np.float64(0.04428300784041782) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(199.82562017440796) - actor/lr:np.float64(1e-06) - training/global_step:64 - training/epoch:1 - critic/score/mean:0.765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.009668726474046707 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.009668726474046707 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:272.50390625 - response_length/max:914.0 - response_length/min:101.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:272.50390625 - response_length_non_aborted/max:914.0 - response_length_non_aborted/min:101.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:292.75 - prompt_length/max:495.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010181218385696411 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3763788174837828) - timing_s/agent_loop/generate_sequences/max:np.float64(7.2169532254338264) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.232598840339051) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.2169532254338264) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:200 - timing_s/agent_loop/slowest/response_length:879 - timing_s/gen:14.560809908434749 - timing_s/reward:0.057523222640156746 - timing_s/old_log_prob:2.9381075855344534 - timing_s/adv:0.5264706443995237 - timing_s/update_actor:21.378882113844156 - timing_s/step:39.54835497029126 - timing_s/stop_profile:0.00011551380157470703 - timing_per_token_ms/update_actor:0.14774114311077127 - timing_per_token_ms/gen:0.2087242142233447 - timing_per_token_ms/adv:0.0036382339545939927 - perf/total_num_tokens:144705 - perf/time_per_step:39.54835497029126 - perf/throughput:457.3673168855647
(TaskRunner pid=2255397) Training Progress: 64%|██████▍ | 64/100 [58:41<29:58, 49.97s/it]
(TaskRunner pid=2255397) step:65 - global_seqlen/min:15051 - global_seqlen/max:21236 - global_seqlen/minmax_diff:6185 - global_seqlen/balanced_min:18004 - global_seqlen/balanced_max:18008 - global_seqlen/mean:18006.625 - actor/entropy:0.3955051898956299 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1947164535522461 - training/rollout_probs_diff_mean:0.0055443341843783855 - training/rollout_probs_diff_std:0.012429744005203247 - training/rollout_actor_probs_pearson_corr:0.9989457726478577 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.86328125 - self_distillation/correct_sample_fraction:0.71484375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.86328125 - rollout_corr/rollout_is_mean:0.9999876618385315 - rollout_corr/rollout_is_ratio_fraction_high:1.3995213521411642e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.031123399734497 - rollout_corr/rollout_is_min:0.504069447517395 - rollout_corr/rollout_is_std:0.041325442492961884 - rollout_corr/rollout_is_eff_sample_size:0.9982950004968502 - rollout_corr/rollout_is_seq_mean:0.9999707341194153 - rollout_corr/rollout_is_seq_std:0.002948605688288808 - rollout_corr/rollout_is_seq_max:1.009615182876587 - rollout_corr/rollout_is_seq_min:0.9913778305053711 - rollout_corr/rollout_is_seq_max_deviation:0.009615182876586914 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.525312386918813) - rollout_corr/training_log_ppl:np.float64(0.4125790528487414) - rollout_corr/kl:np.float64(0.00100548441882653) - rollout_corr/k3_kl:np.float64(0.001056205478960237) - rollout_corr/rollout_ppl:np.float64(1.5237079304642975) - rollout_corr/rollout_log_ppl:np.float64(0.4115735680970829) - rollout_corr/log_ppl_diff:np.float64(0.0010054847516585141) - rollout_corr/log_ppl_abs_diff:np.float64(0.0024253172159660608) - rollout_corr/log_ppl_diff_max:np.float64(0.012961804866790771) - rollout_corr/log_ppl_diff_min:np.float64(-0.007540702819824219) - rollout_corr/ppl_ratio:np.float64(1.0010107702109963) - rollout_corr/chi2_token:np.float64(0.0022052584681659937) - rollout_corr/chi2_seq:np.float64(0.9272577015217394) - actor/pg_loss:np.float64(0.003355647437274456) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000539568421572767) - actor/ppo_kl:np.float64(7.455754089491506e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.86328125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.86328125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.71484375) - self_distillation/token_reweight_w_mean:np.float64(178.37819157890044) - self_distillation/token_reweight_w_std:np.float64(3052.0747698339983) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9989271517843008) - self_distillation/token_reweight_w_clip_frac:np.float64(0.07668937157723121) - self_distillation/empty_target_batch:np.float64(0.13671875) - actor/grad_norm:np.float64(0.31669436022639275) - perf/mfu/actor:np.float64(0.04449943785630652) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.69052124023438) - actor/lr:np.float64(1e-06) - training/global_step:65 - training/epoch:1 - critic/score/mean:0.71484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.013841266743838787 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.013841266743838787 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:279.11328125 - response_length/max:754.0 - response_length/min:94.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:279.11328125 - response_length_non_aborted/max:754.0 - response_length_non_aborted/min:94.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:283.59375 - prompt_length/max:818.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.972788393497467e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2479094993323088) - timing_s/agent_loop/generate_sequences/max:np.float64(6.392431171610951) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.2372494999872288) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.392431171610951) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:228 - timing_s/agent_loop/slowest/response_length:754 - timing_s/gen:13.095486616715789 - timing_s/reward:0.0594562403857708 - timing_s/old_log_prob:2.941354649141431 - timing_s/adv:0.5120643582195044 - timing_s/update_actor:20.915571719408035 - timing_s/step:37.61257514543831 - timing_s/stop_profile:0.00016906112432479858 - timing_per_token_ms/update_actor:0.14519358652307163 - timing_per_token_ms/gen:0.1832741328805759 - timing_per_token_ms/adv:0.003554694162700564 - perf/total_num_tokens:144053 - perf/time_per_step:37.61257514543831 - perf/throughput:478.7394888643741
(TaskRunner pid=2255397) Training Progress: 65%|██████▌ | 65/100 [59:19<26:59, 46.27s/it]
(TaskRunner pid=2255397) step:66 - global_seqlen/min:16554 - global_seqlen/max:19900 - global_seqlen/minmax_diff:3346 - global_seqlen/balanced_min:18653 - global_seqlen/balanced_max:18704 - global_seqlen/mean:18662.625 - actor/entropy:0.3692823350429535 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24191009998321533 - training/rollout_probs_diff_mean:0.005031523294746876 - training/rollout_probs_diff_std:0.011805592104792595 - training/rollout_actor_probs_pearson_corr:0.998999834060669 - self_distillation/success_group_fraction:0.71875 - self_distillation/success_sample_fraction:0.7109375 - self_distillation/correct_sample_fraction:0.6171875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7109375 - rollout_corr/rollout_is_mean:0.9999994039535522 - rollout_corr/rollout_is_ratio_fraction_high:1.3333866263565142e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.2511892318725586 - rollout_corr/rollout_is_min:0.5517903566360474 - rollout_corr/rollout_is_std:0.03924669697880745 - rollout_corr/rollout_is_eff_sample_size:0.998462065654954 - rollout_corr/rollout_is_seq_mean:1.0000253915786743 - rollout_corr/rollout_is_seq_std:0.002698667347431183 - rollout_corr/rollout_is_seq_max:1.0073356628417969 - rollout_corr/rollout_is_seq_min:0.992901086807251 - rollout_corr/rollout_is_seq_max_deviation:0.007335662841796875 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5085271298885345) - rollout_corr/training_log_ppl:np.float64(0.4001465483015636) - rollout_corr/kl:np.float64(0.0009092299301867168) - rollout_corr/k3_kl:np.float64(0.001055825624348472) - rollout_corr/rollout_ppl:np.float64(1.5071750050410628) - rollout_corr/rollout_log_ppl:np.float64(0.3992373197543202) - rollout_corr/log_ppl_diff:np.float64(0.0009092285472434014) - rollout_corr/log_ppl_abs_diff:np.float64(0.002248107542982325) - rollout_corr/log_ppl_diff_max:np.float64(0.008349627256393433) - rollout_corr/log_ppl_diff_min:np.float64(-0.008058488368988037) - rollout_corr/ppl_ratio:np.float64(1.0009135268628597) - rollout_corr/chi2_token:np.float64(0.0025282949209213257) - rollout_corr/chi2_seq:np.float64(0.6103038967121392) - actor/pg_loss:np.float64(0.002379831741563976) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00033230165354325436) - actor/ppo_kl:np.float64(8.482311253033004e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7109375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7109375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6171875) - self_distillation/token_reweight_w_mean:np.float64(14.794726703781635) - self_distillation/token_reweight_w_std:np.float64(315.56329661473865) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9989614612422884) - self_distillation/token_reweight_w_clip_frac:np.float64(0.04611051009851508) - self_distillation/empty_target_batch:np.float64(0.2890625) - actor/grad_norm:np.float64(0.24986116960644722) - perf/mfu/actor:np.float64(0.044863735441974) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.69231033325195) - actor/lr:np.float64(1e-06) - training/global_step:66 - training/epoch:1 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0023150925990194082 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0023150925990194082 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:292.95703125 - response_length/max:1040.0 - response_length/min:134.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:292.95703125 - response_length_non_aborted/max:1040.0 - response_length_non_aborted/min:134.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:290.25 - prompt_length/max:617.0 - prompt_length/min:184.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014759600162506104 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7828350365161896) - timing_s/agent_loop/generate_sequences/max:np.float64(8.071565691381693) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.388918655909947) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.071565691381693) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:197 - timing_s/agent_loop/slowest/response_length:1040 - timing_s/gen:15.357409680262208 - timing_s/reward:0.0539607759565115 - timing_s/old_log_prob:2.9237958528101444 - timing_s/adv:0.5357848312705755 - timing_s/update_actor:21.778983931988478 - timing_s/step:40.74699264019728 - timing_s/stop_profile:0.00011424347758293152 - timing_per_token_ms/update_actor:0.14587299436700676 - timing_per_token_ms/gen:0.2047736533496301 - timing_per_token_ms/adv:0.0035886218529720196 - perf/total_num_tokens:149301 - perf/time_per_step:40.74699264019728 - perf/throughput:458.01232902742254
(TaskRunner pid=2255397) Training Progress: 66%|██████▌ | 66/100 [1:00:00<25:17, 44.63s/it]
(TaskRunner pid=2255397) step:67 - global_seqlen/min:14936 - global_seqlen/max:21018 - global_seqlen/minmax_diff:6082 - global_seqlen/balanced_min:18548 - global_seqlen/balanced_max:18548 - global_seqlen/mean:18548.0 - actor/entropy:0.4010680615901947 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18490827083587646 - training/rollout_probs_diff_mean:0.005276753567159176 - training/rollout_probs_diff_std:0.011970635503530502 - training/rollout_actor_probs_pearson_corr:0.9990053772926331 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.75 - self_distillation/correct_sample_fraction:0.6796875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.75 - rollout_corr/rollout_is_mean:1.0001569986343384 - rollout_corr/rollout_is_ratio_fraction_high:1.357367818854982e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.357367818854982e-05 - rollout_corr/rollout_is_max:2.239859104156494 - rollout_corr/rollout_is_min:0.4357939660549164 - rollout_corr/rollout_is_std:0.03935437649488449 - rollout_corr/rollout_is_eff_sample_size:0.9984541032441931 - rollout_corr/rollout_is_seq_mean:1.0001627206802368 - rollout_corr/rollout_is_seq_std:0.002580940490588546 - rollout_corr/rollout_is_seq_max:1.0100871324539185 - rollout_corr/rollout_is_seq_min:0.9937511682510376 - rollout_corr/rollout_is_seq_max_deviation:0.010087132453918457 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5274818786419928) - rollout_corr/training_log_ppl:np.float64(0.41380137618398294) - rollout_corr/kl:np.float64(0.0010146739747582956) - rollout_corr/k3_kl:np.float64(0.0009835226737777703) - rollout_corr/rollout_ppl:np.float64(1.5259358463808894) - rollout_corr/rollout_log_ppl:np.float64(0.41278669840539806) - rollout_corr/log_ppl_diff:np.float64(0.0010146777785848826) - rollout_corr/log_ppl_abs_diff:np.float64(0.0021784571872558445) - rollout_corr/log_ppl_diff_max:np.float64(0.014455139636993408) - rollout_corr/log_ppl_diff_min:np.float64(-0.0060721635818481445) - rollout_corr/ppl_ratio:np.float64(1.0010192152112722) - rollout_corr/chi2_token:np.float64(0.0019758078269660473) - rollout_corr/chi2_seq:np.float64(0.4387206360697746) - actor/pg_loss:np.float64(0.0029250025982037187) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00039720970880807727) - actor/ppo_kl:np.float64(0.0003566578742226767) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.75) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.75) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6796875) - self_distillation/token_reweight_w_mean:np.float64(547.347474707989) - self_distillation/token_reweight_w_std:np.float64(8326.106724163285) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9982913499698043) - self_distillation/token_reweight_w_clip_frac:np.float64(0.06155556399608031) - self_distillation/empty_target_batch:np.float64(0.25) - actor/grad_norm:np.float64(0.2857639491558075) - perf/mfu/actor:np.float64(0.04548089224569089) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.682861328125) - actor/lr:np.float64(1e-06) - training/global_step:67 - training/epoch:1 - critic/score/mean:0.6796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0016373249236494303 - critic/advantages/max:0.625 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0016373249236494303 - critic/returns/max:0.625 - critic/returns/min:-0.875 - response_length/mean:287.78125 - response_length/max:673.0 - response_length/min:110.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:287.78125 - response_length_non_aborted/max:673.0 - response_length_non_aborted/min:110.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:291.84375 - prompt_length/max:576.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013326294720172882 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3205897863954306) - timing_s/agent_loop/generate_sequences/max:np.float64(5.870759850367904) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.2840450326475548) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(5.870759850367904) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:228 - timing_s/agent_loop/slowest/response_length:673 - timing_s/gen:12.662429416552186 - timing_s/reward:0.05886896140873432 - timing_s/old_log_prob:2.852442506700754 - timing_s/adv:0.49736564978957176 - timing_s/update_actor:21.174225766211748 - timing_s/step:37.33046619966626 - timing_s/stop_profile:2.7060508728027344e-05 - timing_per_token_ms/update_actor:0.14269884735693705 - timing_per_token_ms/gen:0.1718757386327531 - timing_per_token_ms/adv:0.00335188194003108 - perf/total_num_tokens:148384 - perf/time_per_step:37.33046619966626 - perf/throughput:496.8595865048645
(TaskRunner pid=2255397) Training Progress: 67%|██████▋ | 67/100 [1:00:37<23:20, 42.45s/it]
(TaskRunner pid=2255397) step:68 - global_seqlen/min:15394 - global_seqlen/max:21848 - global_seqlen/minmax_diff:6454 - global_seqlen/balanced_min:19111 - global_seqlen/balanced_max:19113 - global_seqlen/mean:19112.0 - actor/entropy:0.454579621553421 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2294895052909851 - training/rollout_probs_diff_mean:0.005558677017688751 - training/rollout_probs_diff_std:0.011987541802227497 - training/rollout_actor_probs_pearson_corr:0.999093234539032 - self_distillation/success_group_fraction:0.65625 - self_distillation/success_sample_fraction:0.6484375 - self_distillation/correct_sample_fraction:0.46875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.6484375 - rollout_corr/rollout_is_mean:0.999990701675415 - rollout_corr/rollout_is_ratio_fraction_high:2.770083119685296e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.329737901687622 - rollout_corr/rollout_is_min:0.5801910758018494 - rollout_corr/rollout_is_std:0.04091522842645645 - rollout_corr/rollout_is_eff_sample_size:0.9983287417268669 - rollout_corr/rollout_is_seq_mean:0.9999210834503174 - rollout_corr/rollout_is_seq_std:0.0029238073620945215 - rollout_corr/rollout_is_seq_max:1.010511040687561 - rollout_corr/rollout_is_seq_min:0.9896543025970459 - rollout_corr/rollout_is_seq_max_deviation:0.010511040687561035 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6329940822906792) - rollout_corr/training_log_ppl:np.float64(0.4769620407023467) - rollout_corr/kl:np.float64(0.0010583482470849503) - rollout_corr/k3_kl:np.float64(0.0009939358607766735) - rollout_corr/rollout_ppl:np.float64(1.631263299845159) - rollout_corr/rollout_log_ppl:np.float64(0.47590369096724316) - rollout_corr/log_ppl_diff:np.float64(0.0010583497351035476) - rollout_corr/log_ppl_abs_diff:np.float64(0.0024261822691187263) - rollout_corr/log_ppl_diff_max:np.float64(0.012225866317749023) - rollout_corr/log_ppl_diff_min:np.float64(-0.008922696113586426) - rollout_corr/ppl_ratio:np.float64(1.0010634786449373) - rollout_corr/chi2_token:np.float64(0.0018958586733788252) - rollout_corr/chi2_seq:np.float64(0.7360206483863294) - actor/pg_loss:np.float64(0.0039513889933004975) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004996858078811783) - actor/ppo_kl:np.float64(9.023317768352968e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.6484375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.6484375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.46875) - self_distillation/token_reweight_w_mean:np.float64(16211.263574679615) - self_distillation/token_reweight_w_std:np.float64(247760.0435439156) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0013059759512544) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09038829631754197) - self_distillation/empty_target_batch:np.float64(0.3515625) - actor/grad_norm:np.float64(0.35276055335998535) - perf/mfu/actor:np.float64(0.04664085001944063) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.4664192199707) - actor/lr:np.float64(1e-06) - training/global_step:68 - training/epoch:1 - critic/score/mean:0.46875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.46875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0022922437638044357 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0022922437638044357 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:282.03125 - response_length/max:777.0 - response_length/min:93.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:282.03125 - response_length_non_aborted/max:777.0 - response_length_non_aborted/min:93.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:315.21875 - prompt_length/max:704.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.814541459083557e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1940552610903978) - timing_s/agent_loop/generate_sequences/max:np.float64(6.472350174561143) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.2505520836712094) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.472350174561143) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:229 - timing_s/agent_loop/slowest/response_length:777 - timing_s/gen:13.590019196271896 - timing_s/reward:0.05382108315825462 - timing_s/old_log_prob:3.070949576795101 - timing_s/adv:0.5018815603107214 - timing_s/update_actor:21.207215555012226 - timing_s/step:38.45727911219001 - timing_s/stop_profile:2.8835609555244446e-05 - timing_per_token_ms/update_actor:0.13870353413439349 - timing_per_token_ms/gen:0.1882274126907465 - timing_per_token_ms/adv:0.003282502879805367 - perf/total_num_tokens:152896 - perf/time_per_step:38.45727911219001 - perf/throughput:496.9670356617083
(TaskRunner pid=2255397) Training Progress: 68%|██████▊ | 68/100 [1:01:16<22:00, 41.26s/it]
(TaskRunner pid=2255397) step:69 - global_seqlen/min:17581 - global_seqlen/max:21958 - global_seqlen/minmax_diff:4377 - global_seqlen/balanced_min:20257 - global_seqlen/balanced_max:20260 - global_seqlen/mean:20258.25 - actor/entropy:0.46467649936676025 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24135446548461914 - training/rollout_probs_diff_mean:0.00523096090182662 - training/rollout_probs_diff_std:0.011185083538293839 - training/rollout_actor_probs_pearson_corr:0.999234676361084 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.8359375 - self_distillation/correct_sample_fraction:0.6796875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8359375 - rollout_corr/rollout_is_mean:1.0000218152999878 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6324684619903564 - rollout_corr/rollout_is_min:0.5706769227981567 - rollout_corr/rollout_is_std:0.03825926035642624 - rollout_corr/rollout_is_eff_sample_size:0.998538606349212 - rollout_corr/rollout_is_seq_mean:1.0000791549682617 - rollout_corr/rollout_is_seq_std:0.002675000810995698 - rollout_corr/rollout_is_seq_max:1.0090395212173462 - rollout_corr/rollout_is_seq_min:0.989310622215271 - rollout_corr/rollout_is_seq_max_deviation:0.010689377784729004 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.646835464052856) - rollout_corr/training_log_ppl:np.float64(0.4856712990440428) - rollout_corr/kl:np.float64(0.0008948077715515979) - rollout_corr/k3_kl:np.float64(0.0010960273730233894) - rollout_corr/rollout_ppl:np.float64(1.6454020515084267) - rollout_corr/rollout_log_ppl:np.float64(0.48477648955304176) - rollout_corr/log_ppl_diff:np.float64(0.0008948094910010695) - rollout_corr/log_ppl_abs_diff:np.float64(0.0023694130359217525) - rollout_corr/log_ppl_diff_max:np.float64(0.012593626976013184) - rollout_corr/log_ppl_diff_min:np.float64(-0.010994911193847656) - rollout_corr/ppl_ratio:np.float64(1.0008999034762383) - rollout_corr/chi2_token:np.float64(0.0027323926333338022) - rollout_corr/chi2_seq:np.float64(1.6403908263891935) - actor/pg_loss:np.float64(0.004149097250774503) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006782121126889251) - actor/ppo_kl:np.float64(0.0001744897412141455) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8359375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8359375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6796875) - self_distillation/token_reweight_w_mean:np.float64(112.03770920354873) - self_distillation/token_reweight_w_std:np.float64(1537.8677468675305) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9990846202708781) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10416805295972154) - self_distillation/empty_target_batch:np.float64(0.1640625) - actor/grad_norm:np.float64(0.3852573037147522) - perf/mfu/actor:np.float64(0.04884632323523585) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.50916624069214) - actor/lr:np.float64(1e-06) - training/global_step:69 - training/epoch:1 - critic/score/mean:0.6796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.001025630161166191 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.001025630161166191 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:323.2578125 - response_length/max:740.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:323.2578125 - response_length_non_aborted/max:740.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:309.8125 - prompt_length/max:597.0 - prompt_length/min:168.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.5704503059387207e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7049627471715212) - timing_s/agent_loop/generate_sequences/max:np.float64(6.463043302297592) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.702042857927154) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.463043302297592) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:225 - timing_s/agent_loop/slowest/response_length:740 - timing_s/gen:13.723623352125287 - timing_s/reward:0.06722629256546497 - timing_s/old_log_prob:3.223280254751444 - timing_s/adv:0.5192670281976461 - timing_s/update_actor:21.539511624723673 - timing_s/step:39.17174446769059 - timing_s/stop_profile:0.00010973773896694183 - timing_per_token_ms/update_actor:0.13290580149274786 - timing_per_token_ms/gen:0.16583637470243476 - timing_per_token_ms/adv:0.003204046673562907 - perf/total_num_tokens:162066 - perf/time_per_step:39.17174446769059 - perf/throughput:517.1648665458158
(TaskRunner pid=2255397) Training Progress: 69%|██████▉ | 69/100 [1:01:55<21:00, 40.65s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 70}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1", we must analyze the structure and functional groups present in both the product and the options.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product contains:
(TaskRunner pid=2255397) - A nitrogen-containing ring (likely a pyrrole or similar heterocycle).
(TaskRunner pid=2255397) - A chlorinated aromatic ring (–Cl group).
(TaskRunner pid=2255397) - A trifluoromethyl group (–C(F)(F)F).
(TaskRunner pid=2255397) - A nitrogen atom bonded to a carbon chain that includes the chlorinated aromatic ring.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Now, let's examine the options:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option A**: Contains a chlorinated aromatic ring, a trifluoromethyl group, and a nitrogen-containing ring with a nitrogen atom bonded to a carbon chain that includes the chlorinated aromatic ring. It closely matches the structure of the product and is a strong candidate.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option B**: Contains a chlorinated aromatic ring but lacks the trifluoromethyl group and the correct nitrogen-containing ring structure. It is not a match.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option C**: Contains a chlorinated aromatic ring and a trifluoromethyl group, but the nitrogen-containing ring is not correctly structured. It does not match the product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option D**: Contains a trifluoromethyl group and a carbon chain with a chlorinated aromatic ring, but the nitrogen-containing ring is not correctly structured. It also does not match the product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Thus, **Option A** is the only one that contains all the necessary functional groups and structural features required to synthesize the product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_70
(TaskRunner pid=2255397) step:70 - global_seqlen/min:14508 - global_seqlen/max:21779 - global_seqlen/minmax_diff:7271 - global_seqlen/balanced_min:19295 - global_seqlen/balanced_max:19376 - global_seqlen/mean:19309.25 - actor/entropy:0.48265349864959717 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2931375205516815 - training/rollout_probs_diff_mean:0.005450577940791845 - training/rollout_probs_diff_std:0.011540403589606285 - training/rollout_actor_probs_pearson_corr:0.9992024302482605 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.796875 - self_distillation/correct_sample_fraction:0.62890625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.796875 - rollout_corr/rollout_is_mean:1.0000245571136475 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6136518716812134 - rollout_corr/rollout_is_min:0.5989248752593994 - rollout_corr/rollout_is_std:0.03922086954116821 - rollout_corr/rollout_is_eff_sample_size:0.9984642048317262 - rollout_corr/rollout_is_seq_mean:1.0000163316726685 - rollout_corr/rollout_is_seq_std:0.002710202941671014 - rollout_corr/rollout_is_seq_max:1.0095508098602295 - rollout_corr/rollout_is_seq_min:0.9921322464942932 - rollout_corr/rollout_is_seq_max_deviation:0.009550809860229492 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6673680869862437) - rollout_corr/training_log_ppl:np.float64(0.4960062170866877) - rollout_corr/kl:np.float64(0.0012155248211556113) - rollout_corr/k3_kl:np.float64(0.0012827583032049006) - rollout_corr/rollout_ppl:np.float64(1.6654691863805056) - rollout_corr/rollout_log_ppl:np.float64(0.4947906881570816) - rollout_corr/log_ppl_diff:np.float64(0.00121552892960608) - rollout_corr/log_ppl_abs_diff:np.float64(0.0026022420497611165) - rollout_corr/log_ppl_diff_max:np.float64(0.016035795211791992) - rollout_corr/log_ppl_diff_min:np.float64(-0.00719606876373291) - rollout_corr/ppl_ratio:np.float64(1.0012216775212437) - rollout_corr/chi2_token:np.float64(0.002924366621300578) - rollout_corr/chi2_seq:np.float64(1.3708638157695532) - actor/pg_loss:np.float64(0.0026991143822669983) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0006329814200398687) - actor/ppo_kl:np.float64(0.0004067609955740181) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.796875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.796875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.62890625) - self_distillation/token_reweight_w_mean:np.float64(1099.6124461798463) - self_distillation/token_reweight_w_std:np.float64(17668.276791150623) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0005380015354604) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08962540709762834) - self_distillation/empty_target_batch:np.float64(0.203125) - actor/grad_norm:np.float64(0.30823376774787903) - perf/mfu/actor:np.float64(0.04660907978159935) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.58329391479492) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6967261904761904) - val-aux/sciknoweval/reward/std@16:np.float64(0.15836902316671747) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7580857142857141) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.12446919093953822) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6347571428571429) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.1415950973799327) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6969619047619046) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.1573615579638144) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8046666666666668) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.09049082267851498) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5808857142857142) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.11888321084630435) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.7069904761904763) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.1207083032051214) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8401380952380952) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.0621350178422806) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.533304761904762) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.09702903481875863) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7099809523809524) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.08266567659967641) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8649238095238094) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.037984835978200744) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.4909904761904762) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.07165352375405533) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7088047619047618) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.057017736779803374) - val-aux/sciknoweval/score/mean@16:np.float64(0.6967261904761904) - val-aux/sciknoweval/score/std@16:np.float64(0.15836902316671747) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7580857142857141) - val-aux/sciknoweval/score/best@2/std:np.float64(0.12446919093953822) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6347571428571429) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.1415950973799327) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6969619047619046) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.1573615579638144) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8046666666666668) - val-aux/sciknoweval/score/best@4/std:np.float64(0.09049082267851498) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5808857142857142) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.11888321084630435) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.7069904761904763) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.1207083032051214) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8401380952380952) - val-aux/sciknoweval/score/best@8/std:np.float64(0.0621350178422806) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.533304761904762) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.09702903481875863) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7099809523809524) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.08266567659967641) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8649238095238094) - val-aux/sciknoweval/score/best@16/std:np.float64(0.037984835978200744) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.4909904761904762) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.07165352375405533) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7088047619047618) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.057017736779803374) - val-core/sciknoweval/acc/mean@16:np.float64(0.6967261904761904) - val-aux/sciknoweval/acc/std@16:np.float64(0.15836902316671747) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7580857142857141) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.12446919093953822) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6347571428571429) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.1415950973799327) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6969619047619046) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.1573615579638144) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8046666666666668) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.09049082267851498) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5808857142857142) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.11888321084630435) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.7069904761904763) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.1207083032051214) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8401380952380952) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.0621350178422806) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.533304761904762) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.09702903481875863) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7099809523809524) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.08266567659967641) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8649238095238094) - val-core/sciknoweval/acc/best@16/std:np.float64(0.037984835978200744) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.4909904761904762) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.07165352375405533) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7088047619047618) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.057017736779803374) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.999702380952381) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0011526736149426837) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9999809523809524) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.000300566358820181) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9993666666666667) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0016170225430592812) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9996809523809523) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0011905809477718342) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9988428571428571) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.002042357581276089) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9998476190476191) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0008380952380952382) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9980571428571428) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.0023403005795668832) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9999428571428571) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0005185012414787124) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9969666666666667) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.002289832599412746) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:70 - training/epoch:1 - critic/score/mean:0.62890625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62890625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.011535261757671833 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.011535261757671833 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:305.1953125 - response_length/max:1167.0 - response_length/min:134.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:305.1953125 - response_length_non_aborted/max:1167.0 - response_length_non_aborted/min:134.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:298.21875 - prompt_length/max:701.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.000107618048787117 - timing_s/agent_loop/generate_sequences/min:np.float64(1.561738733202219) - timing_s/agent_loop/generate_sequences/max:np.float64(8.901570096611977) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.487381806007761) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.901570096611977) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:191 - timing_s/agent_loop/slowest/response_length:1167 - timing_s/gen:15.990489525720477 - timing_s/reward:0.05796127952635288 - timing_s/old_log_prob:2.9395081773400307 - timing_s/adv:0.5521378461271524 - timing_s/update_actor:21.566690308973193 - timing_s/step:41.193913504481316 - timing_s/testing:129.7496616318822 - timing_s/save_checkpoint:13.531663544476032 - timing_s/stop_profile:0.00015689432621002197 - timing_per_token_ms/update_actor:0.13961372340311765 - timing_per_token_ms/gen:0.20466516735851117 - timing_per_token_ms/adv:0.0035743092438025328 - perf/total_num_tokens:154474 - perf/time_per_step:41.193913504481316 - perf/throughput:468.7403637408577
(TaskRunner pid=2255397) Training Progress: 70%|███████ | 70/100 [1:04:59<41:54, 83.82s/it]
(TaskRunner pid=2255397) step:71 - global_seqlen/min:19243 - global_seqlen/max:22825 - global_seqlen/minmax_diff:3582 - global_seqlen/balanced_min:21239 - global_seqlen/balanced_max:21249 - global_seqlen/mean:21244.625 - actor/entropy:0.48243123292922974 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1782420575618744 - training/rollout_probs_diff_mean:0.004945038352161646 - training/rollout_probs_diff_std:0.010490105487406254 - training/rollout_actor_probs_pearson_corr:0.9993298649787903 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.73828125 - self_distillation/correct_sample_fraction:0.58984375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.73828125 - rollout_corr/rollout_is_mean:1.0001386404037476 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.646081805229187 - rollout_corr/rollout_is_min:0.6092609167098999 - rollout_corr/rollout_is_std:0.036472681909799576 - rollout_corr/rollout_is_eff_sample_size:0.998671986340567 - rollout_corr/rollout_is_seq_mean:1.00009286403656 - rollout_corr/rollout_is_seq_std:0.0023478141520172358 - rollout_corr/rollout_is_seq_max:1.0064475536346436 - rollout_corr/rollout_is_seq_min:0.9918783903121948 - rollout_corr/rollout_is_seq_max_deviation:0.008121609687805176 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6814591218717396) - rollout_corr/training_log_ppl:np.float64(0.5044128161389381) - rollout_corr/kl:np.float64(0.0009273746180333831) - rollout_corr/k3_kl:np.float64(0.0009558818163668548) - rollout_corr/rollout_ppl:np.float64(1.679951566271484) - rollout_corr/rollout_log_ppl:np.float64(0.5034854415280279) - rollout_corr/log_ppl_diff:np.float64(0.0009273746109101921) - rollout_corr/log_ppl_abs_diff:np.float64(0.0021467784827109426) - rollout_corr/log_ppl_diff_max:np.float64(0.009796887636184692) - rollout_corr/log_ppl_diff_min:np.float64(-0.007009267807006836) - rollout_corr/ppl_ratio:np.float64(1.0009312871843576) - rollout_corr/chi2_token:np.float64(0.002058735117316246) - rollout_corr/chi2_seq:np.float64(0.9684193118009716) - actor/pg_loss:np.float64(0.002505662152543664) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00036506839978756034) - actor/ppo_kl:np.float64(0.0002983230811750559) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.73828125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.73828125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.58984375) - self_distillation/token_reweight_w_mean:np.float64(1927.8832940307911) - self_distillation/token_reweight_w_std:np.float64(37339.10144847393) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0009577083401382) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08062945469282568) - self_distillation/empty_target_batch:np.float64(0.26171875) - actor/grad_norm:np.float64(0.5023428723216057) - perf/mfu/actor:np.float64(0.051698067264255104) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(201.09897232055664) - actor/lr:np.float64(1e-06) - training/global_step:71 - training/epoch:1 - critic/score/mean:0.58984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0013479345943778753 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0013479345943778753 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:361.51953125 - response_length/max:967.0 - response_length/min:141.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:361.51953125 - response_length_non_aborted/max:967.0 - response_length_non_aborted/min:141.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:302.375 - prompt_length/max:666.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.134435534477234e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8741174843162298) - timing_s/agent_loop/generate_sequences/max:np.float64(8.054633025079966) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.1346120286834775) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.054633025079966) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:264 - timing_s/agent_loop/slowest/response_length:967 - timing_s/gen:14.367894859984517 - timing_s/reward:0.06300288252532482 - timing_s/old_log_prob:3.210338668897748 - timing_s/adv:0.5408081188797951 - timing_s/update_actor:21.329918820410967 - timing_s/step:39.616243589669466 - timing_s/stop_profile:8.219294250011444e-05 - timing_per_token_ms/update_actor:0.12550185529522742 - timing_per_token_ms/gen:0.15524635447151797 - timing_per_token_ms/adv:0.0031820290948875013 - perf/total_num_tokens:169957 - perf/time_per_step:39.616243589669466 - perf/throughput:536.260459725663
(TaskRunner pid=2255397) Training Progress: 71%|███████ | 71/100 [1:05:39<34:06, 70.57s/it]
(TaskRunner pid=2255397) step:72 - global_seqlen/min:16380 - global_seqlen/max:21611 - global_seqlen/minmax_diff:5231 - global_seqlen/balanced_min:19858 - global_seqlen/balanced_max:19861 - global_seqlen/mean:19860.0 - actor/entropy:0.5429559946060181 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.15169164538383484 - training/rollout_probs_diff_mean:0.005359958857297897 - training/rollout_probs_diff_std:0.010805261321365833 - training/rollout_actor_probs_pearson_corr:0.9993027448654175 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83984375 - self_distillation/correct_sample_fraction:0.671875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83984375 - rollout_corr/rollout_is_mean:1.0000475645065308 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.644222617149353 - rollout_corr/rollout_is_min:0.6008924245834351 - rollout_corr/rollout_is_std:0.03811253234744072 - rollout_corr/rollout_is_eff_sample_size:0.9985496605558143 - rollout_corr/rollout_is_seq_mean:0.9999592304229736 - rollout_corr/rollout_is_seq_std:0.0024577102158218622 - rollout_corr/rollout_is_seq_max:1.0064175128936768 - rollout_corr/rollout_is_seq_min:0.9917982220649719 - rollout_corr/rollout_is_seq_max_deviation:0.008201777935028076 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7471187515184283) - rollout_corr/training_log_ppl:np.float64(0.5459236325696111) - rollout_corr/kl:np.float64(0.0010623894131924772) - rollout_corr/k3_kl:np.float64(0.0008694290827975237) - rollout_corr/rollout_ppl:np.float64(1.745131317526102) - rollout_corr/rollout_log_ppl:np.float64(0.5448612435720861) - rollout_corr/log_ppl_diff:np.float64(0.0010623889975249767) - rollout_corr/log_ppl_abs_diff:np.float64(0.0021515846019610763) - rollout_corr/log_ppl_diff_max:np.float64(0.010035932064056396) - rollout_corr/log_ppl_diff_min:np.float64(-0.00523415207862854) - rollout_corr/ppl_ratio:np.float64(1.0010662653949112) - rollout_corr/chi2_token:np.float64(0.001382074085995555) - rollout_corr/chi2_seq:np.float64(0.6654008245095611) - actor/pg_loss:np.float64(0.005195157486014068) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003912408333235362) - actor/ppo_kl:np.float64(0.0002666896097824889) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83984375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83984375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.671875) - self_distillation/token_reweight_w_mean:np.float64(7515.255449632183) - self_distillation/token_reweight_w_std:np.float64(120734.30576296372) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9992036235053092) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1153781593311578) - self_distillation/empty_target_batch:np.float64(0.16015625) - actor/grad_norm:np.float64(0.3598603457212448) - perf/mfu/actor:np.float64(0.048463804802359345) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.67909240722656) - actor/lr:np.float64(1e-06) - training/global_step:72 - training/epoch:1 - critic/score/mean:0.671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0006963299238123 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0006963299238123 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:333.78125 - response_length/max:799.0 - response_length/min:152.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:333.78125 - response_length_non_aborted/max:799.0 - response_length_non_aborted/min:152.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.84375 - prompt_length/max:620.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0005877390503883362 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6762614008039236) - timing_s/agent_loop/generate_sequences/max:np.float64(6.705963581800461) - timing_s/agent_loop/generate_sequences/mean:np.float64(3.7834387093244004) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(6.705963581800461) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:258 - timing_s/agent_loop/slowest/response_length:767 - timing_s/gen:14.802263040095568 - timing_s/reward:0.06146353296935558 - timing_s/old_log_prob:3.034574717283249 - timing_s/adv:0.524946503341198 - timing_s/update_actor:21.136499028652906 - timing_s/step:39.64624011516571 - timing_s/stop_profile:2.850964665412903e-05 - timing_per_token_ms/update_actor:0.13303435944519704 - timing_per_token_ms/gen:0.17323124052166894 - timing_per_token_ms/adv:0.003304043953557389 - perf/total_num_tokens:158880 - perf/time_per_step:39.64624011516571 - perf/throughput:500.9302254718232
(TaskRunner pid=2255397) Training Progress: 72%|███████▏ | 72/100 [1:06:19<28:36, 61.30s/it]
(TaskRunner pid=2255397) step:73 - global_seqlen/min:19647 - global_seqlen/max:22925 - global_seqlen/minmax_diff:3278 - global_seqlen/balanced_min:21192 - global_seqlen/balanced_max:21202 - global_seqlen/mean:21199.0 - actor/entropy:0.5374614596366882 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.301607221364975 - training/rollout_probs_diff_mean:0.004932303912937641 - training/rollout_probs_diff_std:0.010089674964547157 - training/rollout_actor_probs_pearson_corr:0.9993661642074585 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.7734375 - self_distillation/correct_sample_fraction:0.640625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7734375 - rollout_corr/rollout_is_mean:0.9999767541885376 - rollout_corr/rollout_is_ratio_fraction_high:1.021909702103585e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.1998536586761475 - rollout_corr/rollout_is_min:0.6301840543746948 - rollout_corr/rollout_is_std:0.03611796349287033 - rollout_corr/rollout_is_eff_sample_size:0.9986969544688866 - rollout_corr/rollout_is_seq_mean:1.0000014305114746 - rollout_corr/rollout_is_seq_std:0.0019163991091772914 - rollout_corr/rollout_is_seq_max:1.0062246322631836 - rollout_corr/rollout_is_seq_min:0.9927696585655212 - rollout_corr/rollout_is_seq_max_deviation:0.00723034143447876 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7496894244104624) - rollout_corr/training_log_ppl:np.float64(0.5468992951209657) - rollout_corr/kl:np.float64(0.0007367991016540287) - rollout_corr/k3_kl:np.float64(0.0007361613289731395) - rollout_corr/rollout_ppl:np.float64(1.7483539679087698) - rollout_corr/rollout_log_ppl:np.float64(0.5461624904419295) - rollout_corr/log_ppl_diff:np.float64(0.0007368046790361404) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016733732772991061) - rollout_corr/log_ppl_diff_max:np.float64(0.009091198444366455) - rollout_corr/log_ppl_diff_min:np.float64(-0.005505651235580444) - rollout_corr/ppl_ratio:np.float64(1.0007393239066005) - rollout_corr/chi2_token:np.float64(0.0014872262254357338) - rollout_corr/chi2_seq:np.float64(0.43628761847503483) - actor/pg_loss:np.float64(0.0029139972757548094) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00018539705752118607) - actor/ppo_kl:np.float64(6.380892117796577e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7734375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7734375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.640625) - self_distillation/token_reweight_w_mean:np.float64(1624.953690484399) - self_distillation/token_reweight_w_std:np.float64(32560.52424334915) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9991668974980712) - self_distillation/token_reweight_w_clip_frac:np.float64(0.06964316591620445) - self_distillation/empty_target_batch:np.float64(0.2265625) - actor/grad_norm:np.float64(0.21447288990020752) - perf/mfu/actor:np.float64(0.050516435918862526) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.0999493598938) - actor/lr:np.float64(1e-06) - training/global_step:73 - training/epoch:1 - critic/score/mean:0.640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007218515034765005 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007218515034765005 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:382.25 - response_length/max:996.0 - response_length/min:130.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:382.25 - response_length_non_aborted/max:996.0 - response_length_non_aborted/min:130.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.21875 - prompt_length/max:557.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.8507784008979797e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6397324707359076) - timing_s/agent_loop/generate_sequences/max:np.float64(8.402389820665121) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.477701659416198) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.402389820665121) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:216 - timing_s/agent_loop/slowest/response_length:996 - timing_s/gen:15.686312183737755 - timing_s/reward:0.06267971731722355 - timing_s/old_log_prob:2.8734097313135862 - timing_s/adv:0.5590537320822477 - timing_s/update_actor:21.74830207787454 - timing_s/step:41.018087424337864 - timing_s/stop_profile:0.00012700818479061127 - timing_per_token_ms/update_actor:0.12823896220266606 - timing_per_token_ms/gen:0.16029995282596626 - timing_per_token_ms/adv:0.003296462876092314 - perf/total_num_tokens:169592 - perf/time_per_step:41.018087424337864 - perf/throughput:516.8207815418933
(TaskRunner pid=2255397) Training Progress: 73%|███████▎ | 73/100 [1:07:00<24:51, 55.23s/it]
(TaskRunner pid=2255397) step:74 - global_seqlen/min:18287 - global_seqlen/max:25091 - global_seqlen/minmax_diff:6804 - global_seqlen/balanced_min:20904 - global_seqlen/balanced_max:20912 - global_seqlen/mean:20910.0 - actor/entropy:0.5127137303352356 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.27330654859542847 - training/rollout_probs_diff_mean:0.004899736028164625 - training/rollout_probs_diff_std:0.010343815200030804 - training/rollout_actor_probs_pearson_corr:0.9993671774864197 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83203125 - self_distillation/correct_sample_fraction:0.703125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83203125 - rollout_corr/rollout_is_mean:1.000099539756775 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.0278759873472154e-05 - rollout_corr/rollout_is_max:1.7854251861572266 - rollout_corr/rollout_is_min:0.48449382185935974 - rollout_corr/rollout_is_std:0.0365249402821064 - rollout_corr/rollout_is_eff_sample_size:0.9986679439991228 - rollout_corr/rollout_is_seq_mean:1.000190258026123 - rollout_corr/rollout_is_seq_std:0.0022744936868548393 - rollout_corr/rollout_is_seq_max:1.0073591470718384 - rollout_corr/rollout_is_seq_min:0.9917874336242676 - rollout_corr/rollout_is_seq_max_deviation:0.008212566375732422 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7097132932394743) - rollout_corr/training_log_ppl:np.float64(0.5229829574818723) - rollout_corr/kl:np.float64(0.0006870645115757412) - rollout_corr/k3_kl:np.float64(0.0007633993515128168) - rollout_corr/rollout_ppl:np.float64(1.7085823002271354) - rollout_corr/rollout_log_ppl:np.float64(0.5222958918311633) - rollout_corr/log_ppl_diff:np.float64(0.0006870656507089734) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019122406374663115) - rollout_corr/log_ppl_diff_max:np.float64(0.008660197257995605) - rollout_corr/log_ppl_diff_min:np.float64(-0.005268260836601257) - rollout_corr/ppl_ratio:np.float64(1.000690019223839) - rollout_corr/chi2_token:np.float64(0.0016905386000871658) - rollout_corr/chi2_seq:np.float64(1.4013629674445838) - actor/pg_loss:np.float64(0.0029980618273839355) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00017913388501256122) - actor/ppo_kl:np.float64(0.00016305825718454514) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83203125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83203125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.703125) - self_distillation/token_reweight_w_mean:np.float64(15.748328928370029) - self_distillation/token_reweight_w_std:np.float64(289.3728471599752) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0004254090599716) - self_distillation/token_reweight_w_clip_frac:np.float64(0.06948782474501058) - self_distillation/empty_target_batch:np.float64(0.16796875) - actor/grad_norm:np.float64(0.2460213266313076) - perf/mfu/actor:np.float64(0.0504502030203242) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.804443359375) - actor/lr:np.float64(1e-06) - training/global_step:74 - training/epoch:1 - critic/score/mean:0.703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005830626469105482 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005830626469105482 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:380.03125 - response_length/max:953.0 - response_length/min:134.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:380.03125 - response_length_non_aborted/max:953.0 - response_length_non_aborted/min:134.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.40625 - prompt_length/max:665.0 - prompt_length/min:162.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.984895586967468e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.763165319338441) - timing_s/agent_loop/generate_sequences/max:np.float64(8.103548426181078) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.310445136674389) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.103548426181078) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:291 - timing_s/agent_loop/slowest/response_length:953 - timing_s/gen:14.785902259871364 - timing_s/reward:0.07314063049852848 - timing_s/old_log_prob:2.9278498981148005 - timing_s/adv:0.5283288117498159 - timing_s/update_actor:21.43048808723688 - timing_s/step:39.835200319066644 - timing_s/stop_profile:0.00012897513806819916 - timing_per_token_ms/update_actor:0.1281114782833386 - timing_per_token_ms/gen:0.15198074027497085 - timing_per_token_ms/adv:0.0031583501419764222 - perf/total_num_tokens:167280 - perf/time_per_step:39.835200319066644 - perf/throughput:524.9126358727428
(TaskRunner pid=2255397) Training Progress: 74%|███████▍ | 74/100 [1:07:40<21:56, 50.63s/it]
(TaskRunner pid=2255397) step:75 - global_seqlen/min:16481 - global_seqlen/max:26987 - global_seqlen/minmax_diff:10506 - global_seqlen/balanced_min:22439 - global_seqlen/balanced_max:22447 - global_seqlen/mean:22444.625 - actor/entropy:0.5642927289009094 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2400461733341217 - training/rollout_probs_diff_mean:0.0051166643388569355 - training/rollout_probs_diff_std:0.010140416212379932 - training/rollout_actor_probs_pearson_corr:0.9994109869003296 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.8046875 - self_distillation/correct_sample_fraction:0.71484375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8046875 - rollout_corr/rollout_is_mean:0.9999061822891235 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.0068769370263908e-05 - rollout_corr/rollout_is_max:1.5713963508605957 - rollout_corr/rollout_is_min:0.4978005588054657 - rollout_corr/rollout_is_std:0.03701609745621681 - rollout_corr/rollout_is_eff_sample_size:0.9986314455154365 - rollout_corr/rollout_is_seq_mean:0.999901294708252 - rollout_corr/rollout_is_seq_std:0.0019736075773835182 - rollout_corr/rollout_is_seq_max:1.0060852766036987 - rollout_corr/rollout_is_seq_min:0.9931412935256958 - rollout_corr/rollout_is_seq_max_deviation:0.006858706474304199 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.791621611919254) - rollout_corr/training_log_ppl:np.float64(0.570267406059429) - rollout_corr/kl:np.float64(0.0008821988329259511) - rollout_corr/k3_kl:np.float64(0.0008163573830870519) - rollout_corr/rollout_ppl:np.float64(1.7901061396114528) - rollout_corr/rollout_log_ppl:np.float64(0.56938520638505) - rollout_corr/log_ppl_diff:np.float64(0.0008821996743790805) - rollout_corr/log_ppl_abs_diff:np.float64(0.00185499427607283) - rollout_corr/log_ppl_diff_max:np.float64(0.00786745548248291) - rollout_corr/log_ppl_diff_min:np.float64(-0.005721062421798706) - rollout_corr/ppl_ratio:np.float64(1.000884982291609) - rollout_corr/chi2_token:np.float64(0.001510836649686098) - rollout_corr/chi2_seq:np.float64(0.9038489798549563) - actor/pg_loss:np.float64(0.002076247939839959) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00023045666375764995) - actor/ppo_kl:np.float64(5.9166708297198056e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8046875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8046875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.71484375) - self_distillation/token_reweight_w_mean:np.float64(6473.88584861136) - self_distillation/token_reweight_w_std:np.float64(110608.01085398125) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9989379325415939) - self_distillation/token_reweight_w_clip_frac:np.float64(0.0619521141925361) - self_distillation/empty_target_batch:np.float64(0.1953125) - actor/grad_norm:np.float64(0.19757572747766972) - perf/mfu/actor:np.float64(0.053976922098505964) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.7879409790039) - actor/lr:np.float64(1e-06) - training/global_step:75 - training/epoch:1 - critic/score/mean:0.71484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0026480865199118853 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0026480865199118853 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:387.95703125 - response_length/max:918.0 - response_length/min:166.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:387.95703125 - response_length_non_aborted/max:918.0 - response_length_non_aborted/min:166.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:313.4375 - prompt_length/max:608.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013186223804950714 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1773367393761873) - timing_s/agent_loop/generate_sequences/max:np.float64(8.078486654907465) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.527875997897354) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.078486654907465) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:311 - timing_s/agent_loop/slowest/response_length:918 - timing_s/gen:14.925510486587882 - timing_s/reward:0.05783644877374172 - timing_s/old_log_prob:2.9777278639376163 - timing_s/adv:0.5886598229408264 - timing_s/update_actor:21.493691984564066 - timing_s/step:40.15656405314803 - timing_s/stop_profile:7.43325799703598e-05 - timing_per_token_ms/update_actor:0.11970400477042982 - timing_per_token_ms/gen:0.1502815277000703 - timing_per_token_ms/adv:0.003278400858450667 - perf/total_num_tokens:179557 - perf/time_per_step:40.15656405314803 - perf/throughput:558.9279244681911
(TaskRunner pid=2255397) Training Progress: 75%|███████▌ | 75/100 [1:08:20<19:47, 47.50s/it]
(TaskRunner pid=2255397) step:76 - global_seqlen/min:20385 - global_seqlen/max:23735 - global_seqlen/minmax_diff:3350 - global_seqlen/balanced_min:22458 - global_seqlen/balanced_max:22461 - global_seqlen/mean:22459.875 - actor/entropy:0.5038673281669617 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.17917871475219727 - training/rollout_probs_diff_mean:0.004544694907963276 - training/rollout_probs_diff_std:0.009781488217413425 - training/rollout_actor_probs_pearson_corr:0.9994357228279114 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.7734375 - self_distillation/correct_sample_fraction:0.6171875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7734375 - rollout_corr/rollout_is_mean:0.999882698059082 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.531444787979126 - rollout_corr/rollout_is_min:0.6298530101776123 - rollout_corr/rollout_is_std:0.03470920771360397 - rollout_corr/rollout_is_eff_sample_size:0.9987963637779437 - rollout_corr/rollout_is_seq_mean:0.9998737573623657 - rollout_corr/rollout_is_seq_std:0.001968828495591879 - rollout_corr/rollout_is_seq_max:1.0092756748199463 - rollout_corr/rollout_is_seq_min:0.9928333163261414 - rollout_corr/rollout_is_seq_max_deviation:0.009275674819946289 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7229888113215566) - rollout_corr/training_log_ppl:np.float64(0.5273766612517647) - rollout_corr/kl:np.float64(0.0006758117173214506) - rollout_corr/k3_kl:np.float64(0.0006499772979395857) - rollout_corr/rollout_ppl:np.float64(1.72180867055431) - rollout_corr/rollout_log_ppl:np.float64(0.526700847200118) - rollout_corr/log_ppl_diff:np.float64(0.0006758140516467392) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016295489040203393) - rollout_corr/log_ppl_diff_max:np.float64(0.007842868566513062) - rollout_corr/log_ppl_diff_min:np.float64(-0.005845367908477783) - rollout_corr/ppl_ratio:np.float64(1.0006780109833926) - rollout_corr/chi2_token:np.float64(0.0012538495939224958) - rollout_corr/chi2_seq:np.float64(0.6266895108856261) - actor/pg_loss:np.float64(0.0024852192727848887) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00022367794190358836) - actor/ppo_kl:np.float64(-8.031681899822729e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7734375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7734375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6171875) - self_distillation/token_reweight_w_mean:np.float64(11.586508150445297) - self_distillation/token_reweight_w_std:np.float64(205.42800806532614) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9991163620725274) - self_distillation/token_reweight_w_clip_frac:np.float64(0.07528238961822353) - self_distillation/empty_target_batch:np.float64(0.2265625) - actor/grad_norm:np.float64(0.21515963971614838) - perf/mfu/actor:np.float64(0.05377171741192728) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.86333084106445) - actor/lr:np.float64(1e-06) - training/global_step:76 - training/epoch:1 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0009749584714882076 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0009749584714882076 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:416.18359375 - response_length/max:835.0 - response_length/min:146.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:416.18359375 - response_length_non_aborted/max:835.0 - response_length_non_aborted/min:146.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:285.6875 - prompt_length/max:526.0 - prompt_length/min:182.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015104003250598907 - timing_s/agent_loop/generate_sequences/min:np.float64(1.69934743642807) - timing_s/agent_loop/generate_sequences/max:np.float64(7.779057312756777) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.804315655819664) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.779057312756777) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:198 - timing_s/agent_loop/slowest/response_length:835 - timing_s/gen:14.658727264031768 - timing_s/reward:0.06544599309563637 - timing_s/old_log_prob:3.1654940266162157 - timing_s/adv:0.5449143052101135 - timing_s/update_actor:21.7036225348711 - timing_s/step:40.22667368501425 - timing_s/stop_profile:0.00010753795504570007 - timing_per_token_ms/update_actor:0.1207910915291776 - timing_per_token_ms/gen:0.13758508080335421 - timing_per_token_ms/adv:0.003032710028495893 - perf/total_num_tokens:179679 - perf/time_per_step:40.22667368501425 - perf/throughput:558.3328906552629
(TaskRunner pid=2255397) Training Progress: 76%|███████▌ | 76/100 [1:09:00<18:08, 45.33s/it]
(TaskRunner pid=2255397) step:77 - global_seqlen/min:19528 - global_seqlen/max:26165 - global_seqlen/minmax_diff:6637 - global_seqlen/balanced_min:22384 - global_seqlen/balanced_max:22391 - global_seqlen/mean:22389.375 - actor/entropy:0.5504021048545837 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2723144292831421 - training/rollout_probs_diff_mean:0.005151774734258652 - training/rollout_probs_diff_std:0.01037499401718378 - training/rollout_actor_probs_pearson_corr:0.9993544816970825 - self_distillation/success_group_fraction:0.90625 - self_distillation/success_sample_fraction:0.8984375 - self_distillation/correct_sample_fraction:0.7421875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8984375 - rollout_corr/rollout_is_mean:0.9999526739120483 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6774260997772217 - rollout_corr/rollout_is_min:0.6362783908843994 - rollout_corr/rollout_is_std:0.037128645926713943 - rollout_corr/rollout_is_eff_sample_size:0.9986231237591031 - rollout_corr/rollout_is_seq_mean:0.9999860525131226 - rollout_corr/rollout_is_seq_std:0.002109500812366605 - rollout_corr/rollout_is_seq_max:1.0075759887695312 - rollout_corr/rollout_is_seq_min:0.993508517742157 - rollout_corr/rollout_is_seq_max_deviation:0.00757598876953125 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7856992213055491) - rollout_corr/training_log_ppl:np.float64(0.5679072652128525) - rollout_corr/kl:np.float64(0.0009176925997458341) - rollout_corr/k3_kl:np.float64(0.0007757482836723284) - rollout_corr/rollout_ppl:np.float64(1.784037644509226) - rollout_corr/rollout_log_ppl:np.float64(0.5669895681203343) - rollout_corr/log_ppl_diff:np.float64(0.0009176970925182104) - rollout_corr/log_ppl_abs_diff:np.float64(0.0019323080778121948) - rollout_corr/log_ppl_diff_max:np.float64(0.007754087448120117) - rollout_corr/log_ppl_diff_min:np.float64(-0.00577545166015625) - rollout_corr/ppl_ratio:np.float64(1.00092074717395) - rollout_corr/chi2_token:np.float64(0.00127337034791708) - rollout_corr/chi2_seq:np.float64(0.6404874103609473) - actor/pg_loss:np.float64(0.0034807646879926324) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002253228935842344) - actor/ppo_kl:np.float64(0.00018203446227005315) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8984375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8984375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.7421875) - self_distillation/token_reweight_w_mean:np.float64(130.1160973869264) - self_distillation/token_reweight_w_std:np.float64(2325.1732160223182) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0002648527733982) - self_distillation/token_reweight_w_clip_frac:np.float64(0.091672543407185) - self_distillation/empty_target_batch:np.float64(0.1015625) - actor/grad_norm:np.float64(0.2861635349690914) - perf/mfu/actor:np.float64(0.053185240362257795) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.9845733642578) - actor/lr:np.float64(1e-06) - training/global_step:77 - training/epoch:1 - critic/score/mean:0.7421875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.7421875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0007428093231283128 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0007428093231283128 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:382.57421875 - response_length/max:872.0 - response_length/min:172.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:382.57421875 - response_length_non_aborted/max:872.0 - response_length_non_aborted/min:172.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:317.09375 - prompt_length/max:716.0 - prompt_length/min:173.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.42423939704895e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.215482749044895) - timing_s/agent_loop/generate_sequences/max:np.float64(7.765327328816056) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.492655532936624) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.765327328816056) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:368 - timing_s/agent_loop/slowest/response_length:857 - timing_s/gen:14.60610330849886 - timing_s/reward:0.06456474959850311 - timing_s/old_log_prob:2.9403065145015717 - timing_s/adv:0.5702805798500776 - timing_s/update_actor:21.948025058954954 - timing_s/step:40.21948102675378 - timing_s/stop_profile:0.00011724233627319336 - timing_per_token_ms/update_actor:0.12253594092596909 - timing_per_token_ms/gen:0.14913469923624767 - timing_per_token_ms/adv:0.003183879517907923 - perf/total_num_tokens:179115 - perf/time_per_step:40.21948102675378 - perf/throughput:556.6798583280254
(TaskRunner pid=2255397) Training Progress: 77%|███████▋ | 77/100 [1:09:40<16:47, 43.82s/it]
(TaskRunner pid=2255397) step:78 - global_seqlen/min:20134 - global_seqlen/max:31492 - global_seqlen/minmax_diff:11358 - global_seqlen/balanced_min:22482 - global_seqlen/balanced_max:29618 - global_seqlen/mean:23385.375 - actor/entropy:0.48590248823165894 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.40464895963668823 - training/rollout_probs_diff_mean:0.004600349348038435 - training/rollout_probs_diff_std:0.009926710277795792 - training/rollout_actor_probs_pearson_corr:0.9993968605995178 - self_distillation/success_group_fraction:0.90625 - self_distillation/success_sample_fraction:0.89453125 - self_distillation/correct_sample_fraction:0.68359375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.89453125 - rollout_corr/rollout_is_mean:0.9998959302902222 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.656884347146843e-06 - rollout_corr/rollout_is_max:1.8092979192733765 - rollout_corr/rollout_is_min:0.48850229382514954 - rollout_corr/rollout_is_std:0.03525106608867645 - rollout_corr/rollout_is_eff_sample_size:0.9987587856840187 - rollout_corr/rollout_is_seq_mean:0.9998641014099121 - rollout_corr/rollout_is_seq_std:0.0020814789459109306 - rollout_corr/rollout_is_seq_max:1.0097109079360962 - rollout_corr/rollout_is_seq_min:0.9942805171012878 - rollout_corr/rollout_is_seq_max_deviation:0.009710907936096191 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7207122039981186) - rollout_corr/training_log_ppl:np.float64(0.5288956113836321) - rollout_corr/kl:np.float64(0.0008265634318824766) - rollout_corr/k3_kl:np.float64(0.0007540007430009155) - rollout_corr/rollout_ppl:np.float64(1.7192450258880854) - rollout_corr/rollout_log_ppl:np.float64(0.5280690497056639) - rollout_corr/log_ppl_diff:np.float64(0.0008265616779681295) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017743823991622776) - rollout_corr/log_ppl_diff_max:np.float64(0.009331107139587402) - rollout_corr/log_ppl_diff_min:np.float64(-0.004447638988494873) - rollout_corr/ppl_ratio:np.float64(1.0008291078265756) - rollout_corr/chi2_token:np.float64(0.0013703966978937387) - rollout_corr/chi2_seq:np.float64(0.8413228269200772) - actor/pg_loss:np.float64(0.004617663682438433) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00030698799764650175) - actor/ppo_kl:np.float64(-3.2745020313029727e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.89453125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.89453125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.68359375) - self_distillation/token_reweight_w_mean:np.float64(38.89375674771145) - self_distillation/token_reweight_w_std:np.float64(700.807301894878) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.001390076475218) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09478579316055402) - self_distillation/empty_target_batch:np.float64(0.10546875) - actor/grad_norm:np.float64(0.28953084722161293) - perf/mfu/actor:np.float64(0.0530806510376341) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.00171661376953) - actor/lr:np.float64(1e-06) - training/global_step:78 - training/epoch:1 - critic/score/mean:0.68359375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.68359375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.025150412693619728 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.025150412693619728 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:451.23046875 - response_length/max:8192.0 - response_length/min:163.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:451.23046875 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:163.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:279.5625 - prompt_length/max:449.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.3435564041137695e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.152110392227769) - timing_s/agent_loop/generate_sequences/max:np.float64(59.49123938381672) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.095185146856238) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(59.49123938381672) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:398 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:66.28803966008127 - timing_s/reward:0.0661248117685318 - timing_s/old_log_prob:3.3414545375853777 - timing_s/adv:0.5907729119062424 - timing_s/update_actor:23.06786256469786 - timing_s/step:93.44836739264429 - timing_s/stop_profile:0.00011118501424789429 - timing_per_token_ms/update_actor:0.12330282582970052 - timing_per_token_ms/gen:0.5738478955986778 - timing_per_token_ms/adv:0.003157811837025504 - perf/total_num_tokens:187083 - perf/time_per_step:93.44836739264429 - perf/throughput:250.24915525534115
(TaskRunner pid=2255397) Training Progress: 78%|███████▊ | 78/100 [1:11:14<21:31, 58.72s/it]
(TaskRunner pid=2255397) step:79 - global_seqlen/min:20379 - global_seqlen/max:23902 - global_seqlen/minmax_diff:3523 - global_seqlen/balanced_min:22483 - global_seqlen/balanced_max:22828 - global_seqlen/mean:22531.875 - actor/entropy:0.47975319623947144 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.999380886554718 - training/rollout_probs_diff_mean:0.004774122033268213 - training/rollout_probs_diff_std:0.010841231793165207 - training/rollout_actor_probs_pearson_corr:0.9993086457252502 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.83203125 - self_distillation/correct_sample_fraction:0.609375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.83203125 - rollout_corr/rollout_is_mean:0.9999167919158936 - rollout_corr/rollout_is_ratio_fraction_high:8.872878424881492e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.7745756849762984e-05 - rollout_corr/rollout_is_max:2.256157159805298 - rollout_corr/rollout_is_min:1.8565188497632334e-07 - rollout_corr/rollout_is_std:0.035090066492557526 - rollout_corr/rollout_is_eff_sample_size:0.9987699636823175 - rollout_corr/rollout_is_seq_mean:0.9999032020568848 - rollout_corr/rollout_is_seq_std:0.002013471210375428 - rollout_corr/rollout_is_seq_max:1.0075592994689941 - rollout_corr/rollout_is_seq_min:0.9934999942779541 - rollout_corr/rollout_is_seq_max_deviation:0.007559299468994141 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6734817675314844) - rollout_corr/training_log_ppl:np.float64(0.5028578019700944) - rollout_corr/kl:np.float64(0.0007867071553491201) - rollout_corr/k3_kl:np.float64(0.0008091878738127889) - rollout_corr/rollout_ppl:np.float64(1.6721355388872325) - rollout_corr/rollout_log_ppl:np.float64(0.5020710933895316) - rollout_corr/log_ppl_diff:np.float64(0.0007867085805628449) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017737957823555917) - rollout_corr/log_ppl_diff_max:np.float64(0.022169530391693115) - rollout_corr/log_ppl_diff_min:np.float64(-0.006566405296325684) - rollout_corr/ppl_ratio:np.float64(1.000790162710473) - rollout_corr/chi2_token:np.float64(0.0015391185879707336) - rollout_corr/chi2_seq:np.float64(1.4496739441528916) - actor/pg_loss:np.float64(0.005306927720084786) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00039829218349041184) - actor/ppo_kl:np.float64(-4.736486153222863e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.83203125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.83203125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.609375) - self_distillation/token_reweight_w_mean:np.float64(76.99012057809159) - self_distillation/token_reweight_w_std:np.float64(1541.8618622039212) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9995152873452753) - self_distillation/token_reweight_w_clip_frac:np.float64(0.11978079983964562) - self_distillation/empty_target_batch:np.float64(0.16796875) - actor/grad_norm:np.float64(0.2994599714875221) - perf/mfu/actor:np.float64(0.052974969811297) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.06073379516602) - actor/lr:np.float64(1e-06) - training/global_step:79 - training/epoch:1 - critic/score/mean:0.609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.009753511287271976 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.009753511287271976 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:440.24609375 - response_length/max:1635.0 - response_length/min:188.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:440.24609375 - response_length_non_aborted/max:1635.0 - response_length_non_aborted/min:188.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.875 - prompt_length/max:766.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010463036596775055 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4277297761291265) - timing_s/agent_loop/generate_sequences/max:np.float64(12.42059437930584) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.095716175761481) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(12.42059437930584) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:234 - timing_s/agent_loop/slowest/response_length:1635 - timing_s/gen:19.97506350465119 - timing_s/reward:0.06424454413354397 - timing_s/old_log_prob:2.966035883873701 - timing_s/adv:0.6315709538757801 - timing_s/update_actor:21.955910548567772 - timing_s/step:45.67770639620721 - timing_s/stop_profile:3.654882311820984e-05 - timing_per_token_ms/update_actor:0.12180472413285497 - timing_per_token_ms/gen:0.17723630697187465 - timing_per_token_ms/adv:0.003503763856069347 - perf/total_num_tokens:180255 - perf/time_per_step:45.67770639620721 - perf/throughput:493.2794743361042
(TaskRunner pid=2255397) Training Progress: 79%|███████▉ | 79/100 [1:12:00<19:11, 54.82s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 80}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1", we must analyze the structure and functional groups of the target molecule and compare it to the structures of the given options.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The target molecule contains:
(TaskRunner pid=2255397) - A nitrogen-containing ring (likely an azole or similar).
(TaskRunner pid=2255397) - A chlorinated aromatic ring (–Cl).
(TaskRunner pid=2255397) - A trifluoromethyl group (–C(F)(F)F).
(TaskRunner pid=2255397) - A nitrogen-containing side chain that is connected to a carbon chain with a chlorinated aromatic group and a trifluoromethyl group.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Now, we analyze the options:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option A:**
(TaskRunner pid=2255397) "O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"
(TaskRunner pid=2255397) - Contains a chlorinated aromatic ring (–Cl).
(TaskRunner pid=2255397) - Contains a trifluoromethyl group (–C(F)(F)F).
(TaskRunner pid=2255397) - Contains a nitrogen-containing ring with a side chain that connects to the chlorinated aromatic ring and the trifluoromethyl group.
(TaskRunner pid=2255397) - Matches the structure of the target molecule closely.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option B:**
(TaskRunner pid=2255397) "O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1"
(TaskRunner pid=2255397) - Contains a chlorinated ring, but the side chain and functional groups do not match the target molecule.
(TaskRunner pid=2255397) - The nitrogen-containing ring is different and does not include the trifluoromethyl group.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option C:**
(TaskRunner pid=2255397) "O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1"
(TaskRunner pid=2255397) - Contains a chlorinated aromatic ring and a trifluoromethyl group.
(TaskRunner pid=2255397) - However, the functional group arrangement and the nitrogen-containing ring do not match the target molecule's structure.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option D:**
(TaskRunner pid=2255397) "O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F"
(TaskRunner pid=2255397) - Contains a trifluoromethyl group and a chlorinated ring.
(TaskRunner pid=2255397) - However, the nitrogen-containing ring and the side chain are not structured correctly to match the target molecule.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Conclusion:**
(TaskRunner pid=2255397) Only **Option A** contains all the necessary functional groups and structural elements that match the target molecule "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1".
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_80
(TaskRunner pid=2255397) step:80 - global_seqlen/min:21379 - global_seqlen/max:27736 - global_seqlen/minmax_diff:6357 - global_seqlen/balanced_min:24041 - global_seqlen/balanced_max:24046 - global_seqlen/mean:24044.0 - actor/entropy:0.5321111679077148 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.22554826736450195 - training/rollout_probs_diff_mean:0.005127668380737305 - training/rollout_probs_diff_std:0.010481650941073895 - training/rollout_actor_probs_pearson_corr:0.999333381652832 - self_distillation/success_group_fraction:0.90625 - self_distillation/success_sample_fraction:0.90234375 - self_distillation/correct_sample_fraction:0.7109375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.90234375 - rollout_corr/rollout_is_mean:0.9999201893806458 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.784020185470581 - rollout_corr/rollout_is_min:0.5773587822914124 - rollout_corr/rollout_is_std:0.03691612929105759 - rollout_corr/rollout_is_eff_sample_size:0.9986386974447631 - rollout_corr/rollout_is_seq_mean:0.9999967813491821 - rollout_corr/rollout_is_seq_std:0.002032036194577813 - rollout_corr/rollout_is_seq_max:1.0090333223342896 - rollout_corr/rollout_is_seq_min:0.99375981092453 - rollout_corr/rollout_is_seq_max_deviation:0.00903332233428955 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.7447218438610435) - rollout_corr/training_log_ppl:np.float64(0.5451616903301328) - rollout_corr/kl:np.float64(0.0007389033336671069) - rollout_corr/k3_kl:np.float64(0.0007813431365661927) - rollout_corr/rollout_ppl:np.float64(1.743419658858329) - rollout_corr/rollout_log_ppl:np.float64(0.5444227863335982) - rollout_corr/log_ppl_diff:np.float64(0.0007389039965346456) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016596049536019564) - rollout_corr/log_ppl_diff_max:np.float64(0.005907297134399414) - rollout_corr/log_ppl_diff_min:np.float64(-0.0042874813079833984) - rollout_corr/ppl_ratio:np.float64(1.0007411232218146) - rollout_corr/chi2_token:np.float64(0.001654008636251092) - rollout_corr/chi2_seq:np.float64(0.4709114315919578) - actor/pg_loss:np.float64(0.004708616062998772) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004071622186074819) - actor/ppo_kl:np.float64(2.530917598608795e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.90234375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.90234375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.7109375) - self_distillation/token_reweight_w_mean:np.float64(13418.10702541261) - self_distillation/token_reweight_w_std:np.float64(249965.62142065133) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0001714508980513) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10478754635551013) - self_distillation/empty_target_batch:np.float64(0.09765625) - actor/grad_norm:np.float64(0.277141023427248) - perf/mfu/actor:np.float64(0.05686537883473178) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.12682485580444) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.7098214285714286) - val-aux/sciknoweval/reward/std@16:np.float64(0.14355997006676555) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7664952380952381) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.10898051269342317) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6531333333333332) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.13319323391175897) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.7104857142857143) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.14460069804004294) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8059285714285714) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.07362869726753447) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.6003619047619048) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.11450096751541655) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.7241619047619048) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.10970231087023842) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8335476190476191) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.05060468808975912) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.5525523809523809) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.0905650570350032) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7320380952380953) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.07469910045616918) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8541285714285713) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.032361125070489466) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.5139714285714285) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.0607653237770218) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7368714285714285) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.0520517104444629) - val-aux/sciknoweval/score/mean@16:np.float64(0.7098214285714286) - val-aux/sciknoweval/score/std@16:np.float64(0.14355997006676555) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7664952380952381) - val-aux/sciknoweval/score/best@2/std:np.float64(0.10898051269342317) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6531333333333332) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.13319323391175897) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.7104857142857143) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.14460069804004294) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8059285714285714) - val-aux/sciknoweval/score/best@4/std:np.float64(0.07362869726753447) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.6003619047619048) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.11450096751541655) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.7241619047619048) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.10970231087023842) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8335476190476191) - val-aux/sciknoweval/score/best@8/std:np.float64(0.05060468808975912) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.5525523809523809) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.0905650570350032) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7320380952380953) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.07469910045616918) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8541285714285713) - val-aux/sciknoweval/score/best@16/std:np.float64(0.032361125070489466) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.5139714285714285) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.0607653237770218) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7368714285714285) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.0520517104444629) - val-core/sciknoweval/acc/mean@16:np.float64(0.7098214285714286) - val-aux/sciknoweval/acc/std@16:np.float64(0.14355997006676555) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7664952380952381) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.10898051269342317) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6531333333333332) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.13319323391175897) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.7104857142857143) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.14460069804004294) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8059285714285714) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.07362869726753447) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.6003619047619048) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.11450096751541655) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.7241619047619048) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.10970231087023842) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8335476190476191) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.05060468808975912) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.5525523809523809) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.0905650570350032) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7320380952380953) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.07469910045616918) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8541285714285713) - val-core/sciknoweval/acc/best@16/std:np.float64(0.032361125070489466) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.5139714285714285) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.0607653237770218) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7368714285714285) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.0520517104444629) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9991071428571429) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0034580208448280513) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9999428571428571) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0008968697662193748) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9982333333333334) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.004700687579230329) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9991285714285715) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.0034158694143255103) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9967) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.006019070521218273) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9996714285714285) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.002136984020463583) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9942619047619048) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.007002323475158229) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9999476190476191) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0008225042333482549) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9909714285714285) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.006886708259921708) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0001505093393264677) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:80 - training/epoch:1 - critic/score/mean:0.7109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.7109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.000642581726424396 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.000642581726424396 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:437.6875 - response_length/max:1165.0 - response_length/min:201.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:437.6875 - response_length_non_aborted/max:1165.0 - response_length_non_aborted/min:201.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:313.6875 - prompt_length/max:950.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:3.976747393608093e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.5183244552463293) - timing_s/agent_loop/generate_sequences/max:np.float64(9.816529558971524) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.047968282407965) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.816529558971524) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:286 - timing_s/agent_loop/slowest/response_length:1165 - timing_s/gen:16.67618520371616 - timing_s/reward:0.06666623428463936 - timing_s/old_log_prob:3.0512595120817423 - timing_s/adv:0.615724191069603 - timing_s/update_actor:21.976544301956892 - timing_s/step:42.47468729875982 - timing_s/testing:132.552817100659 - timing_s/save_checkpoint:13.726791623979807 - timing_s/stop_profile:0.00016298145055770874 - timing_per_token_ms/update_actor:0.11425170677693443 - timing_per_token_ms/gen:0.14883072615054405 - timing_per_token_ms/adv:0.003201028276646996 - perf/total_num_tokens:192352 - perf/time_per_step:42.47468729875982 - perf/throughput:566.0783287438596
(TaskRunner pid=2255397) Training Progress: 80%|████████ | 80/100 [1:15:09<31:40, 95.02s/it]
(TaskRunner pid=2255397) step:81 - global_seqlen/min:19063 - global_seqlen/max:27921 - global_seqlen/minmax_diff:8858 - global_seqlen/balanced_min:22607 - global_seqlen/balanced_max:22618 - global_seqlen/mean:22615.0 - actor/entropy:0.4837610721588135 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18462881445884705 - training/rollout_probs_diff_mean:0.004895538557320833 - training/rollout_probs_diff_std:0.010500817559659481 - training/rollout_actor_probs_pearson_corr:0.9993140697479248 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.7734375 - self_distillation/correct_sample_fraction:0.63671875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7734375 - rollout_corr/rollout_is_mean:0.9999274611473083 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.790866897674277e-06 - rollout_corr/rollout_is_max:1.9132018089294434 - rollout_corr/rollout_is_min:0.4658448398113251 - rollout_corr/rollout_is_std:0.036682888865470886 - rollout_corr/rollout_is_eff_sample_size:0.998655936060176 - rollout_corr/rollout_is_seq_mean:0.9999848008155823 - rollout_corr/rollout_is_seq_std:0.002200948540121317 - rollout_corr/rollout_is_seq_max:1.0080451965332031 - rollout_corr/rollout_is_seq_min:0.9924934506416321 - rollout_corr/rollout_is_seq_max_deviation:0.008045196533203125 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6707927286624908) - rollout_corr/training_log_ppl:np.float64(0.496672905923333) - rollout_corr/kl:np.float64(0.0007028676274636325) - rollout_corr/k3_kl:np.float64(0.0008774436023486487) - rollout_corr/rollout_ppl:np.float64(1.6695120050571859) - rollout_corr/rollout_log_ppl:np.float64(0.4959700393374078) - rollout_corr/log_ppl_diff:np.float64(0.0007028665859252214) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018403253052383661) - rollout_corr/log_ppl_diff_max:np.float64(0.008645951747894287) - rollout_corr/log_ppl_diff_min:np.float64(-0.005466446280479431) - rollout_corr/ppl_ratio:np.float64(1.0007056852336973) - rollout_corr/chi2_token:np.float64(0.002068758476525545) - rollout_corr/chi2_seq:np.float64(0.5465720517095178) - actor/pg_loss:np.float64(0.0029664566973224282) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00035411453472988796) - actor/ppo_kl:np.float64(-4.833459730235745e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7734375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7734375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.63671875) - self_distillation/token_reweight_w_mean:np.float64(46.35069849877618) - self_distillation/token_reweight_w_std:np.float64(809.1466785562807) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0000212090089917) - self_distillation/token_reweight_w_clip_frac:np.float64(0.07078566148993559) - self_distillation/empty_target_batch:np.float64(0.2265625) - actor/grad_norm:np.float64(0.2487758994102478) - perf/mfu/actor:np.float64(0.0539087676600922) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.76457595825195) - actor/lr:np.float64(1e-06) - training/global_step:81 - training/epoch:1 - critic/score/mean:0.63671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.63671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.008355281315743923 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.008355281315743923 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:398.96875 - response_length/max:1130.0 - response_length/min:140.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:398.96875 - response_length_non_aborted/max:1130.0 - response_length_non_aborted/min:140.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:307.75 - prompt_length/max:735.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.358565926551819e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.753862602636218) - timing_s/agent_loop/generate_sequences/max:np.float64(8.9995809327811) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.538052094459999) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.9995809327811) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:266 - timing_s/agent_loop/slowest/response_length:1130 - timing_s/gen:15.53846220485866 - timing_s/reward:0.06122434884309769 - timing_s/old_log_prob:3.0600726436823606 - timing_s/adv:0.5756961815059185 - timing_s/update_actor:21.892810126766562 - timing_s/step:41.233498902991414 - timing_s/stop_profile:0.00010543875396251678 - timing_per_token_ms/update_actor:0.12100823638495778 - timing_per_token_ms/gen:0.15213501806276591 - timing_per_token_ms/adv:0.0031820483169683754 - perf/total_num_tokens:180920 - perf/time_per_step:41.233498902991414 - perf/throughput:548.461823557722
(TaskRunner pid=2255397) Training Progress: 81%|████████ | 81/100 [1:15:50<24:59, 78.90s/it]
(TaskRunner pid=2255397) step:82 - global_seqlen/min:20637 - global_seqlen/max:24388 - global_seqlen/minmax_diff:3751 - global_seqlen/balanced_min:22213 - global_seqlen/balanced_max:22222 - global_seqlen/mean:22220.125 - actor/entropy:0.44993990659713745 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24135524034500122 - training/rollout_probs_diff_mean:0.0047300467267632484 - training/rollout_probs_diff_std:0.010585080832242966 - training/rollout_actor_probs_pearson_corr:0.9992892146110535 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.828125 - self_distillation/correct_sample_fraction:0.6015625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.828125 - rollout_corr/rollout_is_mean:0.9999090433120728 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.8240019926452078e-05 - rollout_corr/rollout_is_max:1.7895359992980957 - rollout_corr/rollout_is_min:0.000389088352676481 - rollout_corr/rollout_is_std:0.03580636531114578 - rollout_corr/rollout_is_eff_sample_size:0.9987193079476842 - rollout_corr/rollout_is_seq_mean:0.9998470544815063 - rollout_corr/rollout_is_seq_std:0.0019274608930572867 - rollout_corr/rollout_is_seq_max:1.0058091878890991 - rollout_corr/rollout_is_seq_min:0.9948349595069885 - rollout_corr/rollout_is_seq_max_deviation:0.005809187889099121 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6126991645433009) - rollout_corr/training_log_ppl:np.float64(0.4650081274448894) - rollout_corr/kl:np.float64(0.0009749815301454134) - rollout_corr/k3_kl:np.float64(0.0008126581537908351) - rollout_corr/rollout_ppl:np.float64(1.6110614440403879) - rollout_corr/rollout_log_ppl:np.float64(0.46403314505005255) - rollout_corr/log_ppl_diff:np.float64(0.000974982394836843) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017385364044457674) - rollout_corr/log_ppl_diff_max:np.float64(0.01138317584991455) - rollout_corr/log_ppl_diff_min:np.float64(-0.004374325275421143) - rollout_corr/ppl_ratio:np.float64(1.0009776803199202) - rollout_corr/chi2_token:np.float64(0.0012213955633342266) - rollout_corr/chi2_seq:np.float64(0.3682185474317521) - actor/pg_loss:np.float64(0.0047809198731556535) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00044549755830303184) - actor/ppo_kl:np.float64(9.73689185599369e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.828125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.828125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6015625) - self_distillation/token_reweight_w_mean:np.float64(8763.06257886719) - self_distillation/token_reweight_w_std:np.float64(183308.58723631455) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9998282515443861) - self_distillation/token_reweight_w_clip_frac:np.float64(0.10694929157034494) - self_distillation/empty_target_batch:np.float64(0.171875) - actor/grad_norm:np.float64(0.27595510706305504) - perf/mfu/actor:np.float64(0.053128164780379857) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.69163131713867) - actor/lr:np.float64(1e-06) - training/global_step:82 - training/epoch:1 - critic/score/mean:0.6015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0036035440862178802 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0036035440862178802 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:428.31640625 - response_length/max:948.0 - response_length/min:190.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:428.31640625 - response_length_non_aborted/max:948.0 - response_length_non_aborted/min:190.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.0625 - prompt_length/max:408.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010885484516620636 - timing_s/agent_loop/generate_sequences/min:np.float64(2.409843420609832) - timing_s/agent_loop/generate_sequences/max:np.float64(8.630525501444936) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.044243809614272) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.630525501444936) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:336 - timing_s/agent_loop/slowest/response_length:948 - timing_s/gen:15.308033773675561 - timing_s/reward:0.06256439350545406 - timing_s/old_log_prob:2.968469714745879 - timing_s/adv:0.5528020858764648 - timing_s/update_actor:21.71131898276508 - timing_s/step:40.68941228836775 - timing_s/stop_profile:3.694556653499603e-05 - timing_per_token_ms/update_actor:0.12213769602311575 - timing_per_token_ms/gen:0.13960942437847643 - timing_per_token_ms/adv:0.0031098052209228393 - perf/total_num_tokens:177761 - perf/time_per_step:40.68941228836775 - perf/throughput:546.0910775148323
(TaskRunner pid=2255397) Training Progress: 82%|████████▏ | 82/100 [1:16:31<20:13, 67.44s/it]
(TaskRunner pid=2255397) step:83 - global_seqlen/min:17994 - global_seqlen/max:23375 - global_seqlen/minmax_diff:5381 - global_seqlen/balanced_min:21160 - global_seqlen/balanced_max:21170 - global_seqlen/mean:21165.25 - actor/entropy:0.4580913484096527 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1788254976272583 - training/rollout_probs_diff_mean:0.005244065076112747 - training/rollout_probs_diff_std:0.011341297999024391 - training/rollout_actor_probs_pearson_corr:0.9992040395736694 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.7578125 - self_distillation/correct_sample_fraction:0.49609375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.7578125 - rollout_corr/rollout_is_mean:1.0000450611114502 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.6948878765106201 - rollout_corr/rollout_is_min:0.5044571757316589 - rollout_corr/rollout_is_std:0.03826860338449478 - rollout_corr/rollout_is_eff_sample_size:0.9985378931829969 - rollout_corr/rollout_is_seq_mean:0.9999917149543762 - rollout_corr/rollout_is_seq_std:0.0024205187801271677 - rollout_corr/rollout_is_seq_max:1.007947325706482 - rollout_corr/rollout_is_seq_min:0.9913325309753418 - rollout_corr/rollout_is_seq_max_deviation:0.008667469024658203 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6229573977179825) - rollout_corr/training_log_ppl:np.float64(0.4732308021048084) - rollout_corr/kl:np.float64(0.0009288024647275961) - rollout_corr/k3_kl:np.float64(0.0008689218968811474) - rollout_corr/rollout_ppl:np.float64(1.6214703912846744) - rollout_corr/rollout_log_ppl:np.float64(0.47230199666228145) - rollout_corr/log_ppl_diff:np.float64(0.0009288054425269365) - rollout_corr/log_ppl_abs_diff:np.float64(0.002043650601990521) - rollout_corr/log_ppl_diff_max:np.float64(0.01018601655960083) - rollout_corr/log_ppl_diff_min:np.float64(-0.0065596699714660645) - rollout_corr/ppl_ratio:np.float64(1.0009324771817774) - rollout_corr/chi2_token:np.float64(0.0016192824114114046) - rollout_corr/chi2_seq:np.float64(0.7770539077464491) - actor/pg_loss:np.float64(0.005605395417660475) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005950954305262712) - actor/ppo_kl:np.float64(0.00014756420982209306) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.7578125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.7578125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.49609375) - self_distillation/token_reweight_w_mean:np.float64(43651.46375838155) - self_distillation/token_reweight_w_std:np.float64(663337.3462570815) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.001920053968206) - self_distillation/token_reweight_w_clip_frac:np.float64(0.12145704132854007) - self_distillation/empty_target_batch:np.float64(0.2421875) - actor/grad_norm:np.float64(0.37171755731105804) - perf/mfu/actor:np.float64(0.051307628909952094) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.40789794921875) - actor/lr:np.float64(1e-06) - training/global_step:83 - training/epoch:1 - critic/score/mean:0.49609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.49609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0006963750929571688 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0006963750929571688 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:362.5078125 - response_length/max:903.0 - response_length/min:145.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:362.5078125 - response_length_non_aborted/max:903.0 - response_length_non_aborted/min:145.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:298.90625 - prompt_length/max:563.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010122545063495636 - timing_s/agent_loop/generate_sequences/min:np.float64(1.972249450162053) - timing_s/agent_loop/generate_sequences/max:np.float64(8.24346206523478) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.390775169689732) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.24346206523478) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:312 - timing_s/agent_loop/slowest/response_length:903 - timing_s/gen:14.806242102757096 - timing_s/reward:0.06265864707529545 - timing_s/old_log_prob:3.0254632886499166 - timing_s/adv:0.5114385951310396 - timing_s/update_actor:21.41834749095142 - timing_s/step:39.9120793864131 - timing_s/stop_profile:0.00010847300291061401 - timing_per_token_ms/update_actor:0.12649477026583325 - timing_per_token_ms/gen:0.15954658415505157 - timing_per_token_ms/adv:0.0030205088241990977 - perf/total_num_tokens:169322 - perf/time_per_step:39.9120793864131 - perf/throughput:530.2968506122257
(TaskRunner pid=2255397) Training Progress: 83%|████████▎ | 83/100 [1:17:10<16:46, 59.20s/it]
(TaskRunner pid=2255397) step:84 - global_seqlen/min:16326 - global_seqlen/max:23634 - global_seqlen/minmax_diff:7308 - global_seqlen/balanced_min:19956 - global_seqlen/balanced_max:19966 - global_seqlen/mean:19962.0 - actor/entropy:0.4355461001396179 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23105639219284058 - training/rollout_probs_diff_mean:0.0052682701498270035 - training/rollout_probs_diff_std:0.011572654359042645 - training/rollout_actor_probs_pearson_corr:0.9991452097892761 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.8671875 - self_distillation/correct_sample_fraction:0.6875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8671875 - rollout_corr/rollout_is_mean:0.9999979138374329 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.7088184356689453 - rollout_corr/rollout_is_min:0.6109715700149536 - rollout_corr/rollout_is_std:0.038198452442884445 - rollout_corr/rollout_is_eff_sample_size:0.9985430042300492 - rollout_corr/rollout_is_seq_mean:0.999947190284729 - rollout_corr/rollout_is_seq_std:0.0022522625513374805 - rollout_corr/rollout_is_seq_max:1.0068590641021729 - rollout_corr/rollout_is_seq_min:0.9928377866744995 - rollout_corr/rollout_is_seq_max_deviation:0.007162213325500488 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5662654708139598) - rollout_corr/training_log_ppl:np.float64(0.4385948544368148) - rollout_corr/kl:np.float64(0.0010095660052041877) - rollout_corr/k3_kl:np.float64(0.0008661396312845682) - rollout_corr/rollout_ppl:np.float64(1.5646833986975253) - rollout_corr/rollout_log_ppl:np.float64(0.4375852865341585) - rollout_corr/log_ppl_diff:np.float64(0.001009567902656272) - rollout_corr/log_ppl_abs_diff:np.float64(0.002128819382051006) - rollout_corr/log_ppl_diff_max:np.float64(0.010426610708236694) - rollout_corr/log_ppl_diff_min:np.float64(-0.0056244730949401855) - rollout_corr/ppl_ratio:np.float64(1.0010133925825357) - rollout_corr/chi2_token:np.float64(0.0014718861784785986) - rollout_corr/chi2_seq:np.float64(0.8280399544164538) - actor/pg_loss:np.float64(0.005005670711398125) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003053363407161669) - actor/ppo_kl:np.float64(0.00018772196818872544) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8671875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8671875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.6875) - self_distillation/token_reweight_w_mean:np.float64(1562.93544895784) - self_distillation/token_reweight_w_std:np.float64(29233.69878470525) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9984272990841419) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09582373814191669) - self_distillation/empty_target_batch:np.float64(0.1328125) - actor/grad_norm:np.float64(0.27201734483242035) - perf/mfu/actor:np.float64(0.04843784064306516) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.41874313354492) - actor/lr:np.float64(1e-06) - training/global_step:84 - training/epoch:1 - critic/score/mean:0.6875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0011461590183898807 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0011461590183898807 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:369.78125 - response_length/max:898.0 - response_length/min:147.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:369.78125 - response_length_non_aborted/max:898.0 - response_length_non_aborted/min:147.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:254.03125 - prompt_length/max:466.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014925189316272736 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8480602335184813) - timing_s/agent_loop/generate_sequences/max:np.float64(7.672212071716785) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.319992882228689) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.672212071716785) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:273 - timing_s/agent_loop/slowest/response_length:898 - timing_s/gen:14.744418485090137 - timing_s/reward:0.05119308829307556 - timing_s/old_log_prob:3.006163891404867 - timing_s/adv:0.5809233784675598 - timing_s/update_actor:21.276869893074036 - timing_s/step:39.69830561801791 - timing_s/stop_profile:2.6756897568702698e-05 - timing_per_token_ms/update_actor:0.13323358063491908 - timing_per_token_ms/gen:0.15575528696326096 - timing_per_token_ms/adv:0.0036376827125761436 - perf/total_num_tokens:159696 - perf/time_per_step:39.69830561801791 - perf/throughput:502.8426198356392
(TaskRunner pid=2255397) Training Progress: 84%|████████▍ | 84/100 [1:17:50<14:13, 53.36s/it]
(TaskRunner pid=2255397) step:85 - global_seqlen/min:18150 - global_seqlen/max:26353 - global_seqlen/minmax_diff:8203 - global_seqlen/balanced_min:22089 - global_seqlen/balanced_max:22097 - global_seqlen/mean:22095.125 - actor/entropy:0.5015257596969604 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.22547149658203125 - training/rollout_probs_diff_mean:0.005411901045590639 - training/rollout_probs_diff_std:0.011227205395698547 - training/rollout_actor_probs_pearson_corr:0.9992418885231018 - self_distillation/success_group_fraction:0.75 - self_distillation/success_sample_fraction:0.75 - self_distillation/correct_sample_fraction:0.71484375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.75 - rollout_corr/rollout_is_mean:1.0000245571136475 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.9998200514237396e-05 - rollout_corr/rollout_is_max:1.8690942525863647 - rollout_corr/rollout_is_min:0.3950064182281494 - rollout_corr/rollout_is_std:0.038822196424007416 - rollout_corr/rollout_is_eff_sample_size:0.9984952239250111 - rollout_corr/rollout_is_seq_mean:1.0000362396240234 - rollout_corr/rollout_is_seq_std:0.0021999897435307503 - rollout_corr/rollout_is_seq_max:1.0113379955291748 - rollout_corr/rollout_is_seq_min:0.9936926364898682 - rollout_corr/rollout_is_seq_max_deviation:0.011337995529174805 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6718262596987188) - rollout_corr/training_log_ppl:np.float64(0.5036372784525156) - rollout_corr/kl:np.float64(0.0008671445817096668) - rollout_corr/k3_kl:np.float64(0.0008546028941509576) - rollout_corr/rollout_ppl:np.float64(1.670389695558697) - rollout_corr/rollout_log_ppl:np.float64(0.5027701300568879) - rollout_corr/log_ppl_diff:np.float64(0.000867148395627737) - rollout_corr/log_ppl_abs_diff:np.float64(0.00184359191916883) - rollout_corr/log_ppl_diff_max:np.float64(0.007470607757568359) - rollout_corr/log_ppl_diff_min:np.float64(-0.005036354064941406) - rollout_corr/ppl_ratio:np.float64(1.000870036194101) - rollout_corr/chi2_token:np.float64(0.001706940121948719) - rollout_corr/chi2_seq:np.float64(0.803199248155579) - actor/pg_loss:np.float64(0.0017579543637111783) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(4.799221642315388e-05) - actor/ppo_kl:np.float64(0.00011437771347289072) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.75) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.75) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.71484375) - self_distillation/token_reweight_w_mean:np.float64(531.5800662350375) - self_distillation/token_reweight_w_std:np.float64(6761.905646110186) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.998828696552664) - self_distillation/token_reweight_w_clip_frac:np.float64(0.02915754570858553) - self_distillation/empty_target_batch:np.float64(0.25) - actor/grad_norm:np.float64(0.16114498674869537) - perf/mfu/actor:np.float64(0.052797853671240076) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.15685653686523) - actor/lr:np.float64(1e-06) - training/global_step:85 - training/epoch:1 - critic/score/mean:0.71484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0009824115550145507 - critic/advantages/max:0.5 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0009824115550145507 - critic/returns/max:0.5 - critic/returns/min:-0.875 - response_length/mean:390.66015625 - response_length/max:984.0 - response_length/min:164.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:390.66015625 - response_length_non_aborted/max:984.0 - response_length_non_aborted/min:164.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:299.8125 - prompt_length/max:706.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.7002766728401184e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1650554779917) - timing_s/agent_loop/generate_sequences/max:np.float64(8.331863490864635) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.615080809744541) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.331863490864635) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:235 - timing_s/agent_loop/slowest/response_length:984 - timing_s/gen:15.139516476541758 - timing_s/reward:0.0630094688385725 - timing_s/old_log_prob:2.9453230183571577 - timing_s/adv:0.5465950388461351 - timing_s/update_actor:21.74274236895144 - timing_s/step:40.526393212378025 - timing_s/stop_profile:0.0001103375107049942 - timing_per_token_ms/update_actor:0.12300644581639299 - timing_per_token_ms/gen:0.15138154042677918 - timing_per_token_ms/adv:0.0030922830197053375 - perf/total_num_tokens:176761 - perf/time_per_step:40.526393212378025 - perf/throughput:545.2033415411727
(TaskRunner pid=2255397) Training Progress: 85%|████████▌ | 85/100 [1:18:31<12:22, 49.52s/it]
(TaskRunner pid=2255397) step:86 - global_seqlen/min:19172 - global_seqlen/max:23707 - global_seqlen/minmax_diff:4535 - global_seqlen/balanced_min:21178 - global_seqlen/balanced_max:21185 - global_seqlen/mean:21182.625 - actor/entropy:0.4417911767959595 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2772989869117737 - training/rollout_probs_diff_mean:0.004951337352395058 - training/rollout_probs_diff_std:0.011015241034328938 - training/rollout_actor_probs_pearson_corr:0.999230682849884 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.80078125 - self_distillation/correct_sample_fraction:0.609375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.80078125 - rollout_corr/rollout_is_mean:1.0001946687698364 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.0157130418519955e-05 - rollout_corr/rollout_is_max:1.913230538368225 - rollout_corr/rollout_is_min:0.48568272590637207 - rollout_corr/rollout_is_std:0.03722003847360611 - rollout_corr/rollout_is_eff_sample_size:0.9986170608525367 - rollout_corr/rollout_is_seq_mean:1.000307321548462 - rollout_corr/rollout_is_seq_std:0.0024071792140603065 - rollout_corr/rollout_is_seq_max:1.0088176727294922 - rollout_corr/rollout_is_seq_min:0.9944450259208679 - rollout_corr/rollout_is_seq_max_deviation:0.008817672729492188 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.608684377744794) - rollout_corr/training_log_ppl:np.float64(0.4590825633786153) - rollout_corr/kl:np.float64(0.0005738776003347823) - rollout_corr/k3_kl:np.float64(0.0008272058389025005) - rollout_corr/rollout_ppl:np.float64(1.6077406979165971) - rollout_corr/rollout_log_ppl:np.float64(0.45850868528941646) - rollout_corr/log_ppl_diff:np.float64(0.0005738780891988426) - rollout_corr/log_ppl_abs_diff:np.float64(0.0020339481707196683) - rollout_corr/log_ppl_diff_max:np.float64(0.008264422416687012) - rollout_corr/log_ppl_diff_min:np.float64(-0.009043440222740173) - rollout_corr/ppl_ratio:np.float64(1.00057721324265) - rollout_corr/chi2_token:np.float64(0.0023612575605511665) - rollout_corr/chi2_seq:np.float64(1.4658026506658643) - actor/pg_loss:np.float64(0.00446285936050117) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004066334154231299) - actor/ppo_kl:np.float64(0.00014794760728875644) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.80078125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.80078125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.609375) - self_distillation/token_reweight_w_mean:np.float64(132.06133578903973) - self_distillation/token_reweight_w_std:np.float64(2865.135968480725) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9988646688871086) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1047386541031301) - self_distillation/empty_target_batch:np.float64(0.19921875) - actor/grad_norm:np.float64(0.297544926404953) - perf/mfu/actor:np.float64(0.05125659080467514) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.21656799316406) - actor/lr:np.float64(1e-06) - training/global_step:86 - training/epoch:1 - critic/score/mean:0.609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.011623566970229149 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.011623566970229149 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:384.58203125 - response_length/max:958.0 - response_length/min:166.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:384.58203125 - response_length_non_aborted/max:958.0 - response_length_non_aborted/min:166.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.375 - prompt_length/max:503.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012109242379665375 - timing_s/agent_loop/generate_sequences/min:np.float64(2.020683040842414) - timing_s/agent_loop/generate_sequences/max:np.float64(8.49462386034429) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.405667071034259) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.49462386034429) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:232 - timing_s/agent_loop/slowest/response_length:958 - timing_s/gen:15.48012137785554 - timing_s/reward:0.06186640448868275 - timing_s/old_log_prob:2.9315725062042475 - timing_s/adv:0.5595468077808619 - timing_s/update_actor:21.52425908483565 - timing_s/step:40.646166034042835 - timing_s/stop_profile:0.00010807625949382782 - timing_per_token_ms/update_actor:0.12701600418288367 - timing_per_token_ms/gen:0.15723361784664297 - timing_per_token_ms/adv:0.0033019208418507023 - perf/total_num_tokens:169461 - perf/time_per_step:40.646166034042835 - perf/throughput:521.1469387360835
(TaskRunner pid=2255397) Training Progress: 86%|████████▌ | 86/100 [1:19:11<10:56, 46.87s/it]
(TaskRunner pid=2255397) step:87 - global_seqlen/min:19881 - global_seqlen/max:22805 - global_seqlen/minmax_diff:2924 - global_seqlen/balanced_min:21579 - global_seqlen/balanced_max:21581 - global_seqlen/mean:21580.5 - actor/entropy:0.4654737114906311 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3751887083053589 - training/rollout_probs_diff_mean:0.0050046672113239765 - training/rollout_probs_diff_std:0.010888464748859406 - training/rollout_actor_probs_pearson_corr:0.9992407560348511 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.86328125 - self_distillation/correct_sample_fraction:0.62109375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.86328125 - rollout_corr/rollout_is_mean:1.0000450611114502 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.0308853234164417e-05 - rollout_corr/rollout_is_max:1.854941487312317 - rollout_corr/rollout_is_min:0.3242693543434143 - rollout_corr/rollout_is_std:0.03734952583909035 - rollout_corr/rollout_is_eff_sample_size:0.998607075049263 - rollout_corr/rollout_is_seq_mean:1.0000066757202148 - rollout_corr/rollout_is_seq_std:0.00233814911916852 - rollout_corr/rollout_is_seq_max:1.0092549324035645 - rollout_corr/rollout_is_seq_min:0.9934799671173096 - rollout_corr/rollout_is_seq_max_deviation:0.009254932403564453 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6343936491757631) - rollout_corr/training_log_ppl:np.float64(0.47929042315809056) - rollout_corr/kl:np.float64(0.0010372451878630784) - rollout_corr/k3_kl:np.float64(0.00085206899655077) - rollout_corr/rollout_ppl:np.float64(1.6326408791355789) - rollout_corr/rollout_log_ppl:np.float64(0.4782531792880036) - rollout_corr/log_ppl_diff:np.float64(0.001037243870086968) - rollout_corr/log_ppl_abs_diff:np.float64(0.002030018251389265) - rollout_corr/log_ppl_diff_max:np.float64(0.011031702160835266) - rollout_corr/log_ppl_diff_min:np.float64(-0.006544649600982666) - rollout_corr/ppl_ratio:np.float64(1.0010407408699393) - rollout_corr/chi2_token:np.float64(0.0013177578803151846) - rollout_corr/chi2_seq:np.float64(0.7552365991286933) - actor/pg_loss:np.float64(0.006197207723744214) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000917987231787265) - actor/ppo_kl:np.float64(0.00031247709554804715) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.86328125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.86328125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.62109375) - self_distillation/token_reweight_w_mean:np.float64(8798.193575591547) - self_distillation/token_reweight_w_std:np.float64(137248.53065139498) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0011895855423063) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1116473093861714) - self_distillation/empty_target_batch:np.float64(0.13671875) - actor/grad_norm:np.float64(0.3543427586555481) - perf/mfu/actor:np.float64(0.052530135237403436) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.04389572143555) - actor/lr:np.float64(1e-06) - training/global_step:87 - training/epoch:1 - critic/score/mean:0.62109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004306523595005274 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004306523595005274 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:378.921875 - response_length/max:813.0 - response_length/min:167.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:378.921875 - response_length_non_aborted/max:813.0 - response_length_non_aborted/min:167.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:295.46875 - prompt_length/max:845.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001322329044342041 - timing_s/agent_loop/generate_sequences/min:np.float64(2.250560261309147) - timing_s/agent_loop/generate_sequences/max:np.float64(7.587926117703319) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.51870351490652) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(7.587926117703319) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:198 - timing_s/agent_loop/slowest/response_length:800 - timing_s/gen:14.453906053677201 - timing_s/reward:0.06278987228870392 - timing_s/old_log_prob:3.1354457326233387 - timing_s/adv:0.5562533307820559 - timing_s/update_actor:21.309262923896313 - timing_s/step:39.60505427233875 - timing_s/stop_profile:0.00011653825640678406 - timing_per_token_ms/update_actor:0.12342892266106156 - timing_per_token_ms/gen:0.14900319629785577 - timing_per_token_ms/adv:0.0032219673477332307 - perf/total_num_tokens:172644 - perf/time_per_step:39.60505427233875 - perf/throughput:544.8925748619012
(TaskRunner pid=2255397) Training Progress: 87%|████████▋ | 87/100 [1:19:51<09:41, 44.71s/it]
(TaskRunner pid=2255397) step:88 - global_seqlen/min:18137 - global_seqlen/max:23978 - global_seqlen/minmax_diff:5841 - global_seqlen/balanced_min:21618 - global_seqlen/balanced_max:21626 - global_seqlen/mean:21623.75 - actor/entropy:0.4558359980583191 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24135497212409973 - training/rollout_probs_diff_mean:0.005039794836193323 - training/rollout_probs_diff_std:0.01103699766099453 - training/rollout_actor_probs_pearson_corr:0.9992441534996033 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.84765625 - self_distillation/correct_sample_fraction:0.58984375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.84765625 - rollout_corr/rollout_is_mean:0.9999773502349854 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.879797284549568e-05 - rollout_corr/rollout_is_max:1.6429718732833862 - rollout_corr/rollout_is_min:0.34945860505104065 - rollout_corr/rollout_is_std:0.03734952583909035 - rollout_corr/rollout_is_eff_sample_size:0.9986068372944764 - rollout_corr/rollout_is_seq_mean:0.9999490976333618 - rollout_corr/rollout_is_seq_std:0.00212614587508142 - rollout_corr/rollout_is_seq_max:1.007338047027588 - rollout_corr/rollout_is_seq_min:0.9925395846366882 - rollout_corr/rollout_is_seq_max_deviation:0.007460415363311768 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6032785079441965) - rollout_corr/training_log_ppl:np.float64(0.46080948115559295) - rollout_corr/kl:np.float64(0.0011248015301781322) - rollout_corr/k3_kl:np.float64(0.0008873714502897201) - rollout_corr/rollout_ppl:np.float64(1.601451831869781) - rollout_corr/rollout_log_ppl:np.float64(0.4596846750937402) - rollout_corr/log_ppl_diff:np.float64(0.0011248060618527234) - rollout_corr/log_ppl_abs_diff:np.float64(0.002022003580350429) - rollout_corr/log_ppl_diff_max:np.float64(0.012501776218414307) - rollout_corr/log_ppl_diff_min:np.float64(-0.007802844047546387) - rollout_corr/ppl_ratio:np.float64(1.0011287301313132) - rollout_corr/chi2_token:np.float64(0.0012870903592556715) - rollout_corr/chi2_seq:np.float64(0.6421570053789765) - actor/pg_loss:np.float64(0.004205821780487895) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004930502600473119) - actor/ppo_kl:np.float64(0.00033077471513121015) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.84765625) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.84765625) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.58984375) - self_distillation/token_reweight_w_mean:np.float64(21107.031607169192) - self_distillation/token_reweight_w_std:np.float64(309542.6647549386) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0036525861360133) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08997728512622416) - self_distillation/empty_target_batch:np.float64(0.15234375) - actor/grad_norm:np.float64(0.29456714540719986) - perf/mfu/actor:np.float64(0.05240681242673272) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.06198120117188) - actor/lr:np.float64(1e-06) - training/global_step:88 - training/epoch:1 - critic/score/mean:0.58984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0007283487357199192 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0007283487357199192 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:406.9296875 - response_length/max:943.0 - response_length/min:157.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:406.9296875 - response_length_non_aborted/max:943.0 - response_length_non_aborted/min:157.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:268.8125 - prompt_length/max:482.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010753795504570007 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9089515656232834) - timing_s/agent_loop/generate_sequences/max:np.float64(8.329806737601757) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.679670039906341) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.329806737601757) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:329 - timing_s/agent_loop/slowest/response_length:943 - timing_s/gen:15.13785439543426 - timing_s/reward:0.06380515545606613 - timing_s/old_log_prob:3.0357294902205467 - timing_s/adv:0.5423688981682062 - timing_s/update_actor:21.611202096566558 - timing_s/step:40.480039244517684 - timing_s/stop_profile:0.00011552684009075165 - timing_per_token_ms/update_actor:0.12492746457348146 - timing_per_token_ms/gen:0.14531317214884962 - timing_per_token_ms/adv:0.0031352615652246152 - perf/total_num_tokens:172990 - perf/time_per_step:40.480039244517684 - perf/throughput:534.1830295514983
(TaskRunner pid=2255397) Training Progress: 88%|████████▊ | 88/100 [1:20:32<08:41, 43.45s/it]
(TaskRunner pid=2255397) step:89 - global_seqlen/min:20192 - global_seqlen/max:26260 - global_seqlen/minmax_diff:6068 - global_seqlen/balanced_min:23178 - global_seqlen/balanced_max:23186 - global_seqlen/mean:23183.75 - actor/entropy:0.4386472702026367 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7019415497779846 - training/rollout_probs_diff_mean:0.0047370451502501965 - training/rollout_probs_diff_std:0.010689225047826767 - training/rollout_actor_probs_pearson_corr:0.999261736869812 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.86328125 - self_distillation/correct_sample_fraction:0.59765625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.86328125 - rollout_corr/rollout_is_mean:0.9999846816062927 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.8036542314803228e-05 - rollout_corr/rollout_is_max:1.5554907321929932 - rollout_corr/rollout_is_min:0.03433068096637726 - rollout_corr/rollout_is_std:0.0357397198677063 - rollout_corr/rollout_is_eff_sample_size:0.9987241830413971 - rollout_corr/rollout_is_seq_mean:0.9999765157699585 - rollout_corr/rollout_is_seq_std:0.0020081165712326765 - rollout_corr/rollout_is_seq_max:1.0105236768722534 - rollout_corr/rollout_is_seq_min:0.9940011501312256 - rollout_corr/rollout_is_seq_max_deviation:0.010523676872253418 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6203870358876884) - rollout_corr/training_log_ppl:np.float64(0.47031728448928334) - rollout_corr/kl:np.float64(0.0010279387461658018) - rollout_corr/k3_kl:np.float64(0.0008464122184363987) - rollout_corr/rollout_ppl:np.float64(1.6186228357255459) - rollout_corr/rollout_log_ppl:np.float64(0.4692893385363277) - rollout_corr/log_ppl_diff:np.float64(0.0010279459529556334) - rollout_corr/log_ppl_abs_diff:np.float64(0.00189788214629516) - rollout_corr/log_ppl_diff_max:np.float64(0.0077957212924957275) - rollout_corr/log_ppl_diff_min:np.float64(-0.004509687423706055) - rollout_corr/ppl_ratio:np.float64(1.0010310297366232) - rollout_corr/chi2_token:np.float64(0.0013277053367346525) - rollout_corr/chi2_seq:np.float64(0.5938672970514745) - actor/pg_loss:np.float64(0.0077381625305861235) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0008514890976130118) - actor/ppo_kl:np.float64(0.0002891868470378256) - actor/pg_clipfrac_lower:np.float64(8.172070920409169e-06) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.86328125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.86328125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.59765625) - self_distillation/token_reweight_w_mean:np.float64(47.27642107871361) - self_distillation/token_reweight_w_std:np.float64(801.2714838348329) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.000362384133041) - self_distillation/token_reweight_w_clip_frac:np.float64(0.13113477459410205) - self_distillation/empty_target_batch:np.float64(0.13671875) - actor/grad_norm:np.float64(0.3602307364344597) - perf/mfu/actor:np.float64(0.05579510012189607) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.08685684204102) - actor/lr:np.float64(1e-06) - training/global_step:89 - training/epoch:1 - critic/score/mean:0.59765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.59765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0041562956757843494 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0041562956757843494 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:433.1484375 - response_length/max:1235.0 - response_length/min:169.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:433.1484375 - response_length_non_aborted/max:1235.0 - response_length_non_aborted/min:169.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:291.34375 - prompt_length/max:720.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.681098163127899e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.020791845396161) - timing_s/agent_loop/generate_sequences/max:np.float64(9.594862563535571) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.899216158031777) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.594862563535571) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:223 - timing_s/agent_loop/slowest/response_length:1053 - timing_s/gen:16.61686352454126 - timing_s/reward:0.055103445425629616 - timing_s/old_log_prob:2.937300179153681 - timing_s/adv:0.5702405758202076 - timing_s/update_actor:21.570211308076978 - timing_s/step:41.78375179320574 - timing_s/stop_profile:2.8584152460098267e-05 - timing_per_token_ms/update_actor:0.11630027124643866 - timing_per_token_ms/gen:0.1498553787181543 - timing_per_token_ms/adv:0.003074570420122972 - perf/total_num_tokens:185470 - perf/time_per_step:41.78375179320574 - perf/throughput:554.8508452457781
(TaskRunner pid=2255397) Training Progress: 89%|████████▉ | 89/100 [1:21:13<07:52, 42.96s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 90}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**, we need to analyze the structure of the product and compare it with the options provided.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product has the following key features:
(TaskRunner pid=2255397) - A nitrogen-containing ring (likely a pyrrole or similar structure).
(TaskRunner pid=2255397) - A **chlorine-substituted benzene ring** (from the **Cl** in **"ccc(Cl)"**).
(TaskRunner pid=2255397) - A **trifluoromethyl group** (from **"C(F)(F)F"**).
(TaskRunner pid=2255397) - A **nitrogen-containing bridge** connecting the ring to the rest of the molecule.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Now, let’s evaluate each option:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option A**:
(TaskRunner pid=2255397) `O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1`
(TaskRunner pid=2255397) - Contains a **chlorine-substituted benzene ring**.
(TaskRunner pid=2255397) - Contains a **trifluoromethyl group**.
(TaskRunner pid=2255397) - Contains a **nitrogen-containing bridge** with the correct connectivity.
(TaskRunner pid=2255397) - This matches the structure of the product and appears to be a direct precursor.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option B**:
(TaskRunner pid=2255397) `O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1`
(TaskRunner pid=2255397) - Contains **chlorine atoms**, but the **trifluoromethyl group** is missing.
(TaskRunner pid=2255397) - The **nitrogen-containing bridge** is not properly connected to the **trifluoromethyl** group.
(TaskRunner pid=2255397) - This does not match the product structure.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option C**:
(TaskRunner pid=2255397) `O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1`
(TaskRunner pid=2255397) - Contains a **chlorine-substituted benzene ring** and a **trifluoromethyl group**.
(TaskRunner pid=2255397) - However, the **nitrogen-containing bridge** is not properly positioned and is connected via an **oxygen**, which is not part of the product.
(TaskRunner pid=2255397) - This is not a correct match.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option D**:
(TaskRunner pid=2255397) `O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F`
(TaskRunner pid=2255397) - Contains a **trifluoromethyl group**.
(TaskRunner pid=2255397) - However, it **lacks the chlorine-substituted benzene ring** and the **correct nitrogen-containing bridge**.
(TaskRunner pid=2255397) - This also does not match the product structure.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Conclusion**:
(TaskRunner pid=2255397) Only **Option A** contains all the necessary functional groups and the correct connectivity to serve as the only correct reactant in the synthesis of the given product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_90
(TaskRunner pid=2255397) step:90 - global_seqlen/min:18865 - global_seqlen/max:24932 - global_seqlen/minmax_diff:6067 - global_seqlen/balanced_min:22815 - global_seqlen/balanced_max:22831 - global_seqlen/mean:22826.625 - actor/entropy:0.40077415108680725 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5775288939476013 - training/rollout_probs_diff_mean:0.0045609320513904095 - training/rollout_probs_diff_std:0.010716044344007969 - training/rollout_actor_probs_pearson_corr:0.9992610216140747 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.85546875 - self_distillation/correct_sample_fraction:0.62109375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.85546875 - rollout_corr/rollout_is_mean:0.9999971985816956 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.970459930424113e-06 - rollout_corr/rollout_is_max:1.8681867122650146 - rollout_corr/rollout_is_min:0.2344389706850052 - rollout_corr/rollout_is_std:0.03548699617385864 - rollout_corr/rollout_is_eff_sample_size:0.9987421380652595 - rollout_corr/rollout_is_seq_mean:1.0000590085983276 - rollout_corr/rollout_is_seq_std:0.0020531753543764353 - rollout_corr/rollout_is_seq_max:1.0109810829162598 - rollout_corr/rollout_is_seq_min:0.9949836730957031 - rollout_corr/rollout_is_seq_max_deviation:0.010981082916259766 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5583982807584107) - rollout_corr/training_log_ppl:np.float64(0.43078241116018035) - rollout_corr/kl:np.float64(0.0008912494752149058) - rollout_corr/k3_kl:np.float64(0.000791456644748223) - rollout_corr/rollout_ppl:np.float64(1.5569715378805995) - rollout_corr/rollout_log_ppl:np.float64(0.42989116214448586) - rollout_corr/log_ppl_diff:np.float64(0.0008912490156944841) - rollout_corr/log_ppl_abs_diff:np.float64(0.001720420696074143) - rollout_corr/log_ppl_diff_max:np.float64(0.01338726282119751) - rollout_corr/log_ppl_diff_min:np.float64(-0.00902646780014038) - rollout_corr/ppl_ratio:np.float64(1.0008942331187427) - rollout_corr/chi2_token:np.float64(0.0014041787944734097) - rollout_corr/chi2_seq:np.float64(0.752974865725264) - actor/pg_loss:np.float64(0.004781530937179923) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0005209917633237637) - actor/ppo_kl:np.float64(0.0002642208678480529) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.85546875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.85546875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.62109375) - self_distillation/token_reweight_w_mean:np.float64(619.1751045542769) - self_distillation/token_reweight_w_std:np.float64(7680.31531262747) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.999175232835114) - self_distillation/token_reweight_w_clip_frac:np.float64(0.1011948958330322) - self_distillation/empty_target_batch:np.float64(0.14453125) - actor/grad_norm:np.float64(0.260063748806715) - perf/mfu/actor:np.float64(0.05435728539450956) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(196.9617691040039) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.7092261904761905) - val-aux/sciknoweval/reward/std@16:np.float64(0.1437505073441559) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7665238095238094) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.11153257282472556) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6511952380952379) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.13026803458061142) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.7090476190476193) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.14272451357066823) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8092428571428572) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.07643847293889101) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5996714285714285) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.10642069734217646) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.7207095238095238) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.11336797001466493) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8373142857142858) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.051272065112436704) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.5569380952380952) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.07817843871980924) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7271952380952381) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.08253191872957805) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8582761904761905) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.03254491463909445) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.525004761904762) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.05193613577764248) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7298285714285715) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.05986238009450025) - val-aux/sciknoweval/score/mean@16:np.float64(0.7092261904761905) - val-aux/sciknoweval/score/std@16:np.float64(0.1437505073441559) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7665238095238094) - val-aux/sciknoweval/score/best@2/std:np.float64(0.11153257282472556) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6511952380952379) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.13026803458061142) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.7090476190476193) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.14272451357066823) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8092428571428572) - val-aux/sciknoweval/score/best@4/std:np.float64(0.07643847293889101) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5996714285714285) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.10642069734217646) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.7207095238095238) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.11336797001466493) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8373142857142858) - val-aux/sciknoweval/score/best@8/std:np.float64(0.051272065112436704) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.5569380952380952) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.07817843871980924) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7271952380952381) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.08253191872957805) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8582761904761905) - val-aux/sciknoweval/score/best@16/std:np.float64(0.03254491463909445) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.525004761904762) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.05193613577764248) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7298285714285715) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.05986238009450025) - val-core/sciknoweval/acc/mean@16:np.float64(0.7092261904761905) - val-aux/sciknoweval/acc/std@16:np.float64(0.1437505073441559) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7665238095238094) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.11153257282472556) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6511952380952379) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.13026803458061142) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.7090476190476193) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.14272451357066823) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8092428571428572) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.07643847293889101) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5996714285714285) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.10642069734217646) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.7207095238095238) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.11336797001466493) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8373142857142858) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.051272065112436704) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.5569380952380952) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.07817843871980924) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7271952380952381) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.08253191872957805) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8582761904761905) - val-core/sciknoweval/acc/best@16/std:np.float64(0.03254491463909445) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.525004761904762) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.05193613577764248) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7298285714285715) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.05986238009450025) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9967261904761905) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.011218419246342346) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9996714285714285) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0035108104210424323) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9937666666666667) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.014896548825679245) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9966666666666667) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.011281400445573764) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.999995238095238) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0001505093393264677) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9886285714285715) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.018614743576868955) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9989571428571429) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.00637191435060849) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9802904761904762) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.02088499966775818) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9997285714285713) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0030510338258627335) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9701857142857143) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.018986995237275702) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999904761904761) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0003010186786529354) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:90 - training/epoch:1 - critic/score/mean:0.62109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0026496497448533773 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0026496497448533773 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:435.45703125 - response_length/max:1197.0 - response_length/min:135.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:435.45703125 - response_length_non_aborted/max:1197.0 - response_length_non_aborted/min:135.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.875 - prompt_length/max:621.0 - prompt_length/min:180.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.609938383102417e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6878780890256166) - timing_s/agent_loop/generate_sequences/max:np.float64(10.072803135961294) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.094866805477068) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.072803135961294) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:221 - timing_s/agent_loop/slowest/response_length:1197 - timing_s/gen:16.879685267806053 - timing_s/reward:0.06467907875776291 - timing_s/old_log_prob:3.10450641810894 - timing_s/adv:0.582694835960865 - timing_s/update_actor:21.818151073530316 - timing_s/step:42.53507168777287 - timing_s/testing:140.61721943691373 - timing_s/save_checkpoint:13.685923740267754 - timing_s/stop_profile:0.00018855556845664978 - timing_per_token_ms/update_actor:0.11947753486077287 - timing_per_token_ms/gen:0.15141854613782263 - timing_per_token_ms/adv:0.0031908726977863846 - perf/total_num_tokens:182613 - perf/time_per_step:42.53507168777287 - perf/throughput:536.6542030905226
(TaskRunner pid=2255397) Training Progress: 90%|█████████ | 90/100 [1:24:30<14:51, 89.15s/it]
(TaskRunner pid=2255397) step:91 - global_seqlen/min:20110 - global_seqlen/max:28143 - global_seqlen/minmax_diff:8033 - global_seqlen/balanced_min:22954 - global_seqlen/balanced_max:22958 - global_seqlen/mean:22955.625 - actor/entropy:0.4345111846923828 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.318805456161499 - training/rollout_probs_diff_mean:0.00483555207028985 - training/rollout_probs_diff_std:0.010917664505541325 - training/rollout_actor_probs_pearson_corr:0.9992064833641052 - self_distillation/success_group_fraction:0.8125 - self_distillation/success_sample_fraction:0.8046875 - self_distillation/correct_sample_fraction:0.640625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8046875 - rollout_corr/rollout_is_mean:1.0000874996185303 - rollout_corr/rollout_is_ratio_fraction_high:9.058217074198183e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.8116434148396365e-05 - rollout_corr/rollout_is_max:2.0618367195129395 - rollout_corr/rollout_is_min:0.27237245440483093 - rollout_corr/rollout_is_std:0.037339948117733 - rollout_corr/rollout_is_eff_sample_size:0.9986079071919081 - rollout_corr/rollout_is_seq_mean:1.0000126361846924 - rollout_corr/rollout_is_seq_std:0.0018744589760899544 - rollout_corr/rollout_is_seq_max:1.0062435865402222 - rollout_corr/rollout_is_seq_min:0.9926995038986206 - rollout_corr/rollout_is_seq_max_deviation:0.0073004961013793945 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5809713881462812) - rollout_corr/training_log_ppl:np.float64(0.4482057260465808) - rollout_corr/kl:np.float64(0.0007782714664905654) - rollout_corr/k3_kl:np.float64(0.0007742295367734187) - rollout_corr/rollout_ppl:np.float64(1.5796857569366693) - rollout_corr/rollout_log_ppl:np.float64(0.4474274521926418) - rollout_corr/log_ppl_diff:np.float64(0.0007782738539390266) - rollout_corr/log_ppl_abs_diff:np.float64(0.0016364274197258055) - rollout_corr/log_ppl_diff_max:np.float64(0.007638216018676758) - rollout_corr/log_ppl_diff_min:np.float64(-0.005470573902130127) - rollout_corr/ppl_ratio:np.float64(1.0007805426139385) - rollout_corr/chi2_token:np.float64(0.001570558873936534) - rollout_corr/chi2_seq:np.float64(0.9185826345346868) - actor/pg_loss:np.float64(0.004598198342137039) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000316593827392353) - actor/ppo_kl:np.float64(5.939174411295767e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8046875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8046875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.640625) - self_distillation/token_reweight_w_mean:np.float64(7563.319094766863) - self_distillation/token_reweight_w_std:np.float64(150023.32301567192) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.999886890873313) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08767912839539349) - self_distillation/empty_target_batch:np.float64(0.1953125) - actor/grad_norm:np.float64(0.2757617086172104) - perf/mfu/actor:np.float64(0.054246744251757004) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(201.1460475921631) - actor/lr:np.float64(1e-06) - training/global_step:91 - training/epoch:1 - critic/score/mean:0.640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0008684565545991063 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0008684565545991063 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:431.23828125 - response_length/max:1001.0 - response_length/min:187.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:431.23828125 - response_length_non_aborted/max:1001.0 - response_length_non_aborted/min:187.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.125 - prompt_length/max:1009.0 - prompt_length/min:176.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010878220200538635 - timing_s/agent_loop/generate_sequences/min:np.float64(2.29626976698637) - timing_s/agent_loop/generate_sequences/max:np.float64(8.867659989744425) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.9120760589357815) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.867659989744425) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:212 - timing_s/agent_loop/slowest/response_length:1001 - timing_s/gen:15.32518370822072 - timing_s/reward:0.061427073553204536 - timing_s/old_log_prob:3.1340453140437603 - timing_s/adv:0.5956883151084185 - timing_s/update_actor:22.10231014713645 - timing_s/step:41.29857017099857 - timing_s/stop_profile:0.00013570114970207214 - timing_per_token_ms/update_actor:0.12035345447540881 - timing_per_token_ms/gen:0.1388188420719831 - timing_per_token_ms/adv:0.003243694710492627 - perf/total_num_tokens:183645 - perf/time_per_step:41.29857017099857 - perf/throughput:555.8455148677353
(TaskRunner pid=2255397) Training Progress: 91%|█████████ | 91/100 [1:25:12<11:13, 74.81s/it]
(TaskRunner pid=2255397) step:92 - global_seqlen/min:20808 - global_seqlen/max:26065 - global_seqlen/minmax_diff:5257 - global_seqlen/balanced_min:23344 - global_seqlen/balanced_max:23359 - global_seqlen/mean:23353.875 - actor/entropy:0.43259748816490173 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2583375573158264 - training/rollout_probs_diff_mean:0.004716021008789539 - training/rollout_probs_diff_std:0.010604353621602058 - training/rollout_actor_probs_pearson_corr:0.9992642402648926 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.828125 - self_distillation/correct_sample_fraction:0.609375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.828125 - rollout_corr/rollout_is_mean:1.0000042915344238 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.831503691908438e-06 - rollout_corr/rollout_is_max:1.6297050714492798 - rollout_corr/rollout_is_min:0.3157358169555664 - rollout_corr/rollout_is_std:0.03597741574048996 - rollout_corr/rollout_is_eff_sample_size:0.9987072987735173 - rollout_corr/rollout_is_seq_mean:0.9999913573265076 - rollout_corr/rollout_is_seq_std:0.0019688971806317568 - rollout_corr/rollout_is_seq_max:1.0057662725448608 - rollout_corr/rollout_is_seq_min:0.9920455813407898 - rollout_corr/rollout_is_seq_max_deviation:0.007954418659210205 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5944940163753927) - rollout_corr/training_log_ppl:np.float64(0.4538156447233632) - rollout_corr/kl:np.float64(0.0008374510896800302) - rollout_corr/k3_kl:np.float64(0.000751384576176406) - rollout_corr/rollout_ppl:np.float64(1.5930784032680094) - rollout_corr/rollout_log_ppl:np.float64(0.45297819224651903) - rollout_corr/log_ppl_diff:np.float64(0.0008374524768441916) - rollout_corr/log_ppl_abs_diff:np.float64(0.0017089889151975513) - rollout_corr/log_ppl_diff_max:np.float64(0.008830234408378601) - rollout_corr/log_ppl_diff_min:np.float64(-0.005124390125274658) - rollout_corr/ppl_ratio:np.float64(1.0008400096558034) - rollout_corr/chi2_token:np.float64(0.0013158204965293407) - rollout_corr/chi2_seq:np.float64(0.8615647666156292) - actor/pg_loss:np.float64(0.00415497412905097) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000329681670109494) - actor/ppo_kl:np.float64(0.000143634846812013) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.828125) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.828125) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.609375) - self_distillation/token_reweight_w_mean:np.float64(2081.352001107065) - self_distillation/token_reweight_w_std:np.float64(46560.20298913785) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0010203949641436) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09130151267163455) - self_distillation/empty_target_batch:np.float64(0.171875) - actor/grad_norm:np.float64(0.25993651896715164) - perf/mfu/actor:np.float64(0.05560204307595709) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.7599754333496) - actor/lr:np.float64(1e-06) - training/global_step:92 - training/epoch:1 - critic/score/mean:0.609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.011590245179831982 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.011590245179831982 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:442.30859375 - response_length/max:1140.0 - response_length/min:182.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:442.30859375 - response_length_non_aborted/max:1140.0 - response_length_non_aborted/min:182.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:287.5 - prompt_length/max:436.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015668943524360657 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3020109608769417) - timing_s/agent_loop/generate_sequences/max:np.float64(9.68418238312006) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.095685060987307) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.68418238312006) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:260 - timing_s/agent_loop/slowest/response_length:1140 - timing_s/gen:16.275774881243706 - timing_s/reward:0.06520042568445206 - timing_s/old_log_prob:3.0083443708717823 - timing_s/adv:0.5682667158544064 - timing_s/update_actor:22.008645860478282 - timing_s/step:42.016618859022856 - timing_s/stop_profile:0.00013844482600688934 - timing_per_token_ms/update_actor:0.11779975411188873 - timing_per_token_ms/gen:0.1437395667374103 - timing_per_token_ms/adv:0.0030416082762197193 - perf/total_num_tokens:186831 - perf/time_per_step:42.016618859022856 - perf/throughput:555.8247101785744
(TaskRunner pid=2255397) Training Progress: 92%|█████████▏| 92/100 [1:25:54<08:39, 64.99s/it]
(TaskRunner pid=2255397) step:93 - global_seqlen/min:20104 - global_seqlen/max:26536 - global_seqlen/minmax_diff:6432 - global_seqlen/balanced_min:22165 - global_seqlen/balanced_max:22172 - global_seqlen/mean:22170.375 - actor/entropy:0.4773474633693695 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18463870882987976 - training/rollout_probs_diff_mean:0.005192984826862812 - training/rollout_probs_diff_std:0.01093792449682951 - training/rollout_actor_probs_pearson_corr:0.9992780685424805 - self_distillation/success_group_fraction:1.0 - self_distillation/success_sample_fraction:0.9921875 - self_distillation/correct_sample_fraction:0.83203125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.9921875 - rollout_corr/rollout_is_mean:1.0000450611114502 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.661463737487793 - rollout_corr/rollout_is_min:0.516041100025177 - rollout_corr/rollout_is_std:0.037531014531850815 - rollout_corr/rollout_is_eff_sample_size:0.9985935232071051 - rollout_corr/rollout_is_seq_mean:1.0000596046447754 - rollout_corr/rollout_is_seq_std:0.0020964141003787518 - rollout_corr/rollout_is_seq_max:1.0067566633224487 - rollout_corr/rollout_is_seq_min:0.9933486580848694 - rollout_corr/rollout_is_seq_max_deviation:0.0067566633224487305 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6435860991477966) - rollout_corr/training_log_ppl:np.float64(0.48362978905788623) - rollout_corr/kl:np.float64(0.000619826348660979) - rollout_corr/k3_kl:np.float64(0.0007837708870965798) - rollout_corr/rollout_ppl:np.float64(1.6425714306533337) - rollout_corr/rollout_log_ppl:np.float64(0.48300995968747884) - rollout_corr/log_ppl_diff:np.float64(0.0006198293704073876) - rollout_corr/log_ppl_abs_diff:np.float64(0.001762611762387678) - rollout_corr/log_ppl_diff_max:np.float64(0.009479671716690063) - rollout_corr/log_ppl_diff_min:np.float64(-0.005926311016082764) - rollout_corr/ppl_ratio:np.float64(1.0006224988028407) - rollout_corr/chi2_token:np.float64(0.0019295907113701105) - rollout_corr/chi2_seq:np.float64(1.5755573513451964) - actor/pg_loss:np.float64(0.004148329375311732) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002517454886401538) - actor/ppo_kl:np.float64(-5.375289406117645e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.9921875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.9921875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.83203125) - self_distillation/token_reweight_w_mean:np.float64(17544.475337664597) - self_distillation/token_reweight_w_std:np.float64(333058.533660894) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0001345220953226) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08209344942588359) - self_distillation/empty_target_batch:np.float64(0.0078125) - actor/grad_norm:np.float64(0.2505064271390438) - perf/mfu/actor:np.float64(0.05288764926923402) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(200.60185623168945) - actor/lr:np.float64(1e-06) - training/global_step:93 - training/epoch:1 - critic/score/mean:0.83203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.83203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00024851804482750595 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00024851804482750595 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:426.35546875 - response_length/max:917.0 - response_length/min:192.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:426.35546875 - response_length_non_aborted/max:917.0 - response_length_non_aborted/min:192.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.46875 - prompt_length/max:709.0 - prompt_length/min:167.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014604441821575165 - timing_s/agent_loop/generate_sequences/min:np.float64(2.234377281740308) - timing_s/agent_loop/generate_sequences/max:np.float64(8.298982378095388) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.870763707032893) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(8.298982378095388) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:193 - timing_s/agent_loop/slowest/response_length:917 - timing_s/gen:15.365983800962567 - timing_s/reward:0.06459944881498814 - timing_s/old_log_prob:2.9253345616161823 - timing_s/adv:0.5848103743046522 - timing_s/update_actor:22.019390346482396 - timing_s/step:41.04795786552131 - timing_s/stop_profile:0.00011224299669265747 - timing_per_token_ms/update_actor:0.1241487251934304 - timing_per_token_ms/gen:0.1407824658576284 - timing_per_token_ms/adv:0.0032972512547975183 - perf/total_num_tokens:177363 - perf/time_per_step:41.04795786552131 - perf/throughput:540.1090858803052
(TaskRunner pid=2255397) Training Progress: 93%|█████████▎| 93/100 [1:26:35<06:44, 57.82s/it]
(TaskRunner pid=2255397) step:94 - global_seqlen/min:20450 - global_seqlen/max:26778 - global_seqlen/minmax_diff:6328 - global_seqlen/balanced_min:23452 - global_seqlen/balanced_max:23461 - global_seqlen/mean:23457.625 - actor/entropy:0.45124438405036926 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.47588256001472473 - training/rollout_probs_diff_mean:0.004685731139034033 - training/rollout_probs_diff_std:0.010418391786515713 - training/rollout_actor_probs_pearson_corr:0.999316930770874 - self_distillation/success_group_fraction:0.875 - self_distillation/success_sample_fraction:0.875 - self_distillation/correct_sample_fraction:0.76171875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.875 - rollout_corr/rollout_is_mean:0.9999880194664001 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.4401948394370265e-05 - rollout_corr/rollout_is_max:1.8073612451553345 - rollout_corr/rollout_is_min:0.04583345353603363 - rollout_corr/rollout_is_std:0.03505437821149826 - rollout_corr/rollout_is_eff_sample_size:0.9987726987625546 - rollout_corr/rollout_is_seq_mean:0.9999796152114868 - rollout_corr/rollout_is_seq_std:0.0017362802755087614 - rollout_corr/rollout_is_seq_max:1.0060876607894897 - rollout_corr/rollout_is_seq_min:0.9951460361480713 - rollout_corr/rollout_is_seq_max_deviation:0.006087660789489746 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.600627651438117) - rollout_corr/training_log_ppl:np.float64(0.4611963724019006) - rollout_corr/kl:np.float64(0.0007944866175790821) - rollout_corr/k3_kl:np.float64(0.000699135587751698) - rollout_corr/rollout_ppl:np.float64(1.599346871022135) - rollout_corr/rollout_log_ppl:np.float64(0.46040188503684476) - rollout_corr/log_ppl_diff:np.float64(0.0007944873650558293) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015184025396592915) - rollout_corr/log_ppl_diff_max:np.float64(0.007160961627960205) - rollout_corr/log_ppl_diff_min:np.float64(-0.004008620977401733) - rollout_corr/ppl_ratio:np.float64(1.0007963897660375) - rollout_corr/chi2_token:np.float64(0.0012088804505765438) - rollout_corr/chi2_seq:np.float64(0.3547051209025085) - actor/pg_loss:np.float64(0.0030129868537187576) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00016773132483649533) - actor/ppo_kl:np.float64(0.00010856679642934353) - actor/pg_clipfrac_lower:np.float64(7.659314178454224e-06) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.76171875) - self_distillation/token_reweight_w_mean:np.float64(4465.609730498167) - self_distillation/token_reweight_w_std:np.float64(81083.046648157) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9998130949679762) - self_distillation/token_reweight_w_clip_frac:np.float64(0.04856777528766543) - self_distillation/empty_target_batch:np.float64(0.125) - actor/grad_norm:np.float64(0.18460461869835854) - perf/mfu/actor:np.float64(0.05546533287041886) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.40380096435547) - actor/lr:np.float64(1e-06) - training/global_step:94 - training/epoch:1 - critic/score/mean:0.76171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.76171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005895104259252548 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:0.005895104259252548 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:480.23828125 - response_length/max:1044.0 - response_length/min:226.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:480.23828125 - response_length_non_aborted/max:1044.0 - response_length_non_aborted/min:226.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:252.8125 - prompt_length/max:384.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012932159006595612 - timing_s/agent_loop/generate_sequences/min:np.float64(2.888417324051261) - timing_s/agent_loop/generate_sequences/max:np.float64(9.281017616391182) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.676639520912431) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.281017616391182) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:221 - timing_s/agent_loop/slowest/response_length:997 - timing_s/gen:15.90216557495296 - timing_s/reward:0.06385539285838604 - timing_s/old_log_prob:2.9568253997713327 - timing_s/adv:0.5866717882454395 - timing_s/update_actor:21.823786733672023 - timing_s/step:41.424420485273004 - timing_s/stop_profile:0.00012981146574020386 - timing_per_token_ms/update_actor:0.11629367174677756 - timing_per_token_ms/gen:0.12934794393207277 - timing_per_token_ms/adv:0.0031262318129256454 - perf/total_num_tokens:187661 - perf/time_per_step:41.424420485273004 - perf/throughput:566.2752725373559
(TaskRunner pid=2255397) Training Progress: 94%|█████████▍| 94/100 [1:27:16<05:17, 52.92s/it]
(TaskRunner pid=2255397) step:95 - global_seqlen/min:18176 - global_seqlen/max:26822 - global_seqlen/minmax_diff:8646 - global_seqlen/balanced_min:23320 - global_seqlen/balanced_max:23330 - global_seqlen/mean:23326.75 - actor/entropy:0.42938148975372314 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4102376401424408 - training/rollout_probs_diff_mean:0.0046544442884624004 - training/rollout_probs_diff_std:0.010648963041603565 - training/rollout_actor_probs_pearson_corr:0.9992654323577881 - self_distillation/success_group_fraction:0.90625 - self_distillation/success_sample_fraction:0.8984375 - self_distillation/correct_sample_fraction:0.68359375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8984375 - rollout_corr/rollout_is_mean:1.0000568628311157 - rollout_corr/rollout_is_ratio_fraction_high:1.6773175957496278e-05 - rollout_corr/rollout_is_ratio_fraction_low:8.386587978748139e-06 - rollout_corr/rollout_is_max:2.396075487136841 - rollout_corr/rollout_is_min:0.3959798514842987 - rollout_corr/rollout_is_std:0.035107050091028214 - rollout_corr/rollout_is_eff_sample_size:0.9987691312695665 - rollout_corr/rollout_is_seq_mean:1.0000056028366089 - rollout_corr/rollout_is_seq_std:0.0018295958871021867 - rollout_corr/rollout_is_seq_max:1.0062776803970337 - rollout_corr/rollout_is_seq_min:0.9954047203063965 - rollout_corr/rollout_is_seq_max_deviation:0.006277680397033691 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5648417291231453) - rollout_corr/training_log_ppl:np.float64(0.4370654333033599) - rollout_corr/kl:np.float64(0.0006150017093133897) - rollout_corr/k3_kl:np.float64(0.0007087535628897967) - rollout_corr/rollout_ppl:np.float64(1.5638857381418347) - rollout_corr/rollout_log_ppl:np.float64(0.43645042926073074) - rollout_corr/log_ppl_diff:np.float64(0.0006150040426291525) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015464308089576662) - rollout_corr/log_ppl_diff_max:np.float64(0.005993485450744629) - rollout_corr/log_ppl_diff_min:np.float64(-0.004650503396987915) - rollout_corr/ppl_ratio:np.float64(1.000617001671344) - rollout_corr/chi2_token:np.float64(0.0016058182809501886) - rollout_corr/chi2_seq:np.float64(0.5998094449751079) - actor/pg_loss:np.float64(0.005259391153231263) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024053280617408745) - actor/ppo_kl:np.float64(-4.200763549278008e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8984375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8984375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.68359375) - self_distillation/token_reweight_w_mean:np.float64(142.92776992870495) - self_distillation/token_reweight_w_std:np.float64(3127.1786301988177) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0005555916577578) - self_distillation/token_reweight_w_clip_frac:np.float64(0.0809866230993066) - self_distillation/empty_target_batch:np.float64(0.1015625) - actor/grad_norm:np.float64(0.24522369727492332) - perf/mfu/actor:np.float64(0.05459364258201367) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.3312644958496) - actor/lr:np.float64(1e-06) - training/global_step:95 - training/epoch:1 - critic/score/mean:0.68359375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.68359375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.017624415457248688 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:-0.017624415457248688 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:465.7734375 - response_length/max:1081.0 - response_length/min:135.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:465.7734375 - response_length_non_aborted/max:1081.0 - response_length_non_aborted/min:135.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.1875 - prompt_length/max:646.0 - prompt_length/min:159.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012592412531375885 - timing_s/agent_loop/generate_sequences/min:np.float64(1.609277019277215) - timing_s/agent_loop/generate_sequences/max:np.float64(9.628803690895438) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.290896388272813) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.628803690895438) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:297 - timing_s/agent_loop/slowest/response_length:1081 - timing_s/gen:16.404801338911057 - timing_s/reward:0.06480440311133862 - timing_s/old_log_prob:3.627877188846469 - timing_s/adv:0.5782617889344692 - timing_s/update_actor:21.97222731821239 - timing_s/step:42.739498157054186 - timing_s/stop_profile:0.00011696107685565948 - timing_per_token_ms/update_actor:0.1177415805792298 - timing_per_token_ms/gen:0.13758031281060615 - timing_per_token_ms/adv:0.0030987052897128256 - perf/total_num_tokens:186614 - perf/time_per_step:42.739498157054186 - perf/throughput:545.7890477394363
(TaskRunner pid=2255397) Training Progress: 95%|█████████▌| 95/100 [1:27:59<04:09, 49.88s/it]
(TaskRunner pid=2255397) step:96 - global_seqlen/min:20301 - global_seqlen/max:28288 - global_seqlen/minmax_diff:7987 - global_seqlen/balanced_min:23030 - global_seqlen/balanced_max:23033 - global_seqlen/mean:23031.375 - actor/entropy:0.44385668635368347 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2772989869117737 - training/rollout_probs_diff_mean:0.004754771012812853 - training/rollout_probs_diff_std:0.010701586492359638 - training/rollout_actor_probs_pearson_corr:0.99928879737854 - self_distillation/success_group_fraction:0.84375 - self_distillation/success_sample_fraction:0.8359375 - self_distillation/correct_sample_fraction:0.67578125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.8359375 - rollout_corr/rollout_is_mean:1.000027060508728 - rollout_corr/rollout_is_ratio_fraction_high:8.62165597936837e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.084261417388916 - rollout_corr/rollout_is_min:0.5124618411064148 - rollout_corr/rollout_is_std:0.036241523921489716 - rollout_corr/rollout_is_eff_sample_size:0.9986881560335953 - rollout_corr/rollout_is_seq_mean:1.0000017881393433 - rollout_corr/rollout_is_seq_std:0.0018420576816424727 - rollout_corr/rollout_is_seq_max:1.0063093900680542 - rollout_corr/rollout_is_seq_min:0.9944624304771423 - rollout_corr/rollout_is_seq_max_deviation:0.006309390068054199 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.598726301919669) - rollout_corr/training_log_ppl:np.float64(0.45704233669675887) - rollout_corr/kl:np.float64(0.0006188384301958338) - rollout_corr/k3_kl:np.float64(0.0007431540022935224) - rollout_corr/rollout_ppl:np.float64(1.5976339941844344) - rollout_corr/rollout_log_ppl:np.float64(0.45642349659465253) - rollout_corr/log_ppl_diff:np.float64(0.0006188401021063328) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015413255896419287) - rollout_corr/log_ppl_diff_max:np.float64(0.0064958930015563965) - rollout_corr/log_ppl_diff_min:np.float64(-0.0055953264236450195) - rollout_corr/ppl_ratio:np.float64(1.0006208247505128) - rollout_corr/chi2_token:np.float64(0.0018268234562128782) - rollout_corr/chi2_seq:np.float64(1.2716746137011796) - actor/pg_loss:np.float64(0.004606917384080589) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00021080635951875593) - actor/ppo_kl:np.float64(-7.161125205357166e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.8359375) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.8359375) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.67578125) - self_distillation/token_reweight_w_mean:np.float64(99.39291401347145) - self_distillation/token_reweight_w_std:np.float64(2908.174926977954) - self_distillation/token_reweight_w_clipped_mean:np.float64(0.9999677997548133) - self_distillation/token_reweight_w_clip_frac:np.float64(0.08357357300701551) - self_distillation/empty_target_batch:np.float64(0.1640625) - actor/grad_norm:np.float64(0.23777736723423004) - perf/mfu/actor:np.float64(0.05523982050510799) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.2464942932129) - actor/lr:np.float64(1e-06) - training/global_step:96 - training/epoch:1 - critic/score/mean:0.67578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.67578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.003513324772939086 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.003513324772939086 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:453.07421875 - response_length/max:1135.0 - response_length/min:170.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:453.07421875 - response_length_non_aborted/max:1135.0 - response_length_non_aborted/min:170.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.65625 - prompt_length/max:456.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.842030704021454e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1754575334489346) - timing_s/agent_loop/generate_sequences/max:np.float64(9.51023275591433) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.9925034317566315) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(9.51023275591433) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:196 - timing_s/agent_loop/slowest/response_length:1135 - timing_s/gen:16.42838684655726 - timing_s/reward:0.07112030126154423 - timing_s/old_log_prob:3.2094656210392714 - timing_s/adv:0.5936893373727798 - timing_s/update_actor:21.62316256016493 - timing_s/step:42.020951598882675 - timing_s/stop_profile:0.00010878965258598328 - timing_per_token_ms/update_actor:0.11735709743862953 - timing_per_token_ms/gen:0.14163989797612886 - timing_per_token_ms/adv:0.0032221770159878635 - perf/total_num_tokens:184251 - perf/time_per_step:42.020951598882675 - perf/throughput:548.0926567263269
(TaskRunner pid=2255397) Training Progress: 96%|█████████▌| 96/100 [1:28:41<03:10, 47.53s/it]
(TaskRunner pid=2255397) step:97 - global_seqlen/min:21184 - global_seqlen/max:28019 - global_seqlen/minmax_diff:6835 - global_seqlen/balanced_min:24523 - global_seqlen/balanced_max:24538 - global_seqlen/mean:24532.875 - actor/entropy:0.43073078989982605 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5830099582672119 - training/rollout_probs_diff_mean:0.004653396084904671 - training/rollout_probs_diff_std:0.010732254013419151 - training/rollout_actor_probs_pearson_corr:0.9992837905883789 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.765625 - self_distillation/correct_sample_fraction:0.5625 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.765625 - rollout_corr/rollout_is_mean:0.999879002571106 - rollout_corr/rollout_is_ratio_fraction_high:8.271640581369866e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.654328116273973e-05 - rollout_corr/rollout_is_max:2.069552421569824 - rollout_corr/rollout_is_min:0.06639717519283295 - rollout_corr/rollout_is_std:0.03590112552046776 - rollout_corr/rollout_is_eff_sample_size:0.9987124115549366 - rollout_corr/rollout_is_seq_mean:0.9998928308486938 - rollout_corr/rollout_is_seq_std:0.001641205046325922 - rollout_corr/rollout_is_seq_max:1.008506178855896 - rollout_corr/rollout_is_seq_min:0.9953399300575256 - rollout_corr/rollout_is_seq_max_deviation:0.008506178855895996 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.5909629054367542) - rollout_corr/training_log_ppl:np.float64(0.45360937030636705) - rollout_corr/kl:np.float64(0.0007517823871872054) - rollout_corr/k3_kl:np.float64(0.0007286852571155578) - rollout_corr/rollout_ppl:np.float64(1.5898262090049684) - rollout_corr/rollout_log_ppl:np.float64(0.45285758542013355) - rollout_corr/log_ppl_diff:np.float64(0.0007517848862335086) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015345667488873005) - rollout_corr/log_ppl_diff_max:np.float64(0.00910806655883789) - rollout_corr/log_ppl_diff_min:np.float64(-0.007346630096435547) - rollout_corr/ppl_ratio:np.float64(1.0007539454381913) - rollout_corr/chi2_token:np.float64(0.00142996059730649) - rollout_corr/chi2_seq:np.float64(0.9373617491219193) - actor/pg_loss:np.float64(0.0030622705817222595) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003035150989489921) - actor/ppo_kl:np.float64(-6.880035599365897e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.765625) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.765625) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.5625) - self_distillation/token_reweight_w_mean:np.float64(482.64709699410014) - self_distillation/token_reweight_w_std:np.float64(7838.689288595924) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0027755422051996) - self_distillation/token_reweight_w_clip_frac:np.float64(0.06985412852372974) - self_distillation/empty_target_batch:np.float64(0.234375) - actor/grad_norm:np.float64(0.2353973463177681) - perf/mfu/actor:np.float64(0.058007253459953285) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.27971267700195) - actor/lr:np.float64(1e-06) - training/global_step:97 - training/epoch:1 - critic/score/mean:0.5625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0036426237784326077 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0036426237784326077 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:472.24609375 - response_length/max:1302.0 - response_length/min:164.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:472.24609375 - response_length_non_aborted/max:1302.0 - response_length_non_aborted/min:164.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:294.40625 - prompt_length/max:900.0 - prompt_length/min:172.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013850070536136627 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9791246633976698) - timing_s/agent_loop/generate_sequences/max:np.float64(10.536064883694053) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.344745161499304) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.536064883694053) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:219 - timing_s/agent_loop/slowest/response_length:1302 - timing_s/gen:17.246258018538356 - timing_s/reward:0.08357716910541058 - timing_s/old_log_prob:2.9908031839877367 - timing_s/adv:0.5831174645572901 - timing_s/update_actor:22.14829995855689 - timing_s/step:43.14054214023054 - timing_s/stop_profile:0.00011339783668518066 - timing_per_token_ms/update_actor:0.11285010398575834 - timing_per_token_ms/gen:0.14265484940269121 - timing_per_token_ms/adv:0.0029711023705807516 - perf/total_num_tokens:196263 - perf/time_per_step:43.14054214023054 - perf/throughput:568.6733124552454
(TaskRunner pid=2255397) Training Progress: 97%|█████████▋| 97/100 [1:29:24<02:18, 46.23s/it]
(TaskRunner pid=2255397) step:98 - global_seqlen/min:21081 - global_seqlen/max:28015 - global_seqlen/minmax_diff:6934 - global_seqlen/balanced_min:24041 - global_seqlen/balanced_max:24240 - global_seqlen/mean:24067.875 - actor/entropy:0.4481050968170166 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3950059413909912 - training/rollout_probs_diff_mean:0.0048210457898676395 - training/rollout_probs_diff_std:0.010794337838888168 - training/rollout_actor_probs_pearson_corr:0.9992629289627075 - self_distillation/success_group_fraction:0.90625 - self_distillation/success_sample_fraction:0.875 - self_distillation/correct_sample_fraction:0.609375 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.875 - rollout_corr/rollout_is_mean:1.000078797340393 - rollout_corr/rollout_is_ratio_fraction_high:3.446166738285683e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.7230833691428415e-05 - rollout_corr/rollout_is_max:4.591516971588135 - rollout_corr/rollout_is_min:0.39532703161239624 - rollout_corr/rollout_is_std:0.03653962165117264 - rollout_corr/rollout_is_eff_sample_size:0.9986668739730418 - rollout_corr/rollout_is_seq_mean:1.000028133392334 - rollout_corr/rollout_is_seq_std:0.0019362398888915777 - rollout_corr/rollout_is_seq_max:1.0084097385406494 - rollout_corr/rollout_is_seq_min:0.9942269325256348 - rollout_corr/rollout_is_seq_max_deviation:0.008409738540649414 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.616344683803618) - rollout_corr/training_log_ppl:np.float64(0.4696293716551736) - rollout_corr/kl:np.float64(0.0008239057563663721) - rollout_corr/k3_kl:np.float64(0.000780692749458467) - rollout_corr/rollout_ppl:np.float64(1.6150271990336478) - rollout_corr/rollout_log_ppl:np.float64(0.46880546608008444) - rollout_corr/log_ppl_diff:np.float64(0.0008239055750891566) - rollout_corr/log_ppl_abs_diff:np.float64(0.0015701879747211933) - rollout_corr/log_ppl_diff_max:np.float64(0.007200688123703003) - rollout_corr/log_ppl_diff_min:np.float64(-0.004303097724914551) - rollout_corr/ppl_ratio:np.float64(1.0008261122275144) - rollout_corr/chi2_token:np.float64(0.001507394015789032) - rollout_corr/chi2_seq:np.float64(0.24129036953672767) - actor/pg_loss:np.float64(0.003733934834599495) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000350107799476973) - actor/ppo_kl:np.float64(0.00011479198082042785) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.609375) - self_distillation/token_reweight_w_mean:np.float64(2963.129227875499) - self_distillation/token_reweight_w_std:np.float64(42459.57496579323) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0034989113919437) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09955895229359157) - self_distillation/empty_target_batch:np.float64(0.125) - actor/grad_norm:np.float64(0.2265518605709076) - perf/mfu/actor:np.float64(0.057423452397621935) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.2193260192871) - actor/lr:np.float64(1e-06) - training/global_step:98 - training/epoch:1 - critic/score/mean:0.609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.008424799889326096 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.008424799889326096 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:453.40234375 - response_length/max:1533.0 - response_length/min:159.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:453.40234375 - response_length_non_aborted/max:1533.0 - response_length_non_aborted/min:159.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:298.71875 - prompt_length/max:631.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013044476509094238 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0533475428819656) - timing_s/agent_loop/generate_sequences/max:np.float64(10.802223604172468) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.998830246309808) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.802223604172468) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:188 - timing_s/agent_loop/slowest/response_length:1533 - timing_s/gen:17.642000133171678 - timing_s/reward:0.06738298386335373 - timing_s/old_log_prob:3.014816017821431 - timing_s/adv:0.5671375151723623 - timing_s/update_actor:21.815314499661326 - timing_s/step:43.19491817057133 - timing_s/stop_profile:0.00011375732719898224 - timing_per_token_ms/update_actor:0.11330100029427882 - timing_per_token_ms/gen:0.15199317773751994 - timing_per_token_ms/adv:0.002945510951695789 - perf/total_num_tokens:192543 - perf/time_per_step:43.19491817057133 - perf/throughput:557.1922813919678
(TaskRunner pid=2255397) Training Progress: 98%|█████████▊| 98/100 [1:30:08<01:30, 45.33s/it]
(TaskRunner pid=2255397) step:99 - global_seqlen/min:19692 - global_seqlen/max:25383 - global_seqlen/minmax_diff:5691 - global_seqlen/balanced_min:23698 - global_seqlen/balanced_max:23716 - global_seqlen/mean:23712.375 - actor/entropy:0.4742501676082611 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4091079831123352 - training/rollout_probs_diff_mean:0.004931129980832338 - training/rollout_probs_diff_std:0.010619550943374634 - training/rollout_actor_probs_pearson_corr:0.9992989301681519 - self_distillation/success_group_fraction:0.78125 - self_distillation/success_sample_fraction:0.765625 - self_distillation/correct_sample_fraction:0.578125 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.765625 - rollout_corr/rollout_is_mean:0.9998912215232849 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.113195119425654e-06 - rollout_corr/rollout_is_max:1.7866209745407104 - rollout_corr/rollout_is_min:0.43442291021347046 - rollout_corr/rollout_is_std:0.03628097102046013 - rollout_corr/rollout_is_eff_sample_size:0.9986851836242739 - rollout_corr/rollout_is_seq_mean:0.9998824596405029 - rollout_corr/rollout_is_seq_std:0.00225405883975327 - rollout_corr/rollout_is_seq_max:1.0066941976547241 - rollout_corr/rollout_is_seq_min:0.9925881028175354 - rollout_corr/rollout_is_seq_max_deviation:0.0074118971824646 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6540217595174909) - rollout_corr/training_log_ppl:np.float64(0.49091246334137395) - rollout_corr/kl:np.float64(0.0009071645049343324) - rollout_corr/k3_kl:np.float64(0.0007826964022683569) - rollout_corr/rollout_ppl:np.float64(1.6525658327154815) - rollout_corr/rollout_log_ppl:np.float64(0.4900052981101908) - rollout_corr/log_ppl_diff:np.float64(0.0009071652311831713) - rollout_corr/log_ppl_abs_diff:np.float64(0.0018514589173719287) - rollout_corr/log_ppl_diff_max:np.float64(0.008251607418060303) - rollout_corr/log_ppl_diff_min:np.float64(-0.005766570568084717) - rollout_corr/ppl_ratio:np.float64(1.0009100551251322) - rollout_corr/chi2_token:np.float64(0.001357766566798091) - rollout_corr/chi2_seq:np.float64(1.0302318546455353) - actor/pg_loss:np.float64(0.004117259988561273) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00030947469667808036) - actor/ppo_kl:np.float64(9.188319657127408e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.765625) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.765625) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.578125) - self_distillation/token_reweight_w_mean:np.float64(190.36794193601236) - self_distillation/token_reweight_w_std:np.float64(4398.700665255601) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.0010327426716685) - self_distillation/token_reweight_w_clip_frac:np.float64(0.09465871116844937) - self_distillation/empty_target_batch:np.float64(0.234375) - actor/grad_norm:np.float64(0.2390700727701187) - perf/mfu/actor:np.float64(0.055725058096870825) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.29999542236328) - actor/lr:np.float64(1e-06) - training/global_step:99 - training/epoch:1 - critic/score/mean:0.578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.006170772016048431 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.006170772016048431 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:428.63671875 - response_length/max:1227.0 - response_length/min:169.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:428.63671875 - response_length_non_aborted/max:1227.0 - response_length_non_aborted/min:169.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:312.375 - prompt_length/max:836.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001377556473016739 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2593938652426004) - timing_s/agent_loop/generate_sequences/max:np.float64(10.128664135932922) - timing_s/agent_loop/generate_sequences/mean:np.float64(4.985631817675312) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(10.128664135932922) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:317 - timing_s/agent_loop/slowest/response_length:1227 - timing_s/gen:17.193039959296584 - timing_s/reward:0.06556063517928123 - timing_s/old_log_prob:2.973387535661459 - timing_s/adv:0.5816671065986156 - timing_s/update_actor:21.934669038280845 - timing_s/step:42.834324315190315 - timing_s/stop_profile:0.00016217492520809174 - timing_per_token_ms/update_actor:0.11562880689028854 - timing_per_token_ms/gen:0.15668352570646932 - timing_per_token_ms/adv:0.003066263431007099 - perf/total_num_tokens:189699 - perf/time_per_step:42.834324315190315 - perf/throughput:553.5834959252735
(TaskRunner pid=2255397) Training Progress: 99%|█████████▉| 99/100 [1:30:51<00:44, 44.60s/it]
(TaskRunner pid=2255397) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 100}
(TaskRunner pid=2255397) validation generation end
(TaskRunner pid=2255397) [prompt] system
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2255397) <reasoning>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) ...
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) user
(TaskRunner pid=2255397) Which of the following molecules is the only correct reactant used in the synthesis of the product "Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1" ?
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) A: O=C1c2ccccc2C(=O)N1c1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1
(TaskRunner pid=2255397) B: O=C1c2ccccc2C(Cl)N1c1ccc2ccc(Cl)nc2n1
(TaskRunner pid=2255397) C: O=C1c2ccccc2C(=O)N1Cc1ccc(Oc2ccc(Cl)c(C(F)(F)F)c2)cc1
(TaskRunner pid=2255397) D: O=C1c2ccccc2C(=O)N1CC=C(Cc1ccccc1)C(F)(F)F
(TaskRunner pid=2255397) Please reason step by step.
(TaskRunner pid=2255397) assistant
(TaskRunner pid=2255397) <think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </think>
(TaskRunner pid=2255397)
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) [response] <reasoning>
(TaskRunner pid=2255397) To determine the correct reactant for the synthesis of the product **"Nc1ccn(Cc2ccc(Cl)cc2C(F)(F)F)n1"**, we need to analyze the structure of the product and compare it with the structures of the given options.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) The product contains:
(TaskRunner pid=2255397) - A **nitrogen-containing ring** (likely an imine or azole ring),
(TaskRunner pid=2255397) - A **chlorine atom** attached to a benzene ring,
(TaskRunner pid=2255397) - A **trifluoromethyl group** (C(F)(F)F),
(TaskRunner pid=2255397) - A **nitrogen atom** connected to a carbon chain containing a **chlorinated benzene ring** and a **fluorinated group**.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Now, let’s examine the options:
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option A:**
(TaskRunner pid=2255397) - Contains a **nitrogen-containing ring** (likely a pyrrole or similar),
(TaskRunner pid=2255397) - A **chlorinated benzene ring** (with Cl),
(TaskRunner pid=2255397) - A **trifluoromethyl group** (C(F)(F)F),
(TaskRunner pid=2255397) - A **nitrogen atom** connected to a carbon chain that includes the **chlorinated benzene** and **fluorinated group**.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) This matches the structure of the target product very closely, including all the key functional groups and their positions.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option B:**
(TaskRunner pid=2255397) - Contains a **chlorine** and **nitrogen**, but the **fluorinated group** is not present in the correct position, and the **nitrogen ring** is not properly structured.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option C:**
(TaskRunner pid=2255397) - Contains a **fluorinated group**, but the **chlorine** is attached to an **oxygen**, not directly to the benzene ring. The **fluorinated group** is in a different position than in the product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) **Option D:**
(TaskRunner pid=2255397) - Contains a **fluorinated group** and a **chlorinated benzene ring**, but the **nitrogen-containing ring** is not present in the correct structure.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) Only **Option A** contains all the required functional groups in the correct positions to serve as the **only correct reactant** for the synthesis of the given product.
(TaskRunner pid=2255397)
(TaskRunner pid=2255397) </reasoning>
(TaskRunner pid=2255397) <answer>
(TaskRunner pid=2255397) A
(TaskRunner pid=2255397) </answer>
(TaskRunner pid=2255397) [ground_truth] A
(TaskRunner pid=2255397) [score] 1.0
(TaskRunner pid=2255397) [acc] 1.0
(TaskRunner pid=2255397) [pred] A
(TaskRunner pid=2255397) [incorrect_format] 1
(TaskRunner pid=2255397) [feedback]
(TaskRunner pid=2255397) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/chemistry/qwen3gen-chemistry-RLSD_TR-Qwen-Qwen3-8B-mbs8-decay0-tr0.1-train32-rollout8-lr1e-6-vllm0.8/global_step_100
(TaskRunner pid=2255397) step:100 - global_seqlen/min:23741 - global_seqlen/max:31057 - global_seqlen/minmax_diff:7316 - global_seqlen/balanced_min:24739 - global_seqlen/balanced_max:31757 - global_seqlen/mean:25617.625 - actor/entropy:0.4436495304107666 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24008721113204956 - training/rollout_probs_diff_mean:0.004401609301567078 - training/rollout_probs_diff_std:0.009758861735463142 - training/rollout_actor_probs_pearson_corr:0.9993872046470642 - self_distillation/success_group_fraction:0.9375 - self_distillation/success_sample_fraction:0.91796875 - self_distillation/correct_sample_fraction:0.69921875 - self_distillation/feedback_available_fraction:0.0 - self_distillation/feedback_used_fraction:0.0 - self_distillation/reprompt_sample_fraction:0.91796875 - rollout_corr/rollout_is_mean:1.0000593662261963 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.7518070936203003 - rollout_corr/rollout_is_min:0.5589723587036133 - rollout_corr/rollout_is_std:0.03378680348396301 - rollout_corr/rollout_is_eff_sample_size:0.9988599913647978 - rollout_corr/rollout_is_seq_mean:1.0000724792480469 - rollout_corr/rollout_is_seq_std:0.0018252491718158126 - rollout_corr/rollout_is_seq_max:1.0057400465011597 - rollout_corr/rollout_is_seq_min:0.9935486912727356 - rollout_corr/rollout_is_seq_max_deviation:0.006451308727264404 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.6480820523574948) - rollout_corr/training_log_ppl:np.float64(0.4857633382116546) - rollout_corr/kl:np.float64(0.0007351505751955756) - rollout_corr/k3_kl:np.float64(0.0006855184932135217) - rollout_corr/rollout_ppl:np.float64(1.6468097120523453) - rollout_corr/rollout_log_ppl:np.float64(0.48502818830229444) - rollout_corr/log_ppl_diff:np.float64(0.0007351499093601888) - rollout_corr/log_ppl_abs_diff:np.float64(0.001427958528893214) - rollout_corr/log_ppl_diff_max:np.float64(0.007762610912322998) - rollout_corr/log_ppl_diff_min:np.float64(-0.004005610942840576) - rollout_corr/ppl_ratio:np.float64(1.0007370123639703) - rollout_corr/chi2_token:np.float64(0.0012815699446946383) - rollout_corr/chi2_seq:np.float64(0.31913869245909154) - actor/pg_loss:np.float64(0.003367550321854651) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003017752374034899) - actor/ppo_kl:np.float64(0.00018170115079385596) - actor/pg_clipfrac_lower:np.float64(0.0) - self_distillation/token_reweight_lambda:np.float64(0.5) - self_distillation/token_reweight_eps_w:np.float64(0.20000000000000004) - self_distillation/token_reweight_active_token_fraction:np.float64(0.91796875) - self_distillation/token_reweight_context_sample_fraction:np.float64(0.91796875) - self_distillation/token_reweight_correct_sample_fraction:np.float64(0.69921875) - self_distillation/token_reweight_w_mean:np.float64(1570.6249493323267) - self_distillation/token_reweight_w_std:np.float64(40484.064735737746) - self_distillation/token_reweight_w_clipped_mean:np.float64(1.001023317920044) - self_distillation/token_reweight_w_clip_frac:np.float64(0.0961508322507143) - self_distillation/empty_target_batch:np.float64(0.08203125) - actor/grad_norm:np.float64(0.2478880025446415) - perf/mfu/actor:np.float64(0.05716674172282474) - perf/max_memory_allocated_gb:np.float64(134.57307815551758) - perf/max_memory_reserved_gb:np.float64(149.07421875) - perf/cpu_memory_used_gb:np.float64(197.12733459472656) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.7065476190476191) - val-aux/sciknoweval/reward/std@16:np.float64(0.14963967195739616) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7672380952380953) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.12522806043858783) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.6460952380952382) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.12793485078945213) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.7068333333333333) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.14978365629442147) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8158333333333333) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.09352166658220648) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.5964190476190476) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.09674390853009297) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.7151904761904762) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.1197973344827459) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8520904761904763) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.06467821428517558) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.5596142857142857) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.06871965760347537) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7202857142857142) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.08852864730736695) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8776952380952381) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.0403808452308748) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.5314904761904762) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.04527152807209723) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7233190476190475) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.06487826170016521) - val-aux/sciknoweval/score/mean@16:np.float64(0.7065476190476191) - val-aux/sciknoweval/score/std@16:np.float64(0.14963967195739616) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7672380952380953) - val-aux/sciknoweval/score/best@2/std:np.float64(0.12522806043858783) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.6460952380952382) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.12793485078945213) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.7068333333333333) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.14978365629442147) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8158333333333333) - val-aux/sciknoweval/score/best@4/std:np.float64(0.09352166658220648) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.5964190476190476) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.09674390853009297) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.7151904761904762) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.1197973344827459) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8520904761904763) - val-aux/sciknoweval/score/best@8/std:np.float64(0.06467821428517558) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.5596142857142857) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.06871965760347537) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7202857142857142) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.08852864730736695) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8776952380952381) - val-aux/sciknoweval/score/best@16/std:np.float64(0.0403808452308748) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.5314904761904762) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.04527152807209723) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7233190476190475) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.06487826170016521) - val-core/sciknoweval/acc/mean@16:np.float64(0.7065476190476191) - val-aux/sciknoweval/acc/std@16:np.float64(0.14963967195739616) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7672380952380953) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.12522806043858783) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.6460952380952382) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.12793485078945213) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.7068333333333333) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.14978365629442147) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8158333333333333) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.09352166658220648) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.5964190476190476) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.09674390853009297) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.7151904761904762) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.1197973344827459) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8520904761904763) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.06467821428517558) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.5596142857142857) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.06871965760347537) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7202857142857142) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.08852864730736695) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8776952380952381) - val-core/sciknoweval/acc/best@16/std:np.float64(0.0403808452308748) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.5314904761904762) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.04527152807209723) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7233190476190475) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.06487826170016521) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9982142857142857) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.006916041689656103) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.999904761904762) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.0016180983939162673) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9966190476190476) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.009226171602153282) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9982380952380953) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.006862312067166463) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9934619047619048) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.012001753208129275) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9993142857142857) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.0043365672892029085) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.988647619047619) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.01397823789672266) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9998380952380953) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.0019314678388328032) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.9817619047619048) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.013705261025670075) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9999714285714286) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.0006015850374731164) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:100 - training/epoch:1 - critic/score/mean:0.69921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.69921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0026495642960071564 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0026495642960071564 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:504.39453125 - response_length/max:8192.0 - response_length/min:195.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:504.39453125 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:195.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:296.15625 - prompt_length/max:625.0 - prompt_length/min:165.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013913773000240326 - timing_s/agent_loop/generate_sequences/min:np.float64(2.3870061337947845) - timing_s/agent_loop/generate_sequences/max:np.float64(60.303343798965216) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.73472642654815) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(60.303343798965216) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:326 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:67.0998271573335 - timing_s/reward:0.0683623980730772 - timing_s/old_log_prob:3.3458864334970713 - timing_s/adv:0.5801629964262247 - timing_s/update_actor:23.488009363412857 - timing_s/step:94.66876287944615 - timing_s/testing:143.36135754734278 - timing_s/save_checkpoint:13.737380957230926 - timing_s/stop_profile:0.00018646568059921265 - timing_per_token_ms/update_actor:0.11460864035704353 - timing_per_token_ms/gen:0.5196501619154579 - timing_per_token_ms/adv:0.002830878137738299 - perf/total_num_tokens:204941 - perf/time_per_step:94.66876287944615 - perf/throughput:270.6027228075453
(TaskRunner pid=2255397) ("Final validation metrics: {'val-aux/sciknoweval/reward/mean@16': "
(TaskRunner pid=2255397) "np.float64(0.7065476190476191), 'val-aux/sciknoweval/reward/std@16': "
(TaskRunner pid=2255397) "np.float64(0.14963967195739616), 'val-aux/sciknoweval/reward/best@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.7672380952380953), 'val-aux/sciknoweval/reward/best@2/std': "
(TaskRunner pid=2255397) "np.float64(0.12522806043858783), 'val-aux/sciknoweval/reward/worst@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.6460952380952382), 'val-aux/sciknoweval/reward/worst@2/std': "
(TaskRunner pid=2255397) "np.float64(0.12793485078945213), 'val-aux/sciknoweval/reward/maj@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.7068333333333333), 'val-aux/sciknoweval/reward/maj@2/std': "
(TaskRunner pid=2255397) "np.float64(0.14978365629442147), 'val-aux/sciknoweval/reward/best@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.8158333333333333), 'val-aux/sciknoweval/reward/best@4/std': "
(TaskRunner pid=2255397) "np.float64(0.09352166658220648), 'val-aux/sciknoweval/reward/worst@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.5964190476190476), 'val-aux/sciknoweval/reward/worst@4/std': "
(TaskRunner pid=2255397) "np.float64(0.09674390853009297), 'val-aux/sciknoweval/reward/maj@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.7151904761904762), 'val-aux/sciknoweval/reward/maj@4/std': "
(TaskRunner pid=2255397) "np.float64(0.1197973344827459), 'val-aux/sciknoweval/reward/best@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.8520904761904763), 'val-aux/sciknoweval/reward/best@8/std': "
(TaskRunner pid=2255397) "np.float64(0.06467821428517558), 'val-aux/sciknoweval/reward/worst@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.5596142857142857), 'val-aux/sciknoweval/reward/worst@8/std': "
(TaskRunner pid=2255397) "np.float64(0.06871965760347537), 'val-aux/sciknoweval/reward/maj@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.7202857142857142), 'val-aux/sciknoweval/reward/maj@8/std': "
(TaskRunner pid=2255397) "np.float64(0.08852864730736695), 'val-aux/sciknoweval/reward/best@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.8776952380952381), 'val-aux/sciknoweval/reward/best@16/std': "
(TaskRunner pid=2255397) "np.float64(0.0403808452308748), 'val-aux/sciknoweval/reward/worst@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.5314904761904762), 'val-aux/sciknoweval/reward/worst@16/std': "
(TaskRunner pid=2255397) "np.float64(0.04527152807209723), 'val-aux/sciknoweval/reward/maj@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.7233190476190475), 'val-aux/sciknoweval/reward/maj@16/std': "
(TaskRunner pid=2255397) "np.float64(0.06487826170016521), 'val-aux/sciknoweval/score/mean@16': "
(TaskRunner pid=2255397) "np.float64(0.7065476190476191), 'val-aux/sciknoweval/score/std@16': "
(TaskRunner pid=2255397) "np.float64(0.14963967195739616), 'val-aux/sciknoweval/score/best@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.7672380952380953), 'val-aux/sciknoweval/score/best@2/std': "
(TaskRunner pid=2255397) "np.float64(0.12522806043858783), 'val-aux/sciknoweval/score/worst@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.6460952380952382), 'val-aux/sciknoweval/score/worst@2/std': "
(TaskRunner pid=2255397) "np.float64(0.12793485078945213), 'val-aux/sciknoweval/score/maj@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.7068333333333333), 'val-aux/sciknoweval/score/maj@2/std': "
(TaskRunner pid=2255397) "np.float64(0.14978365629442147), 'val-aux/sciknoweval/score/best@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.8158333333333333), 'val-aux/sciknoweval/score/best@4/std': "
(TaskRunner pid=2255397) "np.float64(0.09352166658220648), 'val-aux/sciknoweval/score/worst@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.5964190476190476), 'val-aux/sciknoweval/score/worst@4/std': "
(TaskRunner pid=2255397) "np.float64(0.09674390853009297), 'val-aux/sciknoweval/score/maj@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.7151904761904762), 'val-aux/sciknoweval/score/maj@4/std': "
(TaskRunner pid=2255397) "np.float64(0.1197973344827459), 'val-aux/sciknoweval/score/best@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.8520904761904763), 'val-aux/sciknoweval/score/best@8/std': "
(TaskRunner pid=2255397) "np.float64(0.06467821428517558), 'val-aux/sciknoweval/score/worst@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.5596142857142857), 'val-aux/sciknoweval/score/worst@8/std': "
(TaskRunner pid=2255397) "np.float64(0.06871965760347537), 'val-aux/sciknoweval/score/maj@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.7202857142857142), 'val-aux/sciknoweval/score/maj@8/std': "
(TaskRunner pid=2255397) "np.float64(0.08852864730736695), 'val-aux/sciknoweval/score/best@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.8776952380952381), 'val-aux/sciknoweval/score/best@16/std': "
(TaskRunner pid=2255397) "np.float64(0.0403808452308748), 'val-aux/sciknoweval/score/worst@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.5314904761904762), 'val-aux/sciknoweval/score/worst@16/std': "
(TaskRunner pid=2255397) "np.float64(0.04527152807209723), 'val-aux/sciknoweval/score/maj@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.7233190476190475), 'val-aux/sciknoweval/score/maj@16/std': "
(TaskRunner pid=2255397) "np.float64(0.06487826170016521), 'val-core/sciknoweval/acc/mean@16': "
(TaskRunner pid=2255397) "np.float64(0.7065476190476191), 'val-aux/sciknoweval/acc/std@16': "
(TaskRunner pid=2255397) "np.float64(0.14963967195739616), 'val-aux/sciknoweval/acc/best@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.7672380952380953), 'val-aux/sciknoweval/acc/best@2/std': "
(TaskRunner pid=2255397) "np.float64(0.12522806043858783), 'val-aux/sciknoweval/acc/worst@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.6460952380952382), 'val-aux/sciknoweval/acc/worst@2/std': "
(TaskRunner pid=2255397) "np.float64(0.12793485078945213), 'val-aux/sciknoweval/acc/maj@2/mean': "
(TaskRunner pid=2255397) "np.float64(0.7068333333333333), 'val-aux/sciknoweval/acc/maj@2/std': "
(TaskRunner pid=2255397) "np.float64(0.14978365629442147), 'val-aux/sciknoweval/acc/best@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.8158333333333333), 'val-aux/sciknoweval/acc/best@4/std': "
(TaskRunner pid=2255397) "np.float64(0.09352166658220648), 'val-aux/sciknoweval/acc/worst@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.5964190476190476), 'val-aux/sciknoweval/acc/worst@4/std': "
(TaskRunner pid=2255397) "np.float64(0.09674390853009297), 'val-aux/sciknoweval/acc/maj@4/mean': "
(TaskRunner pid=2255397) "np.float64(0.7151904761904762), 'val-aux/sciknoweval/acc/maj@4/std': "
(TaskRunner pid=2255397) "np.float64(0.1197973344827459), 'val-aux/sciknoweval/acc/best@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.8520904761904763), 'val-aux/sciknoweval/acc/best@8/std': "
(TaskRunner pid=2255397) "np.float64(0.06467821428517558), 'val-aux/sciknoweval/acc/worst@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.5596142857142857), 'val-aux/sciknoweval/acc/worst@8/std': "
(TaskRunner pid=2255397) "np.float64(0.06871965760347537), 'val-aux/sciknoweval/acc/maj@8/mean': "
(TaskRunner pid=2255397) "np.float64(0.7202857142857142), 'val-aux/sciknoweval/acc/maj@8/std': "
(TaskRunner pid=2255397) "np.float64(0.08852864730736695), 'val-core/sciknoweval/acc/best@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.8776952380952381), 'val-core/sciknoweval/acc/best@16/std': "
(TaskRunner pid=2255397) "np.float64(0.0403808452308748), 'val-aux/sciknoweval/acc/worst@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.5314904761904762), 'val-aux/sciknoweval/acc/worst@16/std': "
(TaskRunner pid=2255397) "np.float64(0.04527152807209723), 'val-core/sciknoweval/acc/maj@16/mean': "
(TaskRunner pid=2255397) "np.float64(0.7233190476190475), 'val-core/sciknoweval/acc/maj@16/std': "
(TaskRunner pid=2255397) 'np.float64(0.06487826170016521), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/mean@16': "
(TaskRunner pid=2255397) 'np.float64(0.9982142857142857), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/std@16': "
(TaskRunner pid=2255397) 'np.float64(0.006916041689656103), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/best@2/mean': "
(TaskRunner pid=2255397) 'np.float64(0.999904761904762), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/best@2/std': "
(TaskRunner pid=2255397) 'np.float64(0.0016180983939162673), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/worst@2/mean': "
(TaskRunner pid=2255397) 'np.float64(0.9966190476190476), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/worst@2/std': "
(TaskRunner pid=2255397) 'np.float64(0.009226171602153282), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/maj@2/mean': "
(TaskRunner pid=2255397) 'np.float64(0.9982380952380953), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/maj@2/std': "
(TaskRunner pid=2255397) 'np.float64(0.006862312067166463), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/best@4/mean': np.float64(1.0), "
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/best@4/std': np.float64(0.0), "
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/worst@4/mean': "
(TaskRunner pid=2255397) 'np.float64(0.9934619047619048), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/worst@4/std': "
(TaskRunner pid=2255397) 'np.float64(0.012001753208129275), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/maj@4/mean': "
(TaskRunner pid=2255397) 'np.float64(0.9993142857142857), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/maj@4/std': "
(TaskRunner pid=2255397) 'np.float64(0.0043365672892029085), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/best@8/mean': np.float64(1.0), "
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/best@8/std': np.float64(0.0), "
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/worst@8/mean': "
(TaskRunner pid=2255397) 'np.float64(0.988647619047619), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/worst@8/std': "
(TaskRunner pid=2255397) 'np.float64(0.01397823789672266), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/maj@8/mean': "
(TaskRunner pid=2255397) 'np.float64(0.9998380952380953), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/maj@8/std': "
(TaskRunner pid=2255397) 'np.float64(0.0019314678388328032), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/best@16/mean': np.float64(1.0), "
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/best@16/std': np.float64(0.0), "
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/worst@16/mean': "
(TaskRunner pid=2255397) 'np.float64(0.9817619047619048), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/worst@16/std': "
(TaskRunner pid=2255397) 'np.float64(0.013705261025670075), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/maj@16/mean': "
(TaskRunner pid=2255397) 'np.float64(0.9999714285714286), '
(TaskRunner pid=2255397) "'val-aux/sciknoweval/incorrect_format/maj@16/std': "
(TaskRunner pid=2255397) "np.float64(0.0006015850374731164), 'val-aux/num_turns/min': np.int32(2), "
(TaskRunner pid=2255397) "'val-aux/num_turns/max': np.int32(2), 'val-aux/num_turns/mean': "
(TaskRunner pid=2255397) 'np.float64(2.0)}')
(TaskRunner pid=2255397) Training Progress: 100%|██████████| 100/100 [1:35:02<00:00, 106.78s/it] Training Progress: 100%|██████████| 100/100 [1:35:02<00:00, 57.03s/it]
(TaskRunner pid=2255397) Traceback (most recent call last):
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py", line 322, in <module>
(TaskRunner pid=2255397) worker.main_loop()
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/worker.py", line 1031, in main_loop
(TaskRunner pid=2255397) self.core_worker.run_task_loop()
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
(TaskRunner pid=2255397) _error_if_any_worker_fails()
(TaskRunner pid=2255397) RuntimeError: DataLoader worker (pid 2270092) is killed by signal: Killed.
(TaskRunner pid=2255397) Traceback (most recent call last):
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/workers/default_worker.py", line 322, in <module>
(TaskRunner pid=2255397) worker.main_loop()
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/worker.py", line 1031, in main_loop
(TaskRunner pid=2255397) self.core_worker.run_task_loop()
(TaskRunner pid=2255397) Exception ignored in: <function Tracking.__del__ at 0x7c4b4a93e020>
(TaskRunner pid=2255397) Traceback (most recent call last):
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/verl/utils/tracking.py", line 169, in __del__
(TaskRunner pid=2255397) self.logger["wandb"].finish(exit_code=0)
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_run.py", line 4097, in finish
(TaskRunner pid=2255397) wandb.run.finish(exit_code=exit_code, quiet=quiet)
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_run.py", line 400, in wrapper
(TaskRunner pid=2255397) return func(self, *args, **kwargs)
(TaskRunner pid=2255397) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_run.py", line 445, in wrapper
(TaskRunner pid=2255397) return func(self, *args, **kwargs)
(TaskRunner pid=2255397) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_run.py", line 2275, in finish
(TaskRunner pid=2255397) return self._finish(exit_code)
(TaskRunner pid=2255397) ^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_run.py", line 400, in wrapper
(TaskRunner pid=2255397) return func(self, *args, **kwargs)
(TaskRunner pid=2255397) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_run.py", line 2288, in _finish
(TaskRunner pid=2255397) with telemetry.context(run=self) as tel:
(TaskRunner pid=2255397) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/telemetry.py", line 42, in __exit__
(TaskRunner pid=2255397) self._run._telemetry_callback(self._obj)
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_run.py", line 752, in _telemetry_callback
(TaskRunner pid=2255397) self._telemetry_flush()
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/wandb_run.py", line 765, in _telemetry_flush
(TaskRunner pid=2255397) self._backend.interface._publish_telemetry(self._telemetry_obj)
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/interface/interface_shared.py", line 103, in _publish_telemetry
(TaskRunner pid=2255397) self._publish(rec)
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/interface/interface_sock.py", line 46, in _publish
(TaskRunner pid=2255397) self._asyncer.run(lambda: self._client.publish(request))
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 136, in run
(TaskRunner pid=2255397) return future.result()
(TaskRunner pid=2255397) ^^^^^^^^^^^^^^^
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/concurrent/futures/_base.py", line 456, in result
(TaskRunner pid=2255397) return self.__get_result()
(TaskRunner pid=2255397) ^^^^^^^^^^^^^^^^^^^
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(TaskRunner pid=2255397) raise self._exception
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 219, in _wrap
(TaskRunner pid=2255397) return await fn()
(TaskRunner pid=2255397) ^^^^^^^^^^
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
(TaskRunner pid=2255397) await self._send_server_request(request)
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/wandb/sdk/lib/service/service_client.py", line 59, in _send_server_request
(TaskRunner pid=2255397) self._writer.write(header)
(TaskRunner pid=2255397) File "/mnt/mole/SDPO/L2T/.venv/lib/python3.12/asyncio/streams.py", line 346, in write
(TaskRunner pid=2255397) self._transport.write(data)
(TaskRunner pid=2255397) File "uvloop/handles/stream.pyx", line 675, in uvloop.loop.UVStream.write
(TaskRunner pid=2255397) File "uvloop/handles/handle.pyx", line 159, in uvloop.loop.UVHandle._ensure_alive
(TaskRunner pid=2255397) RuntimeError: unable to perform operation on <UnixTransport closed=True reading=False 0x7c4aaf7460c0>; the handler is closed
main_ppo exit code: 0
[2026-07-03 03:18:43] Finished Qwen/Qwen3-8B chemistry rlsd_tr.
[2026-07-03 03:18:43] Starting Qwen/Qwen3-8B physics grpo.
Experiment: qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
Dataset: physics
Method: grpo
Config: baseline_grpo
Model: Qwen/Qwen3-8B
Train: /mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/train.parquet
Val: /mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/test.parquet
Ray tmp: /tmp/ray_q3g_physics_grpo_Qwen3_8B
2026-07-03 03:18:44,953 INFO scripts.py:1364 -- Did not find any active Ray processes.
Experiment: qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
Config: baseline_grpo
Task: datasets/sciknoweval/physics
Arguments: max_model_len=10240 data.train_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/train.parquet] data.val_files=[/mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/test.parquet] data.train_batch_size=32 data.train_max_samples=3200 data.max_prompt_length=2048 data.max_response_length=8192 data.apply_chat_template_kwargs.enable_thinking=false trainer.group_name=QWEN3-GRPO-generalization trainer.n_gpus_per_node=8 trainer.total_training_steps=100 trainer.val_before_train=False trainer.save_freq=10 trainer.test_freq=10 trainer.default_local_dir=/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8 custom_reward_function.path=/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py +ray_kwargs.ray_init._temp_dir=/tmp/ray_q3g_physics_grpo_Qwen3_8B +ray_kwargs.ray_init.include_dashboard=False +ray_kwargs.ray_init.runtime_env.env_vars.USER=root +ray_kwargs.ray_init.runtime_env.env_vars.TASK=datasets/sciknoweval/physics +ray_kwargs.ray_init.runtime_env.env_vars.EXPERIMENT=qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8 actor_rollout_ref.model.path=Qwen/Qwen3-8B actor_rollout_ref.actor.optim.lr=1e-6 actor_rollout_ref.actor.optim.lr_warmup_steps=10 actor_rollout_ref.actor.optim.weight_decay=0.01 actor_rollout_ref.actor.grad_clip=1.0 actor_rollout_ref.actor.ppo_mini_batch_size=8 actor_rollout_ref.actor.clip_ratio_high=0.28 actor_rollout_ref.actor.clip_ratio_low=0.2 actor_rollout_ref.actor.ppo_max_token_len_per_gpu=10240 actor_rollout_ref.rollout.n=8 actor_rollout_ref.rollout.temperature=1.0 actor_rollout_ref.rollout.top_p=1.0 actor_rollout_ref.rollout.max_model_len=10240 actor_rollout_ref.rollout.max_num_batched_tokens=10240 actor_rollout_ref.rollout.gpu_memory_utilization=0.8 actor_rollout_ref.rollout.calculate_log_probs=True actor_rollout_ref.rollout.val_kwargs.n=16 actor_rollout_ref.rollout.val_kwargs.temperature=0.6 actor_rollout_ref.rollout.val_kwargs.top_p=0.95 actor_rollout_ref.rollout.val_kwargs.do_sample=True algorithm.norm_adv_by_std_in_grpo=False algorithm.rollout_correction.rollout_is=token algorithm.rollout_correction.rollout_is_threshold=2.0 trainer.max_actor_ckpt_to_keep=100 actor_rollout_ref.actor.policy_loss.loss_mode=vanilla actor_rollout_ref.actor.kl_loss_coef=0.0
ray init kwargs: {'num_cpus': None, '_temp_dir': '/tmp/ray_q3g_physics_grpo_Qwen3_8B', 'include_dashboard': False, 'runtime_env': {'env_vars': {'TOKENIZERS_PARALLELISM': 'true', 'NCCL_DEBUG': 'WARN', 'VLLM_LOGGING_LEVEL': 'WARN', 'VLLM_ALLOW_RUNTIME_LORA_UPDATING': 'true', 'VLLM_ALLREDUCE_USE_SYMM_MEM': '0', 'CUDA_DEVICE_MAX_CONNECTIONS': '1', 'NCCL_CUMEM_ENABLE': '0', 'USER': 'root', 'TASK': 'datasets/sciknoweval/physics', 'EXPERIMENT': 'qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8'}, 'working_dir': None}}
2026-07-03 03:18:54,758 INFO worker.py:2007 -- Started a local Ray instance.
/mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/ray/_private/worker.py:2046: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
warnings.warn(
(TaskRunner pid=2281424) TaskRunner hostname: mole-workspace-rw, PID: 2281424
(TaskRunner pid=2281424) {'actor_rollout_ref': {'actor': {'_target_': 'verl.workers.config.FSDPActorConfig',
(TaskRunner pid=2281424) 'calculate_entropy': False,
(TaskRunner pid=2281424) 'calculate_sum_pi_squared': False,
(TaskRunner pid=2281424) 'checkpoint': {'_target_': 'verl.trainer.config.CheckpointConfig',
(TaskRunner pid=2281424) 'async_save': False,
(TaskRunner pid=2281424) 'load_contents': ['model',
(TaskRunner pid=2281424) 'optimizer',
(TaskRunner pid=2281424) 'extra'],
(TaskRunner pid=2281424) 'save_contents': ['model',
(TaskRunner pid=2281424) 'optimizer',
(TaskRunner pid=2281424) 'extra']},
(TaskRunner pid=2281424) 'clip_ratio': 0.2,
(TaskRunner pid=2281424) 'clip_ratio_c': 3.0,
(TaskRunner pid=2281424) 'clip_ratio_high': 0.28,
(TaskRunner pid=2281424) 'clip_ratio_low': 0.2,
(TaskRunner pid=2281424) 'data_loader_seed': 42,
(TaskRunner pid=2281424) 'entropy_checkpointing': False,
(TaskRunner pid=2281424) 'entropy_coeff': 0,
(TaskRunner pid=2281424) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2281424) 'freeze_vision_tower': False,
(TaskRunner pid=2281424) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2281424) 'dtype': 'bfloat16',
(TaskRunner pid=2281424) 'entropy_checkpointing': False,
(TaskRunner pid=2281424) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2281424) 'forward_only': False,
(TaskRunner pid=2281424) 'forward_prefetch': False,
(TaskRunner pid=2281424) 'fsdp_size': -1,
(TaskRunner pid=2281424) 'full_determinism': False,
(TaskRunner pid=2281424) 'model_dtype': 'fp32',
(TaskRunner pid=2281424) 'offload_policy': False,
(TaskRunner pid=2281424) 'optimizer_offload': False,
(TaskRunner pid=2281424) 'param_offload': False,
(TaskRunner pid=2281424) 'reshard_after_forward': True,
(TaskRunner pid=2281424) 'seed': 42,
(TaskRunner pid=2281424) 'strategy': 'fsdp',
(TaskRunner pid=2281424) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2281424) 'use_orig_params': False,
(TaskRunner pid=2281424) 'use_torch_compile': True,
(TaskRunner pid=2281424) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2281424) 'grad_clip': 1.0,
(TaskRunner pid=2281424) 'kl_loss_coef': 0.0,
(TaskRunner pid=2281424) 'kl_loss_type': 'low_var_kl',
(TaskRunner pid=2281424) 'loss_agg_mode': 'token-mean',
(TaskRunner pid=2281424) 'loss_scale_factor': None,
(TaskRunner pid=2281424) 'optim': {'_target_': 'verl.workers.config.FSDPOptimizerConfig',
(TaskRunner pid=2281424) 'betas': [0.9, 0.999],
(TaskRunner pid=2281424) 'clip_grad': 1.0,
(TaskRunner pid=2281424) 'lr': 1e-06,
(TaskRunner pid=2281424) 'lr_scheduler_type': 'constant',
(TaskRunner pid=2281424) 'lr_warmup_steps': 10,
(TaskRunner pid=2281424) 'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=2281424) 'min_lr_ratio': 0.0,
(TaskRunner pid=2281424) 'num_cycles': 0.5,
(TaskRunner pid=2281424) 'optimizer': 'AdamW',
(TaskRunner pid=2281424) 'optimizer_impl': 'torch.optim',
(TaskRunner pid=2281424) 'override_optimizer_config': None,
(TaskRunner pid=2281424) 'total_training_steps': -1,
(TaskRunner pid=2281424) 'warmup_style': None,
(TaskRunner pid=2281424) 'weight_decay': 0.01},
(TaskRunner pid=2281424) 'policy_loss': {'_target_': 'verl.workers.config.PolicyLossConfig',
(TaskRunner pid=2281424) 'clip_cov_lb': 1.0,
(TaskRunner pid=2281424) 'clip_cov_ratio': 0.0002,
(TaskRunner pid=2281424) 'clip_cov_ub': 5.0,
(TaskRunner pid=2281424) 'kl_cov_ratio': 0.0002,
(TaskRunner pid=2281424) 'loss_mode': 'vanilla',
(TaskRunner pid=2281424) 'ppo_kl_coef': 0.1},
(TaskRunner pid=2281424) 'ppo_epochs': 1,
(TaskRunner pid=2281424) 'ppo_max_token_len_per_gpu': 10240,
(TaskRunner pid=2281424) 'ppo_micro_batch_size': None,
(TaskRunner pid=2281424) 'ppo_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2281424) 'ppo_mini_batch_size': 8,
(TaskRunner pid=2281424) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2281424) 'all_ranks': False,
(TaskRunner pid=2281424) 'enable': False,
(TaskRunner pid=2281424) 'ranks': [],
(TaskRunner pid=2281424) 'save_path': 'outputs/profile',
(TaskRunner pid=2281424) 'tool': None,
(TaskRunner pid=2281424) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2281424) 'analysis': True,
(TaskRunner pid=2281424) 'contents': [],
(TaskRunner pid=2281424) 'discrete': False,
(TaskRunner pid=2281424) 'level': 'level0'},
(TaskRunner pid=2281424) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2281424) 'discrete': False},
(TaskRunner pid=2281424) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2281424) 'step_end': None,
(TaskRunner pid=2281424) 'step_start': 0},
(TaskRunner pid=2281424) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2281424) 'stack_depth': 32,
(TaskRunner pid=2281424) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2281424) 'rollout_n': 8,
(TaskRunner pid=2281424) 'router_replay': {'_target_': 'verl.workers.config.RouterReplayConfig',
(TaskRunner pid=2281424) 'mode': 'disabled',
(TaskRunner pid=2281424) 'record_file': None,
(TaskRunner pid=2281424) 'replay_file': None},
(TaskRunner pid=2281424) 'self_distillation': {'_target_': 'verl.workers.config.SelfDistillationConfig',
(TaskRunner pid=2281424) 'alpha': 0.5,
(TaskRunner pid=2281424) 'distillation_add_tail': True,
(TaskRunner pid=2281424) 'distillation_topk': 100,
(TaskRunner pid=2281424) 'dont_reprompt_on_self_success': True,
(TaskRunner pid=2281424) 'environment_feedback_only_without_solution': True,
(TaskRunner pid=2281424) 'evolving_teacher': {'_target_': 'verl.workers.config.EvolvingTeacherConfig',
(TaskRunner pid=2281424) 'enable': False,
(TaskRunner pid=2281424) 'loss_weight': 0.0,
(TaskRunner pid=2281424) 'mask': 'all'},
(TaskRunner pid=2281424) 'feedback_template': '\n'
(TaskRunner pid=2281424) 'The '
(TaskRunner pid=2281424) 'following '
(TaskRunner pid=2281424) 'is '
(TaskRunner pid=2281424) 'feedback '
(TaskRunner pid=2281424) 'from '
(TaskRunner pid=2281424) 'your '
(TaskRunner pid=2281424) 'unsuccessful '
(TaskRunner pid=2281424) 'earlier '
(TaskRunner pid=2281424) 'attempt:\n'
(TaskRunner pid=2281424) '\n'
(TaskRunner pid=2281424) '{feedback_raw}',
(TaskRunner pid=2281424) 'full_logit_distillation': True,
(TaskRunner pid=2281424) 'include_environment_feedback': True,
(TaskRunner pid=2281424) 'is_clip': 2,
(TaskRunner pid=2281424) 'max_reprompt_len': 22528,
(TaskRunner pid=2281424) 'remove_thinking_from_demonstration': False,
(TaskRunner pid=2281424) 'reprompt_template': '{prompt}{solution}{feedback}\n'
(TaskRunner pid=2281424) '\n'
(TaskRunner pid=2281424) 'Correctly '
(TaskRunner pid=2281424) 'solve '
(TaskRunner pid=2281424) 'the '
(TaskRunner pid=2281424) 'original '
(TaskRunner pid=2281424) 'question.',
(TaskRunner pid=2281424) 'reprompt_truncation': 'right',
(TaskRunner pid=2281424) 'solution_template': '\n'
(TaskRunner pid=2281424) 'Correct '
(TaskRunner pid=2281424) 'solution:\n'
(TaskRunner pid=2281424) '\n'
(TaskRunner pid=2281424) '{successful_previous_attempt}',
(TaskRunner pid=2281424) 'srpo_dynamic_weighting': False,
(TaskRunner pid=2281424) 'srpo_dynamic_weighting_temperature': 1.0,
(TaskRunner pid=2281424) 'success_reward_threshold': 0.5,
(TaskRunner pid=2281424) 'teacher_regularization': 'ema',
(TaskRunner pid=2281424) 'teacher_update_rate': 0.0,
(TaskRunner pid=2281424) 'token_reweight_decay_steps': None,
(TaskRunner pid=2281424) 'token_reweight_eps_w': 0.2,
(TaskRunner pid=2281424) 'token_reweight_lambda': 0.5},
(TaskRunner pid=2281424) 'shuffle': False,
(TaskRunner pid=2281424) 'strategy': 'fsdp',
(TaskRunner pid=2281424) 'sum_pi_squared_checkpointing': False,
(TaskRunner pid=2281424) 'tau_neg': 1.05,
(TaskRunner pid=2281424) 'tau_pos': 1.0,
(TaskRunner pid=2281424) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2281424) 'use_dynamic_bsz': False,
(TaskRunner pid=2281424) 'use_fused_kernels': False,
(TaskRunner pid=2281424) 'use_kl_loss': False,
(TaskRunner pid=2281424) 'use_prefix_grouper': False,
(TaskRunner pid=2281424) 'use_remove_padding': True,
(TaskRunner pid=2281424) 'use_torch_compile': True},
(TaskRunner pid=2281424) 'hybrid_engine': True,
(TaskRunner pid=2281424) 'model': {'_target_': 'verl.workers.config.HFModelConfig',
(TaskRunner pid=2281424) 'custom_chat_template': None,
(TaskRunner pid=2281424) 'enable_activation_offload': False,
(TaskRunner pid=2281424) 'enable_gradient_checkpointing': True,
(TaskRunner pid=2281424) 'exclude_modules': None,
(TaskRunner pid=2281424) 'external_lib': None,
(TaskRunner pid=2281424) 'fused_kernel_options': {'impl_backend': 'torch'},
(TaskRunner pid=2281424) 'hf_config_path': None,
(TaskRunner pid=2281424) 'lora_adapter_path': None,
(TaskRunner pid=2281424) 'lora_alpha': 16,
(TaskRunner pid=2281424) 'lora_rank': 0,
(TaskRunner pid=2281424) 'override_config': {},
(TaskRunner pid=2281424) 'path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2281424) 'target_modules': 'all-linear',
(TaskRunner pid=2281424) 'tiled_mlp': {'enabled': False,
(TaskRunner pid=2281424) 'num_shards': 4},
(TaskRunner pid=2281424) 'tokenizer_path': None,
(TaskRunner pid=2281424) 'trust_remote_code': True,
(TaskRunner pid=2281424) 'use_fused_kernels': False,
(TaskRunner pid=2281424) 'use_liger': False,
(TaskRunner pid=2281424) 'use_remove_padding': True,
(TaskRunner pid=2281424) 'use_shm': False},
(TaskRunner pid=2281424) 'nccl_timeout': 600,
(TaskRunner pid=2281424) 'ref': {'_target_': 'verl.workers.config.FSDPActorConfig',
(TaskRunner pid=2281424) 'entropy_checkpointing': False,
(TaskRunner pid=2281424) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2281424) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2281424) 'dtype': 'bfloat16',
(TaskRunner pid=2281424) 'entropy_checkpointing': False,
(TaskRunner pid=2281424) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2281424) 'forward_only': True,
(TaskRunner pid=2281424) 'forward_prefetch': False,
(TaskRunner pid=2281424) 'fsdp_size': -1,
(TaskRunner pid=2281424) 'full_determinism': False,
(TaskRunner pid=2281424) 'model_dtype': 'fp32',
(TaskRunner pid=2281424) 'offload_policy': False,
(TaskRunner pid=2281424) 'optimizer_offload': False,
(TaskRunner pid=2281424) 'param_offload': False,
(TaskRunner pid=2281424) 'reshard_after_forward': True,
(TaskRunner pid=2281424) 'seed': 42,
(TaskRunner pid=2281424) 'strategy': 'fsdp',
(TaskRunner pid=2281424) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2281424) 'use_orig_params': False,
(TaskRunner pid=2281424) 'use_torch_compile': True,
(TaskRunner pid=2281424) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2281424) 'log_prob_max_token_len_per_gpu': 10240,
(TaskRunner pid=2281424) 'log_prob_micro_batch_size': None,
(TaskRunner pid=2281424) 'log_prob_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2281424) 'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=2281424) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2281424) 'all_ranks': False,
(TaskRunner pid=2281424) 'enable': False,
(TaskRunner pid=2281424) 'ranks': [],
(TaskRunner pid=2281424) 'save_path': 'outputs/profile',
(TaskRunner pid=2281424) 'tool': None,
(TaskRunner pid=2281424) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2281424) 'analysis': True,
(TaskRunner pid=2281424) 'contents': [],
(TaskRunner pid=2281424) 'discrete': False,
(TaskRunner pid=2281424) 'level': 'level0'},
(TaskRunner pid=2281424) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2281424) 'discrete': False},
(TaskRunner pid=2281424) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2281424) 'step_end': None,
(TaskRunner pid=2281424) 'step_start': 0},
(TaskRunner pid=2281424) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2281424) 'stack_depth': 32,
(TaskRunner pid=2281424) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2281424) 'rollout_n': 8,
(TaskRunner pid=2281424) 'router_replay': {'_target_': 'verl.workers.config.RouterReplayConfig',
(TaskRunner pid=2281424) 'mode': 'disabled',
(TaskRunner pid=2281424) 'record_file': None,
(TaskRunner pid=2281424) 'replay_file': None},
(TaskRunner pid=2281424) 'strategy': 'fsdp',
(TaskRunner pid=2281424) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2281424) 'use_torch_compile': True},
(TaskRunner pid=2281424) 'rollout': {'_target_': 'verl.workers.config.RolloutConfig',
(TaskRunner pid=2281424) 'agent': {'_target_': 'verl.workers.config.AgentLoopConfig',
(TaskRunner pid=2281424) 'agent_loop_config_path': None,
(TaskRunner pid=2281424) 'custom_async_server': {'_target_': 'verl.workers.config.CustomAsyncServerConfig',
(TaskRunner pid=2281424) 'name': None,
(TaskRunner pid=2281424) 'path': None},
(TaskRunner pid=2281424) 'default_agent_loop': 'single_turn_agent',
(TaskRunner pid=2281424) 'num_workers': 8},
(TaskRunner pid=2281424) 'calculate_log_probs': True,
(TaskRunner pid=2281424) 'cudagraph_capture_sizes': None,
(TaskRunner pid=2281424) 'data_parallel_size': 1,
(TaskRunner pid=2281424) 'disable_log_stats': True,
(TaskRunner pid=2281424) 'do_sample': True,
(TaskRunner pid=2281424) 'dtype': 'bfloat16',
(TaskRunner pid=2281424) 'enable_chunked_prefill': True,
(TaskRunner pid=2281424) 'enable_prefix_caching': True,
(TaskRunner pid=2281424) 'enable_rollout_routing_replay': False,
(TaskRunner pid=2281424) 'enforce_eager': False,
(TaskRunner pid=2281424) 'engine_kwargs': {'sglang': {}, 'vllm': {}},
(TaskRunner pid=2281424) 'expert_parallel_size': 1,
(TaskRunner pid=2281424) 'free_cache_engine': True,
(TaskRunner pid=2281424) 'gpu_memory_utilization': 0.8,
(TaskRunner pid=2281424) 'ignore_eos': False,
(TaskRunner pid=2281424) 'layered_summon': False,
(TaskRunner pid=2281424) 'load_format': 'dummy',
(TaskRunner pid=2281424) 'log_prob_max_token_len_per_gpu': 10240,
(TaskRunner pid=2281424) 'log_prob_micro_batch_size': None,
(TaskRunner pid=2281424) 'log_prob_micro_batch_size_per_gpu': 1,
(TaskRunner pid=2281424) 'log_prob_use_dynamic_bsz': False,
(TaskRunner pid=2281424) 'logprobs_mode': 'processed_logprobs',
(TaskRunner pid=2281424) 'max_model_len': 10240,
(TaskRunner pid=2281424) 'max_num_batched_tokens': 10240,
(TaskRunner pid=2281424) 'max_num_seqs': 1024,
(TaskRunner pid=2281424) 'mode': 'async',
(TaskRunner pid=2281424) 'multi_stage_wake_up': False,
(TaskRunner pid=2281424) 'multi_turn': {'_target_': 'verl.workers.config.MultiTurnConfig',
(TaskRunner pid=2281424) 'enable': False,
(TaskRunner pid=2281424) 'format': 'hermes',
(TaskRunner pid=2281424) 'interaction_config_path': None,
(TaskRunner pid=2281424) 'max_assistant_turns': None,
(TaskRunner pid=2281424) 'max_parallel_calls': 1,
(TaskRunner pid=2281424) 'max_tool_response_length': 256,
(TaskRunner pid=2281424) 'max_user_turns': None,
(TaskRunner pid=2281424) 'num_repeat_rollouts': None,
(TaskRunner pid=2281424) 'tokenization_sanity_check_mode': 'strict',
(TaskRunner pid=2281424) 'tool_config_path': None,
(TaskRunner pid=2281424) 'tool_response_truncate_side': 'middle',
(TaskRunner pid=2281424) 'use_inference_chat_template': False},
(TaskRunner pid=2281424) 'n': 8,
(TaskRunner pid=2281424) 'name': 'vllm',
(TaskRunner pid=2281424) 'over_sample_rate': 0,
(TaskRunner pid=2281424) 'pipeline_model_parallel_size': 1,
(TaskRunner pid=2281424) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2281424) 'all_ranks': False,
(TaskRunner pid=2281424) 'enable': False,
(TaskRunner pid=2281424) 'ranks': [],
(TaskRunner pid=2281424) 'save_path': 'outputs/profile',
(TaskRunner pid=2281424) 'tool': None,
(TaskRunner pid=2281424) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2281424) 'analysis': True,
(TaskRunner pid=2281424) 'contents': [],
(TaskRunner pid=2281424) 'discrete': False,
(TaskRunner pid=2281424) 'level': 'level0'},
(TaskRunner pid=2281424) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2281424) 'discrete': False},
(TaskRunner pid=2281424) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2281424) 'step_end': None,
(TaskRunner pid=2281424) 'step_start': 0},
(TaskRunner pid=2281424) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2281424) 'stack_depth': 32,
(TaskRunner pid=2281424) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2281424) 'prometheus': {'_target_': 'verl.workers.config.PrometheusConfig',
(TaskRunner pid=2281424) 'enable': False,
(TaskRunner pid=2281424) 'file': '/tmp/ray/session_latest/metrics/prometheus/prometheus.yml',
(TaskRunner pid=2281424) 'port': 9090,
(TaskRunner pid=2281424) 'served_model_name': 'Qwen/Qwen3-8B'},
(TaskRunner pid=2281424) 'prompt_length': 2048,
(TaskRunner pid=2281424) 'quantization': None,
(TaskRunner pid=2281424) 'quantization_config_file': None,
(TaskRunner pid=2281424) 'response_length': 8192,
(TaskRunner pid=2281424) 'scheduling_policy': 'fcfs',
(TaskRunner pid=2281424) 'skip_dump_dir': '/tmp/rollout_dump',
(TaskRunner pid=2281424) 'skip_rollout': False,
(TaskRunner pid=2281424) 'skip_tokenizer_init': True,
(TaskRunner pid=2281424) 'temperature': 1.0,
(TaskRunner pid=2281424) 'tensor_model_parallel_size': 2,
(TaskRunner pid=2281424) 'top_k': -1,
(TaskRunner pid=2281424) 'top_p': 1.0,
(TaskRunner pid=2281424) 'trace': {'_target_': 'verl.workers.config.TraceConfig',
(TaskRunner pid=2281424) 'backend': None,
(TaskRunner pid=2281424) 'max_samples_per_step_per_worker': None,
(TaskRunner pid=2281424) 'token2text': False},
(TaskRunner pid=2281424) 'update_weights_bucket_megabytes': 512,
(TaskRunner pid=2281424) 'val_kwargs': {'_target_': 'verl.workers.config.SamplingConfig',
(TaskRunner pid=2281424) 'do_sample': True,
(TaskRunner pid=2281424) 'n': 16,
(TaskRunner pid=2281424) 'temperature': 0.6,
(TaskRunner pid=2281424) 'top_k': -1,
(TaskRunner pid=2281424) 'top_p': 0.95}}},
(TaskRunner pid=2281424) 'algorithm': {'_target_': 'verl.trainer.config.AlgoConfig',
(TaskRunner pid=2281424) 'adv_estimator': 'grpo',
(TaskRunner pid=2281424) 'gamma': 1.0,
(TaskRunner pid=2281424) 'kl_ctrl': {'_target_': 'verl.trainer.config.KLControlConfig',
(TaskRunner pid=2281424) 'horizon': 10000,
(TaskRunner pid=2281424) 'kl_coef': 0.001,
(TaskRunner pid=2281424) 'target_kl': 0.1,
(TaskRunner pid=2281424) 'type': 'fixed'},
(TaskRunner pid=2281424) 'kl_penalty': 'kl',
(TaskRunner pid=2281424) 'lam': 1.0,
(TaskRunner pid=2281424) 'norm_adv_by_std_in_grpo': False,
(TaskRunner pid=2281424) 'pf_ppo': {'reweight_method': 'pow', 'weight_pow': 2.0},
(TaskRunner pid=2281424) 'rollout_correction': {'bypass_mode': False,
(TaskRunner pid=2281424) 'loss_type': 'ppo_clip',
(TaskRunner pid=2281424) 'rollout_is': 'token',
(TaskRunner pid=2281424) 'rollout_is_batch_normalize': False,
(TaskRunner pid=2281424) 'rollout_is_threshold': 2.0,
(TaskRunner pid=2281424) 'rollout_rs': None,
(TaskRunner pid=2281424) 'rollout_rs_threshold': None},
(TaskRunner pid=2281424) 'use_kl_in_reward': False,
(TaskRunner pid=2281424) 'use_pf_ppo': False},
(TaskRunner pid=2281424) 'critic': {'_target_': 'verl.workers.config.FSDPCriticConfig',
(TaskRunner pid=2281424) 'checkpoint': {'_target_': 'verl.trainer.config.CheckpointConfig',
(TaskRunner pid=2281424) 'async_save': False,
(TaskRunner pid=2281424) 'load_contents': ['model', 'optimizer', 'extra'],
(TaskRunner pid=2281424) 'save_contents': ['model', 'optimizer', 'extra']},
(TaskRunner pid=2281424) 'cliprange_value': 0.5,
(TaskRunner pid=2281424) 'data_loader_seed': 42,
(TaskRunner pid=2281424) 'enable': None,
(TaskRunner pid=2281424) 'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=2281424) 'forward_micro_batch_size': None,
(TaskRunner pid=2281424) 'forward_micro_batch_size_per_gpu': None,
(TaskRunner pid=2281424) 'grad_clip': 1.0,
(TaskRunner pid=2281424) 'loss_agg_mode': 'token-mean',
(TaskRunner pid=2281424) 'model': {'_target_': 'verl.workers.config.FSDPCriticModelCfg',
(TaskRunner pid=2281424) 'enable_activation_offload': False,
(TaskRunner pid=2281424) 'enable_gradient_checkpointing': True,
(TaskRunner pid=2281424) 'external_lib': None,
(TaskRunner pid=2281424) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2281424) 'dtype': 'bfloat16',
(TaskRunner pid=2281424) 'entropy_checkpointing': False,
(TaskRunner pid=2281424) 'entropy_from_logits_with_chunking': False,
(TaskRunner pid=2281424) 'forward_only': False,
(TaskRunner pid=2281424) 'forward_prefetch': False,
(TaskRunner pid=2281424) 'fsdp_size': -1,
(TaskRunner pid=2281424) 'full_determinism': False,
(TaskRunner pid=2281424) 'model_dtype': 'fp32',
(TaskRunner pid=2281424) 'offload_policy': False,
(TaskRunner pid=2281424) 'optimizer_offload': False,
(TaskRunner pid=2281424) 'param_offload': False,
(TaskRunner pid=2281424) 'reshard_after_forward': True,
(TaskRunner pid=2281424) 'seed': 42,
(TaskRunner pid=2281424) 'strategy': 'fsdp',
(TaskRunner pid=2281424) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2281424) 'use_orig_params': False,
(TaskRunner pid=2281424) 'use_torch_compile': True,
(TaskRunner pid=2281424) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2281424) 'lora_alpha': 16,
(TaskRunner pid=2281424) 'lora_rank': 0,
(TaskRunner pid=2281424) 'override_config': {},
(TaskRunner pid=2281424) 'path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2281424) 'target_modules': 'all-linear',
(TaskRunner pid=2281424) 'tiled_mlp': {'enabled': False, 'num_shards': 4},
(TaskRunner pid=2281424) 'tokenizer_path': 'Qwen/Qwen3-8B',
(TaskRunner pid=2281424) 'trust_remote_code': True,
(TaskRunner pid=2281424) 'use_remove_padding': False,
(TaskRunner pid=2281424) 'use_shm': False},
(TaskRunner pid=2281424) 'optim': {'_target_': 'verl.workers.config.FSDPOptimizerConfig',
(TaskRunner pid=2281424) 'betas': [0.9, 0.999],
(TaskRunner pid=2281424) 'clip_grad': 1.0,
(TaskRunner pid=2281424) 'lr': 1e-05,
(TaskRunner pid=2281424) 'lr_scheduler_type': 'constant',
(TaskRunner pid=2281424) 'lr_warmup_steps': -1,
(TaskRunner pid=2281424) 'lr_warmup_steps_ratio': 0.0,
(TaskRunner pid=2281424) 'min_lr_ratio': 0.0,
(TaskRunner pid=2281424) 'num_cycles': 0.5,
(TaskRunner pid=2281424) 'optimizer': 'AdamW',
(TaskRunner pid=2281424) 'optimizer_impl': 'torch.optim',
(TaskRunner pid=2281424) 'override_optimizer_config': None,
(TaskRunner pid=2281424) 'total_training_steps': -1,
(TaskRunner pid=2281424) 'warmup_style': None,
(TaskRunner pid=2281424) 'weight_decay': 0.01},
(TaskRunner pid=2281424) 'ppo_epochs': 1,
(TaskRunner pid=2281424) 'ppo_max_token_len_per_gpu': 32768,
(TaskRunner pid=2281424) 'ppo_micro_batch_size': None,
(TaskRunner pid=2281424) 'ppo_micro_batch_size_per_gpu': None,
(TaskRunner pid=2281424) 'ppo_mini_batch_size': 8,
(TaskRunner pid=2281424) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2281424) 'all_ranks': False,
(TaskRunner pid=2281424) 'enable': False,
(TaskRunner pid=2281424) 'ranks': [],
(TaskRunner pid=2281424) 'save_path': 'outputs/profile',
(TaskRunner pid=2281424) 'tool': None,
(TaskRunner pid=2281424) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2281424) 'analysis': True,
(TaskRunner pid=2281424) 'contents': [],
(TaskRunner pid=2281424) 'discrete': False,
(TaskRunner pid=2281424) 'level': 'level0'},
(TaskRunner pid=2281424) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2281424) 'discrete': False},
(TaskRunner pid=2281424) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2281424) 'step_end': None,
(TaskRunner pid=2281424) 'step_start': 0},
(TaskRunner pid=2281424) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2281424) 'stack_depth': 32,
(TaskRunner pid=2281424) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2281424) 'rollout_n': 8,
(TaskRunner pid=2281424) 'shuffle': False,
(TaskRunner pid=2281424) 'strategy': 'fsdp',
(TaskRunner pid=2281424) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2281424) 'use_dynamic_bsz': False},
(TaskRunner pid=2281424) 'custom_reward_function': {'name': 'compute_score',
(TaskRunner pid=2281424) 'path': '/mnt/mole/SDPO/L2T/verl/utils/reward_score/feedback/__init__.py'},
(TaskRunner pid=2281424) 'data': {'apply_chat_template_kwargs': {'enable_thinking': False},
(TaskRunner pid=2281424) 'custom_cls': {'name': None, 'path': None},
(TaskRunner pid=2281424) 'datagen': {'name': None, 'path': None},
(TaskRunner pid=2281424) 'dataloader_num_workers': 8,
(TaskRunner pid=2281424) 'filter_overlong_prompts': True,
(TaskRunner pid=2281424) 'filter_overlong_prompts_workers': 1,
(TaskRunner pid=2281424) 'image_key': 'images',
(TaskRunner pid=2281424) 'image_patch_size': 14,
(TaskRunner pid=2281424) 'max_prompt_length': 2048,
(TaskRunner pid=2281424) 'max_response_length': 8192,
(TaskRunner pid=2281424) 'prompt_key': 'prompt',
(TaskRunner pid=2281424) 'return_full_prompt': False,
(TaskRunner pid=2281424) 'return_multi_modal_inputs': True,
(TaskRunner pid=2281424) 'return_raw_chat': True,
(TaskRunner pid=2281424) 'return_raw_input_ids': False,
(TaskRunner pid=2281424) 'reward_fn_key': 'data_source',
(TaskRunner pid=2281424) 'sampler': {'class_name': None, 'class_path': None},
(TaskRunner pid=2281424) 'seed': None,
(TaskRunner pid=2281424) 'shuffle': True,
(TaskRunner pid=2281424) 'tokenizer': None,
(TaskRunner pid=2281424) 'tool_config_path': None,
(TaskRunner pid=2281424) 'train_batch_size': 32,
(TaskRunner pid=2281424) 'train_files': ['/mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/train.parquet'],
(TaskRunner pid=2281424) 'train_max_samples': 3200,
(TaskRunner pid=2281424) 'truncation': 'error',
(TaskRunner pid=2281424) 'trust_remote_code': True,
(TaskRunner pid=2281424) 'use_shm': False,
(TaskRunner pid=2281424) 'val_batch_size': None,
(TaskRunner pid=2281424) 'val_files': ['/mnt/mole/SDPO/L2T/datasets/sciknoweval/physics/test.parquet'],
(TaskRunner pid=2281424) 'val_max_samples': -1,
(TaskRunner pid=2281424) 'validation_shuffle': False,
(TaskRunner pid=2281424) 'video_key': 'videos'},
(TaskRunner pid=2281424) 'global_profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2281424) 'global_tool_config': {'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2281424) 'controller_nsight_options': {'cuda-graph-trace': 'graph',
(TaskRunner pid=2281424) 'cuda-memory-usage': 'true',
(TaskRunner pid=2281424) 'trace': 'cuda,nvtx,cublas,ucx'},
(TaskRunner pid=2281424) 'discrete': False,
(TaskRunner pid=2281424) 'worker_nsight_options': {'capture-range': 'cudaProfilerApi',
(TaskRunner pid=2281424) 'capture-range-end': None,
(TaskRunner pid=2281424) 'cuda-graph-trace': 'graph',
(TaskRunner pid=2281424) 'cuda-memory-usage': 'true',
(TaskRunner pid=2281424) 'kill': 'none',
(TaskRunner pid=2281424) 'trace': 'cuda,nvtx,cublas,ucx'}},
(TaskRunner pid=2281424) 'torch_memory': {'context': 'all',
(TaskRunner pid=2281424) 'kw_args': {},
(TaskRunner pid=2281424) 'stack_depth': 32,
(TaskRunner pid=2281424) 'stacks': 'all',
(TaskRunner pid=2281424) 'trace_alloc_max_entries': 100000}},
(TaskRunner pid=2281424) 'profile_continuous_steps': False,
(TaskRunner pid=2281424) 'save_path': 'outputs/profile',
(TaskRunner pid=2281424) 'steps': None,
(TaskRunner pid=2281424) 'tool': None},
(TaskRunner pid=2281424) 'max_model_len': 10240,
(TaskRunner pid=2281424) 'ray_kwargs': {'ray_init': {'_temp_dir': '/tmp/ray_q3g_physics_grpo_Qwen3_8B',
(TaskRunner pid=2281424) 'include_dashboard': False,
(TaskRunner pid=2281424) 'num_cpus': None,
(TaskRunner pid=2281424) 'runtime_env': {'env_vars': {'EXPERIMENT': 'qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2281424) 'TASK': 'datasets/sciknoweval/physics',
(TaskRunner pid=2281424) 'USER': 'root'}}},
(TaskRunner pid=2281424) 'timeline_json_file': None},
(TaskRunner pid=2281424) 'reward_manager': {'_target_': 'verl.trainer.config.config.RewardManagerConfig',
(TaskRunner pid=2281424) 'module': {'_target_': 'verl.trainer.config.config.ModuleConfig',
(TaskRunner pid=2281424) 'name': 'custom_reward_manager',
(TaskRunner pid=2281424) 'path': None},
(TaskRunner pid=2281424) 'name': 'naive',
(TaskRunner pid=2281424) 'source': 'register'},
(TaskRunner pid=2281424) 'reward_model': {'enable': False,
(TaskRunner pid=2281424) 'enable_resource_pool': False,
(TaskRunner pid=2281424) 'forward_max_token_len_per_gpu': 32768,
(TaskRunner pid=2281424) 'launch_reward_fn_async': False,
(TaskRunner pid=2281424) 'max_length': None,
(TaskRunner pid=2281424) 'micro_batch_size': None,
(TaskRunner pid=2281424) 'micro_batch_size_per_gpu': None,
(TaskRunner pid=2281424) 'model': {'external_lib': None,
(TaskRunner pid=2281424) 'fsdp_config': {'_target_': 'verl.workers.config.FSDPEngineConfig',
(TaskRunner pid=2281424) 'forward_prefetch': False,
(TaskRunner pid=2281424) 'fsdp_size': -1,
(TaskRunner pid=2281424) 'param_offload': False,
(TaskRunner pid=2281424) 'reshard_after_forward': True,
(TaskRunner pid=2281424) 'wrap_policy': {'min_num_params': 0}},
(TaskRunner pid=2281424) 'input_tokenizer': 'Qwen/Qwen3-8B',
(TaskRunner pid=2281424) 'override_config': {},
(TaskRunner pid=2281424) 'path': '~/models/FsfairX-LLaMA3-RM-v0.1',
(TaskRunner pid=2281424) 'trust_remote_code': False,
(TaskRunner pid=2281424) 'use_fused_kernels': False,
(TaskRunner pid=2281424) 'use_remove_padding': False,
(TaskRunner pid=2281424) 'use_shm': False},
(TaskRunner pid=2281424) 'n_gpus_per_node': 8,
(TaskRunner pid=2281424) 'nnodes': 0,
(TaskRunner pid=2281424) 'num_workers': 1,
(TaskRunner pid=2281424) 'profiler': {'_target_': 'verl.utils.profiler.ProfilerConfig',
(TaskRunner pid=2281424) 'all_ranks': False,
(TaskRunner pid=2281424) 'enable': False,
(TaskRunner pid=2281424) 'ranks': [],
(TaskRunner pid=2281424) 'save_path': 'outputs/profile',
(TaskRunner pid=2281424) 'tool': None,
(TaskRunner pid=2281424) 'tool_config': {'npu': {'_target_': 'verl.utils.profiler.config.NPUToolConfig',
(TaskRunner pid=2281424) 'analysis': True,
(TaskRunner pid=2281424) 'contents': [],
(TaskRunner pid=2281424) 'discrete': False,
(TaskRunner pid=2281424) 'level': 'level0'},
(TaskRunner pid=2281424) 'nsys': {'_target_': 'verl.utils.profiler.config.NsightToolConfig',
(TaskRunner pid=2281424) 'discrete': False},
(TaskRunner pid=2281424) 'torch': {'_target_': 'verl.utils.profiler.config.TorchProfilerToolConfig',
(TaskRunner pid=2281424) 'step_end': None,
(TaskRunner pid=2281424) 'step_start': 0},
(TaskRunner pid=2281424) 'torch_memory': {'_target_': 'verl.utils.profiler.config.TorchMemoryToolConfig',
(TaskRunner pid=2281424) 'stack_depth': 32,
(TaskRunner pid=2281424) 'trace_alloc_max_entries': 100000}}},
(TaskRunner pid=2281424) 'reward_loop_class_name': None,
(TaskRunner pid=2281424) 'reward_loop_module_path': None,
(TaskRunner pid=2281424) 'reward_loop_source': 'register',
(TaskRunner pid=2281424) 'reward_manager': 'naive',
(TaskRunner pid=2281424) 'rollout': {'_target_': 'verl.workers.config.RolloutConfig',
(TaskRunner pid=2281424) 'cudagraph_capture_sizes': None,
(TaskRunner pid=2281424) 'data_parallel_size': 1,
(TaskRunner pid=2281424) 'disable_log_stats': True,
(TaskRunner pid=2281424) 'dtype': 'bfloat16',
(TaskRunner pid=2281424) 'enable_chunked_prefill': True,
(TaskRunner pid=2281424) 'enable_prefix_caching': True,
(TaskRunner pid=2281424) 'enforce_eager': True,
(TaskRunner pid=2281424) 'engine_kwargs': {},
(TaskRunner pid=2281424) 'expert_parallel_size': 1,
(TaskRunner pid=2281424) 'free_cache_engine': True,
(TaskRunner pid=2281424) 'gpu_memory_utilization': 0.5,
(TaskRunner pid=2281424) 'limit_images': None,
(TaskRunner pid=2281424) 'load_format': 'auto',
(TaskRunner pid=2281424) 'max_model_len': None,
(TaskRunner pid=2281424) 'max_num_batched_tokens': 8192,
(TaskRunner pid=2281424) 'max_num_seqs': 1024,
(TaskRunner pid=2281424) 'name': '???',
(TaskRunner pid=2281424) 'prompt_length': 2048,
(TaskRunner pid=2281424) 'response_length': 2048,
(TaskRunner pid=2281424) 'skip_tokenizer_init': False,
(TaskRunner pid=2281424) 'tensor_model_parallel_size': 2},
(TaskRunner pid=2281424) 'sandbox_fusion': {'max_concurrent': 64,
(TaskRunner pid=2281424) 'memory_limit_mb': 1024,
(TaskRunner pid=2281424) 'url': None},
(TaskRunner pid=2281424) 'strategy': 'fsdp',
(TaskRunner pid=2281424) 'ulysses_sequence_parallel_size': 1,
(TaskRunner pid=2281424) 'use_dynamic_bsz': False,
(TaskRunner pid=2281424) 'use_reward_loop': False},
(TaskRunner pid=2281424) 'trainer': {'balance_batch': True,
(TaskRunner pid=2281424) 'critic_warmup': 0,
(TaskRunner pid=2281424) 'default_hdfs_dir': None,
(TaskRunner pid=2281424) 'default_local_dir': '/mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2281424) 'del_local_ckpt_after_load': False,
(TaskRunner pid=2281424) 'device': 'cuda',
(TaskRunner pid=2281424) 'esi_redundant_time': 0,
(TaskRunner pid=2281424) 'experiment_name': 'qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8',
(TaskRunner pid=2281424) 'group_name': 'QWEN3-GRPO-generalization',
(TaskRunner pid=2281424) 'log_val_generations': 0,
(TaskRunner pid=2281424) 'logger': ['console', 'wandb'],
(TaskRunner pid=2281424) 'max_actor_ckpt_to_keep': 100,
(TaskRunner pid=2281424) 'max_critic_ckpt_to_keep': None,
(TaskRunner pid=2281424) 'n_gpus_per_node': 8,
(TaskRunner pid=2281424) 'nnodes': 1,
(TaskRunner pid=2281424) 'project_name': 'SDPO-root',
(TaskRunner pid=2281424) 'ray_wait_register_center_timeout': 300,
(TaskRunner pid=2281424) 'resume_from_path': None,
(TaskRunner pid=2281424) 'resume_mode': 'auto',
(TaskRunner pid=2281424) 'rollout_data_dir': None,
(TaskRunner pid=2281424) 'save_freq': 10,
(TaskRunner pid=2281424) 'test_freq': 10,
(TaskRunner pid=2281424) 'total_epochs': 30,
(TaskRunner pid=2281424) 'total_training_steps': 100,
(TaskRunner pid=2281424) 'use_legacy_worker_impl': 'auto',
(TaskRunner pid=2281424) 'val_before_train': False,
(TaskRunner pid=2281424) 'val_only': False,
(TaskRunner pid=2281424) 'validation_data_dir': None},
(TaskRunner pid=2281424) 'transfer_queue': {'enable': False},
(TaskRunner pid=2281424) 'vars': {'ckpt_dir': '/capstor/scratch/cscs/root/ttrl_runs/datasets/sciknoweval/physics',
(TaskRunner pid=2281424) 'dir': '/users/root/SDPO',
(TaskRunner pid=2281424) 'log_dir': '/users/root/output',
(TaskRunner pid=2281424) 'task': 'datasets/sciknoweval/physics'}}
(TaskRunner pid=2281424) /mnt/mole/SDPO/L2T/verl/trainer/main_ppo.py:326: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=2281424) use_critic=need_critic(config),
(TaskRunner pid=2281424) [validate_config] All configuration checks passed successfully!
(TaskRunner pid=2281424) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=2281424) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(TaskRunner pid=2281424) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(TaskRunner pid=2281424) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(TaskRunner pid=2281424) Using dataset class: RLHFDataset
(TaskRunner pid=2281424) dataset len: 720
(TaskRunner pid=2281424) Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2281424) WARNING:2026-07-03 03:19:03,686:Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2281424) Filtering prompts longer than 2048 tokens (num_proc=1): 0%| | 0/720 [00:00<?, ? examples/s]
(TaskRunner pid=2281424) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 720/720 [00:00<00:00, 854.26 examples/s]
(TaskRunner pid=2281424) filter dataset len: 720
(TaskRunner pid=2281424) Using dataset class: RLHFDataset
(TaskRunner pid=2281424) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 720/720 [00:00<00:00, 764.92 examples/s]
(TaskRunner pid=2281424) dataset len: 80
(TaskRunner pid=2281424) Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2281424) WARNING:2026-07-03 03:19:05,039:Setting TOKENIZERS_PARALLELISM=false for forked processes.
(TaskRunner pid=2281424) Filtering prompts longer than 2048 tokens (num_proc=1): 0%| | 0/80 [00:00<?, ? examples/s]
(TaskRunner pid=2281424) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 80/80 [00:00<00:00, 145.81 examples/s]
(TaskRunner pid=2281424) filter dataset len: 80
(TaskRunner pid=2281424) Size of train dataloader: 22, Size of val dataloader: 1
(TaskRunner pid=2281424) Total training steps: 100
(TaskRunner pid=2281424) colocated worker base class <class 'verl.single_controller.base.worker.Worker'>
(TaskRunner pid=2281424) bind role actor_rollout method chat_completion to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2281424) bind role actor_rollout method generate to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2281424) bind role actor_rollout method get_zeromq_address to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2281424) bind role actor_rollout method sleep to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2281424) bind role actor_rollout method wake_up to class <class 'verl.single_controller.ray.base.create_colocated_worker_cls.<locals>.WorkerDict'>
(TaskRunner pid=2281424) Filtering prompts longer than 2048 tokens (num_proc=1): 100%|██████████| 80/80 [00:00<00:00, 122.84 examples/s]
(TaskRunner pid=2281424) /mnt/mole/SDPO/L2T/verl/trainer/ppo/ray_trainer.py:354: UserWarning: Disabled critic as algorithm.adv_estimator != gae. If it is not intended, please set critic.enable=True
(TaskRunner pid=2281424) self.use_critic = need_critic(self.config)
(WorkerDict pid=2281814) [W703 03:19:12.382603555 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [::ffff:198.19.44.212]:53237 (errno: 97 - Address family not supported by protocol).
(WorkerDict pid=2281810) [Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
(WorkerDict pid=2281810) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(WorkerDict pid=2281810) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(WorkerDict pid=2281817) [W703 03:19:13.439209755 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [::ffff:198.19.44.212]:53237 (errno: 97 - Address family not supported by protocol). [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(WorkerDict pid=2281810) Model config after override: Qwen3Config {
(WorkerDict pid=2281810) "architectures": [
(WorkerDict pid=2281810) "Qwen3ForCausalLM"
(WorkerDict pid=2281810) ],
(WorkerDict pid=2281810) "attention_bias": false,
(WorkerDict pid=2281810) "attention_dropout": 0.0,
(WorkerDict pid=2281810) "dtype": "bfloat16",
(WorkerDict pid=2281810) "eos_token_id": 151645,
(WorkerDict pid=2281810) "head_dim": 128,
(WorkerDict pid=2281810) "hidden_act": "silu",
(WorkerDict pid=2281810) "hidden_size": 4096,
(WorkerDict pid=2281810) "initializer_range": 0.02,
(WorkerDict pid=2281810) "intermediate_size": 12288,
(WorkerDict pid=2281810) "layer_types": [
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention",
(WorkerDict pid=2281810) "full_attention"
(WorkerDict pid=2281810) ],
(WorkerDict pid=2281810) "max_position_embeddings": 40960,
(WorkerDict pid=2281810) "max_window_layers": 36,
(WorkerDict pid=2281810) "model_type": "qwen3",
(WorkerDict pid=2281810) "num_attention_heads": 32,
(WorkerDict pid=2281810) "num_hidden_layers": 36,
(WorkerDict pid=2281810) "num_key_value_heads": 8,
(WorkerDict pid=2281810) "pad_token_id": 151643,
(WorkerDict pid=2281810) "rms_norm_eps": 1e-06,
(WorkerDict pid=2281810) "rope_scaling": null,
(WorkerDict pid=2281810) "rope_theta": 1000000,
(WorkerDict pid=2281810) "sliding_window": null,
(WorkerDict pid=2281810) "tie_word_embeddings": false,
(WorkerDict pid=2281810) "transformers_version": "4.57.1",
(WorkerDict pid=2281810) "use_cache": true,
(WorkerDict pid=2281810) "use_sliding_window": false,
(WorkerDict pid=2281810) "vocab_size": 151936
(WorkerDict pid=2281810) }
(WorkerDict pid=2281810)
(WorkerDict pid=2281810) `torch_dtype` is deprecated! Use `dtype` instead!
(WorkerDict pid=2281810) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)`
(WorkerDict pid=2281812) Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
(WorkerDict pid=2281810) Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.98s/it]
(WorkerDict pid=2281817) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(WorkerDict pid=2281817) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2281817) `torch_dtype` is deprecated! Use `dtype` instead! [repeated 7x across cluster]
(WorkerDict pid=2281817) Flash Attention 2 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", dtype=torch.float16)` [repeated 15x across cluster]
(WorkerDict pid=2281817) Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s] [repeated 7x across cluster]
(WorkerDict pid=2281810) Loading checkpoint shards: 60%|██████ | 3/5 [00:08<00:05, 2.96s/it] [repeated 16x across cluster]
(WorkerDict pid=2281810) Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.07s/it] Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.43s/it]
(WorkerDict pid=2281810) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention
(WorkerDict pid=2281810) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch
(WorkerDict pid=2281817) [Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7 [repeated 7x across cluster]
(WorkerDict pid=2281810) Qwen3ForCausalLM contains 8.19B parameters
(WorkerDict pid=2281810) wrap_policy: functools.partial(<function _or_policy at 0x7b9735e84680>, policies=[functools.partial(<function transformer_auto_wrap_policy at 0x7b9735e84540>, transformer_layer_cls={<class 'transformers.models.qwen3.modeling_qwen3.Qwen3DecoderLayer'>})])
(WorkerDict pid=2281810) NCCL version 2.27.5+cuda12.9
(WorkerDict pid=2281811) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(WorkerDict pid=2281811) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(WorkerDict pid=2281817) Loading checkpoint shards: 80%|████████ | 4/5 [00:11<00:02, 2.89s/it] [repeated 15x across cluster]
(WorkerDict pid=2281817) Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.20s/it] Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.58s/it] [repeated 7x across cluster]
(WorkerDict pid=2281810) Total steps: 100, num_warmup_steps: 10
(WorkerDict pid=2281810) Actor use_remove_padding=True
(WorkerDict pid=2281810) Actor use_fused_kernels=False
(WorkerDict pid=2281810) Actor use_prefix_grouper=False
(WorkerDict pid=2281817) Monkey patch _flash_attention_forward in transformers.integrations.flash_attention [repeated 7x across cluster]
(WorkerDict pid=2281817) Skipping monkey patch for Qwen3ForCausalLM as use_fused_kernels is False or fused_kernels_backend is torch [repeated 7x across cluster]
(WorkerDict pid=2281814) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(WorkerDict pid=2281814) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(WorkerDict pid=2281811) [Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3
(WorkerDict pid=2281810) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now. [repeated 7x across cluster]
(WorkerDict pid=2281810) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1) [repeated 7x across cluster]
(WorkerDict pid=2281816) /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
(WorkerDict pid=2281816) warnings.warn(
(WorkerDict pid=2281810) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(WorkerDict pid=2281810) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(TaskRunner pid=2281424) /mnt/mole/SDPO/L2T/verl/utils/profiler/config.py:52: UserWarning: Torch profiler tool config is not fully supported now.
(TaskRunner pid=2281424) warnings.warn("Torch profiler tool config is not fully supported now.", stacklevel=1)
(WorkerDict pid=2281817) /mnt/mole/SDPO/L2T/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . [repeated 7x across cluster]
(WorkerDict pid=2281817) warnings.warn( [repeated 7x across cluster]
(vLLMHttpServer pid=2282697) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(vLLMHttpServer pid=2282697) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(vLLMHttpServer pid=2282700) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(vLLMHttpServer pid=2282700) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(vLLMHttpServer pid=2282697) ['serve',
(vLLMHttpServer pid=2282697) 'Qwen/Qwen3-8B',
(vLLMHttpServer pid=2282697) '--dtype',
(vLLMHttpServer pid=2282697) 'bfloat16',
(vLLMHttpServer pid=2282697) '--load_format',
(vLLMHttpServer pid=2282697) 'dummy',
(vLLMHttpServer pid=2282697) '--trust_remote_code',
(vLLMHttpServer pid=2282697) '--max_model_len',
(vLLMHttpServer pid=2282697) '40960',
(vLLMHttpServer pid=2282697) '--max_num_seqs',
(vLLMHttpServer pid=2282697) '1024',
(vLLMHttpServer pid=2282697) '--enable_chunked_prefill',
(vLLMHttpServer pid=2282697) '--max_num_batched_tokens',
(vLLMHttpServer pid=2282697) '10240',
(vLLMHttpServer pid=2282697) '--enable_prefix_caching',
(vLLMHttpServer pid=2282697) '--enable_sleep_mode',
(vLLMHttpServer pid=2282697) '--logprobs_mode',
(vLLMHttpServer pid=2282697) 'processed_logprobs',
(vLLMHttpServer pid=2282697) '--disable_custom_all_reduce',
(vLLMHttpServer pid=2282697) '--gpu_memory_utilization',
(vLLMHttpServer pid=2282697) '0.8',
(vLLMHttpServer pid=2282697) '--disable_log_stats',
(vLLMHttpServer pid=2282697) '--tensor_parallel_size',
(vLLMHttpServer pid=2282697) '2',
(vLLMHttpServer pid=2282697) '--seed',
(vLLMHttpServer pid=2282697) '0',
(vLLMHttpServer pid=2282697) '--override_generation_config',
(vLLMHttpServer pid=2282697) '{"temperature": 1.0, "top_k": -1, "top_p": 1.0, "repetition_penalty": 1.0, '
(vLLMHttpServer pid=2282697) '"max_new_tokens": 8192}',
(vLLMHttpServer pid=2282697) '--hf_overrides',
(vLLMHttpServer pid=2282697) '{}',
(vLLMHttpServer pid=2282697) '--scheduling_policy',
(vLLMHttpServer pid=2282697) 'fcfs']
(WorkerDict pid=2281817) [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [repeated 23x across cluster]
(vLLMHttpServer pid=2282697) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead.
(vLLMHttpServer pid=2282697) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(vLLMHttpServer pid=2282697) WARNING 07-03 03:20:04 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned
(WorkerDict pid=2281811) WARNING 07-03 03:20:12 [worker_base.py:301] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'.
(vLLMHttpServer pid=2282699) WARNING 07-03 03:20:05 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: In a Ray actor and can only be spawned [repeated 3x across cluster]
(WorkerDict pid=2281812) NCCL version 2.27.5+cuda12.9
(WorkerDict pid=2281810) 2026-07-03 03:20:26,670 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(vLLMHttpServer pid=2282699) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 2x across cluster]
(vLLMHttpServer pid=2282699) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 2x across cluster]
(vLLMHttpServer pid=2282699) Using blocking ray.get inside async actor. This blocks the event loop. Please use `await` on object ref with asyncio.gather if you want to yield execution to the event loop instead. [repeated 3x across cluster]
(vLLMHttpServer pid=2282699) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [repeated 3x across cluster]
(WorkerDict pid=2281814) 2026-07-03 03:20:26,669 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00<?, ?it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 4%|▍ | 2/51 [00:00<00:02, 17.29it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 8%|▊ | 4/51 [00:00<00:02, 18.21it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 12%|█▏ | 6/51 [00:00<00:02, 18.03it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 16%|█▌ | 8/51 [00:00<00:02, 17.87it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|█▉ | 10/51 [00:00<00:02, 17.90it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 24%|██▎ | 12/51 [00:00<00:02, 17.93it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 14/51 [00:00<00:02, 17.97it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 31%|███▏ | 16/51 [00:00<00:02, 17.24it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 35%|███▌ | 18/51 [00:01<00:01, 17.10it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 39%|███▉ | 20/51 [00:01<00:01, 16.58it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 43%|████▎ | 22/51 [00:01<00:01, 16.64it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 47%|████▋ | 24/51 [00:01<00:01, 16.63it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 51%|█████ | 26/51 [00:01<00:01, 16.46it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 28/51 [00:01<00:01, 15.96it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 59%|█████▉ | 30/51 [00:01<00:01, 15.69it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 63%|██████▎ | 32/51 [00:01<00:01, 15.83it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 67%|██████▋ | 34/51 [00:02<00:01, 15.41it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████ | 36/51 [00:02<00:01, 14.99it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 75%|███████▍ | 38/51 [00:02<00:00, 14.23it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 78%|███████▊ | 40/51 [00:02<00:00, 14.08it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 42/51 [00:02<00:00, 14.44it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▋ | 44/51 [00:02<00:00, 12.33it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 90%|█████████ | 46/51 [00:02<00:00, 12.82it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 94%|█████████▍| 48/51 [00:03<00:00, 13.05it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 98%|█████████▊| 50/51 [00:03<00:00, 13.79it/s] Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 15.34it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 0%| | 0/51 [00:00<?, ?it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 6%|▌ | 3/51 [00:00<00:02, 21.91it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 12%|█▏ | 6/51 [00:00<00:02, 19.40it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 18%|█▊ | 9/51 [00:00<00:02, 20.78it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 24%|██▎ | 12/51 [00:00<00:01, 21.78it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 29%|██▉ | 15/51 [00:00<00:01, 22.24it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 35%|███▌ | 18/51 [00:00<00:01, 22.56it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 41%|████ | 21/51 [00:00<00:01, 22.72it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 47%|████▋ | 24/51 [00:01<00:01, 22.98it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 53%|█████▎ | 27/51 [00:01<00:01, 23.16it/s]
(WorkerDict pid=2281817) 2026-07-03 03:20:26,984 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ... [repeated 7x across cluster]
(WorkerDict pid=2281817) 2026-07-03 03:20:26,999 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends [repeated 7x across cluster]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 59%|█████▉ | 30/51 [00:01<00:00, 23.30it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 65%|██████▍ | 33/51 [00:01<00:00, 22.65it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 71%|███████ | 36/51 [00:01<00:00, 22.72it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 76%|███████▋ | 39/51 [00:01<00:00, 22.62it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 82%|████████▏ | 42/51 [00:01<00:00, 22.65it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 88%|████████▊ | 45/51 [00:02<00:00, 22.56it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 94%|█████████▍| 48/51 [00:02<00:00, 22.59it/s]
(WorkerDict pid=2281810) Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:02<00:00, 22.48it/s] Capturing CUDA graphs (decode, FULL): 100%|██████████| 51/51 [00:02<00:00, 22.42it/s]
(vLLMHttpServer pid=2282699) WARNING 07-03 03:20:34 [api_server.py:1358] LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development!
(WorkerDict pid=2281815) WARNING 07-03 03:20:13 [worker_base.py:301] Missing `shared_worker_lock` argument from executor. This argument is needed for mm_processor_cache_type='shm'. [repeated 7x across cluster]
(WorkerDict pid=2281816) NCCL version 2.27.5+cuda12.9 [repeated 2x across cluster]
(vLLMHttpServer pid=2282699) WARNING 07-03 03:20:35 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(TaskRunner pid=2281424) AgentLoopManager: ['198.19.44.212:40921', '198.19.44.212:40541', '198.19.44.212:37869', '198.19.44.212:42381']
(TaskRunner pid=2281424) wandb: Currently logged in as: seongryongjung (seongryongjung-chung-ang-university) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
(TaskRunner pid=2281424) wandb: Tracking run with wandb version 0.23.1
(TaskRunner pid=2281424) wandb: Run data is saved locally in /mnt/mole/SDPO/L2T/wandb/run-20260703_032040-6ig3l55l
(TaskRunner pid=2281424) wandb: Run `wandb offline` to turn off syncing.
(TaskRunner pid=2281424) wandb: Syncing run qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8
(TaskRunner pid=2281424) wandb: ⭐️ View project at https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root
(TaskRunner pid=2281424) wandb: 🚀 View run at https://wandb.ai/seongryongjung-chung-ang-university/SDPO-root/runs/6ig3l55l
(TaskRunner pid=2281424) Checkpoint tracker file does not exist: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/latest_checkpointed_iteration.txt
(TaskRunner pid=2281424) Training from scratch
(vLLMHttpServer pid=2282700) WARNING 07-03 03:20:35 [api_server.py:1358] LoRA dynamic loading & unloading is enabled in the API server. This should ONLY be used for local development! [repeated 3x across cluster]
(vLLMHttpServer pid=2282700) WARNING 07-03 03:20:35 [model.py:1576] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. [repeated 3x across cluster]
(TaskRunner pid=2281424) wandb: Detected [openai] in use.
(TaskRunner pid=2281424) wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
(TaskRunner pid=2281424) wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
(TaskRunner pid=2281424) Training Progress: 0%| | 0/100 [00:00<?, ?it/s]
(AgentLoopWorker pid=2283645) Using dataset class: RLHFDataset
(AgentLoopWorker pid=2283645) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing
(AgentLoopWorker pid=2283645) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1)
(AgentLoopWorker pid=2283645) You're using a Qwen2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
(AgentLoopWorker pid=2283648) /mnt/mole/SDPO/L2T/verl/utils/tokenizer.py:107: UserWarning: Failed to create processor: Unsupported processor type: Qwen2TokenizerFast. This may affect multimodal processing [repeated 7x across cluster]
(AgentLoopWorker pid=2283648) warnings.warn(f"Failed to create processor: {e}. This may affect multimodal processing", stacklevel=1) [repeated 7x across cluster]
(AgentLoopWorker pid=2283647) You're using a Qwen2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding. [repeated 7x across cluster]
(TaskRunner pid=2281424) step:1 - global_seqlen/min:21599 - global_seqlen/max:39795 - global_seqlen/minmax_diff:18196 - global_seqlen/balanced_min:27997 - global_seqlen/balanced_max:28074 - global_seqlen/mean:28049.0 - actor/entropy:0.2933548390865326 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3997599184513092 - training/rollout_probs_diff_mean:0.0031216552015393972 - training/rollout_probs_diff_std:0.008815591223537922 - training/rollout_actor_probs_pearson_corr:0.9993525743484497 - rollout_corr/rollout_is_mean:0.9999884963035583 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.3050060260866303e-05 - rollout_corr/rollout_is_max:1.6403627395629883 - rollout_corr/rollout_is_min:0.4432307779788971 - rollout_corr/rollout_is_std:0.02760624885559082 - rollout_corr/rollout_is_eff_sample_size:0.9992384753736182 - rollout_corr/rollout_is_seq_mean:0.9998464584350586 - rollout_corr/rollout_is_seq_std:0.001634377520531416 - rollout_corr/rollout_is_seq_max:1.0051524639129639 - rollout_corr/rollout_is_seq_min:0.9904565215110779 - rollout_corr/rollout_is_seq_max_deviation:0.00954347848892212 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3432956486940384) - rollout_corr/training_log_ppl:np.float64(0.274512951305951) - rollout_corr/kl:np.float64(0.0005876388563024193) - rollout_corr/k3_kl:np.float64(0.0004341341748954619) - rollout_corr/rollout_ppl:np.float64(1.3425641637295485) - rollout_corr/rollout_log_ppl:np.float64(0.2739253109320998) - rollout_corr/log_ppl_diff:np.float64(0.0005876403738511726) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011765340605052188) - rollout_corr/log_ppl_diff_max:np.float64(0.011144682765007019) - rollout_corr/log_ppl_diff_min:np.float64(-0.003832995891571045) - rollout_corr/ppl_ratio:np.float64(1.0005893006455153) - rollout_corr/chi2_token:np.float64(0.0005533059593290091) - rollout_corr/chi2_seq:np.float64(0.49172084289602935) - actor/pg_loss:np.float64(7.330859079957008e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0) - actor/ppo_kl:np.float64(0.0) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1784479133784771) - perf/mfu/actor:np.float64(0.1053086686554278) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(136.935546875) - perf/cpu_memory_used_gb:np.float64(100.71479415893555) - actor/lr:np.float64(0.0) - training/global_step:1 - training/epoch:0 - critic/score/mean:0.6015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004050412215292454 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004050412215292454 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:598.65625 - response_length/max:2273.0 - response_length/min:85.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:598.65625 - response_length_non_aborted/max:2273.0 - response_length_non_aborted/min:85.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.875 - prompt_length/max:375.0 - prompt_length/min:155.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00023187324404716492 - timing_s/agent_loop/generate_sequences/min:np.float64(10.81203643977642) - timing_s/agent_loop/generate_sequences/max:np.float64(31.67797383107245) - timing_s/agent_loop/generate_sequences/mean:np.float64(20.031546598176647) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(31.67797383107245) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:342 - timing_s/agent_loop/slowest/response_length:2273 - timing_s/gen:43.80677997134626 - timing_s/reward:0.06696747615933418 - timing_s/old_log_prob:4.027026003226638 - timing_s/adv:0.17974235117435455 - timing_s/update_actor:14.05974173732102 - timing_s/step:62.25599696300924 - timing_s/stop_profile:0.0001486893743276596 - timing_per_token_ms/gen:0.28584055417958354 - timing_per_token_ms/update_actor:0.06265705433937493 - timing_per_token_ms/adv:0.0008010194266032415 - perf/total_num_tokens:224392 - perf/time_per_step:62.25599696300924 - perf/throughput:450.5429415364744
(AgentLoopWorker pid=2283648) Using dataset class: RLHFDataset [repeated 7x across cluster]
(TaskRunner pid=2281424) Training Progress: 1%| | 1/100 [01:02<1:43:04, 62.47s/it]
(TaskRunner pid=2281424) step:2 - global_seqlen/min:18488 - global_seqlen/max:35485 - global_seqlen/minmax_diff:16997 - global_seqlen/balanced_min:28661 - global_seqlen/balanced_max:28687 - global_seqlen/mean:28676.875 - actor/entropy:0.2852829098701477 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5235563516616821 - training/rollout_probs_diff_mean:0.0032305719796568155 - training/rollout_probs_diff_std:0.009228591807186604 - training/rollout_actor_probs_pearson_corr:0.9993439316749573 - rollout_corr/rollout_is_mean:0.9998864531517029 - rollout_corr/rollout_is_ratio_fraction_high:6.217088412086014e-06 - rollout_corr/rollout_is_ratio_fraction_low:6.217088412086014e-06 - rollout_corr/rollout_is_max:2.544478178024292 - rollout_corr/rollout_is_min:0.39712417125701904 - rollout_corr/rollout_is_std:0.02848818711936474 - rollout_corr/rollout_is_eff_sample_size:0.9991889622635779 - rollout_corr/rollout_is_seq_mean:0.99990314245224 - rollout_corr/rollout_is_seq_std:0.0016033288557082415 - rollout_corr/rollout_is_seq_max:1.0100219249725342 - rollout_corr/rollout_is_seq_min:0.9953435063362122 - rollout_corr/rollout_is_seq_max_deviation:0.01002192497253418 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3009517984464765) - rollout_corr/training_log_ppl:np.float64(0.25215593120083213) - rollout_corr/kl:np.float64(0.0004823018046948846) - rollout_corr/k3_kl:np.float64(0.0004502152170005047) - rollout_corr/rollout_ppl:np.float64(1.3002900667488575) - rollout_corr/rollout_log_ppl:np.float64(0.2516736279503675) - rollout_corr/log_ppl_diff:np.float64(0.00048230325046461076) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011988449696218595) - rollout_corr/log_ppl_diff_max:np.float64(0.005951404571533203) - rollout_corr/log_ppl_diff_min:np.float64(-0.008813902735710144) - rollout_corr/ppl_ratio:np.float64(1.0004837419837713) - rollout_corr/chi2_token:np.float64(0.0008451999165117741) - rollout_corr/chi2_seq:np.float64(0.24925414193421602) - actor/pg_loss:np.float64(-0.0001076720654964447) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001199317946429801) - actor/ppo_kl:np.float64(-5.8633315513034745e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.14331618882715702) - perf/mfu/actor:np.float64(0.11175082545657457) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(136.935546875) - perf/cpu_memory_used_gb:np.float64(100.56762313842773) - actor/lr:np.float64(1e-07) - training/global_step:2 - training/epoch:0 - critic/score/mean:0.5859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.002406013198196888 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.002406013198196888 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:628.30859375 - response_length/max:2049.0 - response_length/min:85.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:628.30859375 - response_length_non_aborted/max:2049.0 - response_length_non_aborted/min:85.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:267.84375 - prompt_length/max:342.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001306328922510147 - timing_s/agent_loop/generate_sequences/min:np.float64(1.0788820944726467) - timing_s/agent_loop/generate_sequences/max:np.float64(16.030704772099853) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.596538595229504) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.030704772099853) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:273 - timing_s/agent_loop/slowest/response_length:2049 - timing_s/gen:22.769220365211368 - timing_s/reward:0.06713957525789738 - timing_s/old_log_prob:3.479922892525792 - timing_s/adv:0.14293243549764156 - timing_s/update_actor:13.602141378447413 - timing_s/step:40.15661580115557 - timing_s/stop_profile:0.00011800043284893036 - timing_per_token_ms/gen:0.1415582532792739 - timing_per_token_ms/update_actor:0.059290549347023574 - timing_per_token_ms/adv:0.0006230300350789685 - perf/total_num_tokens:229415 - perf/time_per_step:40.15661580115557 - perf/throughput:714.1257904301482
(TaskRunner pid=2281424) Training Progress: 2%|▏ | 2/100 [01:42<1:20:39, 49.39s/it]
(TaskRunner pid=2281424) step:3 - global_seqlen/min:17827 - global_seqlen/max:47471 - global_seqlen/minmax_diff:29644 - global_seqlen/balanced_min:29373 - global_seqlen/balanced_max:29478 - global_seqlen/mean:29437.25 - actor/entropy:0.3266855776309967 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2997521162033081 - training/rollout_probs_diff_mean:0.003248069901019335 - training/rollout_probs_diff_std:0.008916190825402737 - training/rollout_actor_probs_pearson_corr:0.999448299407959 - rollout_corr/rollout_is_mean:0.9999076724052429 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.8091252059093677e-05 - rollout_corr/rollout_is_max:1.636006236076355 - rollout_corr/rollout_is_min:0.3921360969543457 - rollout_corr/rollout_is_std:0.0288705974817276 - rollout_corr/rollout_is_eff_sample_size:0.9991670637992444 - rollout_corr/rollout_is_seq_mean:0.9999990463256836 - rollout_corr/rollout_is_seq_std:0.0017653178656473756 - rollout_corr/rollout_is_seq_max:1.0086678266525269 - rollout_corr/rollout_is_seq_min:0.992326557636261 - rollout_corr/rollout_is_seq_max_deviation:0.008667826652526855 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.305179648566991) - rollout_corr/training_log_ppl:np.float64(0.25337072674301453) - rollout_corr/kl:np.float64(0.00037616747434299214) - rollout_corr/k3_kl:np.float64(0.0004263165984070838) - rollout_corr/rollout_ppl:np.float64(1.30465651396662) - rollout_corr/rollout_log_ppl:np.float64(0.2529945558635518) - rollout_corr/log_ppl_diff:np.float64(0.00037617087946273386) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011907302250619978) - rollout_corr/log_ppl_diff_max:np.float64(0.006459161639213562) - rollout_corr/log_ppl_diff_min:np.float64(-0.006830498576164246) - rollout_corr/ppl_ratio:np.float64(1.0003775698132813) - rollout_corr/chi2_token:np.float64(0.0009592922870069742) - rollout_corr/chi2_seq:np.float64(0.461652651662007) - actor/pg_loss:np.float64(6.696488708257675e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00017301455625329254) - actor/ppo_kl:np.float64(-5.806259433693839e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.19614363089203835) - perf/mfu/actor:np.float64(0.11421010958811989) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(136.935546875) - perf/cpu_memory_used_gb:np.float64(99.99443435668945) - actor/lr:np.float64(2e-07) - training/global_step:3 - training/epoch:0 - critic/score/mean:0.52734375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.52734375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004827348981052637 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004827348981052637 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:647.7578125 - response_length/max:3010.0 - response_length/min:73.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:647.7578125 - response_length_non_aborted/max:3010.0 - response_length_non_aborted/min:73.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:272.15625 - prompt_length/max:365.0 - prompt_length/min:158.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011022016406059265 - timing_s/agent_loop/generate_sequences/min:np.float64(0.9544193036854267) - timing_s/agent_loop/generate_sequences/max:np.float64(22.102071948349476) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.615853465111286) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(22.102071948349476) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:251 - timing_s/agent_loop/slowest/response_length:3010 - timing_s/gen:29.6279108915478 - timing_s/reward:0.07157537713646889 - timing_s/old_log_prob:3.4585841801017523 - timing_s/adv:0.1487460471689701 - timing_s/update_actor:13.637943856418133 - timing_s/step:47.031133998185396 - timing_s/stop_profile:0.00012046471238136292 - timing_per_token_ms/gen:0.17866867012137902 - timing_per_token_ms/update_actor:0.05791108143771129 - timing_per_token_ms/adv:0.0006316233987930688 - perf/total_num_tokens:235498 - perf/time_per_step:47.031133998185396 - perf/throughput:625.909849444323
(TaskRunner pid=2281424) Training Progress: 3%|▎ | 3/100 [02:29<1:18:08, 48.33s/it]
(TaskRunner pid=2281424) step:4 - global_seqlen/min:19941 - global_seqlen/max:38325 - global_seqlen/minmax_diff:18384 - global_seqlen/balanced_min:28308 - global_seqlen/balanced_max:28336 - global_seqlen/mean:28325.875 - actor/entropy:0.2808186411857605 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2771974205970764 - training/rollout_probs_diff_mean:0.0030747849959880114 - training/rollout_probs_diff_std:0.008857703767716885 - training/rollout_actor_probs_pearson_corr:0.9993070363998413 - rollout_corr/rollout_is_mean:1.0000015497207642 - rollout_corr/rollout_is_ratio_fraction_high:1.2669855095737148e-05 - rollout_corr/rollout_is_ratio_fraction_low:6.334927547868574e-06 - rollout_corr/rollout_is_max:2.03224778175354 - rollout_corr/rollout_is_min:0.4781637489795685 - rollout_corr/rollout_is_std:0.02775697410106659 - rollout_corr/rollout_is_eff_sample_size:0.9992299054457299 - rollout_corr/rollout_is_seq_mean:0.9999985694885254 - rollout_corr/rollout_is_seq_std:0.0015701246447861195 - rollout_corr/rollout_is_seq_max:1.0103716850280762 - rollout_corr/rollout_is_seq_min:0.9926313161849976 - rollout_corr/rollout_is_seq_max_deviation:0.010371685028076172 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2888889200985432) - rollout_corr/training_log_ppl:np.float64(0.24067001044750214) - rollout_corr/kl:np.float64(0.00041824670213985726) - rollout_corr/k3_kl:np.float64(0.0004014854371803267) - rollout_corr/rollout_ppl:np.float64(1.2883517127484083) - rollout_corr/rollout_log_ppl:np.float64(0.24025176231953083) - rollout_corr/log_ppl_diff:np.float64(0.00041824812797131017) - rollout_corr/log_ppl_abs_diff:np.float64(0.001111818324716296) - rollout_corr/log_ppl_diff_max:np.float64(0.010206438601016998) - rollout_corr/log_ppl_diff_min:np.float64(-0.007612407207489014) - rollout_corr/ppl_ratio:np.float64(1.0004195608198643) - rollout_corr/chi2_token:np.float64(0.0007662170100957155) - rollout_corr/chi2_seq:np.float64(0.5071867927908897) - actor/pg_loss:np.float64(-1.1301599442958832e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00016651269106660038) - actor/ppo_kl:np.float64(1.1325929784034017e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.19906450808048248) - perf/mfu/actor:np.float64(0.11065199852544017) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(136.935546875) - perf/cpu_memory_used_gb:np.float64(99.7738823890686) - actor/lr:np.float64(3e-07) - training/global_step:4 - training/epoch:0 - critic/score/mean:0.63671875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.63671875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0001662918512010947 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0001662918512010947 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:616.62109375 - response_length/max:2113.0 - response_length/min:89.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:616.62109375 - response_length_non_aborted/max:2113.0 - response_length_non_aborted/min:89.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:268.5625 - prompt_length/max:361.0 - prompt_length/min:185.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014886818826198578 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1396879702806473) - timing_s/agent_loop/generate_sequences/max:np.float64(16.492680540308356) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.509170034310955) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(16.492680540308356) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:268 - timing_s/agent_loop/slowest/response_length:2113 - timing_s/gen:23.51997653208673 - timing_s/reward:0.07057112827897072 - timing_s/old_log_prob:3.281124971807003 - timing_s/adv:0.14427484013140202 - timing_s/update_actor:13.601994905620813 - timing_s/step:40.70403571240604 - timing_s/stop_profile:0.0001428518444299698 - timing_per_token_ms/gen:0.1489973490360567 - timing_per_token_ms/update_actor:0.060024601647878546 - timing_per_token_ms/adv:0.000636674242770091 - perf/total_num_tokens:226607 - perf/time_per_step:40.70403571240604 - perf/throughput:695.8984411309038
(TaskRunner pid=2281424) Training Progress: 4%|▍ | 4/100 [03:10<1:12:33, 45.35s/it]
(TaskRunner pid=2281424) step:5 - global_seqlen/min:23939 - global_seqlen/max:37277 - global_seqlen/minmax_diff:13338 - global_seqlen/balanced_min:28100 - global_seqlen/balanced_max:29575 - global_seqlen/mean:28450.375 - actor/entropy:0.23747068643569946 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.7566088438034058 - training/rollout_probs_diff_mean:0.0028660795651376247 - training/rollout_probs_diff_std:0.009149988181889057 - training/rollout_actor_probs_pearson_corr:0.9992654919624329 - rollout_corr/rollout_is_mean:1.0000168085098267 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.0813411285635084e-05 - rollout_corr/rollout_is_max:1.8653022050857544 - rollout_corr/rollout_is_min:0.1207408457994461 - rollout_corr/rollout_is_std:0.026950737461447716 - rollout_corr/rollout_is_eff_sample_size:0.9992743040251587 - rollout_corr/rollout_is_seq_mean:1.0000309944152832 - rollout_corr/rollout_is_seq_std:0.0014656533021479845 - rollout_corr/rollout_is_seq_max:1.007324457168579 - rollout_corr/rollout_is_seq_min:0.9938392043113708 - rollout_corr/rollout_is_seq_max_deviation:0.0073244571685791016 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2311981176026165) - rollout_corr/training_log_ppl:np.float64(0.20067966838541906) - rollout_corr/kl:np.float64(0.00032477841484368497) - rollout_corr/k3_kl:np.float64(0.0004131826744924183) - rollout_corr/rollout_ppl:np.float64(1.230793579481542) - rollout_corr/rollout_log_ppl:np.float64(0.2003548895881977) - rollout_corr/log_ppl_diff:np.float64(0.00032477879722137004) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011101132404292002) - rollout_corr/log_ppl_diff_max:np.float64(0.005720242857933044) - rollout_corr/log_ppl_diff_min:np.float64(-0.003920510411262512) - rollout_corr/ppl_ratio:np.float64(1.0003259072545916) - rollout_corr/chi2_token:np.float64(0.0010012227576225996) - rollout_corr/chi2_seq:np.float64(0.8521349627990276) - actor/pg_loss:np.float64(8.134916424751282e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019610086025068085) - actor/ppo_kl:np.float64(-6.217524514084971e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.17139575630426407) - perf/mfu/actor:np.float64(0.10970825383409716) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(99.72949981689453) - actor/lr:np.float64(4e-07) - training/global_step:5 - training/epoch:0 - critic/score/mean:0.73046875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.73046875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0029827384278178215 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0029827384278178215 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:633.85546875 - response_length/max:4229.0 - response_length/min:75.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:633.85546875 - response_length_non_aborted/max:4229.0 - response_length_non_aborted/min:75.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:255.21875 - prompt_length/max:350.0 - prompt_length/min:160.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011628493666648865 - timing_s/agent_loop/generate_sequences/min:np.float64(0.9417712222784758) - timing_s/agent_loop/generate_sequences/max:np.float64(27.8466968331486) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.5835448407015065) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(27.8466968331486) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:259 - timing_s/agent_loop/slowest/response_length:4229 - timing_s/gen:34.45733902230859 - timing_s/reward:0.06979488581418991 - timing_s/old_log_prob:3.3189262840896845 - timing_s/adv:0.15610580332577229 - timing_s/update_actor:13.891692074015737 - timing_s/step:51.97962950356305 - timing_s/stop_profile:0.0001263655722141266 - timing_per_token_ms/gen:0.21234963992868905 - timing_per_token_ms/update_actor:0.06103474942780076 - timing_per_token_ms/adv:0.00068586883004957 - perf/total_num_tokens:227603 - perf/time_per_step:51.97962950356305 - perf/throughput:547.337010127204
(TaskRunner pid=2281424) Training Progress: 5%|▌ | 5/100 [04:02<1:15:36, 47.76s/it]
(TaskRunner pid=2281424) step:6 - global_seqlen/min:22147 - global_seqlen/max:43094 - global_seqlen/minmax_diff:20947 - global_seqlen/balanced_min:31606 - global_seqlen/balanced_max:32026 - global_seqlen/mean:31664.375 - actor/entropy:0.27507346868515015 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5490837097167969 - training/rollout_probs_diff_mean:0.002936719683930278 - training/rollout_probs_diff_std:0.008653098717331886 - training/rollout_actor_probs_pearson_corr:0.9993214011192322 - rollout_corr/rollout_is_mean:1.0000293254852295 - rollout_corr/rollout_is_ratio_fraction_high:1.1111419553344604e-05 - rollout_corr/rollout_is_ratio_fraction_low:2.2222839106689207e-05 - rollout_corr/rollout_is_max:2.245166063308716 - rollout_corr/rollout_is_min:0.017338119447231293 - rollout_corr/rollout_is_std:0.027201691642403603 - rollout_corr/rollout_is_eff_sample_size:0.9992606150343992 - rollout_corr/rollout_is_seq_mean:1.000042200088501 - rollout_corr/rollout_is_seq_std:0.0012306083226576447 - rollout_corr/rollout_is_seq_max:1.0037089586257935 - rollout_corr/rollout_is_seq_min:0.9957495927810669 - rollout_corr/rollout_is_seq_max_deviation:0.0042504072189331055 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.313299536705017) - rollout_corr/training_log_ppl:np.float64(0.26095605584851) - rollout_corr/kl:np.float64(0.00043005530495854316) - rollout_corr/k3_kl:np.float64(0.00047133550992839446) - rollout_corr/rollout_ppl:np.float64(1.3127399231307209) - rollout_corr/rollout_log_ppl:np.float64(0.2605260002455907) - rollout_corr/log_ppl_diff:np.float64(0.00043005560291931033) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010077479528263211) - rollout_corr/log_ppl_diff_max:np.float64(0.00828494131565094) - rollout_corr/log_ppl_diff_min:np.float64(-0.004989057779312134) - rollout_corr/ppl_ratio:np.float64(1.0004311946686357) - rollout_corr/chi2_token:np.float64(0.000960806617513299) - rollout_corr/chi2_seq:np.float64(0.3135292446240783) - actor/pg_loss:np.float64(8.26478935778141e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00020431413167898427) - actor/ppo_kl:np.float64(4.7137278594192034e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.15935598872601986) - perf/mfu/actor:np.float64(0.12093475083929373) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(99.71675491333008) - actor/lr:np.float64(5e-07) - training/global_step:6 - training/epoch:0 - critic/score/mean:0.65625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.65625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.008676629513502121 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.008676629513502121 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:703.10546875 - response_length/max:2812.0 - response_length/min:115.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:703.10546875 - response_length_non_aborted/max:2812.0 - response_length_non_aborted/min:115.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:286.40625 - prompt_length/max:375.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001495908945798874 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4465590845793486) - timing_s/agent_loop/generate_sequences/max:np.float64(21.05473268404603) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.725667599523149) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(21.05473268404603) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:314 - timing_s/agent_loop/slowest/response_length:2812 - timing_s/gen:27.992402544245124 - timing_s/reward:0.07044400088489056 - timing_s/old_log_prob:3.3681033942848444 - timing_s/adv:0.14429624751210213 - timing_s/update_actor:13.960568750277162 - timing_s/step:45.63073200918734 - timing_s/stop_profile:0.00011079013347625732 - timing_per_token_ms/gen:0.1555176674032341 - timing_per_token_ms/update_actor:0.05511149655676593 - timing_per_token_ms/adv:0.0005696316740504989 - perf/total_num_tokens:253315 - perf/time_per_step:45.63073200918734 - perf/throughput:693.9265185056566
(TaskRunner pid=2281424) Training Progress: 6%|▌ | 6/100 [04:48<1:13:43, 47.06s/it]
(TaskRunner pid=2281424) step:7 - global_seqlen/min:20814 - global_seqlen/max:34501 - global_seqlen/minmax_diff:13687 - global_seqlen/balanced_min:26181 - global_seqlen/balanced_max:26228 - global_seqlen/mean:26214.75 - actor/entropy:0.2326890081167221 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35570627450942993 - training/rollout_probs_diff_mean:0.0028703256975859404 - training/rollout_probs_diff_std:0.00889651384204626 - training/rollout_actor_probs_pearson_corr:0.9992586970329285 - rollout_corr/rollout_is_mean:1.0000824928283691 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:7.117336281226017e-06 - rollout_corr/rollout_is_max:1.9689844846725464 - rollout_corr/rollout_is_min:0.4276629388332367 - rollout_corr/rollout_is_std:0.026294749230146408 - rollout_corr/rollout_is_eff_sample_size:0.9993091828880825 - rollout_corr/rollout_is_seq_mean:1.0001779794692993 - rollout_corr/rollout_is_seq_std:0.0015202864306047559 - rollout_corr/rollout_is_seq_max:1.0066863298416138 - rollout_corr/rollout_is_seq_min:0.992588460445404 - rollout_corr/rollout_is_seq_max_deviation:0.007411539554595947 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.243249746505171) - rollout_corr/training_log_ppl:np.float64(0.21141109146992676) - rollout_corr/kl:np.float64(0.000269961312439837) - rollout_corr/k3_kl:np.float64(0.0004097315033675386) - rollout_corr/rollout_ppl:np.float64(1.2429048488847911) - rollout_corr/rollout_log_ppl:np.float64(0.21114112783106975) - rollout_corr/log_ppl_diff:np.float64(0.000269963638857007) - rollout_corr/log_ppl_abs_diff:np.float64(0.001071124483132735) - rollout_corr/log_ppl_diff_max:np.float64(0.005365625023841858) - rollout_corr/log_ppl_diff_min:np.float64(-0.0069854408502578735) - rollout_corr/ppl_ratio:np.float64(1.0002711091656238) - rollout_corr/chi2_token:np.float64(0.0011220404412597418) - rollout_corr/chi2_seq:np.float64(0.38850099057890475) - actor/pg_loss:np.float64(2.9145623557269573e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(9.624894073567702e-05) - actor/ppo_kl:np.float64(5.5716070178846167e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1705261506140232) - perf/mfu/actor:np.float64(0.10573418722291973) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(99.6202507019043) - actor/lr:np.float64(6e-07) - training/global_step:7 - training/epoch:0 - critic/score/mean:0.75390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.75390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005430527497082949 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005430527497082949 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:548.8359375 - response_length/max:1959.0 - response_length/min:89.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:548.8359375 - response_length_non_aborted/max:1959.0 - response_length_non_aborted/min:89.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:270.375 - prompt_length/max:363.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011132284998893738 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1634128130972385) - timing_s/agent_loop/generate_sequences/max:np.float64(14.78265242651105) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.892496629217931) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.78265242651105) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:294 - timing_s/agent_loop/slowest/response_length:1959 - timing_s/gen:21.572919487953186 - timing_s/reward:0.07407674379646778 - timing_s/old_log_prob:3.216347351670265 - timing_s/adv:0.14478853531181812 - timing_s/update_actor:13.154543800279498 - timing_s/step:38.24971301481128 - timing_s/stop_profile:0.00011751428246498108 - timing_per_token_ms/gen:0.1535417252989508 - timing_per_token_ms/update_actor:0.06272491536386718 - timing_per_token_ms/adv:0.0006903963193994703 - perf/total_num_tokens:209718 - perf/time_per_step:38.24971301481128 - perf/throughput:685.3580833364415
(TaskRunner pid=2281424) Training Progress: 7%|▋ | 7/100 [05:26<1:08:30, 44.19s/it]
(TaskRunner pid=2281424) step:8 - global_seqlen/min:24414 - global_seqlen/max:43215 - global_seqlen/minmax_diff:18801 - global_seqlen/balanced_min:30100 - global_seqlen/balanced_max:30612 - global_seqlen/mean:30198.5 - actor/entropy:0.26404014229774475 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.1850457787513733 - training/rollout_probs_diff_mean:0.002853685524314642 - training/rollout_probs_diff_std:0.0084674758836627 - training/rollout_actor_probs_pearson_corr:0.9993511438369751 - rollout_corr/rollout_is_mean:1.0000009536743164 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.1623852515185717e-05 - rollout_corr/rollout_is_max:1.9149731397628784 - rollout_corr/rollout_is_min:0.3131442368030548 - rollout_corr/rollout_is_std:0.026572102680802345 - rollout_corr/rollout_is_eff_sample_size:0.9992943025293132 - rollout_corr/rollout_is_seq_mean:0.99997878074646 - rollout_corr/rollout_is_seq_std:0.0011704006465151906 - rollout_corr/rollout_is_seq_max:1.0035500526428223 - rollout_corr/rollout_is_seq_min:0.996343731880188 - rollout_corr/rollout_is_seq_max_deviation:0.0036562681198120117 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3104631286114454) - rollout_corr/training_log_ppl:np.float64(0.2542687382810982) - rollout_corr/kl:np.float64(0.0003127987160311818) - rollout_corr/k3_kl:np.float64(0.0003740117848280988) - rollout_corr/rollout_ppl:np.float64(1.310010947752744) - rollout_corr/rollout_log_ppl:np.float64(0.25395593749999534) - rollout_corr/log_ppl_diff:np.float64(0.00031280078110285103) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009792603377718478) - rollout_corr/log_ppl_diff_max:np.float64(0.004227757453918457) - rollout_corr/log_ppl_diff_min:np.float64(-0.003130972385406494) - rollout_corr/ppl_ratio:np.float64(1.0003135872539133) - rollout_corr/chi2_token:np.float64(0.0008723102509975433) - rollout_corr/chi2_seq:np.float64(1.1971292272210121) - actor/pg_loss:np.float64(-1.5925150364637375e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001960772210622963) - actor/ppo_kl:np.float64(-8.012754377872966e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.200693067163229) - perf/mfu/actor:np.float64(0.11682536401655431) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(99.71234512329102) - actor/lr:np.float64(7e-07) - training/global_step:8 - training/epoch:0 - critic/score/mean:0.7265625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.7265625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006695338990539312 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006695338990539312 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:672.109375 - response_length/max:3280.0 - response_length/min:183.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:672.109375 - response_length_non_aborted/max:3280.0 - response_length_non_aborted/min:183.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:271.59375 - prompt_length/max:375.0 - prompt_length/min:205.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.800944149494171e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1623081266880035) - timing_s/agent_loop/generate_sequences/max:np.float64(23.256134187802672) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.173095271158672) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(23.256134187802672) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:242 - timing_s/agent_loop/slowest/response_length:3280 - timing_s/gen:30.018665039911866 - timing_s/reward:0.06087653711438179 - timing_s/old_log_prob:3.2937866989523172 - timing_s/adv:0.15990737453103065 - timing_s/update_actor:13.748827634379268 - timing_s/step:47.31482618674636 - timing_s/stop_profile:3.798492252826691e-05 - timing_per_token_ms/gen:0.17446626200111512 - timing_per_token_ms/update_actor:0.056910225815765964 - timing_per_token_ms/adv:0.0006619011479503562 - perf/total_num_tokens:241588 - perf/time_per_step:47.31482618674636 - perf/throughput:638.2460305530846
(TaskRunner pid=2281424) Training Progress: 8%|▊ | 8/100 [06:13<1:09:17, 45.19s/it]
(TaskRunner pid=2281424) step:9 - global_seqlen/min:19363 - global_seqlen/max:38902 - global_seqlen/minmax_diff:19539 - global_seqlen/balanced_min:27273 - global_seqlen/balanced_max:27330 - global_seqlen/mean:27308.125 - actor/entropy:0.278557687997818 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2980630397796631 - training/rollout_probs_diff_mean:0.0030410082545131445 - training/rollout_probs_diff_std:0.008712326176464558 - training/rollout_actor_probs_pearson_corr:0.9993500709533691 - rollout_corr/rollout_is_mean:1.0001354217529297 - rollout_corr/rollout_is_ratio_fraction_high:6.632663371419767e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.3265326742839534e-05 - rollout_corr/rollout_is_max:2.081381320953369 - rollout_corr/rollout_is_min:0.375669002532959 - rollout_corr/rollout_is_std:0.02771829441189766 - rollout_corr/rollout_is_eff_sample_size:0.9992325240192108 - rollout_corr/rollout_is_seq_mean:1.000201940536499 - rollout_corr/rollout_is_seq_std:0.001359231653623283 - rollout_corr/rollout_is_seq_max:1.0056883096694946 - rollout_corr/rollout_is_seq_min:0.9940919280052185 - rollout_corr/rollout_is_seq_max_deviation:0.005908071994781494 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3391040116548538) - rollout_corr/training_log_ppl:np.float64(0.27259969667647965) - rollout_corr/kl:np.float64(0.00025990279606591393) - rollout_corr/k3_kl:np.float64(0.0004612994821684424) - rollout_corr/rollout_ppl:np.float64(1.3387267710641026) - rollout_corr/rollout_log_ppl:np.float64(0.2723397923691664) - rollout_corr/log_ppl_diff:np.float64(0.0002599043073132634) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011203358299098909) - rollout_corr/log_ppl_diff_max:np.float64(0.0065671950578689575) - rollout_corr/log_ppl_diff_min:np.float64(-0.004946023225784302) - rollout_corr/ppl_ratio:np.float64(1.000261099776253) - rollout_corr/chi2_token:np.float64(0.0013498091138899326) - rollout_corr/chi2_seq:np.float64(0.847581728361547) - actor/pg_loss:np.float64(-4.3141539208590984e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003405330644454807) - actor/ppo_kl:np.float64(4.417660383637667e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.26127634942531586) - perf/mfu/actor:np.float64(0.10783332125209669) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(99.68988418579102) - actor/lr:np.float64(8e-07) - training/global_step:9 - training/epoch:0 - critic/score/mean:0.58984375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.58984375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.010788856074213982 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:-0.010788856074213982 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:588.94140625 - response_length/max:2145.0 - response_length/min:103.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:588.94140625 - response_length_non_aborted/max:2145.0 - response_length_non_aborted/min:103.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:264.4375 - prompt_length/max:362.0 - prompt_length/min:180.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011837668716907501 - timing_s/agent_loop/generate_sequences/min:np.float64(0.915661258623004) - timing_s/agent_loop/generate_sequences/max:np.float64(15.536512579768896) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.094332097287406) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.536512579768896) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:213 - timing_s/agent_loop/slowest/response_length:2145 - timing_s/gen:22.36645532399416 - timing_s/reward:0.062141528353095055 - timing_s/old_log_prob:3.2151599191129208 - timing_s/adv:0.15049505233764648 - timing_s/update_actor:13.28318246267736 - timing_s/step:39.18115637823939 - timing_s/stop_profile:0.00013834238052368164 - timing_per_token_ms/gen:0.14834916543847979 - timing_per_token_ms/update_actor:0.06080233658790818 - timing_per_token_ms/adv:0.000688874887682908 - perf/total_num_tokens:218465 - perf/time_per_step:39.18115637823939 - perf/throughput:696.9708789699354
(TaskRunner pid=2281424) Training Progress: 9%|▉ | 9/100 [06:53<1:05:43, 43.33s/it]
(TaskRunner pid=2281424) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 10}
(TaskRunner pid=2281424) validation generation end
(TaskRunner pid=2281424) [prompt] system
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2281424) <reasoning>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </reasoning>
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) A
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) user
(TaskRunner pid=2281424) A charged particle produces an electric field with a magnitude of $2.0 \mathrm{~N} / \mathrm{C}$ at a point that is $50 \mathrm{~cm}$ away from the particle. What is the magnitude of the particle's charge? Recall that the electric field $E$ due to a point charge $q$ at a distance $r$ is given by $E = k\frac{q}{r^2}$, where $k = 8.99 \times 10^9 \mathrm{N} \mathrm{m}^2 / \mathrm{C}^2$.
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) A: 50 pC
(TaskRunner pid=2281424) B: 56 pC
(TaskRunner pid=2281424) C: 60 pC
(TaskRunner pid=2281424) D: 64 pC
(TaskRunner pid=2281424) Please reason step by step.
(TaskRunner pid=2281424) assistant
(TaskRunner pid=2281424) <think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) </think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) [response] <reasoning>
(TaskRunner pid=2281424) We are given the electric field $ E = 2.0 \, \mathrm{N/C} $, the distance $ r = 50 \, \mathrm{cm} = 0.50 \, \mathrm{m} $, and the constant $ k = 8.99 \times 10^9 \, \mathrm{N \cdot m^2/C^2} $. The formula for the electric field due to a point charge is:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) E = k \frac{q}{r^2}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) We need to solve for $ q $, the charge. Rearranging the formula:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{E r^2}{k}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Substitute the known values:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{(2.0 \, \mathrm{N/C}) \times (0.50 \, \mathrm{m})^2}{8.99 \times 10^9 \, \mathrm{N \cdot m^2/C^2}}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{2.0 \times 0.25}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{0.5}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 5.56 \times 10^{-11} \, \mathrm{C}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Convert to picocoulombs (1 pC = $ 10^{-12} \, \mathrm{C} $):
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 55.6 \, \mathrm{pC}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) The closest option is **B: 56 pC**.
(TaskRunner pid=2281424) </reasoning>
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) B
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424) [ground_truth] B
(TaskRunner pid=2281424) [score] 1.0
(TaskRunner pid=2281424) [acc] 1.0
(TaskRunner pid=2281424) [pred] B
(TaskRunner pid=2281424) [incorrect_format] 1
(TaskRunner pid=2281424) [feedback]
(TaskRunner pid=2281424) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_10
(TaskRunner pid=2281424) step:10 - global_seqlen/min:18370 - global_seqlen/max:31673 - global_seqlen/minmax_diff:13303 - global_seqlen/balanced_min:26421 - global_seqlen/balanced_max:26444 - global_seqlen/mean:26434.75 - actor/entropy:0.32802483439445496 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.45646193623542786 - training/rollout_probs_diff_mean:0.0033578479196876287 - training/rollout_probs_diff_std:0.00903936568647623 - training/rollout_actor_probs_pearson_corr:0.9993472099304199 - rollout_corr/rollout_is_mean:0.9999197721481323 - rollout_corr/rollout_is_ratio_fraction_high:7.013114554865751e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.4026229109731503e-05 - rollout_corr/rollout_is_max:3.0499253273010254 - rollout_corr/rollout_is_min:0.36710941791534424 - rollout_corr/rollout_is_std:0.029133660718798637 - rollout_corr/rollout_is_eff_sample_size:0.999151830651037 - rollout_corr/rollout_is_seq_mean:0.9999980330467224 - rollout_corr/rollout_is_seq_std:0.0017985748127102852 - rollout_corr/rollout_is_seq_max:1.008771300315857 - rollout_corr/rollout_is_seq_min:0.9932235479354858 - rollout_corr/rollout_is_seq_max_deviation:0.008771300315856934 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.377936966251582) - rollout_corr/training_log_ppl:np.float64(0.3002843804570148) - rollout_corr/kl:np.float64(0.0005245888745832872) - rollout_corr/k3_kl:np.float64(0.000498560940002335) - rollout_corr/rollout_ppl:np.float64(1.377146857790649) - rollout_corr/rollout_log_ppl:np.float64(0.29975979137816466) - rollout_corr/log_ppl_diff:np.float64(0.0005245890788501129) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013371909590205178) - rollout_corr/log_ppl_diff_max:np.float64(0.0069696009159088135) - rollout_corr/log_ppl_diff_min:np.float64(-0.004811465740203857) - rollout_corr/ppl_ratio:np.float64(1.0005262338090688) - rollout_corr/chi2_token:np.float64(0.0009716309141367674) - rollout_corr/chi2_seq:np.float64(1.0234597465023398) - actor/pg_loss:np.float64(2.4606240913271904e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00024767978288764425) - actor/ppo_kl:np.float64(4.621054888964693e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.18785973638296127) - perf/mfu/actor:np.float64(0.10560325767310388) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(99.76485443115234) - actor/lr:np.float64(9e-07) - val-aux/sciknoweval/reward/mean@16:np.float64(0.58359375) - val-aux/sciknoweval/reward/std@16:np.float64(0.21906251159087287) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.67415) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.19020325207669364) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.4956125) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.17791978642388112) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.5849125000000001) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.2188371622304282) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7478) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.14805144079456356) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.425825) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.12986404819208913) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6028874999999999) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.17818578143360303) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8051999999999999) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.10766830979919102) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.3753875) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.08819187467072624) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6146375) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.13509611267304375) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8500624999999999) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.07012187539545785) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.3417125) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.05316338904072637) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6237) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.09735119649509659) - val-aux/sciknoweval/score/mean@16:np.float64(0.58359375) - val-aux/sciknoweval/score/std@16:np.float64(0.21906251159087287) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.67415) - val-aux/sciknoweval/score/best@2/std:np.float64(0.19020325207669364) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.4956125) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.17791978642388112) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.5849125000000001) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.2188371622304282) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7478) - val-aux/sciknoweval/score/best@4/std:np.float64(0.14805144079456356) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.425825) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.12986404819208913) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6028874999999999) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.17818578143360303) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8051999999999999) - val-aux/sciknoweval/score/best@8/std:np.float64(0.10766830979919102) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.3753875) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.08819187467072624) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6146375) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.13509611267304375) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8500624999999999) - val-aux/sciknoweval/score/best@16/std:np.float64(0.07012187539545785) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.3417125) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.05316338904072637) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6237) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.09735119649509659) - val-core/sciknoweval/acc/mean@16:np.float64(0.58359375) - val-aux/sciknoweval/acc/std@16:np.float64(0.21906251159087287) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.67415) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.19020325207669364) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.4956125) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.17791978642388112) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.5849125000000001) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.2188371622304282) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7478) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.14805144079456356) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.425825) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.12986404819208913) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6028874999999999) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.17818578143360303) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8051999999999999) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.10766830979919102) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.3753875) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.08819187467072624) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6146375) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.13509611267304375) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8500624999999999) - val-core/sciknoweval/acc/best@16/std:np.float64(0.07012187539545785) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.3417125) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.05316338904072637) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6237) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.09735119649509659) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.98125) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.0432113421230519) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.99315) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.016507906298959766) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9690875) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.053154032584486456) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9808874999999999) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.042999991812243614) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9968875) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.005405422624550276) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.9512750000000001) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.06454074729263456) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9880125) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.028221887660256677) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(0.9994) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0026720778431774776) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.9238625) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.07328509472283329) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.992225) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.014819305487201267) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(0.9999750000000001) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0005584576975922169) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.887475) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.06968272194151441) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9943125) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.007892775584975463) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:10 - training/epoch:0 - critic/score/mean:0.6875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.01252454612404108 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.01252454612404108 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:556.9921875 - response_length/max:1781.0 - response_length/min:88.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:556.9921875 - response_length_non_aborted/max:1781.0 - response_length_non_aborted/min:88.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:269.09375 - prompt_length/max:382.0 - prompt_length/min:164.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014546141028404236 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1660982128232718) - timing_s/agent_loop/generate_sequences/max:np.float64(13.751114128157496) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.805240769754164) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(13.751114128157496) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:320 - timing_s/agent_loop/slowest/response_length:1781 - timing_s/gen:20.531365724280477 - timing_s/reward:0.06193559989333153 - timing_s/old_log_prob:3.251886686310172 - timing_s/adv:0.15388510562479496 - timing_s/update_actor:13.160292625427246 - timing_s/step:37.241492161527276 - timing_s/testing:116.28815667517483 - timing_s/save_checkpoint:12.811152569949627 - timing_s/stop_profile:0.00015922822058200836 - timing_per_token_ms/gen:0.1439888191617959 - timing_per_token_ms/update_actor:0.06223007889911596 - timing_per_token_ms/adv:0.0007276648427959171 - perf/total_num_tokens:211478 - perf/time_per_step:37.241492161527276 - perf/throughput:709.8198397997786
(TaskRunner pid=2281424) Training Progress: 10%|█ | 10/100 [09:39<2:02:00, 81.34s/it]
(TaskRunner pid=2281424) step:11 - global_seqlen/min:20995 - global_seqlen/max:35423 - global_seqlen/minmax_diff:14428 - global_seqlen/balanced_min:27571 - global_seqlen/balanced_max:27590 - global_seqlen/mean:27580.5 - actor/entropy:0.24689078330993652 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.6190903186798096 - training/rollout_probs_diff_mean:0.0029323738999664783 - training/rollout_probs_diff_std:0.008981931023299694 - training/rollout_actor_probs_pearson_corr:0.999229907989502 - rollout_corr/rollout_is_mean:0.9999557137489319 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.325087163830176e-05 - rollout_corr/rollout_is_max:1.9537171125411987 - rollout_corr/rollout_is_min:0.08846962451934814 - rollout_corr/rollout_is_std:0.0269706342369318 - rollout_corr/rollout_is_eff_sample_size:0.9992731136632474 - rollout_corr/rollout_is_seq_mean:1.0000543594360352 - rollout_corr/rollout_is_seq_std:0.0017303781351074576 - rollout_corr/rollout_is_seq_max:1.0074354410171509 - rollout_corr/rollout_is_seq_min:0.9947044253349304 - rollout_corr/rollout_is_seq_max_deviation:0.007435441017150879 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2710583903826773) - rollout_corr/training_log_ppl:np.float64(0.22995059203822166) - rollout_corr/kl:np.float64(0.0004979196115613149) - rollout_corr/k3_kl:np.float64(0.00043510435352800414) - rollout_corr/rollout_ppl:np.float64(1.270424968097359) - rollout_corr/rollout_log_ppl:np.float64(0.22945267148315907) - rollout_corr/log_ppl_diff:np.float64(0.000497920555062592) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013768301578238606) - rollout_corr/log_ppl_diff_max:np.float64(0.006137266755104065) - rollout_corr/log_ppl_diff_min:np.float64(-0.007355779409408569) - rollout_corr/ppl_ratio:np.float64(1.000499751418829) - rollout_corr/chi2_token:np.float64(0.0007434843573719263) - rollout_corr/chi2_seq:np.float64(0.47075980668887496) - actor/pg_loss:np.float64(4.087574779987335e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001813313299408037) - actor/ppo_kl:np.float64(0.00013543530537063475) - actor/pg_clipfrac_lower:np.float64(4.278477717889473e-06) - actor/grad_norm:np.float64(0.17897598445415497) - perf/mfu/actor:np.float64(0.10733546751479987) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(104.49744415283203) - actor/lr:np.float64(1e-06) - training/global_step:11 - training/epoch:0 - critic/score/mean:0.62890625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62890625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004445641301572323 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:0.004445641301572323 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:587.390625 - response_length/max:1781.0 - response_length/min:81.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:587.390625 - response_length_non_aborted/max:1781.0 - response_length_non_aborted/min:81.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:274.5 - prompt_length/max:437.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.45193862915039e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(0.9449637047946453) - timing_s/agent_loop/generate_sequences/max:np.float64(13.539711721241474) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.947522593545727) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(13.539711721241474) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:334 - timing_s/agent_loop/slowest/response_length:1671 - timing_s/gen:20.11647711135447 - timing_s/reward:0.06534852646291256 - timing_s/old_log_prob:3.4682928640395403 - timing_s/adv:0.13045809976756573 - timing_s/update_actor:13.693839404731989 - timing_s/step:37.579134887084365 - timing_s/stop_profile:0.00012080185115337372 - timing_per_token_ms/gen:0.13377807777614495 - timing_per_token_ms/update_actor:0.06206304909597356 - timing_per_token_ms/adv:0.0005912605816046016 - perf/total_num_tokens:220644 - perf/time_per_step:37.579134887084365 - perf/throughput:733.9312116383815
(TaskRunner pid=2281424) Training Progress: 11%|█ | 11/100 [10:17<1:40:48, 67.96s/it]
(TaskRunner pid=2281424) step:12 - global_seqlen/min:18284 - global_seqlen/max:36197 - global_seqlen/minmax_diff:17913 - global_seqlen/balanced_min:27117 - global_seqlen/balanced_max:27742 - global_seqlen/mean:27219.25 - actor/entropy:0.2918388545513153 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.18463563919067383 - training/rollout_probs_diff_mean:0.00314357434399426 - training/rollout_probs_diff_std:0.008670532144606113 - training/rollout_actor_probs_pearson_corr:0.9993590712547302 - rollout_corr/rollout_is_mean:0.9999263882637024 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.328426969848806e-05 - rollout_corr/rollout_is_max:1.6952860355377197 - rollout_corr/rollout_is_min:0.2599896490573883 - rollout_corr/rollout_is_std:0.02788124419748783 - rollout_corr/rollout_is_eff_sample_size:0.9992230019996114 - rollout_corr/rollout_is_seq_mean:0.9999516010284424 - rollout_corr/rollout_is_seq_std:0.0013419243041425943 - rollout_corr/rollout_is_seq_max:1.0067940950393677 - rollout_corr/rollout_is_seq_min:0.9959049224853516 - rollout_corr/rollout_is_seq_max_deviation:0.006794095039367676 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.284353747498244) - rollout_corr/training_log_ppl:np.float64(0.24118711793562397) - rollout_corr/kl:np.float64(0.0004640274163274505) - rollout_corr/k3_kl:np.float64(0.000438987553025072) - rollout_corr/rollout_ppl:np.float64(1.2837417596019804) - rollout_corr/rollout_log_ppl:np.float64(0.2407230889366474) - rollout_corr/log_ppl_diff:np.float64(0.00046402899897657335) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012675680045504123) - rollout_corr/log_ppl_diff_max:np.float64(0.00536249577999115) - rollout_corr/log_ppl_diff_min:np.float64(-0.005736485123634338) - rollout_corr/ppl_ratio:np.float64(1.0004653770010918) - rollout_corr/chi2_token:np.float64(0.0008103374857455492) - rollout_corr/chi2_seq:np.float64(0.7635552103165537) - actor/pg_loss:np.float64(6.2944600358605385e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00022421280800699606) - actor/ppo_kl:np.float64(8.907836768656807e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1681164726614952) - perf/mfu/actor:np.float64(0.10657234495140916) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.30431365966797) - actor/lr:np.float64(1e-06) - training/global_step:12 - training/epoch:0 - critic/score/mean:0.5234375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5234375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.02310134656727314 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.02310134656727314 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:588.1015625 - response_length/max:3570.0 - response_length/min:116.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:588.1015625 - response_length_non_aborted/max:3570.0 - response_length_non_aborted/min:116.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:262.5 - prompt_length/max:364.0 - prompt_length/min:189.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:7.758848369121552e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(0.9172126762568951) - timing_s/agent_loop/generate_sequences/max:np.float64(22.92699559032917) - timing_s/agent_loop/generate_sequences/mean:np.float64(5.20948198466067) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(22.92699559032917) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:237 - timing_s/agent_loop/slowest/response_length:3570 - timing_s/gen:29.500037664547563 - timing_s/reward:0.06999753788113594 - timing_s/old_log_prob:3.321774547919631 - timing_s/adv:0.16080131568014622 - timing_s/update_actor:13.495587630197406 - timing_s/step:46.61714624799788 - timing_s/stop_profile:0.00011683814227581024 - timing_per_token_ms/gen:0.19594323408576034 - timing_per_token_ms/update_actor:0.06197630183692334 - timing_per_token_ms/adv:0.0007384540154492969 - perf/total_num_tokens:217754 - perf/time_per_step:46.61714624799788 - perf/throughput:583.8892379897454
(TaskRunner pid=2281424) Training Progress: 12%|█▏ | 12/100 [11:03<1:30:10, 61.48s/it]
(TaskRunner pid=2281424) step:13 - global_seqlen/min:24964 - global_seqlen/max:33451 - global_seqlen/minmax_diff:8487 - global_seqlen/balanced_min:28949 - global_seqlen/balanced_max:28972 - global_seqlen/mean:28959.375 - actor/entropy:0.31463027000427246 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24114426970481873 - training/rollout_probs_diff_mean:0.00320067978464067 - training/rollout_probs_diff_std:0.008584943599998951 - training/rollout_actor_probs_pearson_corr:0.9994125962257385 - rollout_corr/rollout_is_mean:1.0001263618469238 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:6.238653440959752e-06 - rollout_corr/rollout_is_max:1.859130859375 - rollout_corr/rollout_is_min:0.4844902455806732 - rollout_corr/rollout_is_std:0.02811329998075962 - rollout_corr/rollout_is_eff_sample_size:0.9992105046242893 - rollout_corr/rollout_is_seq_mean:1.0000969171524048 - rollout_corr/rollout_is_seq_std:0.0011731568956747651 - rollout_corr/rollout_is_seq_max:1.0035706758499146 - rollout_corr/rollout_is_seq_min:0.9957525134086609 - rollout_corr/rollout_is_seq_max_deviation:0.004247486591339111 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3780980156734586) - rollout_corr/training_log_ppl:np.float64(0.2991240085102618) - rollout_corr/kl:np.float64(0.0003781683996528784) - rollout_corr/k3_kl:np.float64(0.0004593968702977236) - rollout_corr/rollout_ppl:np.float64(1.3775346302427351) - rollout_corr/rollout_log_ppl:np.float64(0.2987458378338488) - rollout_corr/log_ppl_diff:np.float64(0.0003781706764129922) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010912520374404266) - rollout_corr/log_ppl_diff_max:np.float64(0.004893004894256592) - rollout_corr/log_ppl_diff_min:np.float64(-0.003899678587913513) - rollout_corr/ppl_ratio:np.float64(1.0003791959024966) - rollout_corr/chi2_token:np.float64(0.001073798630386591) - rollout_corr/chi2_seq:np.float64(0.506177173461765) - actor/pg_loss:np.float64(3.237277269363403e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004119513098430616) - actor/ppo_kl:np.float64(7.283486395426308e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.21577078476548195) - perf/mfu/actor:np.float64(0.11467366616050297) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.31635284423828) - actor/lr:np.float64(1e-06) - training/global_step:13 - training/epoch:0 - critic/score/mean:0.5703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0002386284904787317 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0002386284904787317 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:626.13671875 - response_length/max:1861.0 - response_length/min:160.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:626.13671875 - response_length_non_aborted/max:1861.0 - response_length_non_aborted/min:160.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.84375 - prompt_length/max:363.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013312511146068573 - timing_s/agent_loop/generate_sequences/min:np.float64(1.9897179938852787) - timing_s/agent_loop/generate_sequences/max:np.float64(15.492640441283584) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.851898552646162) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.492640441283584) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:264 - timing_s/agent_loop/slowest/response_length:1861 - timing_s/gen:22.31190705113113 - timing_s/reward:0.0678411666303873 - timing_s/old_log_prob:3.3336705192923546 - timing_s/adv:0.18029628694057465 - timing_s/update_actor:13.361042624339461 - timing_s/step:39.349795043468475 - timing_s/stop_profile:0.00011570192873477936 - timing_per_token_ms/gen:0.1391962558791893 - timing_per_token_ms/update_actor:0.05767149077086203 - timing_per_token_ms/adv:0.0007782293598384575 - perf/total_num_tokens:231675 - perf/time_per_step:39.349795043468475 - perf/throughput:735.9472893825621
(TaskRunner pid=2281424) Training Progress: 13%|█▎ | 13/100 [11:43<1:19:27, 54.80s/it]
(TaskRunner pid=2281424) step:14 - global_seqlen/min:18482 - global_seqlen/max:39950 - global_seqlen/minmax_diff:21468 - global_seqlen/balanced_min:27989 - global_seqlen/balanced_max:28043 - global_seqlen/mean:28025.875 - actor/entropy:0.27694693207740784 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9996616840362549 - training/rollout_probs_diff_mean:0.0032118367962539196 - training/rollout_probs_diff_std:0.009403628297150135 - training/rollout_actor_probs_pearson_corr:0.9992220997810364 - rollout_corr/rollout_is_mean:0.9999675154685974 - rollout_corr/rollout_is_ratio_fraction_high:2.5921677661244757e-05 - rollout_corr/rollout_is_ratio_fraction_low:3.2402098440798e-05 - rollout_corr/rollout_is_max:2.0761139392852783 - rollout_corr/rollout_is_min:0.000335350981913507 - rollout_corr/rollout_is_std:0.028911858797073364 - rollout_corr/rollout_is_eff_sample_size:0.9991645645790151 - rollout_corr/rollout_is_seq_mean:0.9999456405639648 - rollout_corr/rollout_is_seq_std:0.001373259350657463 - rollout_corr/rollout_is_seq_max:1.0052764415740967 - rollout_corr/rollout_is_seq_min:0.995773434638977 - rollout_corr/rollout_is_seq_max_deviation:0.00527644157409668 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3075349098071456) - rollout_corr/training_log_ppl:np.float64(0.2546500996249961) - rollout_corr/kl:np.float64(0.0006805974277988014) - rollout_corr/k3_kl:np.float64(0.0006370248192979489) - rollout_corr/rollout_ppl:np.float64(1.3065974819473922) - rollout_corr/rollout_log_ppl:np.float64(0.2539695030573057) - rollout_corr/log_ppl_diff:np.float64(0.0006805965676903725) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013074282323941588) - rollout_corr/log_ppl_diff_max:np.float64(0.01114201545715332) - rollout_corr/log_ppl_diff_min:np.float64(-0.004195064306259155) - rollout_corr/ppl_ratio:np.float64(1.000682536046952) - rollout_corr/chi2_token:np.float64(0.0010629112366586924) - rollout_corr/chi2_seq:np.float64(0.7319330512546003) - actor/pg_loss:np.float64(-7.280893623828888e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004904413003714581) - actor/ppo_kl:np.float64(0.00013513596405800854) - actor/pg_clipfrac_lower:np.float64(5.243288796918932e-06) - actor/grad_norm:np.float64(0.22375838086009026) - perf/mfu/actor:np.float64(0.11024404131767285) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.50100326538086) - actor/lr:np.float64(1e-06) - training/global_step:14 - training/epoch:0 - critic/score/mean:0.64453125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.64453125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004044591914862394 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004044591914862394 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:602.77734375 - response_length/max:2091.0 - response_length/min:88.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:602.77734375 - response_length_non_aborted/max:2091.0 - response_length_non_aborted/min:88.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:273.03125 - prompt_length/max:375.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.801425039768219e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1079022623598576) - timing_s/agent_loop/generate_sequences/max:np.float64(15.903715757653117) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.53672656758863) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(15.903715757653117) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:249 - timing_s/agent_loop/slowest/response_length:2091 - timing_s/gen:22.835159683600068 - timing_s/reward:0.07095035165548325 - timing_s/old_log_prob:3.195645259693265 - timing_s/adv:0.15754373371601105 - timing_s/update_actor:13.296314042061567 - timing_s/step:39.66534601524472 - timing_s/stop_profile:0.00011271052062511444 - timing_per_token_ms/gen:0.14798141210672 - timing_per_token_ms/update_actor:0.0593037418192187 - timing_per_token_ms/adv:0.0007026708966089866 - perf/total_num_tokens:224207 - perf/time_per_step:39.66534601524472 - perf/throughput:706.5581878254312
(TaskRunner pid=2281424) Training Progress: 14%|█▍ | 14/100 [12:23<1:12:00, 50.24s/it]
(TaskRunner pid=2281424) step:15 - global_seqlen/min:18409 - global_seqlen/max:34972 - global_seqlen/minmax_diff:16563 - global_seqlen/balanced_min:27584 - global_seqlen/balanced_max:27657 - global_seqlen/mean:27632.0 - actor/entropy:0.23792505264282227 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2081579566001892 - training/rollout_probs_diff_mean:0.0028373654931783676 - training/rollout_probs_diff_std:0.008734497241675854 - training/rollout_actor_probs_pearson_corr:0.9992759227752686 - rollout_corr/rollout_is_mean:1.000104308128357 - rollout_corr/rollout_is_ratio_fraction_high:6.5414201344538014e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.0567126274108887 - rollout_corr/rollout_is_min:0.6020901799201965 - rollout_corr/rollout_is_std:0.0269706342369318 - rollout_corr/rollout_is_eff_sample_size:0.9992733517354029 - rollout_corr/rollout_is_seq_mean:1.0001739263534546 - rollout_corr/rollout_is_seq_std:0.0016109951538965106 - rollout_corr/rollout_is_seq_max:1.0077435970306396 - rollout_corr/rollout_is_seq_min:0.9901288747787476 - rollout_corr/rollout_is_seq_max_deviation:0.009871125221252441 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2825340307317674) - rollout_corr/training_log_ppl:np.float64(0.2307681181409862) - rollout_corr/kl:np.float64(0.00024018173945883348) - rollout_corr/k3_kl:np.float64(0.00046571865181022076) - rollout_corr/rollout_ppl:np.float64(1.2822559108026326) - rollout_corr/rollout_log_ppl:np.float64(0.23052793518581893) - rollout_corr/log_ppl_diff:np.float64(0.0002401829551672563) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012830473860958591) - rollout_corr/log_ppl_diff_max:np.float64(0.011005207896232605) - rollout_corr/log_ppl_diff_min:np.float64(-0.004439152777194977) - rollout_corr/ppl_ratio:np.float64(1.0002417913638055) - rollout_corr/chi2_token:np.float64(0.0013833614066243172) - rollout_corr/chi2_seq:np.float64(0.8579549191053957) - actor/pg_loss:np.float64(0.00013076758477836847) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00022705169044456852) - actor/ppo_kl:np.float64(1.2435915886399584e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1974821835756302) - perf/mfu/actor:np.float64(0.10945352184092855) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.5802116394043) - actor/lr:np.float64(1e-06) - training/global_step:15 - training/epoch:0 - critic/score/mean:0.69140625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.69140625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.011041100136935711 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.011041100136935711 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:597.15625 - response_length/max:2544.0 - response_length/min:91.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:597.15625 - response_length_non_aborted/max:2544.0 - response_length_non_aborted/min:91.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.34375 - prompt_length/max:460.0 - prompt_length/min:157.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001034904271364212 - timing_s/agent_loop/generate_sequences/min:np.float64(1.0850455239415169) - timing_s/agent_loop/generate_sequences/max:np.float64(18.34066752716899) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.139947620045859) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(18.34066752716899) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:263 - timing_s/agent_loop/slowest/response_length:2544 - timing_s/gen:24.80101560242474 - timing_s/reward:0.06913194060325623 - timing_s/old_log_prob:3.2432605400681496 - timing_s/adv:0.15737476013600826 - timing_s/update_actor:13.316460059955716 - timing_s/step:41.67461157031357 - timing_s/stop_profile:0.00011474825441837311 - timing_per_token_ms/gen:0.1622338662568995 - timing_per_token_ms/update_actor:0.06024021089658601 - timing_per_token_ms/adv:0.0007119225903662794 - perf/total_num_tokens:221056 - perf/time_per_step:41.67461157031357 - perf/throughput:663.0415727661715
(TaskRunner pid=2281424) Training Progress: 15%|█▌ | 15/100 [13:04<1:07:32, 47.68s/it]
(TaskRunner pid=2281424) step:16 - global_seqlen/min:19774 - global_seqlen/max:37148 - global_seqlen/minmax_diff:17374 - global_seqlen/balanced_min:26832 - global_seqlen/balanced_max:26869 - global_seqlen/mean:26857.125 - actor/entropy:0.269243985414505 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.29323524236679077 - training/rollout_probs_diff_mean:0.0030834192875772715 - training/rollout_probs_diff_std:0.009029646404087543 - training/rollout_actor_probs_pearson_corr:0.9992733001708984 - rollout_corr/rollout_is_mean:1.000123381614685 - rollout_corr/rollout_is_ratio_fraction_high:1.3476997992256656e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.39180064201355 - rollout_corr/rollout_is_min:0.5188149213790894 - rollout_corr/rollout_is_std:0.028928346931934357 - rollout_corr/rollout_is_eff_sample_size:0.9991640885384842 - rollout_corr/rollout_is_seq_mean:1.0000724792480469 - rollout_corr/rollout_is_seq_std:0.0016198219964280725 - rollout_corr/rollout_is_seq_max:1.0117334127426147 - rollout_corr/rollout_is_seq_min:0.9945037961006165 - rollout_corr/rollout_is_seq_max_deviation:0.011733412742614746 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2698871567845345) - rollout_corr/training_log_ppl:np.float64(0.22661357806646265) - rollout_corr/kl:np.float64(0.000278438638499523) - rollout_corr/k3_kl:np.float64(0.0004574081480726022) - rollout_corr/rollout_ppl:np.float64(1.2695180620066822) - rollout_corr/rollout_log_ppl:np.float64(0.22633513917389791) - rollout_corr/log_ppl_diff:np.float64(0.0002784388925647363) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010494471789570525) - rollout_corr/log_ppl_diff_max:np.float64(0.00465761125087738) - rollout_corr/log_ppl_diff_min:np.float64(-0.009550660848617554) - rollout_corr/ppl_ratio:np.float64(1.000279544852674) - rollout_corr/chi2_token:np.float64(0.0013024122454226017) - rollout_corr/chi2_seq:np.float64(0.9144449539016932) - actor/pg_loss:np.float64(-1.0943855158984661e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00023535769855698163) - actor/ppo_kl:np.float64(-7.823348023361376e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.21825583279132843) - perf/mfu/actor:np.float64(0.10663394115254357) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.46512222290039) - actor/lr:np.float64(1e-06) - training/global_step:16 - training/epoch:0 - critic/score/mean:0.765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005873444024473429 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005873444024473429 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:579.69140625 - response_length/max:1869.0 - response_length/min:101.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:579.69140625 - response_length_non_aborted/max:1869.0 - response_length_non_aborted/min:101.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:259.59375 - prompt_length/max:344.0 - prompt_length/min:186.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001010652631521225 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3836285527795553) - timing_s/agent_loop/generate_sequences/max:np.float64(14.655153315514326) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.240751736462698) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(14.655153315514326) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:334 - timing_s/agent_loop/slowest/response_length:1869 - timing_s/gen:21.27606852352619 - timing_s/reward:0.06742825172841549 - timing_s/old_log_prob:3.215101270005107 - timing_s/adv:0.13886815682053566 - timing_s/update_actor:13.220384620130062 - timing_s/step:38.01398337073624 - timing_s/stop_profile:2.8956681489944458e-05 - timing_per_token_ms/gen:0.14336876788920688 - timing_per_token_ms/update_actor:0.06153108635106169 - timing_per_token_ms/adv:0.0006463282872819394 - perf/total_num_tokens:214857 - perf/time_per_step:38.01398337073624 - perf/throughput:706.5064646888607
(TaskRunner pid=2281424) Training Progress: 16%|█▌ | 16/100 [13:42<1:02:41, 44.77s/it]
(TaskRunner pid=2281424) step:17 - global_seqlen/min:21078 - global_seqlen/max:38139 - global_seqlen/minmax_diff:17061 - global_seqlen/balanced_min:27815 - global_seqlen/balanced_max:28195 - global_seqlen/mean:27876.125 - actor/entropy:0.3084404766559601 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23105865716934204 - training/rollout_probs_diff_mean:0.003280392149463296 - training/rollout_probs_diff_std:0.008956593461334705 - training/rollout_actor_probs_pearson_corr:0.9993877410888672 - rollout_corr/rollout_is_mean:1.0001431703567505 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:6.458905318140751e-06 - rollout_corr/rollout_is_max:1.8297889232635498 - rollout_corr/rollout_is_min:0.49442058801651 - rollout_corr/rollout_is_std:0.02931314706802368 - rollout_corr/rollout_is_eff_sample_size:0.9991418341499507 - rollout_corr/rollout_is_seq_mean:1.0001916885375977 - rollout_corr/rollout_is_seq_std:0.0019043951760977507 - rollout_corr/rollout_is_seq_max:1.009162425994873 - rollout_corr/rollout_is_seq_min:0.9893993139266968 - rollout_corr/rollout_is_seq_max_deviation:0.010600686073303223 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3411333807744086) - rollout_corr/training_log_ppl:np.float64(0.2804811333771795) - rollout_corr/kl:np.float64(0.00039379035426634346) - rollout_corr/k3_kl:np.float64(0.0005168410488067821) - rollout_corr/rollout_ppl:np.float64(1.3405519826337695) - rollout_corr/rollout_log_ppl:np.float64(0.28008734311151784) - rollout_corr/log_ppl_diff:np.float64(0.0003937902656616643) - rollout_corr/log_ppl_abs_diff:np.float64(0.001501921404269524) - rollout_corr/log_ppl_diff_max:np.float64(0.011711418628692627) - rollout_corr/log_ppl_diff_min:np.float64(-0.007628694176673889) - rollout_corr/ppl_ratio:np.float64(1.0003962186165154) - rollout_corr/chi2_token:np.float64(0.001294472487643361) - rollout_corr/chi2_seq:np.float64(1.0754117609467357) - actor/pg_loss:np.float64(-0.00010975566692650318) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00020725219656014815) - actor/ppo_kl:np.float64(0.00011342834537941826) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2514360435307026) - perf/mfu/actor:np.float64(0.10841431290124035) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.30791854858398) - actor/lr:np.float64(1e-06) - training/global_step:17 - training/epoch:0 - critic/score/mean:0.70703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.70703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.012801550328731537 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.012801550328731537 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:604.78515625 - response_length/max:2740.0 - response_length/min:90.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:604.78515625 - response_length_non_aborted/max:2740.0 - response_length_non_aborted/min:90.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.34375 - prompt_length/max:375.0 - prompt_length/min:160.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:6.358139216899872e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.0903181321918964) - timing_s/agent_loop/generate_sequences/max:np.float64(20.050666142255068) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.366174238610256) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(20.050666142255068) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:219 - timing_s/agent_loop/slowest/response_length:2740 - timing_s/gen:27.158069726079702 - timing_s/reward:0.08344477787613869 - timing_s/old_log_prob:3.2945270854979753 - timing_s/adv:0.13789121992886066 - timing_s/update_actor:13.660845940932631 - timing_s/step:44.418967282399535 - timing_s/stop_profile:2.7371570467948914e-05 - timing_per_token_ms/gen:0.17541139819848023 - timing_per_token_ms/update_actor:0.06125692658562045 - timing_per_token_ms/adv:0.0006183213230356652 - perf/total_num_tokens:223009 - perf/time_per_step:44.418967282399535 - perf/throughput:627.5725597755077
(TaskRunner pid=2281424) Training Progress: 17%|█▋ | 17/100 [14:27<1:01:48, 44.68s/it]
(TaskRunner pid=2281424) step:18 - global_seqlen/min:18777 - global_seqlen/max:44614 - global_seqlen/minmax_diff:25837 - global_seqlen/balanced_min:32860 - global_seqlen/balanced_max:32946 - global_seqlen/mean:32927.0 - actor/entropy:0.2818203866481781 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2900807857513428 - training/rollout_probs_diff_mean:0.002938726684078574 - training/rollout_probs_diff_std:0.00846165232360363 - training/rollout_actor_probs_pearson_corr:0.9994096159934998 - rollout_corr/rollout_is_mean:1.0000146627426147 - rollout_corr/rollout_is_ratio_fraction_high:5.105583568365546e-06 - rollout_corr/rollout_is_ratio_fraction_low:2.552791738708038e-05 - rollout_corr/rollout_is_max:2.0839157104492188 - rollout_corr/rollout_is_min:0.4158320426940918 - rollout_corr/rollout_is_std:0.02699493244290352 - rollout_corr/rollout_is_eff_sample_size:0.9992719233041721 - rollout_corr/rollout_is_seq_mean:1.0000157356262207 - rollout_corr/rollout_is_seq_std:0.0014222179306671023 - rollout_corr/rollout_is_seq_max:1.0088167190551758 - rollout_corr/rollout_is_seq_min:0.995079755783081 - rollout_corr/rollout_is_seq_max_deviation:0.008816719055175781 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.289564796257764) - rollout_corr/training_log_ppl:np.float64(0.2442400810541585) - rollout_corr/kl:np.float64(0.0005266999699861663) - rollout_corr/k3_kl:np.float64(0.0004649700730112727) - rollout_corr/rollout_ppl:np.float64(1.2888782764784992) - rollout_corr/rollout_log_ppl:np.float64(0.24371338007040322) - rollout_corr/log_ppl_diff:np.float64(0.0005267009837552905) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011411418090574443) - rollout_corr/log_ppl_diff_max:np.float64(0.010496377944946289) - rollout_corr/log_ppl_diff_min:np.float64(-0.007471442222595215) - rollout_corr/ppl_ratio:np.float64(1.0005280999466777) - rollout_corr/chi2_token:np.float64(0.0008352911099791527) - rollout_corr/chi2_seq:np.float64(0.4274897330906242) - actor/pg_loss:np.float64(9.232200682163239e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002937083892220471) - actor/ppo_kl:np.float64(0.00015827277887936475) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1933615617454052) - perf/mfu/actor:np.float64(0.12426924001280201) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.3476791381836) - actor/lr:np.float64(1e-06) - training/global_step:18 - training/epoch:0 - critic/score/mean:0.41015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.41015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007050810847431421 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007050810847431421 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:765.09375 - response_length/max:3008.0 - response_length/min:78.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:765.09375 - response_length_non_aborted/max:3008.0 - response_length_non_aborted/min:78.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.875 - prompt_length/max:349.0 - prompt_length/min:181.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.749613046646118e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(0.9996219798922539) - timing_s/agent_loop/generate_sequences/max:np.float64(21.299407305195928) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.81618346637697) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(21.299407305195928) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:330 - timing_s/agent_loop/slowest/response_length:3008 - timing_s/gen:28.408122271299362 - timing_s/reward:0.07211213000118732 - timing_s/old_log_prob:3.439273815602064 - timing_s/adv:0.14084554836153984 - timing_s/update_actor:14.02409834600985 - timing_s/step:46.15811971202493 - timing_s/stop_profile:0.0001232251524925232 - timing_per_token_ms/gen:0.14504003937068252 - timing_per_token_ms/update_actor:0.05323935655392934 - timing_per_token_ms/adv:0.0005346886611350102 - perf/total_num_tokens:263416 - perf/time_per_step:46.15811971202493 - perf/throughput:713.352281363012
(TaskRunner pid=2281424) Training Progress: 18%|█▊ | 18/100 [15:13<1:01:41, 45.14s/it]
(TaskRunner pid=2281424) step:19 - global_seqlen/min:21194 - global_seqlen/max:44487 - global_seqlen/minmax_diff:23293 - global_seqlen/balanced_min:30623 - global_seqlen/balanced_max:30828 - global_seqlen/mean:30657.0 - actor/entropy:0.2834802567958832 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35189008712768555 - training/rollout_probs_diff_mean:0.002977204043418169 - training/rollout_probs_diff_std:0.00879612471908331 - training/rollout_actor_probs_pearson_corr:0.9994228482246399 - rollout_corr/rollout_is_mean:1.0000439882278442 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.2517451725434512e-05 - rollout_corr/rollout_is_max:1.6279520988464355 - rollout_corr/rollout_is_min:0.2959691882133484 - rollout_corr/rollout_is_std:0.027716144919395447 - rollout_corr/rollout_is_eff_sample_size:0.9992325240192108 - rollout_corr/rollout_is_seq_mean:1.000134825706482 - rollout_corr/rollout_is_seq_std:0.0016275998204946518 - rollout_corr/rollout_is_seq_max:1.009211540222168 - rollout_corr/rollout_is_seq_min:0.9948177337646484 - rollout_corr/rollout_is_seq_max_deviation:0.009211540222167969 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2855765600688756) - rollout_corr/training_log_ppl:np.float64(0.23977286802255549) - rollout_corr/kl:np.float64(0.0005027599546281181) - rollout_corr/k3_kl:np.float64(0.0004755220272727456) - rollout_corr/rollout_ppl:np.float64(1.2849308247677982) - rollout_corr/rollout_log_ppl:np.float64(0.2392701047792798) - rollout_corr/log_ppl_diff:np.float64(0.0005027632432756945) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011823297390947118) - rollout_corr/log_ppl_diff_max:np.float64(0.009016215801239014) - rollout_corr/log_ppl_diff_min:np.float64(-0.008047908544540405) - rollout_corr/ppl_ratio:np.float64(1.0005042462144047) - rollout_corr/chi2_token:np.float64(0.0009039849974215031) - rollout_corr/chi2_seq:np.float64(0.45300188730470836) - actor/pg_loss:np.float64(8.254800923168659e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003364918597981159) - actor/ppo_kl:np.float64(0.0002041434732245051) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.19276084750890732) - perf/mfu/actor:np.float64(0.11752787272188393) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.36764144897461) - actor/lr:np.float64(1e-06) - training/global_step:19 - training/epoch:0 - critic/score/mean:0.71875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0269034281373024 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0269034281373024 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:693.90625 - response_length/max:3216.0 - response_length/min:101.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:693.90625 - response_length_non_aborted/max:3216.0 - response_length_non_aborted/min:101.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:264.125 - prompt_length/max:339.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013180077075958252 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2591454926878214) - timing_s/agent_loop/generate_sequences/max:np.float64(22.810867067426443) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.210481837464613) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(22.810867067426443) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:278 - timing_s/agent_loop/slowest/response_length:3216 - timing_s/gen:29.960727905854583 - timing_s/reward:0.07135201431810856 - timing_s/old_log_prob:3.375285627320409 - timing_s/adv:0.1431940197944641 - timing_s/update_actor:13.87465051561594 - timing_s/step:47.520265359431505 - timing_s/stop_profile:0.00011952407658100128 - timing_per_token_ms/gen:0.16865980581994247 - timing_per_token_ms/update_actor:0.05657211450735534 - timing_per_token_ms/adv:0.0005838553176862712 - perf/total_num_tokens:245256 - perf/time_per_step:47.520265359431505 - perf/throughput:645.1352863482149
(TaskRunner pid=2281424) Training Progress: 19%|█▉ | 19/100 [16:01<1:01:55, 45.87s/it]
(TaskRunner pid=2281424) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 20}
(TaskRunner pid=2281424) validation generation end
(TaskRunner pid=2281424) [prompt] system
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2281424) <reasoning>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </reasoning>
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) A
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) user
(TaskRunner pid=2281424) A charged particle produces an electric field with a magnitude of $2.0 \mathrm{~N} / \mathrm{C}$ at a point that is $50 \mathrm{~cm}$ away from the particle. What is the magnitude of the particle's charge? Recall that the electric field $E$ due to a point charge $q$ at a distance $r$ is given by $E = k\frac{q}{r^2}$, where $k = 8.99 \times 10^9 \mathrm{N} \mathrm{m}^2 / \mathrm{C}^2$.
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) A: 50 pC
(TaskRunner pid=2281424) B: 56 pC
(TaskRunner pid=2281424) C: 60 pC
(TaskRunner pid=2281424) D: 64 pC
(TaskRunner pid=2281424) Please reason step by step.
(TaskRunner pid=2281424) assistant
(TaskRunner pid=2281424) <think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) </think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) [response] <reasoning>
(TaskRunner pid=2281424) We are given the electric field $ E = 2.0 \, \mathrm{N/C} $, the distance $ r = 50 \, \mathrm{cm} = 0.50 \, \mathrm{m} $, and the constant $ k = 8.99 \times 10^9 \, \mathrm{N \cdot m^2/C^2} $. The electric field due to a point charge is given by:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) E = k \frac{q}{r^2}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) We need to solve for the charge $ q $:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{E \cdot r^2}{k}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Substituting the given values:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{2.0 \cdot (0.50)^2}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{2.0 \cdot 0.25}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{0.50}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 5.56 \times 10^{-11} \, \mathrm{C}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Convert to picocoulombs ($ 1 \, \mathrm{pC} = 10^{-12} \, \mathrm{C} $):
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 55.6 \, \mathrm{pC}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) The closest option is:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) B
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424) [ground_truth] B
(TaskRunner pid=2281424) [score] 1.0
(TaskRunner pid=2281424) [acc] 1.0
(TaskRunner pid=2281424) [pred] B
(TaskRunner pid=2281424) [incorrect_format] 1
(TaskRunner pid=2281424) [feedback]
(TaskRunner pid=2281424) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_20
(TaskRunner pid=2281424) step:20 - global_seqlen/min:19499 - global_seqlen/max:36025 - global_seqlen/minmax_diff:16526 - global_seqlen/balanced_min:27704 - global_seqlen/balanced_max:28173 - global_seqlen/mean:27765.0 - actor/entropy:0.280457079410553 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24135375022888184 - training/rollout_probs_diff_mean:0.003101746318861842 - training/rollout_probs_diff_std:0.008900674059987068 - training/rollout_actor_probs_pearson_corr:0.999406099319458 - rollout_corr/rollout_is_mean:0.9998956918716431 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.9279214029666036e-05 - rollout_corr/rollout_is_max:1.8279300928115845 - rollout_corr/rollout_is_min:0.40598663687705994 - rollout_corr/rollout_is_std:0.028816867619752884 - rollout_corr/rollout_is_eff_sample_size:0.9991700390777238 - rollout_corr/rollout_is_seq_mean:0.999803900718689 - rollout_corr/rollout_is_seq_std:0.001518259639851749 - rollout_corr/rollout_is_seq_max:1.0038286447525024 - rollout_corr/rollout_is_seq_min:0.9914582967758179 - rollout_corr/rollout_is_seq_max_deviation:0.008541703224182129 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2934490512125194) - rollout_corr/training_log_ppl:np.float64(0.24451396349468268) - rollout_corr/kl:np.float64(0.0004959776629758217) - rollout_corr/k3_kl:np.float64(0.0004771725971863816) - rollout_corr/rollout_ppl:np.float64(1.2928053527139127) - rollout_corr/rollout_log_ppl:np.float64(0.2440179859258933) - rollout_corr/log_ppl_diff:np.float64(0.0004959775687893853) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011907425214303657) - rollout_corr/log_ppl_diff_max:np.float64(0.009751290082931519) - rollout_corr/log_ppl_diff_min:np.float64(-0.004419475793838501) - rollout_corr/ppl_ratio:np.float64(1.0004974079784006) - rollout_corr/chi2_token:np.float64(0.0009594245348125696) - rollout_corr/chi2_seq:np.float64(0.41463141702115536) - actor/pg_loss:np.float64(1.230929046869278e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00021071931905680685) - actor/ppo_kl:np.float64(-0.0001457839747551759) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1746744140982628) - perf/mfu/actor:np.float64(0.10426063797607768) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(103.34674835205078) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.59921875) - val-aux/sciknoweval/reward/std@16:np.float64(0.20532643626967628) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.684225) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.18261389709388515) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.514475) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.1643688800388951) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.60015) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.2055792779718459) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7575125) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.14221272724332604) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.4511375) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.11570816324542939) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6215249999999999) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.17291949688631747) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8132375) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.09937488352535821) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.4064375) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.07134373339436899) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6354124999999999) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.1400502136179348) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8518374999999999) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.06232857449090168) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.37925) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.04032492559816903) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6459375) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.10760674780586772) - val-aux/sciknoweval/score/mean@16:np.float64(0.59921875) - val-aux/sciknoweval/score/std@16:np.float64(0.20532643626967628) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.684225) - val-aux/sciknoweval/score/best@2/std:np.float64(0.18261389709388515) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.514475) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.1643688800388951) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.60015) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.2055792779718459) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7575125) - val-aux/sciknoweval/score/best@4/std:np.float64(0.14221272724332604) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.4511375) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.11570816324542939) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6215249999999999) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.17291949688631747) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8132375) - val-aux/sciknoweval/score/best@8/std:np.float64(0.09937488352535821) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.4064375) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.07134373339436899) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6354124999999999) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.1400502136179348) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8518374999999999) - val-aux/sciknoweval/score/best@16/std:np.float64(0.06232857449090168) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.37925) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.04032492559816903) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6459375) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.10760674780586772) - val-core/sciknoweval/acc/mean@16:np.float64(0.59921875) - val-aux/sciknoweval/acc/std@16:np.float64(0.20532643626967628) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.684225) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.18261389709388515) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.514475) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.1643688800388951) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.60015) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.2055792779718459) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7575125) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.14221272724332604) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.4511375) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.11570816324542939) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6215249999999999) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.17291949688631747) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8132375) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.09937488352535821) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.4064375) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.07134373339436899) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6354124999999999) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.1400502136179348) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8518374999999999) - val-core/sciknoweval/acc/best@16/std:np.float64(0.06232857449090168) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.37925) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.04032492559816903) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6459375) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.10760674780586772) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9578125) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.08158753118437541) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9860749999999999) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.03951222065525328) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9282874999999999) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.0973249508439899) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9574) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.08213995093780886) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9966125) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.012924820131308547) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.8906499999999999) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.10540656140217979) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.974175) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.05962872167707507) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(0.999525) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0032780861692795426) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.8437625000000001) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.10035091942020005) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9859375) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.03701549481184431) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(0.9999874999999999) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0003950870157319778) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.7979749999999999) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.08009154869844874) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9934750000000001) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.018079184472962695) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:20 - training/epoch:0 - critic/score/mean:0.69921875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.69921875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.011969178915023804 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:0.011969178915023804 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:607.84375 - response_length/max:2668.0 - response_length/min:136.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:607.84375 - response_length_non_aborted/max:2668.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:259.8125 - prompt_length/max:355.0 - prompt_length/min:184.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.631311357021332e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.728585135191679) - timing_s/agent_loop/generate_sequences/max:np.float64(19.024446960538626) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.399849012530467) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(19.024446960538626) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:341 - timing_s/agent_loop/slowest/response_length:2668 - timing_s/gen:26.06310199201107 - timing_s/reward:0.06901264749467373 - timing_s/old_log_prob:3.238787254318595 - timing_s/adv:0.17001512832939625 - timing_s/update_actor:14.12741300649941 - timing_s/step:43.75453623570502 - timing_s/testing:145.7744435351342 - timing_s/save_checkpoint:13.140905741602182 - timing_s/stop_profile:0.00017121247947216034 - timing_per_token_ms/gen:0.16749204405950252 - timing_per_token_ms/update_actor:0.06360261573248428 - timing_per_token_ms/adv:0.000765420170760833 - perf/total_num_tokens:222120 - perf/time_per_step:43.75453623570502 - perf/throughput:634.5627765411652
(TaskRunner pid=2281424) Training Progress: 20%|██ | 20/100 [19:23<2:03:58, 92.98s/it]
(TaskRunner pid=2281424) step:21 - global_seqlen/min:25301 - global_seqlen/max:34398 - global_seqlen/minmax_diff:9097 - global_seqlen/balanced_min:29031 - global_seqlen/balanced_max:29324 - global_seqlen/mean:29106.375 - actor/entropy:0.24065154790878296 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.37683379650115967 - training/rollout_probs_diff_mean:0.002658553421497345 - training/rollout_probs_diff_std:0.008330690674483776 - training/rollout_actor_probs_pearson_corr:0.9994138479232788 - rollout_corr/rollout_is_mean:1.0000982284545898 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.2177079042885453e-05 - rollout_corr/rollout_is_max:1.7682775259017944 - rollout_corr/rollout_is_min:0.3936323821544647 - rollout_corr/rollout_is_std:0.02563127689063549 - rollout_corr/rollout_is_eff_sample_size:0.9993435879743411 - rollout_corr/rollout_is_seq_mean:1.0001851320266724 - rollout_corr/rollout_is_seq_std:0.0012425798922777176 - rollout_corr/rollout_is_seq_max:1.0056188106536865 - rollout_corr/rollout_is_seq_min:0.9965409636497498 - rollout_corr/rollout_is_seq_max_deviation:0.0056188106536865234 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2498526307754219) - rollout_corr/training_log_ppl:np.float64(0.21081234834855422) - rollout_corr/kl:np.float64(0.00032340780622153176) - rollout_corr/k3_kl:np.float64(0.0003890872453951033) - rollout_corr/rollout_ppl:np.float64(1.2494138525798917) - rollout_corr/rollout_log_ppl:np.float64(0.21048893986153416) - rollout_corr/log_ppl_diff:np.float64(0.0003234084870200604) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010021366470027715) - rollout_corr/log_ppl_diff_max:np.float64(0.004034250974655151) - rollout_corr/log_ppl_diff_min:np.float64(-0.004479348659515381) - rollout_corr/ppl_ratio:np.float64(1.0003242590464652) - rollout_corr/chi2_token:np.float64(0.0009132004342973232) - rollout_corr/chi2_seq:np.float64(0.7536862585693598) - actor/pg_loss:np.float64(3.263808321207762e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.000137664294015849) - actor/ppo_kl:np.float64(0.00017732822421123728) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.13653844129294157) - perf/mfu/actor:np.float64(0.11284239253639325) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(104.92810821533203) - actor/lr:np.float64(1e-06) - training/global_step:21 - training/epoch:0 - critic/score/mean:0.6640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0021431660279631615 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0021431660279631615 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:641.57421875 - response_length/max:3017.0 - response_length/min:180.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:641.57421875 - response_length_non_aborted/max:3017.0 - response_length_non_aborted/min:180.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:268.0 - prompt_length/max:364.0 - prompt_length/min:195.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.9451907873153687e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2093537878245115) - timing_s/agent_loop/generate_sequences/max:np.float64(21.849115535616875) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.7584272636668175) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(21.849115535616875) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:364 - timing_s/agent_loop/slowest/response_length:3017 - timing_s/gen:27.816161528229713 - timing_s/reward:0.0652250312268734 - timing_s/old_log_prob:3.518872758373618 - timing_s/adv:0.15252801775932312 - timing_s/update_actor:13.777937777340412 - timing_s/step:45.40992152877152 - timing_s/stop_profile:0.00012294389307498932 - timing_per_token_ms/gen:0.16935979937184364 - timing_per_token_ms/update_actor:0.059170618882205414 - timing_per_token_ms/adv:0.0006550455774693822 - perf/total_num_tokens:232851 - perf/time_per_step:45.40992152877152 - perf/throughput:640.9695066651532
(TaskRunner pid=2281424) Training Progress: 21%|██ | 21/100 [20:09<1:43:38, 78.71s/it]
(TaskRunner pid=2281424) step:22 - global_seqlen/min:20123 - global_seqlen/max:45886 - global_seqlen/minmax_diff:25763 - global_seqlen/balanced_min:32892 - global_seqlen/balanced_max:32946 - global_seqlen/mean:32919.625 - actor/entropy:0.2954756021499634 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5972118377685547 - training/rollout_probs_diff_mean:0.0029836043249815702 - training/rollout_probs_diff_std:0.008759740740060806 - training/rollout_actor_probs_pearson_corr:0.9994177222251892 - rollout_corr/rollout_is_mean:1.000027060508728 - rollout_corr/rollout_is_ratio_fraction_high:1.0150790330953896e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.5226184586936142e-05 - rollout_corr/rollout_is_max:5.339309215545654 - rollout_corr/rollout_is_min:0.3289409577846527 - rollout_corr/rollout_is_std:0.027539236471056938 - rollout_corr/rollout_is_eff_sample_size:0.999242284277643 - rollout_corr/rollout_is_seq_mean:1.000131607055664 - rollout_corr/rollout_is_seq_std:0.0019331261282786727 - rollout_corr/rollout_is_seq_max:1.0212081670761108 - rollout_corr/rollout_is_seq_min:0.990929365158081 - rollout_corr/rollout_is_seq_max_deviation:0.02120816707611084 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2931092074140906) - rollout_corr/training_log_ppl:np.float64(0.24355409591225907) - rollout_corr/kl:np.float64(0.0005864815489502462) - rollout_corr/k3_kl:np.float64(0.0005580453379678829) - rollout_corr/rollout_ppl:np.float64(1.292321430053562) - rollout_corr/rollout_log_ppl:np.float64(0.24296761279401835) - rollout_corr/log_ppl_diff:np.float64(0.0005864831182407215) - rollout_corr/log_ppl_abs_diff:np.float64(0.001289499065023847) - rollout_corr/log_ppl_diff_max:np.float64(0.019712358713150024) - rollout_corr/log_ppl_diff_min:np.float64(-0.007148668169975281) - rollout_corr/ppl_ratio:np.float64(1.0005892012268305) - rollout_corr/chi2_token:np.float64(0.0012211541179567575) - rollout_corr/chi2_seq:np.float64(0.6782732140272856) - actor/pg_loss:np.float64(0.00016800372395664454) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002511092553731942) - actor/ppo_kl:np.float64(0.00027240615360524245) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.29514269530773163) - perf/mfu/actor:np.float64(0.12364022737507745) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(104.99796676635742) - actor/lr:np.float64(1e-06) - training/global_step:22 - training/epoch:0 - critic/score/mean:0.71484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.005714894738048315 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.005714894738048315 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:769.64453125 - response_length/max:2870.0 - response_length/min:138.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:769.64453125 - response_length_non_aborted/max:2870.0 - response_length_non_aborted/min:138.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:259.09375 - prompt_length/max:375.0 - prompt_length/min:197.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.717978537082672e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7690473347902298) - timing_s/agent_loop/generate_sequences/max:np.float64(21.679362626746297) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.051158084097551) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(21.679362626746297) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:197 - timing_s/agent_loop/slowest/response_length:2870 - timing_s/gen:28.594687478616834 - timing_s/reward:0.07660721056163311 - timing_s/old_log_prob:3.3768766596913338 - timing_s/adv:0.14736606925725937 - timing_s/update_actor:14.125624496489763 - timing_s/step:46.4090982247144 - timing_s/stop_profile:0.00013731606304645538 - timing_per_token_ms/gen:0.1451293336443713 - timing_per_token_ms/update_actor:0.0536367914902196 - timing_per_token_ms/adv:0.0005595676942601084 - perf/total_num_tokens:263357 - perf/time_per_step:46.4090982247144 - perf/throughput:709.3355884788383
(TaskRunner pid=2281424) Training Progress: 22%|██▏ | 22/100 [20:55<1:29:44, 69.04s/it]
(TaskRunner pid=2281424) step:23 - global_seqlen/min:22934 - global_seqlen/max:53809 - global_seqlen/minmax_diff:30875 - global_seqlen/balanced_min:39243 - global_seqlen/balanced_max:39551 - global_seqlen/mean:39306.5 - actor/entropy:0.33397042751312256 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.4463748633861542 - training/rollout_probs_diff_mean:0.003143494250252843 - training/rollout_probs_diff_std:0.008556975051760674 - training/rollout_actor_probs_pearson_corr:0.9994155764579773 - rollout_corr/rollout_is_mean:0.9999798536300659 - rollout_corr/rollout_is_ratio_fraction_high:8.167132364178542e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.6334264728357084e-05 - rollout_corr/rollout_is_max:2.7373318672180176 - rollout_corr/rollout_is_min:0.23669613897800446 - rollout_corr/rollout_is_std:0.028452597558498383 - rollout_corr/rollout_is_eff_sample_size:0.9991909855397003 - rollout_corr/rollout_is_seq_mean:0.9999604821205139 - rollout_corr/rollout_is_seq_std:0.001166414120234549 - rollout_corr/rollout_is_seq_max:1.0046206712722778 - rollout_corr/rollout_is_seq_min:0.9961167573928833 - rollout_corr/rollout_is_seq_max_deviation:0.004620671272277832 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.385243427939713) - rollout_corr/training_log_ppl:np.float64(0.3101144570391625) - rollout_corr/kl:np.float64(0.0005338342346767888) - rollout_corr/k3_kl:np.float64(0.000465180856878078) - rollout_corr/rollout_ppl:np.float64(1.3844622108153999) - rollout_corr/rollout_log_ppl:np.float64(0.30958062058198266) - rollout_corr/log_ppl_diff:np.float64(0.0005338364571798593) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009809565672185272) - rollout_corr/log_ppl_diff_max:np.float64(0.006667211651802063) - rollout_corr/log_ppl_diff_min:np.float64(-0.0032280683517456055) - rollout_corr/ppl_ratio:np.float64(1.0005347987171263) - rollout_corr/chi2_token:np.float64(0.000802481546998024) - rollout_corr/chi2_seq:np.float64(0.3838173961266875) - actor/pg_loss:np.float64(6.97027426213026e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00030751674410112173) - actor/ppo_kl:np.float64(6.243631206404032e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.20673737302422523) - perf/mfu/actor:np.float64(0.1370845535950862) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.68359375) - perf/cpu_memory_used_gb:np.float64(105.6169319152832) - actor/lr:np.float64(1e-06) - training/global_step:23 - training/epoch:1 - critic/score/mean:0.5 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.5 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0067174662835896015 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0067174662835896015 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:956.578125 - response_length/max:4024.0 - response_length/min:152.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:956.578125 - response_length_non_aborted/max:4024.0 - response_length_non_aborted/min:152.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:271.75 - prompt_length/max:375.0 - prompt_length/min:203.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00028231181204319 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8497151639312506) - timing_s/agent_loop/generate_sequences/max:np.float64(29.862245429307222) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.941006362365442) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(29.862245429307222) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:311 - timing_s/agent_loop/slowest/response_length:4024 - timing_s/gen:36.85486473329365 - timing_s/reward:0.08099697157740593 - timing_s/old_log_prob:3.847181210294366 - timing_s/adv:0.1430294755846262 - timing_s/update_actor:15.214192001149058 - timing_s/step:56.236885111778975 - timing_s/stop_profile:0.00012627430260181427 - timing_per_token_ms/gen:0.1504992761196879 - timing_per_token_ms/update_actor:0.04838319362303009 - timing_per_token_ms/adv:0.00045485312729645925 - perf/total_num_tokens:314452 - perf/time_per_step:56.236885111778975 - perf/throughput:698.9451837859195
(TaskRunner pid=2281424) Training Progress: 23%|██▎ | 23/100 [21:52<1:23:49, 65.31s/it]
(TaskRunner pid=2281424) step:24 - global_seqlen/min:21247 - global_seqlen/max:40900 - global_seqlen/minmax_diff:19653 - global_seqlen/balanced_min:30630 - global_seqlen/balanced_max:32071 - global_seqlen/mean:30857.125 - actor/entropy:0.31415796279907227 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3094596862792969 - training/rollout_probs_diff_mean:0.003211991162970662 - training/rollout_probs_diff_std:0.008898185566067696 - training/rollout_actor_probs_pearson_corr:0.9993066191673279 - rollout_corr/rollout_is_mean:0.9999200105667114 - rollout_corr/rollout_is_ratio_fraction_high:5.59005866307416e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.59005866307416e-06 - rollout_corr/rollout_is_max:2.404808521270752 - rollout_corr/rollout_is_min:0.19617219269275665 - rollout_corr/rollout_is_std:0.02825077250599861 - rollout_corr/rollout_is_eff_sample_size:0.9992022922335758 - rollout_corr/rollout_is_seq_mean:0.9998365640640259 - rollout_corr/rollout_is_seq_std:0.001564831123687327 - rollout_corr/rollout_is_seq_max:1.0052552223205566 - rollout_corr/rollout_is_seq_min:0.992292582988739 - rollout_corr/rollout_is_seq_max_deviation:0.007707417011260986 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.303541206754744) - rollout_corr/training_log_ppl:np.float64(0.25006278717773966) - rollout_corr/kl:np.float64(0.0005476736968059726) - rollout_corr/k3_kl:np.float64(0.00043206283888252983) - rollout_corr/rollout_ppl:np.float64(1.3028357764706016) - rollout_corr/rollout_log_ppl:np.float64(0.24951511321705766) - rollout_corr/log_ppl_diff:np.float64(0.0005476739606820047) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011344742961227894) - rollout_corr/log_ppl_diff_max:np.float64(0.008412361145019531) - rollout_corr/log_ppl_diff_min:np.float64(-0.00302673876285553) - rollout_corr/ppl_ratio:np.float64(1.0005489422474056) - rollout_corr/chi2_token:np.float64(0.0006379920523613691) - rollout_corr/chi2_seq:np.float64(0.4201601871754974) - actor/pg_loss:np.float64(-3.0018971301615238e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(6.42312039076387e-05) - actor/ppo_kl:np.float64(-2.9987399832176465e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1152929738163948) - perf/mfu/actor:np.float64(0.11669484239312708) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(138.80078125) - perf/cpu_memory_used_gb:np.float64(105.66312026977539) - actor/lr:np.float64(1e-06) - training/global_step:24 - training/epoch:1 - critic/score/mean:0.74609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.74609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.013332289643585682 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:-0.013332289643585682 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:698.78515625 - response_length/max:4217.0 - response_length/min:85.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:698.78515625 - response_length_non_aborted/max:4217.0 - response_length_non_aborted/min:85.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.5 - prompt_length/max:383.0 - prompt_length/min:161.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001115109771490097 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1027332954108715) - timing_s/agent_loop/generate_sequences/max:np.float64(28.570936573669314) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.08747353394574) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(28.570936573669314) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:353 - timing_s/agent_loop/slowest/response_length:4217 - timing_s/gen:35.1314688809216 - timing_s/reward:0.07224285788834095 - timing_s/old_log_prob:3.3607450742274523 - timing_s/adv:0.1696465015411377 - timing_s/update_actor:13.863193551078439 - timing_s/step:52.68583432585001 - timing_s/stop_profile:0.00012148357927799225 - timing_per_token_ms/gen:0.19638697114367906 - timing_per_token_ms/update_actor:0.05615880267149985 - timing_per_token_ms/adv:0.0006872258090357482 - perf/total_num_tokens:246857 - perf/time_per_step:52.68583432585001 - perf/throughput:585.6816238147742
(TaskRunner pid=2281424) Training Progress: 24%|██▍ | 24/100 [22:45<1:17:57, 61.54s/it]
(TaskRunner pid=2281424) step:25 - global_seqlen/min:23246 - global_seqlen/max:52171 - global_seqlen/minmax_diff:28925 - global_seqlen/balanced_min:34912 - global_seqlen/balanced_max:36421 - global_seqlen/mean:35164.75 - actor/entropy:0.35677048563957214 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5146784782409668 - training/rollout_probs_diff_mean:0.0033477256074547768 - training/rollout_probs_diff_std:0.00895012728869915 - training/rollout_actor_probs_pearson_corr:0.9994227886199951 - rollout_corr/rollout_is_mean:0.9999942779541016 - rollout_corr/rollout_is_ratio_fraction_high:1.4294563698058482e-05 - rollout_corr/rollout_is_ratio_fraction_low:1.4294563698058482e-05 - rollout_corr/rollout_is_max:2.550842046737671 - rollout_corr/rollout_is_min:0.2724951505661011 - rollout_corr/rollout_is_std:0.02962060645222664 - rollout_corr/rollout_is_eff_sample_size:0.9991233887504097 - rollout_corr/rollout_is_seq_mean:1.0000044107437134 - rollout_corr/rollout_is_seq_std:0.001464587403461337 - rollout_corr/rollout_is_seq_max:1.0044803619384766 - rollout_corr/rollout_is_seq_min:0.9913833737373352 - rollout_corr/rollout_is_seq_max_deviation:0.008616626262664795 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3856255286373198) - rollout_corr/training_log_ppl:np.float64(0.3002330707240617) - rollout_corr/kl:np.float64(0.00044336768771691126) - rollout_corr/k3_kl:np.float64(0.0005020772942430085) - rollout_corr/rollout_ppl:np.float64(1.3849827772937715) - rollout_corr/rollout_log_ppl:np.float64(0.2997897020104574) - rollout_corr/log_ppl_diff:np.float64(0.0004433687136042863) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010328854841645807) - rollout_corr/log_ppl_diff_max:np.float64(0.010214567184448242) - rollout_corr/log_ppl_diff_min:np.float64(-0.005682602524757385) - rollout_corr/ppl_ratio:np.float64(1.000444594072178) - rollout_corr/chi2_token:np.float64(0.0011366589460521936) - rollout_corr/chi2_seq:np.float64(0.17277534841559827) - actor/pg_loss:np.float64(-0.00010206212755292654) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0003034779521726705) - actor/ppo_kl:np.float64(-3.511990449567293e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1835123896598816) - perf/mfu/actor:np.float64(0.1283748813468745) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(139.953125) - perf/cpu_memory_used_gb:np.float64(105.58767604827881) - actor/lr:np.float64(1e-06) - training/global_step:25 - training/epoch:1 - critic/score/mean:0.71875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.025912469252943993 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.025912469252943993 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:819.8046875 - response_length/max:5123.0 - response_length/min:93.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:819.8046875 - response_length_non_aborted/max:5123.0 - response_length_non_aborted/min:93.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:279.09375 - prompt_length/max:365.0 - prompt_length/min:189.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001536533236503601 - timing_s/agent_loop/generate_sequences/min:np.float64(1.2272972408682108) - timing_s/agent_loop/generate_sequences/max:np.float64(33.647482462227345) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.377409354601696) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(33.647482462227345) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:251 - timing_s/agent_loop/slowest/response_length:5123 - timing_s/gen:40.239728681743145 - timing_s/reward:0.07882356271147728 - timing_s/old_log_prob:3.5549416448920965 - timing_s/adv:0.13953095860779285 - timing_s/update_actor:14.543323775753379 - timing_s/step:58.64243852160871 - timing_s/stop_profile:0.00014534033834934235 - timing_per_token_ms/gen:0.1917364496199702 - timing_per_token_ms/update_actor:0.05169709643802878 - timing_per_token_ms/adv:0.0004959901556522969 - perf/total_num_tokens:281318 - perf/time_per_step:58.64243852160871 - perf/throughput:599.646789705759
(TaskRunner pid=2281424) Training Progress: 25%|██▌ | 25/100 [23:43<1:15:51, 60.69s/it]
(TaskRunner pid=2281424) step:26 - global_seqlen/min:24712 - global_seqlen/max:39123 - global_seqlen/minmax_diff:14411 - global_seqlen/balanced_min:30297 - global_seqlen/balanced_max:30365 - global_seqlen/mean:30344.375 - actor/entropy:0.2876966595649719 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.20822957158088684 - training/rollout_probs_diff_mean:0.0029764524661004543 - training/rollout_probs_diff_std:0.008548058569431305 - training/rollout_actor_probs_pearson_corr:0.9995021820068359 - rollout_corr/rollout_is_mean:1.0000873804092407 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.9248120784759521 - rollout_corr/rollout_is_min:0.5073025822639465 - rollout_corr/rollout_is_std:0.02774408645927906 - rollout_corr/rollout_is_eff_sample_size:0.9992309766786769 - rollout_corr/rollout_is_seq_mean:1.0000227689743042 - rollout_corr/rollout_is_seq_std:0.0015661150682717562 - rollout_corr/rollout_is_seq_max:1.0046665668487549 - rollout_corr/rollout_is_seq_min:0.9934303760528564 - rollout_corr/rollout_is_seq_max_deviation:0.006569623947143555 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3186388644389808) - rollout_corr/training_log_ppl:np.float64(0.2604680637450656) - rollout_corr/kl:np.float64(0.0005275833006166408) - rollout_corr/k3_kl:np.float64(0.0005207556534685409) - rollout_corr/rollout_ppl:np.float64(1.3179062088020146) - rollout_corr/rollout_log_ppl:np.float64(0.259940478586941) - rollout_corr/log_ppl_diff:np.float64(0.0005275851581245661) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012775544892065227) - rollout_corr/log_ppl_diff_max:np.float64(0.009042024612426758) - rollout_corr/log_ppl_diff_min:np.float64(-0.0037787258625030518) - rollout_corr/ppl_ratio:np.float64(1.0005293833091855) - rollout_corr/chi2_token:np.float64(0.0010106251575052738) - rollout_corr/chi2_seq:np.float64(1.5079653931315988) - actor/pg_loss:np.float64(0.00019484246149659157) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0004149369976857997) - actor/ppo_kl:np.float64(0.00010909117361990184) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.17218420654535294) - perf/mfu/actor:np.float64(0.11675351697015622) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(139.953125) - perf/cpu_memory_used_gb:np.float64(105.56155776977539) - actor/lr:np.float64(1e-06) - training/global_step:26 - training/epoch:1 - critic/score/mean:0.7109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.7109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.007410742342472076 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.007410742342472076 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:688.73046875 - response_length/max:2773.0 - response_length/min:143.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:688.73046875 - response_length_non_aborted/max:2773.0 - response_length_non_aborted/min:143.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:259.53125 - prompt_length/max:363.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011996924877166748 - timing_s/agent_loop/generate_sequences/min:np.float64(1.479305861517787) - timing_s/agent_loop/generate_sequences/max:np.float64(20.65613953024149) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.7965023621873115) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(20.65613953024149) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:295 - timing_s/agent_loop/slowest/response_length:2773 - timing_s/gen:27.100659066811204 - timing_s/reward:0.07199572958052158 - timing_s/old_log_prob:3.321561750024557 - timing_s/adv:0.16026310436427593 - timing_s/update_actor:13.673425955697894 - timing_s/step:44.414213774725795 - timing_s/stop_profile:0.00011827610433101654 - timing_per_token_ms/gen:0.1537059187636401 - timing_per_token_ms/update_actor:0.05632603223702043 - timing_per_token_ms/adv:0.0006601845661851494 - perf/total_num_tokens:242755 - perf/time_per_step:44.414213774725795 - perf/throughput:683.2131523009796
(TaskRunner pid=2281424) Training Progress: 26%|██▌ | 26/100 [24:28<1:08:50, 55.82s/it]
(TaskRunner pid=2281424) step:27 - global_seqlen/min:25977 - global_seqlen/max:48183 - global_seqlen/minmax_diff:22206 - global_seqlen/balanced_min:36093 - global_seqlen/balanced_max:36153 - global_seqlen/mean:36121.125 - actor/entropy:0.30439427495002747 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2310372292995453 - training/rollout_probs_diff_mean:0.002941888989880681 - training/rollout_probs_diff_std:0.008349236100912094 - training/rollout_actor_probs_pearson_corr:0.99952632188797 - rollout_corr/rollout_is_mean:0.9999545216560364 - rollout_corr/rollout_is_ratio_fraction_high:4.5820902414561715e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.3746271179115865e-05 - rollout_corr/rollout_is_max:2.7575740814208984 - rollout_corr/rollout_is_min:0.4973885416984558 - rollout_corr/rollout_is_std:0.027530577033758163 - rollout_corr/rollout_is_eff_sample_size:0.9992425223351087 - rollout_corr/rollout_is_seq_mean:0.9999428987503052 - rollout_corr/rollout_is_seq_std:0.0012769426684826612 - rollout_corr/rollout_is_seq_max:1.004395604133606 - rollout_corr/rollout_is_seq_min:0.9921810030937195 - rollout_corr/rollout_is_seq_max_deviation:0.007818996906280518 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3105780305340886) - rollout_corr/training_log_ppl:np.float64(0.25283126058639027) - rollout_corr/kl:np.float64(0.00047030424940075477) - rollout_corr/k3_kl:np.float64(0.0004522940477897919) - rollout_corr/rollout_ppl:np.float64(1.3099122606217861) - rollout_corr/rollout_log_ppl:np.float64(0.2523609542840859) - rollout_corr/log_ppl_diff:np.float64(0.00047030630230437964) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010534748871577904) - rollout_corr/log_ppl_diff_max:np.float64(0.009107545018196106) - rollout_corr/log_ppl_diff_min:np.float64(-0.00359538197517395) - rollout_corr/ppl_ratio:np.float64(1.0004714394453913) - rollout_corr/chi2_token:np.float64(0.0008890700992196798) - rollout_corr/chi2_seq:np.float64(1.7405338953249156) - actor/pg_loss:np.float64(-9.15073323994875e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002051412569699096) - actor/ppo_kl:np.float64(1.4041290314459332e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.15001486241817474) - perf/mfu/actor:np.float64(0.13349702138291186) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(139.953125) - perf/cpu_memory_used_gb:np.float64(105.56867599487305) - actor/lr:np.float64(1e-06) - training/global_step:27 - training/epoch:1 - critic/score/mean:0.796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.006906355731189251 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.006906355731189251 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:852.50390625 - response_length/max:3307.0 - response_length/min:112.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:852.50390625 - response_length_non_aborted/max:3307.0 - response_length_non_aborted/min:112.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:276.28125 - prompt_length/max:375.0 - prompt_length/min:186.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015197880566120148 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5302176997065544) - timing_s/agent_loop/generate_sequences/max:np.float64(24.74932035431266) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.707113990400103) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(24.74932035431266) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:226 - timing_s/agent_loop/slowest/response_length:3281 - timing_s/gen:31.510395139455795 - timing_s/reward:0.07958226092159748 - timing_s/old_log_prob:3.521709840744734 - timing_s/adv:0.14874472096562386 - timing_s/update_actor:14.358893560245633 - timing_s/step:49.70740628428757 - timing_s/stop_profile:0.00017487630248069763 - timing_per_token_ms/gen:0.144383480370122 - timing_per_token_ms/update_actor:0.049690082881712686 - timing_per_token_ms/adv:0.0005147428304268757 - perf/total_num_tokens:288969 - perf/time_per_step:49.70740628428757 - perf/throughput:726.6749102420544
(TaskRunner pid=2281424) Training Progress: 27%|██▋ | 27/100 [25:18<1:05:42, 54.01s/it]
(TaskRunner pid=2281424) step:28 - global_seqlen/min:21975 - global_seqlen/max:58966 - global_seqlen/minmax_diff:36991 - global_seqlen/balanced_min:38635 - global_seqlen/balanced_max:38827 - global_seqlen/mean:38769.375 - actor/entropy:0.3461316227912903 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.34711864590644836 - training/rollout_probs_diff_mean:0.003255465067923069 - training/rollout_probs_diff_std:0.008680691011250019 - training/rollout_actor_probs_pearson_corr:0.9994996190071106 - rollout_corr/rollout_is_mean:1.00006103515625 - rollout_corr/rollout_is_ratio_fraction_high:8.337119652424008e-06 - rollout_corr/rollout_is_ratio_fraction_low:2.084279913106002e-05 - rollout_corr/rollout_is_max:2.25616192817688 - rollout_corr/rollout_is_min:0.17730121314525604 - rollout_corr/rollout_is_std:0.028798246756196022 - rollout_corr/rollout_is_eff_sample_size:0.9991714672176878 - rollout_corr/rollout_is_seq_mean:1.0000207424163818 - rollout_corr/rollout_is_seq_std:0.0014608351048082113 - rollout_corr/rollout_is_seq_max:1.006380558013916 - rollout_corr/rollout_is_seq_min:0.9955966472625732 - rollout_corr/rollout_is_seq_max_deviation:0.006380558013916016 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3715200861915946) - rollout_corr/training_log_ppl:np.float64(0.2959491069341311) - rollout_corr/kl:np.float64(0.0004920310259990401) - rollout_corr/k3_kl:np.float64(0.000492703045580356) - rollout_corr/rollout_ppl:np.float64(1.3707749480381608) - rollout_corr/rollout_log_ppl:np.float64(0.2954570743313525) - rollout_corr/log_ppl_diff:np.float64(0.0004920326027786359) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011141302093164995) - rollout_corr/log_ppl_diff_max:np.float64(0.00506967306137085) - rollout_corr/log_ppl_diff_min:np.float64(-0.004320159554481506) - rollout_corr/ppl_ratio:np.float64(1.0004931495059282) - rollout_corr/chi2_token:np.float64(0.0010096586775034666) - rollout_corr/chi2_seq:np.float64(1.0986658702604473) - actor/pg_loss:np.float64(-3.765069413930178e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001984241041554924) - actor/ppo_kl:np.float64(4.411382598945579e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1757737323641777) - perf/mfu/actor:np.float64(0.13821070890339499) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(139.953125) - perf/cpu_memory_used_gb:np.float64(105.6130256652832) - actor/lr:np.float64(1e-06) - training/global_step:28 - training/epoch:1 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.01265887450426817 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.01265887450426817 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:937.07421875 - response_length/max:4442.0 - response_length/min:86.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:937.07421875 - response_length_non_aborted/max:4442.0 - response_length_non_aborted/min:86.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:274.46875 - prompt_length/max:363.0 - prompt_length/min:200.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013406015932559967 - timing_s/agent_loop/generate_sequences/min:np.float64(1.040765456855297) - timing_s/agent_loop/generate_sequences/max:np.float64(31.025565603747964) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.124727715730842) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(31.025565603747964) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:340 - timing_s/agent_loop/slowest/response_length:4440 - timing_s/gen:37.81342346780002 - timing_s/reward:0.08299151435494423 - timing_s/old_log_prob:3.631516432389617 - timing_s/adv:0.1339629665017128 - timing_s/update_actor:15.046643575653434 - timing_s/step:56.79524556733668 - timing_s/stop_profile:0.00011165253818035126 - timing_per_token_ms/gen:0.1576275202812945 - timing_per_token_ms/update_actor:0.04851330326982778 - timing_per_token_ms/adv:0.00043192264029827924 - perf/total_num_tokens:310155 - perf/time_per_step:56.79524556733668 - perf/throughput:682.61655729677
(TaskRunner pid=2281424) Training Progress: 28%|██▊ | 28/100 [26:14<1:05:49, 54.86s/it]
(TaskRunner pid=2281424) step:29 - global_seqlen/min:26468 - global_seqlen/max:47121 - global_seqlen/minmax_diff:20653 - global_seqlen/balanced_min:34309 - global_seqlen/balanced_max:34448 - global_seqlen/mean:34354.375 - actor/entropy:0.2958447337150574 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24135512113571167 - training/rollout_probs_diff_mean:0.003082486568018794 - training/rollout_probs_diff_std:0.00864423532038927 - training/rollout_actor_probs_pearson_corr:0.9994667768478394 - rollout_corr/rollout_is_mean:0.9999562501907349 - rollout_corr/rollout_is_ratio_fraction_high:4.822275059268577e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.446682472305838e-05 - rollout_corr/rollout_is_max:2.0740530490875244 - rollout_corr/rollout_is_min:0.37482571601867676 - rollout_corr/rollout_is_std:0.027990059927105904 - rollout_corr/rollout_is_eff_sample_size:0.9992171698522248 - rollout_corr/rollout_is_seq_mean:0.9999083280563354 - rollout_corr/rollout_is_seq_std:0.0016167733119800687 - rollout_corr/rollout_is_seq_max:1.008034586906433 - rollout_corr/rollout_is_seq_min:0.993806004524231 - rollout_corr/rollout_is_seq_max_deviation:0.008034586906433105 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3131270464509726) - rollout_corr/training_log_ppl:np.float64(0.26043001232028473) - rollout_corr/kl:np.float64(0.0006687698600753578) - rollout_corr/k3_kl:np.float64(0.0005304543523152461) - rollout_corr/rollout_ppl:np.float64(1.3122303052805364) - rollout_corr/rollout_log_ppl:np.float64(0.2597612422541715) - rollout_corr/log_ppl_diff:np.float64(0.0006687700661132112) - rollout_corr/log_ppl_abs_diff:np.float64(0.0013376595597947016) - rollout_corr/log_ppl_diff_max:np.float64(0.009146973490715027) - rollout_corr/log_ppl_diff_min:np.float64(-0.004970654845237732) - rollout_corr/ppl_ratio:np.float64(1.0006705992855132) - rollout_corr/chi2_token:np.float64(0.0007638768292963505) - rollout_corr/chi2_seq:np.float64(0.7327017430216074) - actor/pg_loss:np.float64(4.302803426980972e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002477214987948173) - actor/ppo_kl:np.float64(0.00011317849087255638) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1297485176473856) - perf/mfu/actor:np.float64(0.1281224223428748) - perf/max_memory_allocated_gb:np.float64(128.6631588935852) - perf/max_memory_reserved_gb:np.float64(139.953125) - perf/cpu_memory_used_gb:np.float64(105.53421020507812) - actor/lr:np.float64(1e-06) - training/global_step:29 - training/epoch:1 - critic/score/mean:0.70703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.70703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.04238719493150711 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.04238719493150711 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:810.04296875 - response_length/max:3813.0 - response_length/min:108.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:810.04296875 - response_length_non_aborted/max:3813.0 - response_length_non_aborted/min:108.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:263.53125 - prompt_length/max:375.0 - prompt_length/min:164.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014430470764636993 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3452266938984394) - timing_s/agent_loop/generate_sequences/max:np.float64(27.088402163237333) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.822816854313714) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(27.088402163237333) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:314 - timing_s/agent_loop/slowest/response_length:3813 - timing_s/gen:33.822356663644314 - timing_s/reward:0.073737021535635 - timing_s/old_log_prob:3.5272585432976484 - timing_s/adv:0.16648594662547112 - timing_s/update_actor:14.246411005035043 - timing_s/step:51.92763191089034 - timing_s/stop_profile:6.692670285701752e-05 - timing_per_token_ms/gen:0.16310070677020563 - timing_per_token_ms/update_actor:0.051836232667000354 - timing_per_token_ms/adv:0.0006057669024158899 - perf/total_num_tokens:274835 - perf/time_per_step:51.92763191089034 - perf/throughput:661.5817770961196
(TaskRunner pid=2281424) Training Progress: 29%|██▉ | 29/100 [27:06<1:03:53, 53.99s/it]
(TaskRunner pid=2281424) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 30}
(TaskRunner pid=2281424) validation generation end
(TaskRunner pid=2281424) [prompt] system
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2281424) <reasoning>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </reasoning>
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) A
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) user
(TaskRunner pid=2281424) A charged particle produces an electric field with a magnitude of $2.0 \mathrm{~N} / \mathrm{C}$ at a point that is $50 \mathrm{~cm}$ away from the particle. What is the magnitude of the particle's charge? Recall that the electric field $E$ due to a point charge $q$ at a distance $r$ is given by $E = k\frac{q}{r^2}$, where $k = 8.99 \times 10^9 \mathrm{N} \mathrm{m}^2 / \mathrm{C}^2$.
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) A: 50 pC
(TaskRunner pid=2281424) B: 56 pC
(TaskRunner pid=2281424) C: 60 pC
(TaskRunner pid=2281424) D: 64 pC
(TaskRunner pid=2281424) Please reason step by step.
(TaskRunner pid=2281424) assistant
(TaskRunner pid=2281424) <think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) </think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) [response] <reasoning>
(TaskRunner pid=2281424) We are given the electric field $ E = 2.0 \, \mathrm{N/C} $, the distance $ r = 50 \, \mathrm{cm} = 0.50 \, \mathrm{m} $, and the constant $ k = 8.99 \times 10^9 \, \mathrm{N \cdot m^2/C^2} $. The formula for the electric field due to a point charge is:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) E = k \frac{q}{r^2}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) We can solve for $ q $:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{E r^2}{k}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Substituting the values:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{(2.0 \, \mathrm{N/C}) \cdot (0.50 \, \mathrm{m})^2}{8.99 \times 10^9 \, \mathrm{N \cdot m^2/C^2}}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{(2.0) \cdot (0.25)}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{0.50}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 5.56 \times 10^{-11} \, \mathrm{C}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Now, convert coulombs to picocoulombs (1 pC = $1 \times 10^{-12} \, \mathrm{C}$):
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 5.56 \times 10^{-11} \, \mathrm{C} = 55.6 \, \mathrm{pC}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) The closest option is:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) B
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424) [ground_truth] B
(TaskRunner pid=2281424) [score] 1.0
(TaskRunner pid=2281424) [acc] 1.0
(TaskRunner pid=2281424) [pred] B
(TaskRunner pid=2281424) [incorrect_format] 1
(TaskRunner pid=2281424) [feedback]
(TaskRunner pid=2281424) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_30
(TaskRunner pid=2281424) step:30 - global_seqlen/min:21152 - global_seqlen/max:43378 - global_seqlen/minmax_diff:22226 - global_seqlen/balanced_min:32690 - global_seqlen/balanced_max:36508 - global_seqlen/mean:33253.625 - actor/entropy:0.3238883912563324 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3535161018371582 - training/rollout_probs_diff_mean:0.0032308220397681 - training/rollout_probs_diff_std:0.00875203125178814 - training/rollout_actor_probs_pearson_corr:0.9993578195571899 - rollout_corr/rollout_is_mean:1.0000488758087158 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:1.9363911151885986 - rollout_corr/rollout_is_min:0.5356025099754333 - rollout_corr/rollout_is_std:0.028555061668157578 - rollout_corr/rollout_is_eff_sample_size:0.9991853917962915 - rollout_corr/rollout_is_seq_mean:1.0000839233398438 - rollout_corr/rollout_is_seq_std:0.0014182842569425702 - rollout_corr/rollout_is_seq_max:1.00834321975708 - rollout_corr/rollout_is_seq_min:0.9960029721260071 - rollout_corr/rollout_is_seq_max_deviation:0.008343219757080078 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3260084195062518) - rollout_corr/training_log_ppl:np.float64(0.26798260471878166) - rollout_corr/kl:np.float64(0.00034142893949073994) - rollout_corr/k3_kl:np.float64(0.00046080480333898777) - rollout_corr/rollout_ppl:np.float64(1.3255186867900193) - rollout_corr/rollout_log_ppl:np.float64(0.2676411745214864) - rollout_corr/log_ppl_diff:np.float64(0.00034143019729526713) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010471087371115573) - rollout_corr/log_ppl_diff_max:np.float64(0.0044901371002197266) - rollout_corr/log_ppl_diff_min:np.float64(-0.006970256567001343) - rollout_corr/ppl_ratio:np.float64(1.0003424619790167) - rollout_corr/chi2_token:np.float64(0.0011780557688325644) - rollout_corr/chi2_seq:np.float64(9.059563745046034) - actor/pg_loss:np.float64(-7.204711437225342e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002318803169032435) - actor/ppo_kl:np.float64(-2.2715349961899278e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1330014280974865) - perf/mfu/actor:np.float64(0.11964730091498024) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.54463195800781) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.6046875) - val-aux/sciknoweval/reward/std@16:np.float64(0.19331331768802099) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.6854) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.1699013098775457) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5269125000000001) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.1573208332576977) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6056250000000001) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.19436455511085113) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.75095) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.13208485427187178) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.4644124999999999) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.11165691249499834) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6343625) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.16176855495856937) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8039375) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.0971781630054231) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.42205000000000004) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.07114364007214655) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6537749999999999) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.12550218875311617) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.84275) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.061103157160745235) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.39563750000000003) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.038223937353486045) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.666125) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.09165205807525627) - val-aux/sciknoweval/score/mean@16:np.float64(0.6046875) - val-aux/sciknoweval/score/std@16:np.float64(0.19331331768802099) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.6854) - val-aux/sciknoweval/score/best@2/std:np.float64(0.1699013098775457) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5269125000000001) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.1573208332576977) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6056250000000001) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.19436455511085113) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.75095) - val-aux/sciknoweval/score/best@4/std:np.float64(0.13208485427187178) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.4644124999999999) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.11165691249499834) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6343625) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.16176855495856937) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8039375) - val-aux/sciknoweval/score/best@8/std:np.float64(0.0971781630054231) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.42205000000000004) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.07114364007214655) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6537749999999999) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.12550218875311617) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.84275) - val-aux/sciknoweval/score/best@16/std:np.float64(0.061103157160745235) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.39563750000000003) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.038223937353486045) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.666125) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.09165205807525627) - val-core/sciknoweval/acc/mean@16:np.float64(0.6046875) - val-aux/sciknoweval/acc/std@16:np.float64(0.19331331768802099) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.6854) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.1699013098775457) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5269125000000001) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.1573208332576977) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6056250000000001) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.19436455511085113) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.75095) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.13208485427187178) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.4644124999999999) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.11165691249499834) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6343625) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.16176855495856937) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8039375) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.0971781630054231) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.42205000000000004) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.07114364007214655) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6537749999999999) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.12550218875311617) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.84275) - val-core/sciknoweval/acc/best@16/std:np.float64(0.061103157160745235) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.39563750000000003) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.038223937353486045) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.666125) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.09165205807525627) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.9421875) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.11810324679239408) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9847249999999999) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.05478145372165707) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.9000999999999999) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.14152552996328321) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9422875000000002) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.11789115899569932) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9971625) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.012950626299582524) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.8410875000000001) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.15275603349423345) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.965475) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.08685302210496705) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(0.9997) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0022688782927956746) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.7717375) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.1420235663973301) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9818749999999999) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.05503683300798294) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.70755) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.10363469468730849) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9918625000000001) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.02568992930006094) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:30 - training/epoch:1 - critic/score/mean:0.66796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.66796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.00041142108966596425 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.00041142108966596425 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:779.73828125 - response_length/max:8192.0 - response_length/min:130.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:779.73828125 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:130.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:259.4375 - prompt_length/max:330.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00012162141501903534 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5774449706077576) - timing_s/agent_loop/generate_sequences/max:np.float64(62.31554364040494) - timing_s/agent_loop/generate_sequences/mean:np.float64(7.508535967179341) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(62.31554364040494) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:323 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:69.08000029809773 - timing_s/reward:0.07287852838635445 - timing_s/old_log_prob:3.7097500916570425 - timing_s/adv:0.15018161572515965 - timing_s/update_actor:14.847722072154284 - timing_s/step:87.9640430957079 - timing_s/testing:149.87052939273417 - timing_s/save_checkpoint:12.872908538207412 - timing_s/stop_profile:0.00015308894217014313 - timing_per_token_ms/gen:0.346069646255994 - timing_per_token_ms/update_actor:0.05581241921803368 - timing_per_token_ms/adv:0.0005645309937080531 - perf/total_num_tokens:266029 - perf/time_per_step:87.9640430957079 - perf/throughput:378.03656846262646
(TaskRunner pid=2281424) Training Progress: 30%|███ | 30/100 [31:17<2:11:52, 113.03s/it]
(TaskRunner pid=2281424) step:31 - global_seqlen/min:26976 - global_seqlen/max:51365 - global_seqlen/minmax_diff:24389 - global_seqlen/balanced_min:37711 - global_seqlen/balanced_max:37821 - global_seqlen/mean:37765.0 - actor/entropy:0.33241793513298035 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.301638126373291 - training/rollout_probs_diff_mean:0.003111757803708315 - training/rollout_probs_diff_std:0.008627519011497498 - training/rollout_actor_probs_pearson_corr:0.9994876980781555 - rollout_corr/rollout_is_mean:0.9999532699584961 - rollout_corr/rollout_is_ratio_fraction_high:8.658907972858287e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.2988362868782133e-05 - rollout_corr/rollout_is_max:2.167019844055176 - rollout_corr/rollout_is_min:0.3715561032295227 - rollout_corr/rollout_is_std:0.028119659051299095 - rollout_corr/rollout_is_eff_sample_size:0.9992096714770495 - rollout_corr/rollout_is_seq_mean:0.9999525547027588 - rollout_corr/rollout_is_seq_std:0.0012215663446113467 - rollout_corr/rollout_is_seq_max:1.004585862159729 - rollout_corr/rollout_is_seq_min:0.9942855834960938 - rollout_corr/rollout_is_seq_max_deviation:0.00571441650390625 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3133017774671316) - rollout_corr/training_log_ppl:np.float64(0.25711654864426237) - rollout_corr/kl:np.float64(0.0004556128830559558) - rollout_corr/k3_kl:np.float64(0.00039641217307462284) - rollout_corr/rollout_ppl:np.float64(1.3126854803413153) - rollout_corr/rollout_log_ppl:np.float64(0.25666093576001003) - rollout_corr/log_ppl_diff:np.float64(0.00045561288425233215) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010043695365311578) - rollout_corr/log_ppl_diff_max:np.float64(0.006338976323604584) - rollout_corr/log_ppl_diff_min:np.float64(-0.0035679787397384644) - rollout_corr/ppl_ratio:np.float64(1.0004564973060042) - rollout_corr/chi2_token:np.float64(0.0006840897258371115) - rollout_corr/chi2_seq:np.float64(0.9153507652226835) - actor/pg_loss:np.float64(1.1453521437942982e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001992799257095612) - actor/ppo_kl:np.float64(2.699381930071354e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.14760523661971092) - perf/mfu/actor:np.float64(0.13769359300581646) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.38329315185547) - actor/lr:np.float64(1e-06) - training/global_step:31 - training/epoch:1 - critic/score/mean:0.62890625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.62890625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.009976144880056381 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.009976144880056381 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:902.25 - response_length/max:3456.0 - response_length/min:174.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:902.25 - response_length_non_aborted/max:3456.0 - response_length_non_aborted/min:174.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.90625 - prompt_length/max:364.0 - prompt_length/min:188.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00019257515668869019 - timing_s/agent_loop/generate_sequences/min:np.float64(2.1455713659524918) - timing_s/agent_loop/generate_sequences/max:np.float64(25.956687213853) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.312753760583291) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(25.956687213853) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:267 - timing_s/agent_loop/slowest/response_length:3456 - timing_s/gen:32.13823664188385 - timing_s/reward:0.0789014957845211 - timing_s/old_log_prob:3.81612709723413 - timing_s/adv:0.13714739680290222 - timing_s/update_actor:14.64492060430348 - timing_s/step:50.91287592239678 - timing_s/stop_profile:0.00011721253395080566 - timing_per_token_ms/gen:0.13914102175933366 - timing_per_token_ms/update_actor:0.048473853449965174 - timing_per_token_ms/adv:0.00045395007547630815 - perf/total_num_tokens:302120 - perf/time_per_step:50.91287592239678 - perf/throughput:741.7573514716152
(TaskRunner pid=2281424) Training Progress: 31%|███ | 31/100 [32:08<1:48:34, 94.42s/it]
(TaskRunner pid=2281424) step:32 - global_seqlen/min:17749 - global_seqlen/max:40522 - global_seqlen/minmax_diff:22773 - global_seqlen/balanced_min:29749 - global_seqlen/balanced_max:30114 - global_seqlen/mean:29860.25 - actor/entropy:0.3001790940761566 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2931319773197174 - training/rollout_probs_diff_mean:0.0032425629906356335 - training/rollout_probs_diff_std:0.009085338562726974 - training/rollout_actor_probs_pearson_corr:0.9993113279342651 - rollout_corr/rollout_is_mean:1.0000972747802734 - rollout_corr/rollout_is_ratio_fraction_high:5.759240593761206e-06 - rollout_corr/rollout_is_ratio_fraction_low:2.3036962375044823e-05 - rollout_corr/rollout_is_max:2.2561657428741455 - rollout_corr/rollout_is_min:0.11440415680408478 - rollout_corr/rollout_is_std:0.028905672952532768 - rollout_corr/rollout_is_eff_sample_size:0.999165278640662 - rollout_corr/rollout_is_seq_mean:1.0000715255737305 - rollout_corr/rollout_is_seq_std:0.001591670559719205 - rollout_corr/rollout_is_seq_max:1.0057041645050049 - rollout_corr/rollout_is_seq_min:0.9924244284629822 - rollout_corr/rollout_is_seq_max_deviation:0.007575571537017822 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3085072333924472) - rollout_corr/training_log_ppl:np.float64(0.25189579716243315) - rollout_corr/kl:np.float64(0.0004256223451228891) - rollout_corr/k3_kl:np.float64(0.0005207587888378384) - rollout_corr/rollout_ppl:np.float64(1.307915877085179) - rollout_corr/rollout_log_ppl:np.float64(0.25147017439303454) - rollout_corr/log_ppl_diff:np.float64(0.00042562276939861476) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012707264686468989) - rollout_corr/log_ppl_diff_max:np.float64(0.008575737476348877) - rollout_corr/log_ppl_diff_min:np.float64(-0.005954384803771973) - rollout_corr/ppl_ratio:np.float64(1.0004271555226296) - rollout_corr/chi2_token:np.float64(0.0012784942518919706) - rollout_corr/chi2_seq:np.float64(1.3941792568657547) - actor/pg_loss:np.float64(-4.464993253350258e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00011340672926962725) - actor/ppo_kl:np.float64(3.421145366999667e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.10089067462831736) - perf/mfu/actor:np.float64(0.11507569937509035) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.87258911132812) - actor/lr:np.float64(1e-06) - training/global_step:32 - training/epoch:1 - critic/score/mean:0.73828125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.73828125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.02760116197168827 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.02760116197168827 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:678.2578125 - response_length/max:3951.0 - response_length/min:90.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:678.2578125 - response_length_non_aborted/max:3951.0 - response_length_non_aborted/min:90.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:254.875 - prompt_length/max:375.0 - prompt_length/min:143.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010407157242298126 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1292062569409609) - timing_s/agent_loop/generate_sequences/max:np.float64(27.004830926656723) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.73606214641768) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(27.004830926656723) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:208 - timing_s/agent_loop/slowest/response_length:3951 - timing_s/gen:34.06722195446491 - timing_s/reward:0.06839162670075893 - timing_s/old_log_prob:3.350517366081476 - timing_s/adv:0.15078148432075977 - timing_s/update_actor:13.741796219721437 - timing_s/step:51.4739043507725 - timing_s/stop_profile:0.00011109933257102966 - timing_per_token_ms/gen:0.1962013312742027 - timing_per_token_ms/update_actor:0.05752545700271028 - timing_per_token_ms/adv:0.0006311965084048181 - perf/total_num_tokens:238882 - perf/time_per_step:51.4739043507725 - perf/throughput:580.104625375127
(TaskRunner pid=2281424) Training Progress: 32%|███▏ | 32/100 [33:00<1:32:25, 81.55s/it]
(TaskRunner pid=2281424) step:33 - global_seqlen/min:27211 - global_seqlen/max:50719 - global_seqlen/minmax_diff:23508 - global_seqlen/balanced_min:36245 - global_seqlen/balanced_max:36302 - global_seqlen/mean:36278.125 - actor/entropy:0.30930498242378235 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35351723432540894 - training/rollout_probs_diff_mean:0.0030934372916817665 - training/rollout_probs_diff_std:0.00848071463406086 - training/rollout_actor_probs_pearson_corr:0.9993887543678284 - rollout_corr/rollout_is_mean:0.9999690055847168 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.3896545169700403e-05 - rollout_corr/rollout_is_max:1.8591357469558716 - rollout_corr/rollout_is_min:0.3342134654521942 - rollout_corr/rollout_is_std:0.027681713923811913 - rollout_corr/rollout_is_eff_sample_size:0.999234190391299 - rollout_corr/rollout_is_seq_mean:0.9999369382858276 - rollout_corr/rollout_is_seq_std:0.0011744012590497732 - rollout_corr/rollout_is_seq_max:1.003006100654602 - rollout_corr/rollout_is_seq_min:0.9960953593254089 - rollout_corr/rollout_is_seq_max_deviation:0.0039046406745910645 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.354571030009538) - rollout_corr/training_log_ppl:np.float64(0.2829173822829034) - rollout_corr/kl:np.float64(0.0005067846795761) - rollout_corr/k3_kl:np.float64(0.00047523474137278754) - rollout_corr/rollout_ppl:np.float64(1.353815943468362) - rollout_corr/rollout_log_ppl:np.float64(0.28241059606079943) - rollout_corr/log_ppl_diff:np.float64(0.0005067862221039832) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010225145379081368) - rollout_corr/log_ppl_diff_max:np.float64(0.005544401705265045) - rollout_corr/log_ppl_diff_min:np.float64(-0.003694787621498108) - rollout_corr/ppl_ratio:np.float64(1.000507768476382) - rollout_corr/chi2_token:np.float64(0.0008780690841376781) - rollout_corr/chi2_seq:np.float64(0.740443660877645) - actor/pg_loss:np.float64(-8.333136793226004e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002747284138422401) - actor/ppo_kl:np.float64(2.7713305339016614e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.16628609038889408) - perf/mfu/actor:np.float64(0.1349622504868255) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.00208282470703) - actor/lr:np.float64(1e-06) - training/global_step:33 - training/epoch:1 - critic/score/mean:0.70703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.70703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004228602629154921 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004228602629154921 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:843.28515625 - response_length/max:3164.0 - response_length/min:142.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:843.28515625 - response_length_non_aborted/max:3164.0 - response_length_non_aborted/min:142.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:290.40625 - prompt_length/max:437.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013750791549682617 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8266812413930893) - timing_s/agent_loop/generate_sequences/max:np.float64(24.079156171530485) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.841072585782968) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(24.079156171530485) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:292 - timing_s/agent_loop/slowest/response_length:3164 - timing_s/gen:30.762977486476302 - timing_s/reward:0.07941771484911442 - timing_s/old_log_prob:3.5987925808876753 - timing_s/adv:0.14997461810708046 - timing_s/update_actor:14.10401264578104 - timing_s/step:48.78351537883282 - timing_s/stop_profile:0.00021011196076869965 - timing_per_token_ms/gen:0.14249969884555055 - timing_per_token_ms/update_actor:0.04859682193395138 - timing_per_token_ms/adv:0.0005167529265469221 - perf/total_num_tokens:290225 - perf/time_per_step:48.78351537883282 - perf/throughput:743.6554073292776
(TaskRunner pid=2281424) Training Progress: 33%|███▎ | 33/100 [33:48<1:20:06, 71.74s/it]
(TaskRunner pid=2281424) step:34 - global_seqlen/min:25666 - global_seqlen/max:73494 - global_seqlen/minmax_diff:47828 - global_seqlen/balanced_min:35473 - global_seqlen/balanced_max:36595 - global_seqlen/mean:35820.75 - actor/entropy:0.35148361325263977 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.45545026659965515 - training/rollout_probs_diff_mean:0.0031844459008425474 - training/rollout_probs_diff_std:0.00856610108166933 - training/rollout_actor_probs_pearson_corr:0.9994761347770691 - rollout_corr/rollout_is_mean:0.9999700784683228 - rollout_corr/rollout_is_ratio_fraction_high:9.077787581190933e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.8155575162381865e-05 - rollout_corr/rollout_is_max:2.0403316020965576 - rollout_corr/rollout_is_min:0.33807873725891113 - rollout_corr/rollout_is_std:0.02805599570274353 - rollout_corr/rollout_is_eff_sample_size:0.9992133611396591 - rollout_corr/rollout_is_seq_mean:0.9998204112052917 - rollout_corr/rollout_is_seq_std:0.0015267465496435761 - rollout_corr/rollout_is_seq_max:1.003962516784668 - rollout_corr/rollout_is_seq_min:0.9919791221618652 - rollout_corr/rollout_is_seq_max_deviation:0.008020877838134766 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.309415325988084) - rollout_corr/training_log_ppl:np.float64(0.25629871332785115) - rollout_corr/kl:np.float64(0.0007180621188593861) - rollout_corr/k3_kl:np.float64(0.000561066810490729) - rollout_corr/rollout_ppl:np.float64(1.3084843065589666) - rollout_corr/rollout_log_ppl:np.float64(0.2555806517775636) - rollout_corr/log_ppl_diff:np.float64(0.0007180615502875298) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012585615331772715) - rollout_corr/log_ppl_diff_max:np.float64(0.009126394987106323) - rollout_corr/log_ppl_diff_min:np.float64(-0.0031400546431541443) - rollout_corr/ppl_ratio:np.float64(1.0007198799867183) - rollout_corr/chi2_token:np.float64(0.0008239294402301311) - rollout_corr/chi2_seq:np.float64(1.3956339918076992) - actor/pg_loss:np.float64(6.344064604490995e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00036903620963357753) - actor/ppo_kl:np.float64(0.00010281039390358515) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1495772898197174) - perf/mfu/actor:np.float64(0.12881182116660295) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.90487289428711) - actor/lr:np.float64(1e-06) - training/global_step:34 - training/epoch:1 - critic/score/mean:0.6796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0018949881196022034 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:0.0018949881196022034 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:860.6171875 - response_length/max:6530.0 - response_length/min:121.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:860.6171875 - response_length_non_aborted/max:6530.0 - response_length_non_aborted/min:121.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:258.78125 - prompt_length/max:382.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010554678738117218 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5495978090912104) - timing_s/agent_loop/generate_sequences/max:np.float64(42.11380794644356) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.242849451620714) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(42.11380794644356) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:259 - timing_s/agent_loop/slowest/response_length:6530 - timing_s/gen:48.750448731705546 - timing_s/reward:0.07960104756057262 - timing_s/old_log_prob:3.7480743881314993 - timing_s/adv:0.1472205240279436 - timing_s/update_actor:14.973976226523519 - timing_s/step:67.78288715332747 - timing_s/stop_profile:3.6908313632011414e-05 - timing_per_token_ms/gen:0.22127310855992496 - timing_per_token_ms/update_actor:0.05225315015222852 - timing_per_token_ms/adv:0.0005137403740427811 - perf/total_num_tokens:286566 - perf/time_per_step:67.78288715332747 - perf/throughput:528.4630310740837
(TaskRunner pid=2281424) Training Progress: 34%|███▍ | 34/100 [34:56<1:17:37, 70.56s/it]
(TaskRunner pid=2281424) step:35 - global_seqlen/min:22711 - global_seqlen/max:52151 - global_seqlen/minmax_diff:29440 - global_seqlen/balanced_min:33570 - global_seqlen/balanced_max:34828 - global_seqlen/mean:33923.125 - actor/entropy:0.356091171503067 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.23818081617355347 - training/rollout_probs_diff_mean:0.003347822232171893 - training/rollout_probs_diff_std:0.008754725567996502 - training/rollout_actor_probs_pearson_corr:0.9993891716003418 - rollout_corr/rollout_is_mean:0.999956488609314 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.74996237346204e-06 - rollout_corr/rollout_is_max:1.6463863849639893 - rollout_corr/rollout_is_min:0.14987757802009583 - rollout_corr/rollout_is_std:0.02889123558998108 - rollout_corr/rollout_is_eff_sample_size:0.9991658736928141 - rollout_corr/rollout_is_seq_mean:0.9999445676803589 - rollout_corr/rollout_is_seq_std:0.0016311032231897116 - rollout_corr/rollout_is_seq_max:1.0061169862747192 - rollout_corr/rollout_is_seq_min:0.994307279586792 - rollout_corr/rollout_is_seq_max_deviation:0.006116986274719238 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3848898322321475) - rollout_corr/training_log_ppl:np.float64(0.3058403903269209) - rollout_corr/kl:np.float64(0.0004393863629239547) - rollout_corr/k3_kl:np.float64(0.0005089315998816346) - rollout_corr/rollout_ppl:np.float64(1.3842959650792181) - rollout_corr/rollout_log_ppl:np.float64(0.3054010035120882) - rollout_corr/log_ppl_diff:np.float64(0.0004393868148326874) - rollout_corr/log_ppl_abs_diff:np.float64(0.0012084643822163343) - rollout_corr/log_ppl_diff_max:np.float64(0.006951093673706055) - rollout_corr/log_ppl_diff_min:np.float64(-0.005402863025665283) - rollout_corr/ppl_ratio:np.float64(1.0004409453831613) - rollout_corr/chi2_token:np.float64(0.001150362892076373) - rollout_corr/chi2_seq:np.float64(1.6086886166594923) - actor/pg_loss:np.float64(2.0964303985238075e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0002854640780469708) - actor/ppo_kl:np.float64(-8.599662149322285e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.18192359805107117) - perf/mfu/actor:np.float64(0.12649445809002435) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.95371627807617) - actor/lr:np.float64(1e-06) - training/global_step:35 - training/epoch:1 - critic/score/mean:0.75390625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.75390625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.00438565481454134 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.00438565481454134 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:801.28515625 - response_length/max:5012.0 - response_length/min:130.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:801.28515625 - response_length_non_aborted/max:5012.0 - response_length_non_aborted/min:130.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:258.8125 - prompt_length/max:365.0 - prompt_length/min:160.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:3.070756793022156e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.6557252500206232) - timing_s/agent_loop/generate_sequences/max:np.float64(33.535721046850085) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.11258227860526) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(33.535721046850085) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:303 - timing_s/agent_loop/slowest/response_length:5012 - timing_s/gen:40.21909198909998 - timing_s/reward:0.07026992179453373 - timing_s/old_log_prob:3.5517083313316107 - timing_s/adv:0.14980567805469036 - timing_s/update_actor:14.293088844045997 - timing_s/step:58.31905349902809 - timing_s/stop_profile:2.7982518076896667e-05 - timing_per_token_ms/gen:0.19606731368602187 - timing_per_token_ms/update_actor:0.05266720284483666 - timing_per_token_ms/adv:0.0005520042672022786 - perf/total_num_tokens:271385 - perf/time_per_step:58.31905349902809 - perf/throughput:581.6816797372294
(TaskRunner pid=2281424) Training Progress: 35%|███▌ | 35/100 [35:55<1:12:28, 66.89s/it]
(TaskRunner pid=2281424) step:36 - global_seqlen/min:33717 - global_seqlen/max:49536 - global_seqlen/minmax_diff:15819 - global_seqlen/balanced_min:43612 - global_seqlen/balanced_max:43672 - global_seqlen/mean:43658.375 - actor/entropy:0.362861692905426 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.8421130180358887 - training/rollout_probs_diff_mean:0.003083376446738839 - training/rollout_probs_diff_std:0.00820560660213232 - training/rollout_actor_probs_pearson_corr:0.9995745420455933 - rollout_corr/rollout_is_mean:0.9999458193778992 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:7.136867225199239e-06 - rollout_corr/rollout_is_max:1.5991979837417603 - rollout_corr/rollout_is_min:0.010655050165951252 - rollout_corr/rollout_is_std:0.027175385504961014 - rollout_corr/rollout_is_eff_sample_size:0.9992619243999026 - rollout_corr/rollout_is_seq_mean:0.999880313873291 - rollout_corr/rollout_is_seq_std:0.0011923309648409486 - rollout_corr/rollout_is_seq_max:1.0057610273361206 - rollout_corr/rollout_is_seq_min:0.9923911094665527 - rollout_corr/rollout_is_seq_max_deviation:0.007608890533447266 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3939749002456665) - rollout_corr/training_log_ppl:np.float64(0.3143607703241287) - rollout_corr/kl:np.float64(0.0004875935232391271) - rollout_corr/k3_kl:np.float64(0.00043940715849544176) - rollout_corr/rollout_ppl:np.float64(1.3933153660036623) - rollout_corr/rollout_log_ppl:np.float64(0.3138731736544287) - rollout_corr/log_ppl_diff:np.float64(0.0004875966696999967) - rollout_corr/log_ppl_abs_diff:np.float64(0.0008989509078674018) - rollout_corr/log_ppl_diff_max:np.float64(0.008858025074005127) - rollout_corr/log_ppl_diff_min:np.float64(-0.003800034523010254) - rollout_corr/ppl_ratio:np.float64(1.0004885424859822) - rollout_corr/chi2_token:np.float64(0.0007559196092188358) - rollout_corr/chi2_seq:np.float64(0.28486403776332736) - actor/pg_loss:np.float64(4.073767922818661e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001381784221621274) - actor/ppo_kl:np.float64(-3.878047037275678e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.11497985571622849) - perf/mfu/actor:np.float64(0.15230643812180578) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.80973815917969) - actor/lr:np.float64(1e-06) - training/global_step:36 - training/epoch:1 - critic/score/mean:0.578125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.578125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006595357321202755 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006595357321202755 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1094.66796875 - response_length/max:3514.0 - response_length/min:111.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1094.66796875 - response_length_non_aborted/max:3514.0 - response_length_non_aborted/min:111.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:269.65625 - prompt_length/max:375.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.7200207114219666e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3092289455235004) - timing_s/agent_loop/generate_sequences/max:np.float64(27.297075459733605) - timing_s/agent_loop/generate_sequences/mean:np.float64(11.346446073104744) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(27.297075459733605) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:297 - timing_s/agent_loop/slowest/response_length:3514 - timing_s/gen:34.776961313560605 - timing_s/reward:0.08570255897939205 - timing_s/old_log_prob:3.737231995910406 - timing_s/adv:0.14758271723985672 - timing_s/update_actor:15.371759809553623 - timing_s/step:54.21316165663302 - timing_s/stop_profile:2.7824193239212036e-05 - timing_per_token_ms/gen:0.12409927851110891 - timing_per_token_ms/update_actor:0.04401148636874833 - timing_per_token_ms/adv:0.00042254984650670326 - perf/total_num_tokens:349267 - perf/time_per_step:54.21316165663302 - perf/throughput:805.3095164697587
(TaskRunner pid=2281424) Training Progress: 36%|███▌ | 36/100 [36:49<1:07:18, 63.10s/it]
(TaskRunner pid=2281424) step:37 - global_seqlen/min:34704 - global_seqlen/max:44098 - global_seqlen/minmax_diff:9394 - global_seqlen/balanced_min:39539 - global_seqlen/balanced_max:39624 - global_seqlen/mean:39597.875 - actor/entropy:0.35228487849235535 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2413027286529541 - training/rollout_probs_diff_mean:0.003145897062495351 - training/rollout_probs_diff_std:0.0083078658208251 - training/rollout_actor_probs_pearson_corr:0.9995119571685791 - rollout_corr/rollout_is_mean:0.9999536275863647 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.998736247012857e-06 - rollout_corr/rollout_is_max:1.6415951251983643 - rollout_corr/rollout_is_min:0.39738908410072327 - rollout_corr/rollout_is_std:0.027896204963326454 - rollout_corr/rollout_is_eff_sample_size:0.9992224068794121 - rollout_corr/rollout_is_seq_mean:0.9999120235443115 - rollout_corr/rollout_is_seq_std:0.0014193912502378225 - rollout_corr/rollout_is_seq_max:1.0059196949005127 - rollout_corr/rollout_is_seq_min:0.9947477579116821 - rollout_corr/rollout_is_seq_max_deviation:0.005919694900512695 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3658450348302722) - rollout_corr/training_log_ppl:np.float64(0.29802232861402445) - rollout_corr/kl:np.float64(0.0005081376050801722) - rollout_corr/k3_kl:np.float64(0.00047723877818839355) - rollout_corr/rollout_ppl:np.float64(1.365188850555569) - rollout_corr/rollout_log_ppl:np.float64(0.2975141915230779) - rollout_corr/log_ppl_diff:np.float64(0.0005081370909465477) - rollout_corr/log_ppl_abs_diff:np.float64(0.001064729891368188) - rollout_corr/log_ppl_diff_max:np.float64(0.005850955843925476) - rollout_corr/log_ppl_diff_min:np.float64(-0.00405728816986084) - rollout_corr/ppl_ratio:np.float64(1.000509355450049) - rollout_corr/chi2_token:np.float64(0.0009017251431941986) - rollout_corr/chi2_seq:np.float64(1.061360603896901) - actor/pg_loss:np.float64(6.26088585704565e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00021849221991487866) - actor/ppo_kl:np.float64(-2.4668701797736503e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1671565007418394) - perf/mfu/actor:np.float64(0.14026788981271016) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.84344863891602) - actor/lr:np.float64(1e-06) - training/global_step:37 - training/epoch:1 - critic/score/mean:0.6640625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6640625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.011376905255019665 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.011376905255019665 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:976.87109375 - response_length/max:3632.0 - response_length/min:123.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:976.87109375 - response_length_non_aborted/max:3632.0 - response_length_non_aborted/min:123.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:260.5625 - prompt_length/max:350.0 - prompt_length/min:157.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.6365742087364197e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.545783119276166) - timing_s/agent_loop/generate_sequences/max:np.float64(27.446757026016712) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.701898429557332) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(27.446757026016712) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:283 - timing_s/agent_loop/slowest/response_length:3632 - timing_s/gen:33.87001137807965 - timing_s/reward:0.0845334343612194 - timing_s/old_log_prob:3.622224699705839 - timing_s/adv:0.14574186690151691 - timing_s/update_actor:15.075419383123517 - timing_s/step:52.884285079315305 - timing_s/stop_profile:0.00012076087296009064 - timing_per_token_ms/gen:0.13543724734215848 - timing_per_token_ms/update_actor:0.0475891047913667 - timing_per_token_ms/adv:0.0004600684598021892 - perf/total_num_tokens:316783 - perf/time_per_step:52.884285079315305 - perf/throughput:748.7644947948435
(TaskRunner pid=2281424) Training Progress: 37%|███▋ | 37/100 [37:42<1:03:03, 60.05s/it]
(TaskRunner pid=2281424) step:38 - global_seqlen/min:25876 - global_seqlen/max:56353 - global_seqlen/minmax_diff:30477 - global_seqlen/balanced_min:36571 - global_seqlen/balanced_max:36714 - global_seqlen/mean:36659.625 - actor/entropy:0.32807788252830505 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.9003579616546631 - training/rollout_probs_diff_mean:0.002977442229166627 - training/rollout_probs_diff_std:0.008599979802966118 - training/rollout_actor_probs_pearson_corr:0.9994491338729858 - rollout_corr/rollout_is_mean:0.9999948143959045 - rollout_corr/rollout_is_ratio_fraction_high:4.4448197513702326e-06 - rollout_corr/rollout_is_ratio_fraction_low:1.3334459254110698e-05 - rollout_corr/rollout_is_max:3.384216547012329 - rollout_corr/rollout_is_min:0.005166077055037022 - rollout_corr/rollout_is_std:0.027199501171708107 - rollout_corr/rollout_is_eff_sample_size:0.999260734067485 - rollout_corr/rollout_is_seq_mean:0.9999963641166687 - rollout_corr/rollout_is_seq_std:0.0012283420655876398 - rollout_corr/rollout_is_seq_max:1.00580632686615 - rollout_corr/rollout_is_seq_min:0.9936442375183105 - rollout_corr/rollout_is_seq_max_deviation:0.006355762481689453 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3002572190016508) - rollout_corr/training_log_ppl:np.float64(0.24828266966505907) - rollout_corr/kl:np.float64(0.0003652476746376365) - rollout_corr/k3_kl:np.float64(0.00043928875018650615) - rollout_corr/rollout_ppl:np.float64(1.2997558075003326) - rollout_corr/rollout_log_ppl:np.float64(0.2479174185282318) - rollout_corr/log_ppl_diff:np.float64(0.0003652511368272826) - rollout_corr/log_ppl_abs_diff:np.float64(0.000950878209550865) - rollout_corr/log_ppl_diff_max:np.float64(0.008495032787322998) - rollout_corr/log_ppl_diff_min:np.float64(-0.004545837640762329) - rollout_corr/ppl_ratio:np.float64(1.0003662216477096) - rollout_corr/chi2_token:np.float64(0.0010224031284451485) - rollout_corr/chi2_seq:np.float64(0.804062022594735) - actor/pg_loss:np.float64(-4.1241059079766273e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001981328330202814) - actor/ppo_kl:np.float64(-5.4830987617826565e-05) - actor/pg_clipfrac_lower:np.float64(3.98596921513672e-06) - actor/grad_norm:np.float64(0.12819637544453144) - perf/mfu/actor:np.float64(0.13466560382170992) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.90673160552979) - actor/lr:np.float64(1e-06) - training/global_step:38 - training/epoch:1 - critic/score/mean:0.66015625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.66015625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004230912774801254 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004230912774801254 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:878.83203125 - response_length/max:3854.0 - response_length/min:123.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:878.83203125 - response_length_non_aborted/max:3854.0 - response_length_non_aborted/min:123.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.78125 - prompt_length/max:328.0 - prompt_length/min:180.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00014106743037700653 - timing_s/agent_loop/generate_sequences/min:np.float64(1.5907990671694279) - timing_s/agent_loop/generate_sequences/max:np.float64(27.61575059592724) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.814689949009335) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(27.61575059592724) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:308 - timing_s/agent_loop/slowest/response_length:3674 - timing_s/gen:35.19211849011481 - timing_s/reward:0.0795986708253622 - timing_s/old_log_prob:3.5315878726541996 - timing_s/adv:0.1513067688792944 - timing_s/update_actor:14.461832895874977 - timing_s/step:53.504376608878374 - timing_s/stop_profile:0.00011487491428852081 - timing_per_token_ms/gen:0.15642262453324862 - timing_per_token_ms/update_actor:0.04931117304076002 - timing_per_token_ms/adv:0.0005159176099022235 - perf/total_num_tokens:293277 - perf/time_per_step:53.504376608878374 - perf/throughput:685.1705846044154
(TaskRunner pid=2281424) Training Progress: 38%|███▊ | 38/100 [38:35<1:00:02, 58.10s/it]
(TaskRunner pid=2281424) step:39 - global_seqlen/min:19213 - global_seqlen/max:40355 - global_seqlen/minmax_diff:21142 - global_seqlen/balanced_min:28562 - global_seqlen/balanced_max:29340 - global_seqlen/mean:28757.375 - actor/entropy:0.3156418800354004 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.22961103916168213 - training/rollout_probs_diff_mean:0.003309106919914484 - training/rollout_probs_diff_std:0.00906699150800705 - training/rollout_actor_probs_pearson_corr:0.9993242025375366 - rollout_corr/rollout_is_mean:1.0000478029251099 - rollout_corr/rollout_is_ratio_fraction_high:1.219846944877645e-05 - rollout_corr/rollout_is_ratio_fraction_low:6.099234724388225e-06 - rollout_corr/rollout_is_max:3.2739274501800537 - rollout_corr/rollout_is_min:0.49741330742836 - rollout_corr/rollout_is_std:0.029006537050008774 - rollout_corr/rollout_is_eff_sample_size:0.9991593281581235 - rollout_corr/rollout_is_seq_mean:1.0000423192977905 - rollout_corr/rollout_is_seq_std:0.0014898530207574368 - rollout_corr/rollout_is_seq_max:1.010595679283142 - rollout_corr/rollout_is_seq_min:0.9943899512290955 - rollout_corr/rollout_is_seq_max_deviation:0.01059567928314209 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3278274578042328) - rollout_corr/training_log_ppl:np.float64(0.26448297980823554) - rollout_corr/kl:np.float64(0.00044818446750127805) - rollout_corr/k3_kl:np.float64(0.00045055161507434605) - rollout_corr/rollout_ppl:np.float64(1.3272340912371874) - rollout_corr/rollout_log_ppl:np.float64(0.264034794352483) - rollout_corr/log_ppl_diff:np.float64(0.00044818545575253665) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011384486278984696) - rollout_corr/log_ppl_diff_max:np.float64(0.006218463182449341) - rollout_corr/log_ppl_diff_min:np.float64(-0.00863182544708252) - rollout_corr/ppl_ratio:np.float64(1.0004495589528233) - rollout_corr/chi2_token:np.float64(0.0009194693993777037) - rollout_corr/chi2_seq:np.float64(0.4609958524815738) - actor/pg_loss:np.float64(1.4589284546673298e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(4.937741061894485e-05) - actor/ppo_kl:np.float64(4.479127906620306e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.10549908224493265) - perf/mfu/actor:np.float64(0.11212622068994285) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.87326049804688) - actor/lr:np.float64(1e-06) - training/global_step:39 - training/epoch:1 - critic/score/mean:0.87890625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.87890625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.009453050792217255 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.009453050792217255 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:640.44921875 - response_length/max:3306.0 - response_length/min:119.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:640.44921875 - response_length_non_aborted/max:3306.0 - response_length_non_aborted/min:119.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:258.21875 - prompt_length/max:326.0 - prompt_length/min:155.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001562461256980896 - timing_s/agent_loop/generate_sequences/min:np.float64(1.4932099115103483) - timing_s/agent_loop/generate_sequences/max:np.float64(23.016741577535868) - timing_s/agent_loop/generate_sequences/mean:np.float64(6.735508493344241) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(23.016741577535868) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:304 - timing_s/agent_loop/slowest/response_length:3306 - timing_s/gen:30.13746011815965 - timing_s/reward:0.07099534384906292 - timing_s/old_log_prob:3.4354004375636578 - timing_s/adv:0.1418602205812931 - timing_s/update_actor:13.508899280801415 - timing_s/step:47.37992210872471 - timing_s/stop_profile:7.83577561378479e-05 - timing_per_token_ms/gen:0.18381543788331953 - timing_per_token_ms/update_actor:0.05871928192681623 - timing_per_token_ms/adv:0.0006166253899273365 - perf/total_num_tokens:230059 - perf/time_per_step:47.37992210872471 - perf/throughput:606.9527707118056
(TaskRunner pid=2281424) Training Progress: 39%|███▉ | 39/100 [39:23<55:49, 54.90s/it]
(TaskRunner pid=2281424) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 40}
(TaskRunner pid=2281424) validation generation end
(TaskRunner pid=2281424) [prompt] system
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2281424) <reasoning>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </reasoning>
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) A
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) user
(TaskRunner pid=2281424) A charged particle produces an electric field with a magnitude of $2.0 \mathrm{~N} / \mathrm{C}$ at a point that is $50 \mathrm{~cm}$ away from the particle. What is the magnitude of the particle's charge? Recall that the electric field $E$ due to a point charge $q$ at a distance $r$ is given by $E = k\frac{q}{r^2}$, where $k = 8.99 \times 10^9 \mathrm{N} \mathrm{m}^2 / \mathrm{C}^2$.
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) A: 50 pC
(TaskRunner pid=2281424) B: 56 pC
(TaskRunner pid=2281424) C: 60 pC
(TaskRunner pid=2281424) D: 64 pC
(TaskRunner pid=2281424) Please reason step by step.
(TaskRunner pid=2281424) assistant
(TaskRunner pid=2281424) <think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) </think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) [response] To find the magnitude of the particle's charge, we can use the formula for the electric field due to a point charge:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) E = k \frac{q}{r^2}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Where:
(TaskRunner pid=2281424) - $ E = 2.0 \, \mathrm{N/C} $
(TaskRunner pid=2281424) - $ k = 8.99 \times 10^9 \, \mathrm{N \cdot m^2/C^2} $
(TaskRunner pid=2281424) - $ r = 50 \, \mathrm{cm} = 0.50 \, \mathrm{m} $
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) We need to solve for $ q $:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{E \cdot r^2}{k}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Now, substitute the known values:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{2.0 \cdot (0.50)^2}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{2.0 \cdot 0.25}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{0.5}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 5.56 \times 10^{-11} \, \mathrm{C}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Now, convert coulombs to picocoulombs ($1 \, \mathrm{pC} = 10^{-12} \, \mathrm{C}$):
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 5.56 \times 10^{-11} \, \mathrm{C} = 55.6 \, \mathrm{pC}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) So, the closest answer is:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) B
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424) [ground_truth] B
(TaskRunner pid=2281424) [score] 1.0
(TaskRunner pid=2281424) [acc] 1.0
(TaskRunner pid=2281424) [pred] B
(TaskRunner pid=2281424) [incorrect_format] 1
(TaskRunner pid=2281424) [feedback]
(TaskRunner pid=2281424) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_40
(TaskRunner pid=2281424) step:40 - global_seqlen/min:25284 - global_seqlen/max:51779 - global_seqlen/minmax_diff:26495 - global_seqlen/balanced_min:36707 - global_seqlen/balanced_max:38750 - global_seqlen/mean:36970.5 - actor/entropy:0.2809111177921295 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35805636644363403 - training/rollout_probs_diff_mean:0.0029524413403123617 - training/rollout_probs_diff_std:0.008548099547624588 - training/rollout_actor_probs_pearson_corr:0.9994715452194214 - rollout_corr/rollout_is_mean:0.9999876022338867 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.368109330243897e-06 - rollout_corr/rollout_is_max:1.9496772289276123 - rollout_corr/rollout_is_min:0.451233834028244 - rollout_corr/rollout_is_std:0.026926398277282715 - rollout_corr/rollout_is_eff_sample_size:0.999275494389906 - rollout_corr/rollout_is_seq_mean:0.9999891519546509 - rollout_corr/rollout_is_seq_std:0.0013239040272310376 - rollout_corr/rollout_is_seq_max:1.0059114694595337 - rollout_corr/rollout_is_seq_min:0.9929170608520508 - rollout_corr/rollout_is_seq_max_deviation:0.007082939147949219 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.2970258207060397) - rollout_corr/training_log_ppl:np.float64(0.24729529570322484) - rollout_corr/kl:np.float64(0.00048714316951858905) - rollout_corr/k3_kl:np.float64(0.00042656138907659624) - rollout_corr/rollout_ppl:np.float64(1.2963653528131545) - rollout_corr/rollout_log_ppl:np.float64(0.24680815133615397) - rollout_corr/log_ppl_diff:np.float64(0.0004871443670708686) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010458430915605277) - rollout_corr/log_ppl_diff_max:np.float64(0.007907375693321228) - rollout_corr/log_ppl_diff_min:np.float64(-0.00467248260974884) - rollout_corr/ppl_ratio:np.float64(1.0004882379435003) - rollout_corr/chi2_token:np.float64(0.0007523114327341318) - rollout_corr/chi2_seq:np.float64(0.4919047087896615) - actor/pg_loss:np.float64(0.00013426528312265873) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(9.462851380703796e-05) - actor/ppo_kl:np.float64(9.811125503489393e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.10963228065520525) - perf/mfu/actor:np.float64(0.13216087132668086) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(105.88793182373047) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.62109375) - val-aux/sciknoweval/reward/std@16:np.float64(0.20978195620494514) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7098125) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.19512127996631243) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.5332000000000001) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.16365501611769506) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.62185) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.209297903919098) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.7927750000000001) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.15226235422617695) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.47158749999999994) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.10047297589092027) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6512499999999999) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.1840588452797897) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8536250000000001) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.09931525131049548) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.43674999999999997) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.05502894877052068) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.6747375) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.15323121047661067) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.8911) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.05002471514195125) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.416225) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.03088195571501212) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.6931875) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.12136011366811209) - val-aux/sciknoweval/score/mean@16:np.float64(0.62109375) - val-aux/sciknoweval/score/std@16:np.float64(0.20978195620494514) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7098125) - val-aux/sciknoweval/score/best@2/std:np.float64(0.19512127996631243) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.5332000000000001) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.16365501611769506) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.62185) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.209297903919098) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.7927750000000001) - val-aux/sciknoweval/score/best@4/std:np.float64(0.15226235422617695) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.47158749999999994) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.10047297589092027) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6512499999999999) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.1840588452797897) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8536250000000001) - val-aux/sciknoweval/score/best@8/std:np.float64(0.09931525131049548) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.43674999999999997) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.05502894877052068) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.6747375) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.15323121047661067) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.8911) - val-aux/sciknoweval/score/best@16/std:np.float64(0.05002471514195125) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.416225) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.03088195571501212) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.6931875) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.12136011366811209) - val-core/sciknoweval/acc/mean@16:np.float64(0.62109375) - val-aux/sciknoweval/acc/std@16:np.float64(0.20978195620494514) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7098125) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.19512127996631243) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.5332000000000001) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.16365501611769506) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.62185) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.209297903919098) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.7927750000000001) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.15226235422617695) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.47158749999999994) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.10047297589092027) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6512499999999999) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.1840588452797897) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8536250000000001) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.09931525131049548) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.43674999999999997) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.05502894877052068) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.6747375) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.15323121047661067) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.8911) - val-core/sciknoweval/acc/best@16/std:np.float64(0.05002471514195125) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.416225) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.03088195571501212) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.6931875) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.12136011366811209) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.93046875) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.12202913367678088) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9769125000000001) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.06539542499226024) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.8835625) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.13970718792907938) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9306750000000001) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.1211440536861494) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9951250000000001) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.02218746491637611) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.823675) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.14131489814444675) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.956325) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.09419032337440465) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(0.9994624999999999) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.004248820241722828) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.760775) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.12100807936316986) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9767624999999999) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.06242768994051049) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.7088500000000001) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.08327809515343695) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.99055) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.030619804261104205) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:40 - training/epoch:1 - critic/score/mean:0.83203125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.83203125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.0065811024978756905 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.0065811024978756905 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:894.265625 - response_length/max:5469.0 - response_length/min:148.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:894.265625 - response_length_non_aborted/max:5469.0 - response_length_non_aborted/min:148.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:261.0625 - prompt_length/max:337.0 - prompt_length/min:158.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.171664714813232e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.883589567616582) - timing_s/agent_loop/generate_sequences/max:np.float64(36.48411184921861) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.177588642422052) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(36.48411184921861) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:313 - timing_s/agent_loop/slowest/response_length:5469 - timing_s/gen:43.124824890866876 - timing_s/reward:0.08122630603611469 - timing_s/old_log_prob:3.681945353746414 - timing_s/adv:0.15046975389122963 - timing_s/update_actor:14.803198579698801 - timing_s/step:61.928098333999515 - timing_s/testing:162.62985380925238 - timing_s/save_checkpoint:12.925621895119548 - timing_s/stop_profile:0.00014221295714378357 - timing_per_token_ms/gen:0.1883739489930061 - timing_per_token_ms/update_actor:0.0500507113093507 - timing_per_token_ms/adv:0.0005087493876578272 - perf/total_num_tokens:295764 - perf/time_per_step:61.928098333999515 - perf/throughput:596.9907197958088
(TaskRunner pid=2281424) Training Progress: 40%|████ | 40/100 [43:20<1:49:42, 109.70s/it]
(TaskRunner pid=2281424) step:41 - global_seqlen/min:29658 - global_seqlen/max:51620 - global_seqlen/minmax_diff:21962 - global_seqlen/balanced_min:38538 - global_seqlen/balanced_max:38782 - global_seqlen/mean:38637.75 - actor/entropy:0.33072540163993835 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.35299545526504517 - training/rollout_probs_diff_mean:0.0031244975980371237 - training/rollout_probs_diff_std:0.008380025625228882 - training/rollout_actor_probs_pearson_corr:0.9994889497756958 - rollout_corr/rollout_is_mean:1.000058650970459 - rollout_corr/rollout_is_ratio_fraction_high:1.260218232346233e-05 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:4.29278564453125 - rollout_corr/rollout_is_min:0.55013507604599 - rollout_corr/rollout_is_std:0.027836313471198082 - rollout_corr/rollout_is_eff_sample_size:0.999225739561659 - rollout_corr/rollout_is_seq_mean:0.9999604225158691 - rollout_corr/rollout_is_seq_std:0.0012300461530685425 - rollout_corr/rollout_is_seq_max:1.006003737449646 - rollout_corr/rollout_is_seq_min:0.9945253133773804 - rollout_corr/rollout_is_seq_max_deviation:0.006003737449645996 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3499207119457424) - rollout_corr/training_log_ppl:np.float64(0.2855829408799764) - rollout_corr/kl:np.float64(0.0005269990794536383) - rollout_corr/k3_kl:np.float64(0.0004426697283577141) - rollout_corr/rollout_ppl:np.float64(1.349194892682135) - rollout_corr/rollout_log_ppl:np.float64(0.28505594066518825) - rollout_corr/log_ppl_diff:np.float64(0.0005270002147881314) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011361698125256225) - rollout_corr/log_ppl_diff_max:np.float64(0.005944006145000458) - rollout_corr/log_ppl_diff_min:np.float64(-0.005369067192077637) - rollout_corr/ppl_ratio:np.float64(1.0005282380152494) - rollout_corr/chi2_token:np.float64(0.0007350665982812643) - rollout_corr/chi2_seq:np.float64(4.942351899808273) - actor/pg_loss:np.float64(4.62154857814312e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001543072065146589) - actor/ppo_kl:np.float64(0.0001058366084052409) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.12146356515586376) - perf/mfu/actor:np.float64(0.13938522742671805) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.29471588134766) - actor/lr:np.float64(1e-06) - training/global_step:41 - training/epoch:1 - critic/score/mean:0.70703125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.70703125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.004173422697931528 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.004173422697931528 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:929.8984375 - response_length/max:4125.0 - response_length/min:111.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:929.8984375 - response_length_non_aborted/max:4125.0 - response_length_non_aborted/min:111.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.53125 - prompt_length/max:361.0 - prompt_length/min:178.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00015160441398620605 - timing_s/agent_loop/generate_sequences/min:np.float64(1.333501910790801) - timing_s/agent_loop/generate_sequences/max:np.float64(29.424998305737972) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.613472268974874) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(29.424998305737972) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:297 - timing_s/agent_loop/slowest/response_length:4125 - timing_s/gen:35.98993204906583 - timing_s/reward:0.08053253218531609 - timing_s/old_log_prob:3.700353318825364 - timing_s/adv:0.16254926659166813 - timing_s/update_actor:14.687377415597439 - timing_s/step:54.72286199964583 - timing_s/stop_profile:0.00014472007751464844 - timing_per_token_ms/gen:0.15118389965749715 - timing_per_token_ms/update_actor:0.04751628076038796 - timing_per_token_ms/adv:0.0005258758163702213 - perf/total_num_tokens:309102 - perf/time_per_step:54.72286199964583 - perf/throughput:706.0623035441762
(TaskRunner pid=2281424) Training Progress: 41%|████ | 41/100 [44:15<1:31:40, 93.23s/it]
(TaskRunner pid=2281424) step:42 - global_seqlen/min:34807 - global_seqlen/max:57617 - global_seqlen/minmax_diff:22810 - global_seqlen/balanced_min:46153 - global_seqlen/balanced_max:46569 - global_seqlen/mean:46354.0 - actor/entropy:0.4125242531299591 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.27730005979537964 - training/rollout_probs_diff_mean:0.0032492319587618113 - training/rollout_probs_diff_std:0.00808805413544178 - training/rollout_actor_probs_pearson_corr:0.9996025562286377 - rollout_corr/rollout_is_mean:0.9999741911888123 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.913554094964638e-06 - rollout_corr/rollout_is_max:1.8443630933761597 - rollout_corr/rollout_is_min:0.3683568835258484 - rollout_corr/rollout_is_std:0.028246553614735603 - rollout_corr/rollout_is_eff_sample_size:0.9992026492912346 - rollout_corr/rollout_is_seq_mean:1.0000536441802979 - rollout_corr/rollout_is_seq_std:0.0012193896109238267 - rollout_corr/rollout_is_seq_max:1.009535312652588 - rollout_corr/rollout_is_seq_min:0.9968177080154419 - rollout_corr/rollout_is_seq_max_deviation:0.00953531265258789 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4070447161793709) - rollout_corr/training_log_ppl:np.float64(0.3194918569060974) - rollout_corr/kl:np.float64(0.00035904505688222343) - rollout_corr/k3_kl:np.float64(0.00043011571953854855) - rollout_corr/rollout_ppl:np.float64(1.4064954705536366) - rollout_corr/rollout_log_ppl:np.float64(0.3191328103421256) - rollout_corr/log_ppl_diff:np.float64(0.0003590465639717877) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009108688100241125) - rollout_corr/log_ppl_diff_max:np.float64(0.0039010122418403625) - rollout_corr/log_ppl_diff_min:np.float64(-0.003787517547607422) - rollout_corr/ppl_ratio:np.float64(1.0003597426693887) - rollout_corr/chi2_token:np.float64(0.0010020083282142878) - rollout_corr/chi2_seq:np.float64(0.939779143081978) - actor/pg_loss:np.float64(0.00010238576214760542) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001587477142379612) - actor/ppo_kl:np.float64(2.258576988367622e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.11312472820281982) - perf/mfu/actor:np.float64(0.15362174541859897) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.15521240234375) - actor/lr:np.float64(1e-06) - training/global_step:42 - training/epoch:1 - critic/score/mean:0.6171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.01894480176270008 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.01894480176270008 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1182.09375 - response_length/max:6248.0 - response_length/min:160.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1182.09375 - response_length_non_aborted/max:6248.0 - response_length_non_aborted/min:160.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.46875 - prompt_length/max:355.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.07406210899353e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.8849900215864182) - timing_s/agent_loop/generate_sequences/max:np.float64(42.688118137419224) - timing_s/agent_loop/generate_sequences/mean:np.float64(11.67329497004539) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(42.688118137419224) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:237 - timing_s/agent_loop/slowest/response_length:6248 - timing_s/gen:50.097527500241995 - timing_s/reward:0.09398127906024456 - timing_s/old_log_prob:3.954119984060526 - timing_s/adv:0.14543932490050793 - timing_s/update_actor:16.064200446009636 - timing_s/step:70.44157047756016 - timing_s/stop_profile:0.00011380389332771301 - timing_per_token_ms/gen:0.16554817822006104 - timing_per_token_ms/update_actor:0.04331934796891756 - timing_per_token_ms/adv:0.00039219734246372464 - perf/total_num_tokens:370832 - perf/time_per_step:70.44157047756016 - perf/throughput:658.0489288603596
(TaskRunner pid=2281424) Training Progress: 42%|████▏ | 42/100 [45:26<1:23:31, 86.41s/it]
(TaskRunner pid=2281424) step:43 - global_seqlen/min:27945 - global_seqlen/max:57053 - global_seqlen/minmax_diff:29108 - global_seqlen/balanced_min:46139 - global_seqlen/balanced_max:46442 - global_seqlen/mean:46394.125 - actor/entropy:0.3777778148651123 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.24276472628116608 - training/rollout_probs_diff_mean:0.0031969803385436535 - training/rollout_probs_diff_std:0.008178768679499626 - training/rollout_actor_probs_pearson_corr:0.9996250867843628 - rollout_corr/rollout_is_mean:0.9999624490737915 - rollout_corr/rollout_is_ratio_fraction_high:3.310370402687113e-06 - rollout_corr/rollout_is_ratio_fraction_low:6.620740805374226e-06 - rollout_corr/rollout_is_max:3.3981943130493164 - rollout_corr/rollout_is_min:0.40390288829803467 - rollout_corr/rollout_is_std:0.027761269360780716 - rollout_corr/rollout_is_eff_sample_size:0.9992297864199887 - rollout_corr/rollout_is_seq_mean:0.9999164938926697 - rollout_corr/rollout_is_seq_std:0.0013294965028762817 - rollout_corr/rollout_is_seq_max:1.0057275295257568 - rollout_corr/rollout_is_seq_min:0.9940543174743652 - rollout_corr/rollout_is_seq_max_deviation:0.005945682525634766 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.368367766495794) - rollout_corr/training_log_ppl:np.float64(0.297560689446982) - rollout_corr/kl:np.float64(0.0004662561269233034) - rollout_corr/k3_kl:np.float64(0.00041893447971119713) - rollout_corr/rollout_ppl:np.float64(1.3677249136380851) - rollout_corr/rollout_log_ppl:np.float64(0.29709443140018266) - rollout_corr/log_ppl_diff:np.float64(0.00046625804679933935) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009343025594716892) - rollout_corr/log_ppl_diff_max:np.float64(0.006767094135284424) - rollout_corr/log_ppl_diff_min:np.float64(-0.0033786147832870483) - rollout_corr/ppl_ratio:np.float64(1.000467125326395) - rollout_corr/chi2_token:np.float64(0.0007502012886106968) - rollout_corr/chi2_seq:np.float64(0.8961137712467462) - actor/pg_loss:np.float64(7.715250831097364e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00018895000766860903) - actor/ppo_kl:np.float64(-1.2429667751945317e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.12724709697067738) - perf/mfu/actor:np.float64(0.15634425835350255) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.0877799987793) - actor/lr:np.float64(1e-06) - training/global_step:43 - training/epoch:1 - critic/score/mean:0.74609375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.74609375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.015876950696110725 - critic/advantages/max:0.75 - critic/advantages/min:-0.875 - critic/returns/mean:-0.015876950696110725 - critic/returns/max:0.75 - critic/returns/min:-0.875 - response_length/mean:1180.00390625 - response_length/max:5656.0 - response_length/min:122.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1180.00390625 - response_length_non_aborted/max:5656.0 - response_length_non_aborted/min:122.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:269.8125 - prompt_length/max:365.0 - prompt_length/min:186.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011191703379154205 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1672064159065485) - timing_s/agent_loop/generate_sequences/max:np.float64(40.370598992332816) - timing_s/agent_loop/generate_sequences/mean:np.float64(10.840922129631508) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(40.370598992332816) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:277 - timing_s/agent_loop/slowest/response_length:5656 - timing_s/gen:47.32844007387757 - timing_s/reward:0.09296286478638649 - timing_s/old_log_prob:3.898196619004011 - timing_s/adv:0.13054160960018635 - timing_s/update_actor:15.71086062118411 - timing_s/step:67.24753057211637 - timing_s/stop_profile:2.9379501938819885e-05 - timing_per_token_ms/gen:0.15667466697302238 - timing_per_token_ms/update_actor:0.04232987641534384 - timing_per_token_ms/adv:0.00035171912823063896 - perf/total_num_tokens:371153 - perf/time_per_step:67.24753057211637 - perf/throughput:689.9007979221907
(TaskRunner pid=2281424) Training Progress: 43%|████▎ | 43/100 [46:33<1:16:37, 80.66s/it]
(TaskRunner pid=2281424) step:44 - global_seqlen/min:31039 - global_seqlen/max:81483 - global_seqlen/minmax_diff:50444 - global_seqlen/balanced_min:49966 - global_seqlen/balanced_max:50221 - global_seqlen/mean:50039.375 - actor/entropy:0.42505836486816406 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.40381795167922974 - training/rollout_probs_diff_mean:0.0033156690187752247 - training/rollout_probs_diff_std:0.00811709649860859 - training/rollout_actor_probs_pearson_corr:0.9995504021644592 - rollout_corr/rollout_is_mean:0.9999823570251465 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.108491212828085e-06 - rollout_corr/rollout_is_max:1.868242859840393 - rollout_corr/rollout_is_min:0.16989949345588684 - rollout_corr/rollout_is_std:0.028271863237023354 - rollout_corr/rollout_is_eff_sample_size:0.9992012210621306 - rollout_corr/rollout_is_seq_mean:1.0000169277191162 - rollout_corr/rollout_is_seq_std:0.0010278340196236968 - rollout_corr/rollout_is_seq_max:1.0032761096954346 - rollout_corr/rollout_is_seq_min:0.9966418147087097 - rollout_corr/rollout_is_seq_max_deviation:0.003358185291290283 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4229408679530025) - rollout_corr/training_log_ppl:np.float64(0.3282105611579027) - rollout_corr/kl:np.float64(0.0004335679858704644) - rollout_corr/k3_kl:np.float64(0.00045822281549590116) - rollout_corr/rollout_ppl:np.float64(1.422240549698472) - rollout_corr/rollout_log_ppl:np.float64(0.3277769897831604) - rollout_corr/log_ppl_diff:np.float64(0.0004335713747423142) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009576250740792602) - rollout_corr/log_ppl_diff_max:np.float64(0.004134416580200195) - rollout_corr/log_ppl_diff_min:np.float64(-0.004089929163455963) - rollout_corr/ppl_ratio:np.float64(1.000434412388131) - rollout_corr/chi2_token:np.float64(0.0009704537224024534) - rollout_corr/chi2_seq:np.float64(1.733142590848729) - actor/pg_loss:np.float64(2.68502626568079e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00020757744624688712) - actor/ppo_kl:np.float64(6.416326252889348e-05) - actor/pg_clipfrac_lower:np.float64(1.8610052165968227e-06) - actor/grad_norm:np.float64(0.14784160628914833) - perf/mfu/actor:np.float64(0.1614180488812359) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.14054107666016) - actor/lr:np.float64(1e-06) - training/global_step:44 - training/epoch:1 - critic/score/mean:0.66796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.66796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.005857898388057947 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.005857898388057947 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1286.57421875 - response_length/max:7615.0 - response_length/min:164.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1286.57421875 - response_length_non_aborted/max:7615.0 - response_length_non_aborted/min:164.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:277.15625 - prompt_length/max:460.0 - prompt_length/min:184.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:2.9996037483215332e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.0687543358653784) - timing_s/agent_loop/generate_sequences/max:np.float64(51.31303711421788) - timing_s/agent_loop/generate_sequences/mean:np.float64(12.701673694929923) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(51.31303711421788) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:267 - timing_s/agent_loop/slowest/response_length:7615 - timing_s/gen:58.36772031150758 - timing_s/reward:0.09810716658830643 - timing_s/old_log_prob:4.152743987739086 - timing_s/adv:0.15301652811467648 - timing_s/update_actor:16.473277961835265 - timing_s/step:79.3296963069588 - timing_s/stop_profile:3.1422823667526245e-05 - timing_per_token_ms/gen:0.17721395636883192 - timing_per_token_ms/update_actor:0.04115078865851958 - timing_per_token_ms/adv:0.0003822403060456802 - perf/total_num_tokens:400315 - perf/time_per_step:79.3296963069588 - perf/throughput:630.777342275172
(TaskRunner pid=2281424) Training Progress: 44%|████▍ | 44/100 [47:52<1:14:55, 80.27s/it]
(TaskRunner pid=2281424) step:45 - global_seqlen/min:29259 - global_seqlen/max:67289 - global_seqlen/minmax_diff:38030 - global_seqlen/balanced_min:45428 - global_seqlen/balanced_max:45525 - global_seqlen/mean:45499.625 - actor/entropy:0.39985886216163635 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.25675493478775024 - training/rollout_probs_diff_mean:0.0032300371676683426 - training/rollout_probs_diff_std:0.00817596260458231 - training/rollout_actor_probs_pearson_corr:0.9996156692504883 - rollout_corr/rollout_is_mean:0.9999929666519165 - rollout_corr/rollout_is_ratio_fraction_high:3.4166423574788496e-06 - rollout_corr/rollout_is_ratio_fraction_low:3.4166423574788496e-06 - rollout_corr/rollout_is_max:2.063965082168579 - rollout_corr/rollout_is_min:0.2600027620792389 - rollout_corr/rollout_is_std:0.028221219778060913 - rollout_corr/rollout_is_eff_sample_size:0.9992040775244214 - rollout_corr/rollout_is_seq_mean:0.999981701374054 - rollout_corr/rollout_is_seq_std:0.0011321797501295805 - rollout_corr/rollout_is_seq_max:1.00321364402771 - rollout_corr/rollout_is_seq_min:0.9935727119445801 - rollout_corr/rollout_is_seq_max_deviation:0.006427288055419922 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4365097098052502) - rollout_corr/training_log_ppl:np.float64(0.33815656184742693) - rollout_corr/kl:np.float64(0.0006105463397099165) - rollout_corr/k3_kl:np.float64(0.0004829082062087764) - rollout_corr/rollout_ppl:np.float64(1.4356286954134703) - rollout_corr/rollout_log_ppl:np.float64(0.3375460135139292) - rollout_corr/log_ppl_diff:np.float64(0.000610548333497718) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009788531460799277) - rollout_corr/log_ppl_diff_max:np.float64(0.007325723767280579) - rollout_corr/log_ppl_diff_min:np.float64(-0.0033019185066223145) - rollout_corr/ppl_ratio:np.float64(1.0006115036085248) - rollout_corr/chi2_token:np.float64(0.0007435702718794346) - rollout_corr/chi2_seq:np.float64(2.2746842054184526) - actor/pg_loss:np.float64(7.623340934514999e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001888360293378355) - actor/ppo_kl:np.float64(0.0001864508021274247) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.12979234382510185) - perf/mfu/actor:np.float64(0.15628703385179005) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.251708984375) - actor/lr:np.float64(1e-06) - training/global_step:45 - training/epoch:2 - critic/score/mean:0.66796875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.66796875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.008445513434708118 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.008445513434708118 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1143.30078125 - response_length/max:3842.0 - response_length/min:192.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1143.30078125 - response_length_non_aborted/max:3842.0 - response_length_non_aborted/min:192.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:278.5625 - prompt_length/max:437.0 - prompt_length/min:208.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0002876799553632736 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4185115825384855) - timing_s/agent_loop/generate_sequences/max:np.float64(29.632253983989358) - timing_s/agent_loop/generate_sequences/mean:np.float64(11.899439175627776) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(29.632253983989358) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:258 - timing_s/agent_loop/slowest/response_length:3842 - timing_s/gen:36.76520950347185 - timing_s/reward:0.08381190523505211 - timing_s/old_log_prob:3.9696482811123133 - timing_s/adv:0.12938232719898224 - timing_s/update_actor:15.343466022983193 - timing_s/step:56.40911629982293 - timing_s/stop_profile:0.00011620856821537018 - timing_per_token_ms/gen:0.12561357604069853 - timing_per_token_ms/update_actor:0.0421527265966016 - timing_per_token_ms/adv:0.0003554488833671218 - perf/total_num_tokens:363997 - perf/time_per_step:56.40911629982293 - perf/throughput:806.600563606823
(TaskRunner pid=2281424) Training Progress: 45%|████▌ | 45/100 [48:49<1:07:07, 73.23s/it]
(TaskRunner pid=2281424) step:46 - global_seqlen/min:20945 - global_seqlen/max:94884 - global_seqlen/minmax_diff:73939 - global_seqlen/balanced_min:43085 - global_seqlen/balanced_max:44324 - global_seqlen/mean:43425.25 - actor/entropy:0.3762817084789276 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5543509721755981 - training/rollout_probs_diff_mean:0.0032524620182812214 - training/rollout_probs_diff_std:0.008445695042610168 - training/rollout_actor_probs_pearson_corr:0.9995606541633606 - rollout_corr/rollout_is_mean:0.999961793422699 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:1.4132578144199215e-05 - rollout_corr/rollout_is_max:1.6447221040725708 - rollout_corr/rollout_is_min:0.002203931100666523 - rollout_corr/rollout_is_std:0.027791311964392662 - rollout_corr/rollout_is_eff_sample_size:0.999228120062589 - rollout_corr/rollout_is_seq_mean:0.9999802112579346 - rollout_corr/rollout_is_seq_std:0.0012473923852667212 - rollout_corr/rollout_is_seq_max:1.0055023431777954 - rollout_corr/rollout_is_seq_min:0.9959372878074646 - rollout_corr/rollout_is_seq_max_deviation:0.00550234317779541 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3645080188289285) - rollout_corr/training_log_ppl:np.float64(0.28979677851020824) - rollout_corr/kl:np.float64(0.0005104832347480048) - rollout_corr/k3_kl:np.float64(0.00045839784411327855) - rollout_corr/rollout_ppl:np.float64(1.3638080032542348) - rollout_corr/rollout_log_ppl:np.float64(0.2892862949083792) - rollout_corr/log_ppl_diff:np.float64(0.0005104836018290371) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010666771559044719) - rollout_corr/log_ppl_diff_max:np.float64(0.005919963121414185) - rollout_corr/log_ppl_diff_min:np.float64(-0.004801034927368164) - rollout_corr/ppl_ratio:np.float64(1.0005116499960423) - rollout_corr/chi2_token:np.float64(0.0008005544077605009) - rollout_corr/chi2_seq:np.float64(1.0131068772170693) - actor/pg_loss:np.float64(-1.758092548698187e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(8.805006137890814e-05) - actor/ppo_kl:np.float64(7.538383222893685e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.10063287615776062) - perf/mfu/actor:np.float64(0.14549136899234336) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.28108978271484) - actor/lr:np.float64(1e-06) - training/global_step:46 - training/epoch:2 - critic/score/mean:0.7890625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.7890625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.027433983981609344 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.027433983981609344 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1105.6015625 - response_length/max:8192.0 - response_length/min:136.0 - response_length/clip_ratio:0.0078125 - response_length_non_aborted/mean:1105.6015625 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:136.0 - response_length_non_aborted/clip_ratio:0.0078125 - response/aborted_ratio:0.0 - prompt_length/mean:251.4375 - prompt_length/max:337.0 - prompt_length/min:158.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001056082546710968 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1990190539509058) - timing_s/agent_loop/generate_sequences/max:np.float64(59.229342663660645) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.871870818904426) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(59.229342663660645) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:259 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:65.95301722921431 - timing_s/reward:0.08986181393265724 - timing_s/old_log_prob:3.924274370074272 - timing_s/adv:0.1476494651287794 - timing_s/update_actor:16.007332164794207 - timing_s/step:86.20792456716299 - timing_s/stop_profile:5.9446319937705994e-05 - timing_per_token_ms/gen:0.23302153532513517 - timing_per_token_ms/update_actor:0.04607725967263921 - timing_per_token_ms/adv:0.0004250104061829794 - perf/total_num_tokens:347402 - perf/time_per_step:86.20792456716299 - perf/throughput:503.72689306733275
(TaskRunner pid=2281424) Training Progress: 46%|████▌ | 46/100 [50:15<1:09:24, 77.13s/it]
(TaskRunner pid=2281424) step:47 - global_seqlen/min:31035 - global_seqlen/max:67357 - global_seqlen/minmax_diff:36322 - global_seqlen/balanced_min:49399 - global_seqlen/balanced_max:49649 - global_seqlen/mean:49578.625 - actor/entropy:0.34646978974342346 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5756693482398987 - training/rollout_probs_diff_mean:0.0030399137176573277 - training/rollout_probs_diff_std:0.00816922727972269 - training/rollout_actor_probs_pearson_corr:0.9995539784431458 - rollout_corr/rollout_is_mean:0.9999984502792358 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:9.122617484536022e-06 - rollout_corr/rollout_is_max:1.9699045419692993 - rollout_corr/rollout_is_min:0.25761306285858154 - rollout_corr/rollout_is_std:0.026937464252114296 - rollout_corr/rollout_is_eff_sample_size:0.9992746611342851 - rollout_corr/rollout_is_seq_mean:1.0000650882720947 - rollout_corr/rollout_is_seq_std:0.001198867685161531 - rollout_corr/rollout_is_seq_max:1.005571961402893 - rollout_corr/rollout_is_seq_min:0.9964936971664429 - rollout_corr/rollout_is_seq_max_deviation:0.005571961402893066 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3529047267511487) - rollout_corr/training_log_ppl:np.float64(0.28667624859372154) - rollout_corr/kl:np.float64(0.0004597161929975613) - rollout_corr/k3_kl:np.float64(0.00046043861678413123) - rollout_corr/rollout_ppl:np.float64(1.3522660555317998) - rollout_corr/rollout_log_ppl:np.float64(0.2862165296683088) - rollout_corr/log_ppl_diff:np.float64(0.0004597189254127443) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009910539374686778) - rollout_corr/log_ppl_diff_max:np.float64(0.007551088929176331) - rollout_corr/log_ppl_diff_min:np.float64(-0.004575580358505249) - rollout_corr/ppl_ratio:np.float64(1.0004606684669852) - rollout_corr/chi2_token:np.float64(0.0009119359310716391) - rollout_corr/chi2_seq:np.float64(954.7251316532493) - actor/pg_loss:np.float64(-8.133123628795147e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001264397735667444) - actor/ppo_kl:np.float64(0.0001422462675852998) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.11066612228751183) - perf/mfu/actor:np.float64(0.1590812815304962) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.31835174560547) - actor/lr:np.float64(1e-06) - training/global_step:47 - training/epoch:2 - critic/score/mean:0.75 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.75 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.024808576330542564 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.024808576330542564 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1284.58203125 - response_length/max:7270.0 - response_length/min:165.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1284.58203125 - response_length_non_aborted/max:7270.0 - response_length_non_aborted/min:165.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:264.75 - prompt_length/max:356.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:5.761533975601196e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(2.059980768710375) - timing_s/agent_loop/generate_sequences/max:np.float64(50.74220941402018) - timing_s/agent_loop/generate_sequences/mean:np.float64(12.601909340250131) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(50.74220941402018) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:284 - timing_s/agent_loop/slowest/response_length:7270 - timing_s/gen:58.42462229728699 - timing_s/reward:0.09585716761648655 - timing_s/old_log_prob:3.987767066806555 - timing_s/adv:0.1361205969005823 - timing_s/update_actor:16.680994622409344 - timing_s/step:79.40998443961143 - timing_s/stop_profile:0.00011073052883148193 - timing_per_token_ms/gen:0.17766181940650377 - timing_per_token_ms/update_actor:0.04205692125994152 - timing_per_token_ms/adv:0.0003431937576440006 - perf/total_num_tokens:396629 - perf/time_per_step:79.40998443961143 - perf/throughput:624.3374224270607
(TaskRunner pid=2281424) Training Progress: 47%|████▋ | 47/100 [51:35<1:08:44, 77.83s/it]
(TaskRunner pid=2281424) step:48 - global_seqlen/min:22232 - global_seqlen/max:57431 - global_seqlen/minmax_diff:35199 - global_seqlen/balanced_min:41913 - global_seqlen/balanced_max:42517 - global_seqlen/mean:42023.25 - actor/entropy:0.34810274839401245 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.2931787967681885 - training/rollout_probs_diff_mean:0.003345586359500885 - training/rollout_probs_diff_std:0.008651177398860455 - training/rollout_actor_probs_pearson_corr:0.9994678497314453 - rollout_corr/rollout_is_mean:1.0000156164169312 - rollout_corr/rollout_is_ratio_fraction_high:7.471328899555374e-06 - rollout_corr/rollout_is_ratio_fraction_low:7.471328899555374e-06 - rollout_corr/rollout_is_max:2.302506685256958 - rollout_corr/rollout_is_min:0.015659699216485023 - rollout_corr/rollout_is_std:0.02865091897547245 - rollout_corr/rollout_is_eff_sample_size:0.9991797981155129 - rollout_corr/rollout_is_seq_mean:1.0000426769256592 - rollout_corr/rollout_is_seq_std:0.0014892894541844726 - rollout_corr/rollout_is_seq_max:1.0078259706497192 - rollout_corr/rollout_is_seq_min:0.9916344881057739 - rollout_corr/rollout_is_seq_max_deviation:0.008365511894226074 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.361631895415485) - rollout_corr/training_log_ppl:np.float64(0.296743152604904) - rollout_corr/kl:np.float64(0.00044437152225995646) - rollout_corr/k3_kl:np.float64(0.0004750857856947732) - rollout_corr/rollout_ppl:np.float64(1.3610092177987099) - rollout_corr/rollout_log_ppl:np.float64(0.2962987821374554) - rollout_corr/log_ppl_diff:np.float64(0.0004443704674486071) - rollout_corr/log_ppl_abs_diff:np.float64(0.0010612811020109802) - rollout_corr/log_ppl_diff_max:np.float64(0.009809985756874084) - rollout_corr/log_ppl_diff_min:np.float64(-0.005429580807685852) - rollout_corr/ppl_ratio:np.float64(1.0004456255119294) - rollout_corr/chi2_token:np.float64(0.0010020604822784662) - rollout_corr/chi2_seq:np.float64(0.4791911744978279) - actor/pg_loss:np.float64(3.651808947324753e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00019356490452082653) - actor/ppo_kl:np.float64(3.127409698944206e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.2193583231419325) - perf/mfu/actor:np.float64(0.14563750602154726) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.25163650512695) - actor/lr:np.float64(1e-06) - training/global_step:48 - training/epoch:2 - critic/score/mean:0.765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.024314971640706062 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.024314971640706062 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1045.6640625 - response_length/max:5423.0 - response_length/min:153.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1045.6640625 - response_length_non_aborted/max:5423.0 - response_length_non_aborted/min:153.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:267.5625 - prompt_length/max:365.0 - prompt_length/min:166.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001386590301990509 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7352543752640486) - timing_s/agent_loop/generate_sequences/max:np.float64(38.22215409949422) - timing_s/agent_loop/generate_sequences/mean:np.float64(10.838098710140912) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(38.22215409949422) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:219 - timing_s/agent_loop/slowest/response_length:5423 - timing_s/gen:45.5628403685987 - timing_s/reward:0.08806743286550045 - timing_s/old_log_prob:3.732063392177224 - timing_s/adv:0.15191545337438583 - timing_s/update_actor:15.251708237454295 - timing_s/step:64.87329536117613 - timing_s/stop_profile:0.00012004934251308441 - timing_per_token_ms/gen:0.17020748017706563 - timing_per_token_ms/update_actor:0.04536687499614587 - timing_per_token_ms/adv:0.00045187917811683365 - perf/total_num_tokens:336186 - perf/time_per_step:64.87329536117613 - perf/throughput:647.7742461831082
(TaskRunner pid=2281424) Training Progress: 48%|████▊ | 48/100 [52:40<1:04:05, 73.96s/it]
(TaskRunner pid=2281424) step:49 - global_seqlen/min:23092 - global_seqlen/max:51168 - global_seqlen/minmax_diff:28076 - global_seqlen/balanced_min:37111 - global_seqlen/balanced_max:37334 - global_seqlen/mean:37259.625 - actor/entropy:0.3179267346858978 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.20985785126686096 - training/rollout_probs_diff_mean:0.0031237066723406315 - training/rollout_probs_diff_std:0.008550681173801422 - training/rollout_actor_probs_pearson_corr:0.9993894100189209 - rollout_corr/rollout_is_mean:0.9999985098838806 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:8.58763451105915e-06 - rollout_corr/rollout_is_max:1.7586421966552734 - rollout_corr/rollout_is_min:0.3407052159309387 - rollout_corr/rollout_is_std:0.02756519615650177 - rollout_corr/rollout_is_eff_sample_size:0.9992407369068808 - rollout_corr/rollout_is_seq_mean:0.9999875426292419 - rollout_corr/rollout_is_seq_std:0.0011641301680356264 - rollout_corr/rollout_is_seq_max:1.0047152042388916 - rollout_corr/rollout_is_seq_min:0.9961039423942566 - rollout_corr/rollout_is_seq_max_deviation:0.0047152042388916016 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.328462516888976) - rollout_corr/training_log_ppl:np.float64(0.2660138398205163) - rollout_corr/kl:np.float64(0.00038798783518378954) - rollout_corr/k3_kl:np.float64(0.0004385801271951095) - rollout_corr/rollout_ppl:np.float64(1.3279090919531882) - rollout_corr/rollout_log_ppl:np.float64(0.26562585204374045) - rollout_corr/log_ppl_diff:np.float64(0.0003879877767758444) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009870426292764023) - rollout_corr/log_ppl_diff_max:np.float64(0.006948456168174744) - rollout_corr/log_ppl_diff_min:np.float64(-0.004289939999580383) - rollout_corr/ppl_ratio:np.float64(1.0003889380022883) - rollout_corr/chi2_token:np.float64(0.0009887455962598324) - rollout_corr/chi2_seq:np.float64(1.5676374214235693) - actor/pg_loss:np.float64(2.256419975310564e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00012307778933973168) - actor/ppo_kl:np.float64(-3.0272832692190832e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.07936635101214051) - perf/mfu/actor:np.float64(0.13578834962618122) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.31588363647461) - actor/lr:np.float64(1e-06) - training/global_step:49 - training/epoch:2 - critic/score/mean:0.859375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.859375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.006559879519045353 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.006559879519045353 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:909.73828125 - response_length/max:4227.0 - response_length/min:93.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:909.73828125 - response_length_non_aborted/max:4227.0 - response_length_non_aborted/min:93.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:254.625 - prompt_length/max:362.0 - prompt_length/min:171.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:9.641796350479126e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.1448177974671125) - timing_s/agent_loop/generate_sequences/max:np.float64(29.94586299546063) - timing_s/agent_loop/generate_sequences/mean:np.float64(8.983509431694984) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(29.94586299546063) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:303 - timing_s/agent_loop/slowest/response_length:4227 - timing_s/gen:37.325577680021524 - timing_s/reward:0.08239317126572132 - timing_s/old_log_prob:3.49440111592412 - timing_s/adv:0.12574291415512562 - timing_s/update_actor:14.33308171108365 - timing_s/step:55.44903611578047 - timing_s/stop_profile:0.00013232603669166565 - timing_per_token_ms/gen:0.16026921238517913 - timing_per_token_ms/update_actor:0.04808516494423807 - timing_per_token_ms/adv:0.00042184708701149576 - perf/total_num_tokens:298077 - perf/time_per_step:55.44903611578047 - perf/throughput:671.9616355855125
(TaskRunner pid=2281424) Training Progress: 49%|████▉ | 49/100 [53:35<58:09, 68.42s/it]
(TaskRunner pid=2281424) test_gen_batch meta info: {'eos_token_id': 151645, 'pad_token_id': 151643, 'recompute_log_prob': False, 'do_sample': True, 'validate': True, 'global_steps': 50}
(TaskRunner pid=2281424) validation generation end
(TaskRunner pid=2281424) [prompt] system
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Given a question and four options, please select the right answer. Respond in the following format:
(TaskRunner pid=2281424) <reasoning>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </reasoning>
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) ...
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) For the answer, only output the letter corresponding to the correct option (A, B, C, or D), and nothing else. Do not restate the answer text. For example, if the answer is "A", just output:
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) A
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) user
(TaskRunner pid=2281424) A charged particle produces an electric field with a magnitude of $2.0 \mathrm{~N} / \mathrm{C}$ at a point that is $50 \mathrm{~cm}$ away from the particle. What is the magnitude of the particle's charge? Recall that the electric field $E$ due to a point charge $q$ at a distance $r$ is given by $E = k\frac{q}{r^2}$, where $k = 8.99 \times 10^9 \mathrm{N} \mathrm{m}^2 / \mathrm{C}^2$.
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) A: 50 pC
(TaskRunner pid=2281424) B: 56 pC
(TaskRunner pid=2281424) C: 60 pC
(TaskRunner pid=2281424) D: 64 pC
(TaskRunner pid=2281424) Please reason step by step.
(TaskRunner pid=2281424) assistant
(TaskRunner pid=2281424) <think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) </think>
(TaskRunner pid=2281424)
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) [response] To find the magnitude of the charge $ q $, we use the formula for the electric field due to a point charge:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) E = k \frac{q}{r^2}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) We are given:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) - $ E = 2.0 \, \mathrm{N/C} $
(TaskRunner pid=2281424) - $ r = 50 \, \mathrm{cm} = 0.50 \, \mathrm{m} $
(TaskRunner pid=2281424) - $ k = 8.99 \times 10^9 \, \mathrm{N \cdot m^2/C^2} $
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) We need to solve for $ q $:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{E \cdot r^2}{k}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Substituting the given values:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{2.0 \cdot (0.50)^2}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{2.0 \cdot 0.25}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q = \frac{0.5}{8.99 \times 10^9}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 5.56 \times 10^{-11} \, \mathrm{C}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) Now convert to picocoulombs (1 pC = $ 10^{-12} $ C):
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424) q \approx 55.6 \, \mathrm{pC}
(TaskRunner pid=2281424) $$
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) So the closest option is:
(TaskRunner pid=2281424)
(TaskRunner pid=2281424) <answer>
(TaskRunner pid=2281424) B
(TaskRunner pid=2281424) </answer>
(TaskRunner pid=2281424) [ground_truth] B
(TaskRunner pid=2281424) [score] 1.0
(TaskRunner pid=2281424) [acc] 1.0
(TaskRunner pid=2281424) [pred] B
(TaskRunner pid=2281424) [incorrect_format] 1
(TaskRunner pid=2281424) [feedback]
(TaskRunner pid=2281424) local_global_step_folder: /mnt/mole/SDPO/L2T/checkpoints/datasets/sciknoweval/physics/qwen3gen-physics-GRPO-Qwen-Qwen3-8B-mbs8-train32-rollout8-lr1e-6-vllm0.8/global_step_50
(TaskRunner pid=2281424) step:50 - global_seqlen/min:34234 - global_seqlen/max:75152 - global_seqlen/minmax_diff:40918 - global_seqlen/balanced_min:51555 - global_seqlen/balanced_max:51907 - global_seqlen/mean:51805.25 - actor/entropy:0.4125359058380127 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.19821900129318237 - training/rollout_probs_diff_mean:0.0032416246831417084 - training/rollout_probs_diff_std:0.008041433990001678 - training/rollout_actor_probs_pearson_corr:0.999610185623169 - rollout_corr/rollout_is_mean:1.0000464916229248 - rollout_corr/rollout_is_ratio_fraction_high:8.665861969348043e-06 - rollout_corr/rollout_is_ratio_fraction_low:0.0 - rollout_corr/rollout_is_max:2.167043685913086 - rollout_corr/rollout_is_min:0.5511428117752075 - rollout_corr/rollout_is_std:0.028244443237781525 - rollout_corr/rollout_is_eff_sample_size:0.9992028873298154 - rollout_corr/rollout_is_seq_mean:0.9999783635139465 - rollout_corr/rollout_is_seq_std:0.0010451057460159063 - rollout_corr/rollout_is_seq_max:1.0044012069702148 - rollout_corr/rollout_is_seq_min:0.9962003827095032 - rollout_corr/rollout_is_seq_max_deviation:0.004401206970214844 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4209796008653939) - rollout_corr/training_log_ppl:np.float64(0.3332373514131177) - rollout_corr/kl:np.float64(0.0004479175979907879) - rollout_corr/k3_kl:np.float64(0.0004798759525215246) - rollout_corr/rollout_ppl:np.float64(1.4203380569815636) - rollout_corr/rollout_log_ppl:np.float64(0.3327894336543977) - rollout_corr/log_ppl_diff:np.float64(0.0004479177587199956) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009449601930100471) - rollout_corr/log_ppl_diff_max:np.float64(0.008978396654129028) - rollout_corr/log_ppl_diff_min:np.float64(-0.004376143217086792) - rollout_corr/ppl_ratio:np.float64(1.0004488993436098) - rollout_corr/chi2_token:np.float64(0.0010542639065533876) - rollout_corr/chi2_seq:np.float64(2.937424761708826) - actor/pg_loss:np.float64(0.00014853477478027344) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00021279246669791974) - actor/ppo_kl:np.float64(1.2714361647070405e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.1673903912305832) - perf/mfu/actor:np.float64(0.16524905376259622) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.29336929321289) - actor/lr:np.float64(1e-06) - val-aux/sciknoweval/reward/mean@16:np.float64(0.65390625) - val-aux/sciknoweval/reward/std@16:np.float64(0.21345823452859586) - val-aux/sciknoweval/reward/best@2/mean:np.float64(0.7464375000000001) - val-aux/sciknoweval/reward/best@2/std:np.float64(0.19899169203769165) - val-aux/sciknoweval/reward/worst@2/mean:np.float64(0.55885) - val-aux/sciknoweval/reward/worst@2/std:np.float64(0.1654343787430585) - val-aux/sciknoweval/reward/maj@2/mean:np.float64(0.6524) - val-aux/sciknoweval/reward/maj@2/std:np.float64(0.21312019792510917) - val-aux/sciknoweval/reward/best@4/mean:np.float64(0.8329625) - val-aux/sciknoweval/reward/best@4/std:np.float64(0.14911841799011322) - val-aux/sciknoweval/reward/worst@4/mean:np.float64(0.49555000000000005) - val-aux/sciknoweval/reward/worst@4/std:np.float64(0.0995466061193929) - val-aux/sciknoweval/reward/maj@4/mean:np.float64(0.6846875000000001) - val-aux/sciknoweval/reward/maj@4/std:np.float64(0.19169144926745516) - val-aux/sciknoweval/reward/best@8/mean:np.float64(0.8903625) - val-aux/sciknoweval/reward/best@8/std:np.float64(0.08902310096279493) - val-aux/sciknoweval/reward/worst@8/mean:np.float64(0.4632625) - val-aux/sciknoweval/reward/worst@8/std:np.float64(0.04804278474577391) - val-aux/sciknoweval/reward/maj@8/mean:np.float64(0.7100125) - val-aux/sciknoweval/reward/maj@8/std:np.float64(0.16088561753057692) - val-aux/sciknoweval/reward/best@16/mean:np.float64(0.9214874999999999) - val-aux/sciknoweval/reward/best@16/std:np.float64(0.04076217776661745) - val-aux/sciknoweval/reward/worst@16/mean:np.float64(0.448325) - val-aux/sciknoweval/reward/worst@16/std:np.float64(0.02214616706327121) - val-aux/sciknoweval/reward/maj@16/mean:np.float64(0.7325125) - val-aux/sciknoweval/reward/maj@16/std:np.float64(0.12534302205736753) - val-aux/sciknoweval/score/mean@16:np.float64(0.65390625) - val-aux/sciknoweval/score/std@16:np.float64(0.21345823452859586) - val-aux/sciknoweval/score/best@2/mean:np.float64(0.7464375000000001) - val-aux/sciknoweval/score/best@2/std:np.float64(0.19899169203769165) - val-aux/sciknoweval/score/worst@2/mean:np.float64(0.55885) - val-aux/sciknoweval/score/worst@2/std:np.float64(0.1654343787430585) - val-aux/sciknoweval/score/maj@2/mean:np.float64(0.6524) - val-aux/sciknoweval/score/maj@2/std:np.float64(0.21312019792510917) - val-aux/sciknoweval/score/best@4/mean:np.float64(0.8329625) - val-aux/sciknoweval/score/best@4/std:np.float64(0.14911841799011322) - val-aux/sciknoweval/score/worst@4/mean:np.float64(0.49555000000000005) - val-aux/sciknoweval/score/worst@4/std:np.float64(0.0995466061193929) - val-aux/sciknoweval/score/maj@4/mean:np.float64(0.6846875000000001) - val-aux/sciknoweval/score/maj@4/std:np.float64(0.19169144926745516) - val-aux/sciknoweval/score/best@8/mean:np.float64(0.8903625) - val-aux/sciknoweval/score/best@8/std:np.float64(0.08902310096279493) - val-aux/sciknoweval/score/worst@8/mean:np.float64(0.4632625) - val-aux/sciknoweval/score/worst@8/std:np.float64(0.04804278474577391) - val-aux/sciknoweval/score/maj@8/mean:np.float64(0.7100125) - val-aux/sciknoweval/score/maj@8/std:np.float64(0.16088561753057692) - val-aux/sciknoweval/score/best@16/mean:np.float64(0.9214874999999999) - val-aux/sciknoweval/score/best@16/std:np.float64(0.04076217776661745) - val-aux/sciknoweval/score/worst@16/mean:np.float64(0.448325) - val-aux/sciknoweval/score/worst@16/std:np.float64(0.02214616706327121) - val-aux/sciknoweval/score/maj@16/mean:np.float64(0.7325125) - val-aux/sciknoweval/score/maj@16/std:np.float64(0.12534302205736753) - val-core/sciknoweval/acc/mean@16:np.float64(0.65390625) - val-aux/sciknoweval/acc/std@16:np.float64(0.21345823452859586) - val-aux/sciknoweval/acc/best@2/mean:np.float64(0.7464375000000001) - val-aux/sciknoweval/acc/best@2/std:np.float64(0.19899169203769165) - val-aux/sciknoweval/acc/worst@2/mean:np.float64(0.55885) - val-aux/sciknoweval/acc/worst@2/std:np.float64(0.1654343787430585) - val-aux/sciknoweval/acc/maj@2/mean:np.float64(0.6524) - val-aux/sciknoweval/acc/maj@2/std:np.float64(0.21312019792510917) - val-aux/sciknoweval/acc/best@4/mean:np.float64(0.8329625) - val-aux/sciknoweval/acc/best@4/std:np.float64(0.14911841799011322) - val-aux/sciknoweval/acc/worst@4/mean:np.float64(0.49555000000000005) - val-aux/sciknoweval/acc/worst@4/std:np.float64(0.0995466061193929) - val-aux/sciknoweval/acc/maj@4/mean:np.float64(0.6846875000000001) - val-aux/sciknoweval/acc/maj@4/std:np.float64(0.19169144926745516) - val-aux/sciknoweval/acc/best@8/mean:np.float64(0.8903625) - val-aux/sciknoweval/acc/best@8/std:np.float64(0.08902310096279493) - val-aux/sciknoweval/acc/worst@8/mean:np.float64(0.4632625) - val-aux/sciknoweval/acc/worst@8/std:np.float64(0.04804278474577391) - val-aux/sciknoweval/acc/maj@8/mean:np.float64(0.7100125) - val-aux/sciknoweval/acc/maj@8/std:np.float64(0.16088561753057692) - val-core/sciknoweval/acc/best@16/mean:np.float64(0.9214874999999999) - val-core/sciknoweval/acc/best@16/std:np.float64(0.04076217776661745) - val-aux/sciknoweval/acc/worst@16/mean:np.float64(0.448325) - val-aux/sciknoweval/acc/worst@16/std:np.float64(0.02214616706327121) - val-core/sciknoweval/acc/maj@16/mean:np.float64(0.7325125) - val-core/sciknoweval/acc/maj@16/std:np.float64(0.12534302205736753) - val-aux/sciknoweval/incorrect_format/mean@16:np.float64(0.93359375) - val-aux/sciknoweval/incorrect_format/std@16:np.float64(0.11242025595458031) - val-aux/sciknoweval/incorrect_format/best@2/mean:np.float64(0.9786250000000001) - val-aux/sciknoweval/incorrect_format/best@2/std:np.float64(0.06145532564716834) - val-aux/sciknoweval/incorrect_format/worst@2/mean:np.float64(0.8873125) - val-aux/sciknoweval/incorrect_format/worst@2/std:np.float64(0.12815918862605175) - val-aux/sciknoweval/incorrect_format/maj@2/mean:np.float64(0.9337624999999999) - val-aux/sciknoweval/incorrect_format/maj@2/std:np.float64(0.11286336680590714) - val-aux/sciknoweval/incorrect_format/best@4/mean:np.float64(0.9963750000000001) - val-aux/sciknoweval/incorrect_format/best@4/std:np.float64(0.019611055703303716) - val-aux/sciknoweval/incorrect_format/worst@4/mean:np.float64(0.8331125) - val-aux/sciknoweval/incorrect_format/worst@4/std:np.float64(0.12538427780463093) - val-aux/sciknoweval/incorrect_format/maj@4/mean:np.float64(0.9568625) - val-aux/sciknoweval/incorrect_format/maj@4/std:np.float64(0.08973386165612807) - val-aux/sciknoweval/incorrect_format/best@8/mean:np.float64(0.9997875) - val-aux/sciknoweval/incorrect_format/best@8/std:np.float64(0.0029360164781816363) - val-aux/sciknoweval/incorrect_format/worst@8/mean:np.float64(0.7791125) - val-aux/sciknoweval/incorrect_format/worst@8/std:np.float64(0.10222723529017974) - val-aux/sciknoweval/incorrect_format/maj@8/mean:np.float64(0.9782) - val-aux/sciknoweval/incorrect_format/maj@8/std:np.float64(0.06007209345275405) - val-aux/sciknoweval/incorrect_format/best@16/mean:np.float64(1.0) - val-aux/sciknoweval/incorrect_format/best@16/std:np.float64(0.0) - val-aux/sciknoweval/incorrect_format/worst@16/mean:np.float64(0.7367125) - val-aux/sciknoweval/incorrect_format/worst@16/std:np.float64(0.06895219571062065) - val-aux/sciknoweval/incorrect_format/maj@16/mean:np.float64(0.9924125) - val-aux/sciknoweval/incorrect_format/maj@16/std:np.float64(0.03078564025657012) - val-aux/num_turns/min:np.int32(2) - val-aux/num_turns/max:np.int32(2) - val-aux/num_turns/mean:np.float64(2.0) - training/global_step:50 - training/epoch:2 - critic/score/mean:0.6328125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6328125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.012263277545571327 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.012263277545571327 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1352.2890625 - response_length/max:6204.0 - response_length/min:110.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1352.2890625 - response_length_non_aborted/max:6204.0 - response_length_non_aborted/min:110.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:266.625 - prompt_length/max:355.0 - prompt_length/min:174.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00010510161519050598 - timing_s/agent_loop/generate_sequences/min:np.float64(0.9486863240599632) - timing_s/agent_loop/generate_sequences/max:np.float64(45.78752861917019) - timing_s/agent_loop/generate_sequences/mean:np.float64(12.385333957761759) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(45.78752861917019) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:303 - timing_s/agent_loop/slowest/response_length:6204 - timing_s/gen:52.41064179688692 - timing_s/reward:0.09868243709206581 - timing_s/old_log_prob:4.130571527406573 - timing_s/adv:0.1609333734959364 - timing_s/update_actor:16.788390869274735 - timing_s/step:73.67946309968829 - timing_s/testing:173.9143652804196 - timing_s/save_checkpoint:12.895720075815916 - timing_s/stop_profile:0.00021344982087612152 - timing_per_token_ms/gen:0.15139445788358546 - timing_per_token_ms/update_actor:0.040508420645771266 - timing_per_token_ms/adv:0.00038831337918438863 - perf/total_num_tokens:414442 - perf/time_per_step:73.67946309968829 - perf/throughput:703.1165513503744
(TaskRunner pid=2281424) Training Progress: 50%|█████ | 50/100 [57:56<1:45:03, 126.06s/it]
(TaskRunner pid=2281424) step:51 - global_seqlen/min:25192 - global_seqlen/max:76157 - global_seqlen/minmax_diff:50965 - global_seqlen/balanced_min:48271 - global_seqlen/balanced_max:48448 - global_seqlen/mean:48379.375 - actor/entropy:0.41202202439308167 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.37214818596839905 - training/rollout_probs_diff_mean:0.003262621583417058 - training/rollout_probs_diff_std:0.008078004233539104 - training/rollout_actor_probs_pearson_corr:0.9995943903923035 - rollout_corr/rollout_is_mean:0.9999638795852661 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:6.276774456637213e-06 - rollout_corr/rollout_is_max:1.636783480644226 - rollout_corr/rollout_is_min:0.13366740942001343 - rollout_corr/rollout_is_std:0.027932504191994667 - rollout_corr/rollout_is_eff_sample_size:0.999220264452564 - rollout_corr/rollout_is_seq_mean:0.9999167919158936 - rollout_corr/rollout_is_seq_std:0.001000249176286161 - rollout_corr/rollout_is_seq_max:1.0028444528579712 - rollout_corr/rollout_is_seq_min:0.9964448809623718 - rollout_corr/rollout_is_seq_max_deviation:0.003555119037628174 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3898025089874864) - rollout_corr/training_log_ppl:np.float64(0.31166147111798637) - rollout_corr/kl:np.float64(0.0004372116775031021) - rollout_corr/k3_kl:np.float64(0.0003934571469699222) - rollout_corr/rollout_ppl:np.float64(1.3891857019625604) - rollout_corr/rollout_log_ppl:np.float64(0.3112242584757041) - rollout_corr/log_ppl_diff:np.float64(0.00043721264228224754) - rollout_corr/log_ppl_abs_diff:np.float64(0.000884955981746316) - rollout_corr/log_ppl_diff_max:np.float64(0.007482245564460754) - rollout_corr/log_ppl_diff_min:np.float64(-0.002665892243385315) - rollout_corr/ppl_ratio:np.float64(1.0004379730671644) - rollout_corr/chi2_token:np.float64(0.0006916895508766174) - rollout_corr/chi2_seq:np.float64(0.8691392296459526) - actor/pg_loss:np.float64(5.567236803472042e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00015443053536046136) - actor/ppo_kl:np.float64(-3.0327652694950302e-05) - actor/pg_clipfrac_lower:np.float64(8.184056241589133e-07) - actor/grad_norm:np.float64(0.10533490870147943) - perf/mfu/actor:np.float64(0.1597418217861688) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.85708999633789) - actor/lr:np.float64(1e-06) - training/global_step:51 - training/epoch:2 - critic/score/mean:0.7109375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.7109375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.014979914762079716 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.014979914762079716 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1244.66796875 - response_length/max:5675.0 - response_length/min:200.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1244.66796875 - response_length_non_aborted/max:5675.0 - response_length_non_aborted/min:200.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:267.1875 - prompt_length/max:356.0 - prompt_length/min:169.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011126697063446045 - timing_s/agent_loop/generate_sequences/min:np.float64(2.4992719031870365) - timing_s/agent_loop/generate_sequences/max:np.float64(40.4043846745044) - timing_s/agent_loop/generate_sequences/mean:np.float64(11.760603568858642) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(40.4043846745044) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:203 - timing_s/agent_loop/slowest/response_length:5675 - timing_s/gen:46.7182105332613 - timing_s/reward:0.0914076529443264 - timing_s/old_log_prob:4.139765975996852 - timing_s/adv:0.15743766352534294 - timing_s/update_actor:16.27989299967885 - timing_s/step:67.497363332659 - timing_s/stop_profile:0.00012996233999729156 - timing_per_token_ms/gen:0.1466198331421887 - timing_per_token_ms/update_actor:0.04206310281932862 - timing_per_token_ms/adv:0.0004067788792366141 - perf/total_num_tokens:387035 - perf/time_per_step:67.497363332659 - perf/throughput:716.7594793527489
(TaskRunner pid=2281424) Training Progress: 51%|█████ | 51/100 [59:03<1:28:37, 108.52s/it]
(TaskRunner pid=2281424) step:52 - global_seqlen/min:38872 - global_seqlen/max:61135 - global_seqlen/minmax_diff:22263 - global_seqlen/balanced_min:50883 - global_seqlen/balanced_max:50962 - global_seqlen/mean:50922.75 - actor/entropy:0.4251244068145752 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.31729912757873535 - training/rollout_probs_diff_mean:0.0034149810671806335 - training/rollout_probs_diff_std:0.008329342119395733 - training/rollout_actor_probs_pearson_corr:0.9995748996734619 - rollout_corr/rollout_is_mean:0.9999804496765137 - rollout_corr/rollout_is_ratio_fraction_high:2.9626644391100854e-06 - rollout_corr/rollout_is_ratio_fraction_low:5.925328878220171e-06 - rollout_corr/rollout_is_max:2.0289995670318604 - rollout_corr/rollout_is_min:0.38479822874069214 - rollout_corr/rollout_is_std:0.02867795154452324 - rollout_corr/rollout_is_eff_sample_size:0.9991781319248341 - rollout_corr/rollout_is_seq_mean:1.0001072883605957 - rollout_corr/rollout_is_seq_std:0.0014209682121872902 - rollout_corr/rollout_is_seq_max:1.0089370012283325 - rollout_corr/rollout_is_seq_min:0.9960955381393433 - rollout_corr/rollout_is_seq_max_deviation:0.00893700122833252 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4353607962839305) - rollout_corr/training_log_ppl:np.float64(0.34465374518185854) - rollout_corr/kl:np.float64(0.00030015603282895764) - rollout_corr/k3_kl:np.float64(0.00044390009770722827) - rollout_corr/rollout_ppl:np.float64(1.4348906846717) - rollout_corr/rollout_log_ppl:np.float64(0.3443535875703674) - rollout_corr/log_ppl_diff:np.float64(0.0003001576114911586) - rollout_corr/log_ppl_abs_diff:np.float64(0.000979981996351853) - rollout_corr/log_ppl_diff_max:np.float64(0.006529062986373901) - rollout_corr/log_ppl_diff_min:np.float64(-0.006823122501373291) - rollout_corr/ppl_ratio:np.float64(1.0003011787775904) - rollout_corr/chi2_token:np.float64(0.0011839414946734905) - rollout_corr/chi2_seq:np.float64(2.5429790497291833) - actor/pg_loss:np.float64(3.4721335396170616e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(8.84185026279738e-05) - actor/ppo_kl:np.float64(-4.0347963830456024e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.11110754683613777) - perf/mfu/actor:np.float64(0.16599907427815286) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.80552291870117) - actor/lr:np.float64(1e-06) - training/global_step:52 - training/epoch:2 - critic/score/mean:0.71484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.71484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.004099216777831316 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:0.004099216777831316 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1318.4921875 - response_length/max:5029.0 - response_length/min:132.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1318.4921875 - response_length_non_aborted/max:5029.0 - response_length_non_aborted/min:132.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:272.84375 - prompt_length/max:365.0 - prompt_length/min:161.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001096632331609726 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7103512845933437) - timing_s/agent_loop/generate_sequences/max:np.float64(38.38058450073004) - timing_s/agent_loop/generate_sequences/mean:np.float64(13.082279307935096) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(38.38058450073004) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:365 - timing_s/agent_loop/slowest/response_length:4841 - timing_s/gen:45.04192319884896 - timing_s/reward:0.10605371557176113 - timing_s/old_log_prob:4.176968155428767 - timing_s/adv:0.13944151997566223 - timing_s/update_actor:16.25410868227482 - timing_s/step:65.81346314027905 - timing_s/stop_profile:0.00014881044626235962 - timing_per_token_ms/gen:0.13344410696062903 - timing_per_token_ms/update_actor:0.03989893682655301 - timing_per_token_ms/adv:0.00034228689528664064 - perf/total_num_tokens:407382 - perf/time_per_step:65.81346314027905 - perf/throughput:773.7436623181486
(TaskRunner pid=2281424) Training Progress: 52%|█████▏ | 52/100 [1:00:09<1:16:34, 95.73s/it]
(TaskRunner pid=2281424) step:53 - global_seqlen/min:24839 - global_seqlen/max:55902 - global_seqlen/minmax_diff:31063 - global_seqlen/balanced_min:43329 - global_seqlen/balanced_max:43584 - global_seqlen/mean:43520.375 - actor/entropy:0.37126773595809937 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3016398251056671 - training/rollout_probs_diff_mean:0.003143634181469679 - training/rollout_probs_diff_std:0.00820545107126236 - training/rollout_actor_probs_pearson_corr:0.9994932413101196 - rollout_corr/rollout_is_mean:0.9999392628669739 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:7.210143394331681e-06 - rollout_corr/rollout_is_max:1.7527995109558105 - rollout_corr/rollout_is_min:0.30146440863609314 - rollout_corr/rollout_is_std:0.027419937774538994 - rollout_corr/rollout_is_eff_sample_size:0.9992485928388086 - rollout_corr/rollout_is_seq_mean:0.9999774694442749 - rollout_corr/rollout_is_seq_std:0.0011340720811858773 - rollout_corr/rollout_is_seq_max:1.005629062652588 - rollout_corr/rollout_is_seq_min:0.9950371980667114 - rollout_corr/rollout_is_seq_max_deviation:0.005629062652587891 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.3701058248989284) - rollout_corr/training_log_ppl:np.float64(0.2918059763032943) - rollout_corr/kl:np.float64(0.0003194374729091898) - rollout_corr/k3_kl:np.float64(0.00042779503252177165) - rollout_corr/rollout_ppl:np.float64(1.3696815236471593) - rollout_corr/rollout_log_ppl:np.float64(0.29148653568699956) - rollout_corr/log_ppl_diff:np.float64(0.00031944061629474163) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009462947491556406) - rollout_corr/log_ppl_diff_max:np.float64(0.006018996238708496) - rollout_corr/log_ppl_diff_min:np.float64(-0.003216482698917389) - rollout_corr/ppl_ratio:np.float64(1.0003202487714589) - rollout_corr/chi2_token:np.float64(0.0010781094897538424) - rollout_corr/chi2_seq:np.float64(1.596792209893465) - actor/pg_loss:np.float64(3.835768438875675e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00021167921056530759) - actor/ppo_kl:np.float64(-7.656292218705119e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.13869349658489227) - perf/mfu/actor:np.float64(0.14950320003636675) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.62938690185547) - actor/lr:np.float64(1e-06) - training/global_step:53 - training/epoch:2 - critic/score/mean:0.73828125 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.73828125 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.015568501316010952 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.015568501316010952 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1083.54296875 - response_length/max:5024.0 - response_length/min:193.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1083.54296875 - response_length_non_aborted/max:5024.0 - response_length_non_aborted/min:193.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:276.46875 - prompt_length/max:375.0 - prompt_length/min:170.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00011430680751800537 - timing_s/agent_loop/generate_sequences/min:np.float64(2.2967226803302765) - timing_s/agent_loop/generate_sequences/max:np.float64(36.389772513881326) - timing_s/agent_loop/generate_sequences/mean:np.float64(10.762310426936892) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(36.389772513881326) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:294 - timing_s/agent_loop/slowest/response_length:5024 - timing_s/gen:43.48752962425351 - timing_s/reward:0.08610256388783455 - timing_s/old_log_prob:3.7902569361031055 - timing_s/adv:0.1312139704823494 - timing_s/update_actor:15.44841111637652 - timing_s/step:63.040359219536185 - timing_s/stop_profile:0.00011725910007953644 - timing_per_token_ms/gen:0.15677565864389287 - timing_per_token_ms/update_actor:0.04437120290317041 - timing_per_token_ms/adv:0.0003768751144789923 - perf/total_num_tokens:348163 - perf/time_per_step:63.040359219536185 - perf/throughput:690.3573447042328
(TaskRunner pid=2281424) Training Progress: 53%|█████▎ | 53/100 [1:01:12<1:07:19, 85.94s/it]
(TaskRunner pid=2281424) step:54 - global_seqlen/min:35066 - global_seqlen/max:65296 - global_seqlen/minmax_diff:30230 - global_seqlen/balanced_min:48016 - global_seqlen/balanced_max:48715 - global_seqlen/mean:48320.375 - actor/entropy:0.41513416171073914 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.3469814658164978 - training/rollout_probs_diff_mean:0.0032787041272968054 - training/rollout_probs_diff_std:0.008095493540167809 - training/rollout_actor_probs_pearson_corr:0.999546229839325 - rollout_corr/rollout_is_mean:0.9998549818992615 - rollout_corr/rollout_is_ratio_fraction_high:3.120212340945727e-06 - rollout_corr/rollout_is_ratio_fraction_low:9.360637704958208e-06 - rollout_corr/rollout_is_max:3.656163454055786 - rollout_corr/rollout_is_min:0.04287761077284813 - rollout_corr/rollout_is_std:0.027915427461266518 - rollout_corr/rollout_is_eff_sample_size:0.99922121664114 - rollout_corr/rollout_is_seq_mean:0.9998648166656494 - rollout_corr/rollout_is_seq_std:0.0012277925852686167 - rollout_corr/rollout_is_seq_max:1.0036733150482178 - rollout_corr/rollout_is_seq_min:0.9944618940353394 - rollout_corr/rollout_is_seq_max_deviation:0.0055381059646606445 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.386646915692836) - rollout_corr/training_log_ppl:np.float64(0.31141754146665335) - rollout_corr/kl:np.float64(0.00046588320381735215) - rollout_corr/k3_kl:np.float64(0.00041930354774422085) - rollout_corr/rollout_ppl:np.float64(1.3860031738877296) - rollout_corr/rollout_log_ppl:np.float64(0.31095165672013536) - rollout_corr/log_ppl_diff:np.float64(0.00046588474651798606) - rollout_corr/log_ppl_abs_diff:np.float64(0.0009574951836839318) - rollout_corr/log_ppl_diff_max:np.float64(0.0065114498138427734) - rollout_corr/log_ppl_diff_min:np.float64(-0.0028219521045684814) - rollout_corr/ppl_ratio:np.float64(1.0004668282344937) - rollout_corr/chi2_token:np.float64(0.000739572336897254) - rollout_corr/chi2_seq:np.float64(1.1254641045816243) - actor/pg_loss:np.float64(-1.3741431757807732e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(8.72049774898187e-05) - actor/ppo_kl:np.float64(-8.353029136642931e-05) - actor/pg_clipfrac_lower:np.float64(1.2657970955842757e-06) - actor/grad_norm:np.float64(0.09051301795989275) - perf/mfu/actor:np.float64(0.15693746638014805) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.6057014465332) - actor/lr:np.float64(1e-06) - training/global_step:54 - training/epoch:2 - critic/score/mean:0.76171875 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.76171875 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.015124449506402016 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.015124449506402016 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1251.91796875 - response_length/max:8192.0 - response_length/min:151.0 - response_length/clip_ratio:0.00390625 - response_length_non_aborted/mean:1251.91796875 - response_length_non_aborted/max:8192.0 - response_length_non_aborted/min:151.0 - response_length_non_aborted/clip_ratio:0.00390625 - response/aborted_ratio:0.0 - prompt_length/mean:258.09375 - prompt_length/max:363.0 - prompt_length/min:155.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:8.027255535125732e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.3670800887048244) - timing_s/agent_loop/generate_sequences/max:np.float64(57.39985219575465) - timing_s/agent_loop/generate_sequences/mean:np.float64(11.685668693418847) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(57.39985219575465) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:267 - timing_s/agent_loop/slowest/response_length:8192 - timing_s/gen:64.27751941606402 - timing_s/reward:0.09590023942291737 - timing_s/old_log_prob:4.017355827614665 - timing_s/adv:0.11962731927633286 - timing_s/update_actor:16.561819057911634 - timing_s/step:85.16014346294105 - timing_s/stop_profile:0.00010659545660018921 - timing_per_token_ms/gen:0.20055951466987848 - timing_per_token_ms/update_actor:0.04284377723142575 - timing_per_token_ms/adv:0.0003094639664849788 - perf/total_num_tokens:386563 - perf/time_per_step:85.16014346294105 - perf/throughput:567.4059840097318
(TaskRunner pid=2281424) Training Progress: 54%|█████▍ | 54/100 [1:02:38<1:05:43, 85.72s/it]
(TaskRunner pid=2281424) step:55 - global_seqlen/min:28028 - global_seqlen/max:53093 - global_seqlen/minmax_diff:25065 - global_seqlen/balanced_min:38416 - global_seqlen/balanced_max:38451 - global_seqlen/mean:38438.25 - actor/entropy:0.3102551996707916 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5002540349960327 - training/rollout_probs_diff_mean:0.0031136018224060535 - training/rollout_probs_diff_std:0.008566453121602535 - training/rollout_actor_probs_pearson_corr:0.9994353652000427 - rollout_corr/rollout_is_mean:0.9999341368675232 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:4.173030447418569e-06 - rollout_corr/rollout_is_max:1.8234162330627441 - rollout_corr/rollout_is_min:0.46666446328163147 - rollout_corr/rollout_is_std:0.02762567438185215 - rollout_corr/rollout_is_eff_sample_size:0.9992372850970654 - rollout_corr/rollout_is_seq_mean:0.999902606010437 - rollout_corr/rollout_is_seq_std:0.0013777980348095298 - rollout_corr/rollout_is_seq_max:1.0075701475143433 - rollout_corr/rollout_is_seq_min:0.9948606491088867 - rollout_corr/rollout_is_seq_max_deviation:0.007570147514343262 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.308470833580941) - rollout_corr/training_log_ppl:np.float64(0.2564443113806192) - rollout_corr/kl:np.float64(0.0004685787036251554) - rollout_corr/k3_kl:np.float64(0.0005076911484707125) - rollout_corr/rollout_ppl:np.float64(1.3078627600334585) - rollout_corr/rollout_log_ppl:np.float64(0.255975730920909) - rollout_corr/log_ppl_diff:np.float64(0.00046858045971021056) - rollout_corr/log_ppl_abs_diff:np.float64(0.0011701608309522271) - rollout_corr/log_ppl_diff_max:np.float64(0.006679356098175049) - rollout_corr/log_ppl_diff_min:np.float64(-0.0075002312660217285) - rollout_corr/ppl_ratio:np.float64(1.000470134196803) - rollout_corr/chi2_token:np.float64(0.0011455377098172903) - rollout_corr/chi2_seq:np.float64(5.852095796493813) - actor/pg_loss:np.float64(-3.334670327603817e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001705758450043504) - actor/ppo_kl:np.float64(-4.5051564853793025e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.07974306493997574) - perf/mfu/actor:np.float64(0.1378203781579895) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(106.55471420288086) - actor/lr:np.float64(1e-06) - training/global_step:55 - training/epoch:2 - critic/score/mean:0.84765625 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.84765625 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.011153988540172577 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.011153988540172577 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:936.0703125 - response_length/max:3898.0 - response_length/min:146.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:936.0703125 - response_length_non_aborted/max:3898.0 - response_length_non_aborted/min:146.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:265.125 - prompt_length/max:383.0 - prompt_length/min:163.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.0001405421644449234 - timing_s/agent_loop/generate_sequences/min:np.float64(1.978264905512333) - timing_s/agent_loop/generate_sequences/max:np.float64(29.77366792783141) - timing_s/agent_loop/generate_sequences/mean:np.float64(9.763206802737841) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(29.77366792783141) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:341 - timing_s/agent_loop/slowest/response_length:3898 - timing_s/gen:36.85863957554102 - timing_s/reward:0.08288105763494968 - timing_s/old_log_prob:3.500804428011179 - timing_s/adv:0.11686178669333458 - timing_s/update_actor:14.765768744051456 - timing_s/step:55.41219947300851 - timing_s/stop_profile:0.00012343376874923706 - timing_per_token_ms/gen:0.1538122285466212 - timing_per_token_ms/update_actor:0.04801782321012096 - timing_per_token_ms/adv:0.0003800309154726561 - perf/total_num_tokens:307506 - perf/time_per_step:55.41219947300851 - perf/throughput:693.6784745157682
(TaskRunner pid=2281424) Training Progress: 55%|█████▌ | 55/100 [1:03:33<57:28, 76.64s/it]
(TaskRunner pid=2281424) step:56 - global_seqlen/min:35222 - global_seqlen/max:67738 - global_seqlen/minmax_diff:32516 - global_seqlen/balanced_min:48431 - global_seqlen/balanced_max:48616 - global_seqlen/mean:48556.375 - actor/entropy:0.3693101406097412 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.5576332211494446 - training/rollout_probs_diff_mean:0.003098106477409601 - training/rollout_probs_diff_std:0.008025160990655422 - training/rollout_actor_probs_pearson_corr:0.9995522499084473 - rollout_corr/rollout_is_mean:0.999983012676239 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:3.090129894189886e-06 - rollout_corr/rollout_is_max:1.6744037866592407 - rollout_corr/rollout_is_min:0.3452942669391632 - rollout_corr/rollout_is_std:0.02711610123515129 - rollout_corr/rollout_is_eff_sample_size:0.9992651383115954 - rollout_corr/rollout_is_seq_mean:0.9999608993530273 - rollout_corr/rollout_is_seq_std:0.0010731675429269671 - rollout_corr/rollout_is_seq_max:1.006026029586792 - rollout_corr/rollout_is_seq_min:0.9958376288414001 - rollout_corr/rollout_is_seq_max_deviation:0.006026029586791992 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.373812660574913) - rollout_corr/training_log_ppl:np.float64(0.3022938699577935) - rollout_corr/kl:np.float64(0.0004166627330413064) - rollout_corr/k3_kl:np.float64(0.00039845293167672935) - rollout_corr/rollout_ppl:np.float64(1.3732107440009713) - rollout_corr/rollout_log_ppl:np.float64(0.3018772063078359) - rollout_corr/log_ppl_diff:np.float64(0.00041666364995762706) - rollout_corr/log_ppl_abs_diff:np.float64(0.0008859913214109838) - rollout_corr/log_ppl_diff_max:np.float64(0.00528949499130249) - rollout_corr/log_ppl_diff_min:np.float64(-0.005298733711242676) - rollout_corr/ppl_ratio:np.float64(1.0004174057394266) - rollout_corr/chi2_token:np.float64(0.0007626630831509829) - rollout_corr/chi2_seq:np.float64(3.9297358801122755) - actor/pg_loss:np.float64(-1.1299271136522293e-06) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.00014216022026403152) - actor/ppo_kl:np.float64(-9.481449527726227e-06) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.08582276105880737) - perf/mfu/actor:np.float64(0.15478688954187692) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(117.41515350341797) - actor/lr:np.float64(1e-06) - training/global_step:56 - training/epoch:2 - critic/score/mean:0.6484375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.6484375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:0.01221644226461649 - critic/advantages/max:0.875 - critic/advantages/min:-0.75 - critic/returns/mean:0.01221644226461649 - critic/returns/max:0.875 - critic/returns/min:-0.75 - response_length/mean:1264.10546875 - response_length/max:5770.0 - response_length/min:163.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1264.10546875 - response_length_non_aborted/max:5770.0 - response_length_non_aborted/min:163.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:253.28125 - prompt_length/max:355.0 - prompt_length/min:143.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:0.00013057328760623932 - timing_s/agent_loop/generate_sequences/min:np.float64(2.215219648554921) - timing_s/agent_loop/generate_sequences/max:np.float64(45.5247247479856) - timing_s/agent_loop/generate_sequences/mean:np.float64(12.946847598039312) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(45.5247247479856) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:303 - timing_s/agent_loop/slowest/response_length:5770 - timing_s/gen:52.53792614676058 - timing_s/reward:0.09584798477590084 - timing_s/old_log_prob:3.963084263727069 - timing_s/adv:0.13674898445606232 - timing_s/update_actor:16.772162212058902 - timing_s/step:73.59324477799237 - timing_s/stop_profile:3.3739954233169556e-05 - timing_per_token_ms/gen:0.1623490120754875 - timing_per_token_ms/update_actor:0.04317703445752206 - timing_per_token_ms/adv:0.0003520366389996739 - perf/total_num_tokens:388451 - perf/time_per_step:73.59324477799237 - perf/throughput:659.7939137821587
(TaskRunner pid=2281424) Training Progress: 56%|█████▌ | 56/100 [1:04:47<55:32, 75.74s/it]
(TaskRunner pid=2281424) step:57 - global_seqlen/min:50111 - global_seqlen/max:61804 - global_seqlen/minmax_diff:11693 - global_seqlen/balanced_min:54058 - global_seqlen/balanced_max:54380 - global_seqlen/mean:54299.125 - actor/entropy:0.3828109800815582 - perf/mfu/actor_infer:0 - training/rollout_probs_diff_valid:1 - training/rollout_probs_diff_max:0.26735612750053406 - training/rollout_probs_diff_mean:0.003163557732477784 - training/rollout_probs_diff_std:0.007935766130685806 - training/rollout_actor_probs_pearson_corr:0.9995775818824768 - rollout_corr/rollout_is_mean:0.9999682903289795 - rollout_corr/rollout_is_ratio_fraction_high:0.0 - rollout_corr/rollout_is_ratio_fraction_low:2.757548145382316e-06 - rollout_corr/rollout_is_max:1.6177020072937012 - rollout_corr/rollout_is_min:0.4519280195236206 - rollout_corr/rollout_is_std:0.027300119400024414 - rollout_corr/rollout_is_eff_sample_size:0.9992552585748898 - rollout_corr/rollout_is_seq_mean:0.9999948740005493 - rollout_corr/rollout_is_seq_std:0.0008971233037300408 - rollout_corr/rollout_is_seq_max:1.004016637802124 - rollout_corr/rollout_is_seq_min:0.9975495934486389 - rollout_corr/rollout_is_seq_max_deviation:0.0040166378021240234 - rollout_corr/rollout_is_seq_fraction_high:0.0 - rollout_corr/rollout_is_seq_fraction_low:0.0 - rollout_corr/training_ppl:np.float64(1.4292010506615043) - rollout_corr/training_log_ppl:np.float64(0.3405683948658407) - rollout_corr/kl:np.float64(0.00034688604483079644) - rollout_corr/k3_kl:np.float64(0.00040535901956673115) - rollout_corr/rollout_ppl:np.float64(1.4287002789787948) - rollout_corr/rollout_log_ppl:np.float64(0.34022150505916215) - rollout_corr/log_ppl_diff:np.float64(0.00034688980667851865) - rollout_corr/log_ppl_abs_diff:np.float64(0.0007467021641787142) - rollout_corr/log_ppl_diff_max:np.float64(0.0035548806190490723) - rollout_corr/log_ppl_diff_min:np.float64(-0.0031991004943847656) - rollout_corr/ppl_ratio:np.float64(1.0003474049735814) - rollout_corr/chi2_token:np.float64(0.0009404013399034739) - rollout_corr/chi2_seq:np.float64(1.3481947318650782) - actor/pg_loss:np.float64(-1.1662370525300503e-05) - actor/kl_loss:np.float64(0.0) - actor/pg_clipfrac:np.float64(0.0001704909616933037) - actor/ppo_kl:np.float64(-4.2620445977803456e-05) - actor/pg_clipfrac_lower:np.float64(0.0) - actor/grad_norm:np.float64(0.11611825972795486) - perf/mfu/actor:np.float64(0.13975899720323387) - perf/max_memory_allocated_gb:np.float64(131.87133502960205) - perf/max_memory_reserved_gb:np.float64(144.58203125) - perf/cpu_memory_used_gb:np.float64(142.28152465820312) - actor/lr:np.float64(1e-06) - training/global_step:57 - training/epoch:2 - critic/score/mean:0.734375 - critic/score/max:1.0 - critic/score/min:0.0 - critic/rewards/mean:0.734375 - critic/rewards/max:1.0 - critic/rewards/min:0.0 - critic/advantages/mean:-0.0023128935135900974 - critic/advantages/max:0.875 - critic/advantages/min:-0.875 - critic/returns/mean:-0.0023128935135900974 - critic/returns/max:0.875 - critic/returns/min:-0.875 - response_length/mean:1416.56640625 - response_length/max:6110.0 - response_length/min:138.0 - response_length/clip_ratio:0.0 - response_length_non_aborted/mean:1416.56640625 - response_length_non_aborted/max:6110.0 - response_length_non_aborted/min:138.0 - response_length_non_aborted/clip_ratio:0.0 - response/aborted_ratio:0.0 - prompt_length/mean:280.28125 - prompt_length/max:382.0 - prompt_length/min:209.0 - prompt_length/clip_ratio:0.0 - num_turns/min:np.int32(2) - num_turns/max:np.int32(2) - num_turns/mean:np.float64(2.0) - timing_s/start_profile:3.9925798773765564e-05 - timing_s/agent_loop/generate_sequences/min:np.float64(1.7666615955531597) - timing_s/agent_loop/generate_sequences/max:np.float64(44.93153551965952) - timing_s/agent_loop/generate_sequences/mean:np.float64(14.538133056579682) - timing_s/agent_loop/tool_calls/min:np.float64(0.0) - timing_s/agent_loop/tool_calls/max:np.float64(0.0) - timing_s/agent_loop/tool_calls/mean:np.float64(0.0) - timing_s/agent_loop/slowest/generate_sequences:np.float64(44.93153551965952) - timing_s/agent_loop/slowest/tool_calls:np.float64(0.0) - timing_s/agent_loop/slowest/prompt_length:382 - timing_s/agent_loop/slowest/response_length:6110 - timing_s/gen:52.147955069318414 - timing_s/reward:0.10221655108034611 - timing_s/old_log_prob:4.09379106014967 - timing_s/adv:0.1320952270179987 - timing_s/update_actor:20.559559671208262 - timing_s/step:77.12406794913113 - timing_s/stop_profile:0.00012364424765110016 - timing_per_token_ms/gen:0.14380049434376813 - timing_per_token_ms/update_actor:0.04732939911832894 - timing_per_token_ms/adv:0.0003040915185511707 - perf/total_num_tokens:434393 - perf/time_per_step:77.12406794913113 - perf/throughput:704.0490270276482
(TaskRunner pid=2281424) Training Progress: 57%|█████▋ | 57/100 [1:06:04<54:35, 76.17s/it]
Reference in New Issue View Git Blame Copy Permalink